source capture with provenance: a manifest for everything the agent read

An agent that reads the web has a custody problem. The page it saw is not the page that exists now; the extraction it ran is not the page at all; and a claim derived from “something I read” is only as good as the record of what was actually read. Provenance for captured material means a manifest: a durable record of what was fetched, by which route, with what completeness, and which exact bytes were retained.

The question this article answers: how do you keep a verifiable record of everything an agent read? Wordcell’s wordcell clip is the reference implementation: every capture writes a self-contained bundle where the Markdown is the readable artifact and capture.json is the chain of custody.

the bundle

A capture does not write a file; it writes a directory. Under kb/articles/<slug>/ you get the selected Markdown, a capture.json manifest, localized assets/, and an optional evidence/ directory and url-metadata.json sidecar:

<slug>/
  <slug>.md
  capture.json
  url-metadata.json    # after an optional metadata backfill
  assets/
  evidence/            # only when requested

The Markdown records source and capture metadata for a human reader. capture.json records the machine version: the acquisition attempts, the selected extractor, the scope, the status, item counts, warnings, per-asset hashes, and the requested artifact outcomes. Writes stage beside the target and install with an atomic rename; --force replaces only a compatible clip-owned bundle and restores the previous one if installation fails. A bundle is meant to be committed: Git is the capture version history, and Wordcell maintains no second content-history database.

The acquisition side is deliberately bounded. Default resource limits are 30 seconds per request, process, or extraction operation; 500 scoped items; depth 16; 25 MB of HTML; 100 MB per asset; 500 MB across assets. Structured adapters handle platforms with stable surfaces (Hacker News through the official Firebase item API, Bluesky through public AT Protocol resolution, YouTube through yt-dlp metadata and one exact-language transcript) while generic rendered capture covers everything else. When the normal routes produce nothing usable, a single read-only Archive.today lookup on the exact URL is permitted; authentication, paywall, CAPTCHA, or rate-limit responses disable that fallback instead of using an archive to bypass a source’s controls, and an archived result is always reported partial.

the manifest is the chain of custody

Schema v4 of capture.json binds the manifest to the exact bytes it describes:

{
  "document": {
    "path": "<slug>.md",
    "bytes": 14823,
    "sha256": "<lowercase hex digest>"
  }
}

The digest covers the exact credential-redacted, newline-terminated Markdown bytes Wordcell wrote, not the manifest, not the source HTML, not the assets. Each asset in assets/ carries its own byte count and SHA-256. The manifest also records what was attempted: which extractors ran, which were selected, which scope was requested (page, thread, comments), and what counts and warnings the run produced. Chain of custody here means the record of acquisition is itself part of the retained evidence: a reader can reconstruct not only what the document says but how it came to be in the vault.

The sidecar rule keeps the same discipline for enrichment. url-metadata.json is a separate tool-owned record written by the URL-metadata backfill; it adds search-derived titles and descriptions without rewriting acquisition provenance in capture.json, and its records keep categorical provenance and failure states: a zero-result record is not-found only after complete attempted coverage, and any failed engine keeps the result partial.

status is a claim about completeness

The manifest’s status field is a controlled vocabulary, not a success flag:

The semantics matter downstream. partial is an honest claim (it names that something was left behind) while a generic rendered conversation reports capturedItems: 0 because visible prose does not prove a trustworthy per-item tree. Exit codes carry the same taxonomy to automation: complete and partial exit 0; auth, blocked, and unsupported exit 3; argument errors exit 2; environment diagnostic failures exit 4; operational errors exit 1. The guidance to callers is to inspect the structured status and warnings rather than trusting the process code.

verify later

A manifest that cannot be re-checked is decoration. wordcell capture verify reports integrity without printing the document (verification JSON deliberately omits both the Markdown and retained source HTML so an integrity check does not disclose captured content), while wordcell capture show prints the document explicitly labeled as untrusted data. --verify-assets on either command reads and hashes every listed asset; a missing asset, a byte mismatch, or a digest mismatch produces an asset-integrity issue and exit status 3. Bundles captured under schemas one through three lack the v4 document record and report integrity unavailable, which verify refuses to present as success.

Retained source HTML is opt-in evidence (--evidence source or all), sanitized into inert HTML with credential-shaped values redacted and a deny-all content security policy, and still documented as hostile data that may carry private content or text intended to influence an agent. Screenshots are viewport-only pixels and are not structurally sanitized. The bundle’s honesty includes saying which parts were sanitized and which were not.

Time comparison runs through Git: wordcell capture diff compares the current Markdown against a bounded ref (HEAD by default, a branch, tag, or commit name, not a revision expression), reports changed, unchanged, or missing-at-ref, and rereads the capture digest after diff generation so a concurrent change cannot pair one digest with another snapshot’s diff.

beyond the bundle

The same custody discipline extends to evidence that moves between systems. Soulscrape’s source packets are the pattern: an ensoul-source-packet-v1 envelope retains identity binding, authorship, provenance, and a bounded corpus; the X-archive preparer refuses overwrite and symlink traversal, writes the packet at mode 0600, emits only a body-free receipt, caps records at 2,000, and refuses packets above 128 MiB. The dependency-free validator checks the envelope, attribution fields, time bounds, claim bindings, I-JSON constraints, and RFC 8785 canonical-JSON SHA-256 digests, and the docs are explicit that a digest proves integrity, not truth. Publication is idempotent on the packet digest: identical bytes republish as a no-op, changed bytes bump the revision.

PeopleBlade applies it to imports: immutable source runs and records are part of the authoritative local database, raw imports validate completely before their transaction commits, and replaying the same source artifact returns its stored receipt. The source run is retained precisely so a later audit can ask what was imported, when, and from which artifact.

the honest limits

Three limits are worth stating because the tools themselves state them:

1. Capture records what a bounded process saw, not what the page contained. Depth caps, item caps, virtualized replies, and rendered-shell blocks all produce partial or failure: the manifest records the boundary it hit, not the content it missed.
2. Verification is integrity, not endorsement. A matching digest proves the retained bytes are the captured bytes; it says nothing about whether the source was accurate, authorized, or worth keeping. Source HTML and screenshots may still carry private or adversarial content.
3. The trust boundary is local. Bundle inspection rejects linked or aliased paths and re-checks identities around every read, but a malicious process with the same filesystem authority that can swap path components mid-call is outside the pure-JavaScript reader’s boundary: the docs say not to inspect bundles in a directory concurrently writable by an untrusted local process.

The claim the design actually makes is narrower and more useful than “provenance”: nothing enters the vault without a durable record of how it got there, and that record can be re-verified against the bytes without exposing the bytes.

sources

• wordcell: docs/capture.md, docs/reference.md
• soulscrape: source-packet schema, validator, and person-index publication
• peopleblade: docs/ARCHITECTURE.md immutable source runs