Spec: Ingestion Provenance — Sources & Span Ledger¶

File: .ai/specs/core-engine/2026-07-02-ingestion-provenance.md Status: Draft Owners: Storage lane; reviewers: workers lane (producers), planner lane (Phase-3 consumer) Related: PR-032; composite-key-layout spec (sources/spans CFs); claim-verification spec (spans are its evidence unit)

1. Overview¶

Defines source identity and versioning, content-hash dedup, the token span ledger, and the node↔span linkage that makes every ingested fact traceable to the exact bytes it came from — the substrate Phase-3 claim verification stands on.

2. Problem Statement¶

Without pinned provenance: re-ingesting the same PDF duplicates the graph; "where did this fact come from" has no answer below document granularity; claim verification has no evidence unit; and offsets are ambiguous (bytes of what, exactly?) across six ingestion formats.

3. Proposed Solution¶

Every ingestion produces a source version: identity = (tenant, source_id), version keyed by tx-time, carrying content_hash (BLAKE3 of raw bytes), original filename/URL, format, and the canonical extracted text reference (stored as a blob alongside, addressed by hash). Dedup: same (source_id, content_hash) → no new version; same source_id, new hash → new version. Spans are half-open byte offsets into the canonical extracted text of a specific source version — never into raw file bytes (which may be binary) and never token-model-dependent. Every node created by ingestion carries provenance.spans: [(source_id, source_version, start, end)].

The gdpr-erasure spec (2026-07-02) redacts canonical blobs in place and blocklists retired digests. Consequences for this spec, all as-built in PR-081:

Blob redaction records + content-address downgrade. A masked canonical blob keeps its blob_id/canonical_text_hash as an addressing key only; a blob redaction record row lives at reserved chunk index u64::MAX in the sources CF (composite-key-layout addendum) with {ranges, ledger refs, post_redaction_hash}. From then on every integrity check (consistency checker, claim verification, restore drills) verifies a blob carrying a record against post_redaction_hash, not against its address. The canonical-slice read API sets redacted: true with ledger refs when the requested range intersects the record; a whole-blob deletion tombstone (the whole_sources/OQ-1 path) surfaces contentErased: true.
Blocklist on ALL dedup paths. The erased-content blocklist (tenant-scoped, keyed hmac_blake3_v1 digests) is checked BEFORE matching on every dedup path this spec defines: request-level (name, content_hash) dedup, submit-time canonical-hash dedup (single-shot and at streamed finalize), and the §17 child-enqueue dedup (the shared-attachment collapse and the code worker's incremental hash-skip). A hit refuses with INGEST_ERASED_CONTENT (REST 409 + ledger ref); on the child path the child is refused and logged while the parent proceeds.
content_b64 surface note. The jobs-CF IngestJobPayload.content_b64 carries full raw document bytes and is a first-class personal-data surface (gdpr-erasure §5 register row 11): terminal ingest jobs whose content is blocklisted are deleted; retained rows have content_b64 scrubbed for affected sources and payload name/event details identifier-masked.

4. Architecture¶

Worker parses raw → emits canonical text + span-annotated projection via SubmitIngestionResult → core writes source version, canonical text blob, span rows, and nodes/edges (spans embedded in provenance) transactionally in one batch chain per job.

5. Data Models¶

sources key per layout spec; value {source_id, uri/name, format, content_hash, canonical_text_hash, size, worker, job_id}. spans key [scope][sourceId][spanStart 8][spanEnd 8]; value {source_version_tx, linked_nodes: [logicalId], kind: paragraph|heading|cell|code|body} — the reverse index (span→nodes) complementing node-side provenance (node→spans). Canonical text stored in a blobs sub-space of sources CF, chunked at 1MB, addressed by hash (dedup for free).

6. API Contracts¶

POST /v1/ingest (multipart | url | inline json + format hint) → {job_id}; GET /v1/ingest/:jobId → status + resulting source_version on success. SubmitIngestionResult message (grpc-contracts spec) carries: canonical_text, spans[], nodes[] (with span refs by index), edges[], child_jobs[]. Span refs by array index within the submission keep the wire format compact; core resolves to absolute span keys.

7. UI/UX¶

Trace viewer (Phase 3) renders span text by slicing canonical text — GET /v1/sources/:id/versions/:v/text?start&end serves slices (tenant-scoped, range-limited). telha debug span <tenant> <source> <start> <end> prints the slice + linked nodes.

8. Configuration¶

ingest.max_upload (default 200MB), ingest.canonical_chunk (1MB), per-format size ceilings (worker specs may tighten).

9. Alternatives Considered¶

Token-index spans (rejected: tokenizer-dependent, breaks on model change; bytes are forever). Spans into raw file bytes (rejected: meaningless for PDF/binary; canonical text is the shared coordinate system all six formats can produce). Storing canonical text in S3 only (rejected for core: verification path must not depend on app-layer blob store; app layer may additionally archive originals per data-split-routing spec). Line/column spans (rejected: normalization ambiguity; byte offsets are exact).

10. Implementation Approach¶

PR-032: this spec → ingest endpoint + job creation → source/blob/span writes wired into Submit → dedup + linkage integrity tests. Worker specs (docling/web/tabular/code/email) define how each format produces canonical text + spans.

11. Migration Path¶

Greenfield. Canonical-text extraction changes in a worker produce a new source version (new canonical_text_hash) — old spans remain valid against their pinned version forever.

12. Success Metrics¶

Dedup test: byte-identical re-ingest creates zero new rows. Round-trip: every emitted span slices to non-empty text; every ingested node's provenance spans resolve. Reverse-index consistency: span→nodes and node→spans agree (checker tool). Verification readiness: PR-052's matcher consumes spans with no adapter layer.

13. Open Questions¶

OQ-1: Should canonical text be normalized (NFC, newline canonicalization) before hashing/spanning? Stance: yes — NFC + LF-only, defined here as normative; workers must apply it so offsets are cross-worker stable.

14. Changelog¶

2026-07-02 — v0.1: initial draft for peer review.
2026-07-03 — v0.2: (1) IngestJobPayload gained a pass-through options object (REST options → job payload → worker) for format-specific knobs (crawl bounds, tabular header/refs, json policy). (2) Submissions gained stats (surfaced as a "stats" event on job completion), contentHash (hex BLAKE3 of fetched bytes — fetch formats), and childJobs (crawl fan-out; core enqueues). (3) Submit-time dedup: latest version with the same content hash AND canonical hash ⇒ zero new rows (same-job retries exempt so crash recovery re-applies); fetch formats (web) skip request-level dedup entirely. (4) Failure taxonomy on the wire: IngestionResult.permanent dead-letters immediately. (5) §15 addendum: the structured-JSON direct path (PR-037).
2026-07-03 — v0.3: multi-submit streaming (§16 addendum). Submissions gained seq/partial; source versions gained an optional blob_id (provisional blob addressing for streamed canonical text); the job row gained a lease-scoped continuation checkpoint. Wire semantics in the grpc-contracts spec v0.2.
2026-07-04 — v0.4: deterministic worker identity + inline child content (§17 addendum, PR-054/055 enablers): submission nodes gained optional id with upsert semantics; childJobs entries gained optional contentB64; child enqueue applies request-level dedup for non-fetch formats (the code worker's incremental hash-skip).
2026-07-05 — v0.5: shared attachment documents (§18 addendum; OQ-3 of the source-connector-framework spec, ratified 2026-07-04, landing with PR-067). The email worker's attachment child-job names are now content-hash-derived (attachment:{blake3_hex} — previously the message-scoped {message_key}/{filename}), so §17's (name, content-hash) child dedup collapses repeated attachment bytes into ONE shared source/DOCUMENT across messages and mailboxes. No core change — the sharing falls out of name-derived child source identity.
2026-07-06 — v0.6: GDPR erasure interaction (§3.19 addendum, PR-081). Blob redaction records downgrade the content address to addressing-only (integrity checks verify post_redaction_hash); the erased-content blocklist is checked on all dedup paths (request-level, submit-time, child-enqueue) refusing with INGEST_ERASED_CONTENT; content_b64 is flagged as a first-class personal-data surface. No change to ingest identity or dedup semantics for non-erased content.

15. Addendum — Structured JSON Direct Path (PR-037)¶

format=json skips the worker fleet: core parses, projects, and applies in the ingest call. The REST response is final (201 with sourceId, sourceVersionTx, nodeIds, edgeIds) — no job to poll. The stored source version records worker = "core:json-bypass" and the nil UUID as job_id (no job row exists). Request-level content-hash dedup applies as for any non-fetch format.

Input shape. One JSON object (one record) or an array of objects (batch, ≤ ingest.json_max_records, default 10 000). Anything else — scalars, arrays of scalars at top level, mixed arrays anywhere — is INVALID_CONTENT.

Label. options.label (sanitized) or UPPER_SNAKE of the request name minus its extension (the tabular sanitization rule).

Canonical text + spans. NDJSON: one compact serde_json rendering per record, LF-joined; each record's line is its span (kind body). Every node projected from a record — root and children — carries that record's span. NFC is not re-applied (serde output is already UTF-8; offsets are computed against exactly the stored bytes).

Projection. Scalars and arrays-of-scalars become properties via the DSL value rules (datetimes have no JSON form — they arrive as whatever the JSON says, typically strings or µs ints; schema inference applies via the normal NodeOps path). Nested objects follow the policy — ingest.json_nested config, overridable per request via options.nested:

flatten (default): dotted keys (address.city), recursing to ingest.json_max_depth (default 8). A dotted-key collision with a literal key is INVALID_CONTENT (fail fast, never last-wins).
child: the nested object becomes a child node labeled UPPER_SNAKE(key), linked by a HAS_<KEY> edge; recursion applies the policy again below it.

Arrays of objects are ALWAYS child nodes with HAS_<KEY> edges carrying ord — under either policy (there is no meaningful flattening of an object array). Parents precede children in submission order, so node_ids[0] is the first record's root.

16. Addendum — Multi-Submit Streaming (v0.3)¶

A worker may stream one job's parse as several submissions (seq + partial on the Submission JSON; wire protocol in the grpc-contracts spec §5). Storage semantics:

One source version per stream. The first partial pins the version: it writes the source-version row (at a fixed txTime, canonical_text_hash absent) and allocates a provisional blob id recorded in the row's new optional blob_id field. Every partial appends its canonical-text bytes as chunks under that blob id (chunk sizes are per-partial, ≤ 1MB; concatenation in chunk-index order reproduces the text — readers must use blob_id when present and fall back to hash-derived addressing otherwise). Spans and node provenance carry GLOBAL byte offsets into the accumulated text and all reference the pinned version tx.

Finalize. The first partial=false submission appends its (possibly empty) tail, hashes the full canonical text by streaming the stored chunks through BLAKE3, and rewrites the version row with canonical_text_hash (and the fetch-format contentHash override, if any). Streamed chains skip submit-time dedup — the canonical hash is unknown until finalize; request-level dedup still covers inline formats.

Continuation + crash recovery. Between partials the core checkpoints an IngestContinuation (expected seq, pinned tx, blob id, accumulated length, next chunk, last partial's node ids) on the leased job row — fenced by worker + lease generation, and the checkpoint extends the lease. A worker crash reaps the job; re-leasing bumps the generation and clears the continuation, so the retry starts a clean stream (new pinned version). The abandoned version row (no canonical hash) and its nodes/spans are orphaned — the same posture as a crash mid single-shot batch chain, made more visible: orphaned versions are identifiable by canonical_text_hash = None on a non-latest version (cleanup tooling is a recorded follow-up alongside idempotency keys). Retrying the previously acknowledged seq after a lost ack is an idempotent replay: the stored node ids return, nothing is written. A core crash between applying a partial and its checkpoint loses that partial's ack — the worker's retry of that seq then re-applies it (duplicate rows possible in that window; identical to the single-shot retry posture).

17. Addendum — Deterministic Node Ids & Inline Child Content (v0.4, PR-054/055)¶

Workers are tenant-blind and cannot read the graph, yet email PERSON / MESSAGE and code FILE / declaration nodes need identity that survives across jobs. Two additive submission fields close the gap; both are rejected loudly by older cores (deny_unknown_fields), so workers and core deploy together.

nodes[i].id (optional UUID string). A client-supplied deterministic logical id — workers mint uuid5 values (email: person-email:{addr} / email-msg:{msgid} under NAMESPACE_URL; code: code-file:{repoNs}/{path} etc. under uuid5(NAMESPACE_URL, "telha:code-worker"); the exact conventions live in each worker's spec changelog and the Entra connector spec must adopt the person convention). Upsert semantics, enforced under the node write lock (NodeOps::upsert_node):

no existing version → created under this id;
latest version live with identical labels+properties → matched: nothing written, but the submission position still resolves to the id, so relationship refs and span reverse-index rows link it (a new mention adds span linkage without version churn);
latest version live and different → a new version via the normal update path (append-only closure);
latest version tombstoned → skipped: ingestion never resurrects an erased entity (GDPR posture). Span rows may still reference the id; reads validate.

Edges to not-yet-ingested deterministic ids are permitted (EdgeOps does not validate the far node): they dangle invisibly — every read path validates endpoints — and self-heal when the target arrives. This is the email worker's pending-reply mechanism and the code worker's cross-file import mechanism, and it is why the compare engine's endpoint-presence check (snapshot-compare spec) is load-bearing rather than an assertion.

childJobs[i].contentB64 (optional base64 bytes). Inline content for the child job (attachment fan-out, per-file code fan-out). Without it the child's name remains its content (URL-identity, crawl fan-out). The child's stored content_hash is the BLAKE3 of the decoded bytes, and child enqueue now applies the same request-level dedup as POST /v1/ingest for non-fetch formats: a child whose (name, content hash) already has an ingested source version is skipped outright — zero jobs, zero writes. That IS the code worker's unchanged-file incremental skip and the email worker's cross-mailbox attachment no-op.

18. Addendum — Shared Attachment Documents (v0.5, PR-067)¶

Child-job source identity is NAME-derived (source_id_for(name) on the §17 enqueue path), so the (name, content-hash) request dedup shares a source only when two children collide on the SAME name. The email worker's original naming ({message_key}/{filename}) was message-scoped: the same PDF attached to ten messages produced ten sources and ten DOCUMENT projections. Per OQ-3 of the source-connector-framework spec (ratified 2026-07-04: repeated Exchange attachments share a DOCUMENT by content hash), attachment child jobs are now named

attachment:{blake3_hex(attachment_bytes)}

One DOCUMENT per distinct byte content. The first message to carry the bytes enqueues the child and creates the source; every later re-attachment — re-send, forward, cross-mailbox, renamed file — hits the §17 dedup (same name ⇒ same source id; same content hash by construction) and is skipped outright: zero jobs, zero writes. This holds for ANY email ingress (Exchange connector, manual .eml/.msg upload), not just PR-067's sync path.
Per-message context stays message-local. Each message keeps its own ATTACHMENT_REF node (id email-att:{message_key}:{index}, unchanged) carrying filename (as sent on THAT message), content_hash, and child_source_name; HAS_ATTACHMENT edges keep position. All ATTACHMENT_REFs for the same bytes carry the same child_source_name and therefore resolve to the one shared document.
Filename is display metadata, not identity. The shared source's stored name is the opaque attachment:{hash} string; human-readable naming comes from the referencing ATTACHMENT_REF(s). Format routing is unaffected: the child's format field is explicit (magic-sniff / MIME / extension precedence in the email worker) and nothing routes on the name's extension.
"Changed item ⇒ new version" is vacuous here. A same-name child with different bytes would require a BLAKE3 collision; attachment sources are effectively immutable, one version each.

Sources created under the retired {message_key}/{filename} naming remain valid (identity is append-only); they simply stop accumulating versions, and re-syncs of those messages converge on the hash-named shared source.