Skip to content

Provenance to the span

Concept 3 of 8

Every fact Telha derives from a document links back to the exact byte range of the exact source version it came from, not just "this document" or "this paragraph." That precision is what makes generated answers auditable and what lets a correction happen without breaking the citations that already point at the old text.

Normative spec

This concept is defined by ingestion-provenance (source identity, the span ledger, node-to-span linkage) and consumed by claim verification, whose evidence unit is the span.

What a span is

A source version is one ingested copy of a document: identity is (tenant, source_id), versioned by transaction time, and carrying a content_hash (BLAKE3 of the raw bytes) plus the canonical extracted text for that version. A span is a half-open byte range, [start, end), into that canonical text: not into the raw file bytes (which may be binary, like a PDF), and not tied to any particular tokenizer.

Every node or edge created by ingestion carries provenance.spans: [(source_id, source_version, start, end)]. The span ledger also keeps the reverse index, span to nodes, so a click on a piece of source text can answer "what facts did this produce" just as easily as a fact can answer "what text produced me."

flowchart LR
    subgraph source["Source version"]
        TXT["Canonical extracted text<br/>(BLAKE3 content_hash)"]
    end
    subgraph spans["Span ledger"]
        S1["span [412, 498)<br/>kind: paragraph"]
    end
    subgraph facts["Fact version"]
        N["NodeVersion: CLAUSE<br/>provenance.spans: [(source_id, v, 412, 498)]"]
    end
    TXT -->|byte offsets| S1
    S1 <-->|reverse index| N

Bytes, not tokens, were chosen deliberately: token spans break the moment the embedding or tokenizer model changes, and offsets into raw file bytes are meaningless for a binary format like PDF. Canonical text is the one coordinate system all ingestion formats (PDF, spreadsheets, email, web pages, code, JSON) can produce, so it is the one Telha spans against.

Source-record time travels with the span

Provenance is not just "which bytes." It also carries source-record time, the date the source document itself asserts (see the tritemporal model). A scan dated December 28 pins its facts to that date even if Telha does not learn about the fact until February. Provenance and source-record time together are what let a citation say not just "this came from this document" but "this came from this document, dated this way, at these exact bytes."

Provenance survives corrections

Ingesting a changed version of a document does not overwrite the old source version or invalidate its spans. A new document version gets its own source_id/version pair and its own canonical text and spans; the old version's spans keep resolving to the same bytes they always did. This matters because facts derived from the old version remain exactly as citable as they were the day they were created, even after a newer version of the same document exists.

The same append-only discipline applies when a fact itself is corrected: a correction writes a new fact version, and that new version can carry its own, different provenance (for example, a clarification answer becomes the new provenance for a corrected fact), while the superseded fact version and its original provenance remain queryable at their original transaction time.

Where this shows up in the product

  • Trace click-through. The verified answers loop scores every claim against planned spans, and the trace viewer renders each claim's best-matching span by slicing the canonical text and highlighting it, so you can see exactly which words justified a sentence in a generated answer.
  • Deep links back into source systems. A fact's provenance identifies which source system it came from, so an answer can link back into the original system (a SharePoint document, an Exchange message, a Salesforce record) behind that system's own login, rather than exposing the text out of context.
  • telha debug span. Operators can pull the exact slice and its linked nodes from the command line for any (tenant, source, start, end), which is the same mechanism the trace viewer uses.
  • GDPR-aware redaction. When source content is erased, the span ledger degrades gracefully rather than silently: a masked canonical blob is still addressable, integrity checks verify it against a recorded post-redaction hash, and reads over a redacted range come back flagged redacted: true instead of either failing or returning stale plaintext.

Spans are the evidence unit, not documents or paragraphs

Claim verification, described in Verified answers, scores generated claims against spans specifically because spans are precise enough to distinguish "this sentence is supported" from "this document is generally relevant." Document- or paragraph-level provenance could not make that distinction.