Skip to content

Spec: Email Worker — Envelope Projection & Attachment Fan-out

File: .ai/specs/core-engine/2026-07-02-email-projection.md Status: Approved Owners: Workers lane Related: PR-055; ingestion-provenance spec; job-queue-leases spec (child jobs); field-encryption spec (addresses may be classified per app-layer policy)

1. Overview

Defines .eml/.msg projection: person/message node taxonomy, header-derived edges, thread reconstruction, body/quote handling, attachment fan-out into child ingestion jobs, and dedup by Message-ID.

2. Problem Statement

Email is a graph pretending to be documents: the value is in who-said-what-to-whom-when, threads reconstructed across messy clients, and attachments that are themselves first-class documents. Without pinned rules: the same message ingested from two mailboxes duplicates; quoted text pollutes spans; threading breaks on missing In-Reply-To; and attachments vanish into properties.

3. Proposed Solution

Nodes: MESSAGE (subject, message_id, sent_at → validTimeStart, sourceRecordTime = ingest time), PERSON (keyed per tenant by normalized address; display names accumulate as a property set), ATTACHMENT_REF (filename, mime, hash, child_job_id). Edges: SENT_BY (message→person), TO / CC (message→person, ord), IN_REPLY_TO (message→message when the referenced Message-ID exists in-tenant; else stored as pending-ref property, resolved opportunistically when the parent later arrives), HAS_ATTACHMENT. Threading fallback: References header chain, then subject-normalization + time-window heuristic emitting resolution: heuristic edges (mirroring code-projection honesty). Body: text/plain preferred, html→text fallback; quoted-reply blocks detected and span-kinded quote (excluded from embedding by default — vector-storage kind handling), fresh content span-kinded body. Dedup: (tenant, Message-ID) — re-ingest of a known id with identical content hash is a no-op; different hash (client mangling) becomes a new source version of the same source.

4. Architecture

PollJobs(kind=ingest:email) → parse envelope → project message/person/edges → extract attachments → child_jobs entries (kind by sniffed format, payload = attachment bytes ref + parent linkage) → Submit. Child completion links ATTACHMENT_REF → child's DOCUMENT node via a linkage job step.

5. Data Models

PERSON identity: lowercase address, plus-tag stripped optionally (config, default keep tags — they carry meaning in ops teams). Canonical text = headers block + fresh body + quoted blocks in order (NFC/LF); spans per block. MESSAGE validTime end open; corrections/recalls are new versions like everything else.

6. API Contracts

Job payload: {format: eml|msg, bytes_ref, mailbox_ns?}. Result stats: persons created/merged, thread edges (resolved vs heuristic vs pending), attachments spawned. Failure taxonomy: corrupt_input, unsupported_encoding (dead with tag), resource_limit, transient.

7. UI/UX

Downstream chat: "show the thread where the Q2 delay was first raised" rides IN_REPLY_TO traversal + semanticSearch; no worker-specific surface.

8. Configuration

email.max_message_mb (50), email.max_attachments (50), email.strip_plus_tags (false), email.heuristic_thread_window_h (168).

9. Alternatives Considered

Global cross-tenant person identity (rejected instantly: tenant isolation is absolute; identical addresses in two tenants are two nodes). Attachments inline as properties (rejected: attachments are documents deserving their own provenance/spans; fan-out reuses every existing worker). Dropping quoted text (rejected: quotes carry evidentiary value — "he was told on May 3rd" — kinded spans keep them retrievable without polluting embeddings by default).

10. Implementation Approach

PR-055 commits as planned: this spec → envelope parsing + projection → threading (resolved/pending/heuristic tiers) → attachment fan-out (proto child_jobs field, buf-checked) → thread-reconstruction fixtures + email→PDF→docling fan-out e2e.

11. Migration Path

Greenfield. Pending-ref resolution is idempotent and retroactive: late-arriving parents attach without rewriting children (new edge versions only).

12. Success Metrics

Thread fixtures: mangled real-world corpora reconstruct expected trees, heuristic edges correctly tagged. Dedup: cross-mailbox same-id ingest yields one MESSAGE. Fan-out e2e: attachment PDF lands as full docling projection linked via ATTACHMENT_REF. Quote spans excluded from embedding assembly (adversarial check with vector-storage tests).

13. Open Questions

  • OQ-1: Calendar invites (.ics parts) — project as EVENT nodes v1 or treat as attachments? Stance: attachments v1; EVENT projection is a small follow-up spec once demand is real.

14. Changelog

  • 2026-07-05 — v0.4: shared attachment documents (PR-067; OQ-3 of the source-connector-framework spec, ratified 2026-07-04; normative mechanics in the ingestion-provenance spec §18 addendum). Attachment child-job names are now CONTENT-HASH-derived — attachment:{blake3_hex(bytes)} — replacing v0.2 pin 6's message-scoped {message_key}/{filename|attachment-<i>}: core's (name, content-hash) child dedup then collapses the same bytes attached to any number of messages/mailboxes into one shared source/DOCUMENT. ATTACHMENT_REF ids, its filename/content_hash properties, and child_source_name linkage are unchanged (the value is now the hash-derived name, shared by every message carrying the bytes); the per-message filename remains display metadata on the ref node.
  • 2026-07-05 — v0.3: temporal ambiguity detection (§15 addendum, PR-061; temporal-clarification spec §5/§6 is the wire + lifecycle authority). Envelope-date integrity only: Date-header vs transport-receipt skew, missing/unparseable Date headers. Body-content date detection is explicitly deferred (needs fact-node extraction the email projection does not do). Gated on options.clarify; no severity hints in v1.
  • 2026-07-04 — v0.2: as-built (PR-055). Deviations and pinned conventions:
  • Single sniffed job format. One format email (jobs kind ingest:email), job content = raw message bytes; .msg vs .eml is sniffed by magic (OLE compound header D0 CF 11 E0 A1 B1 1A E1 → .msg, else .eml) — replaces §6's {format: eml|msg, bytes_ref}. options.mailboxNs is accepted and surfaced as a stat only (never a node property — cross-mailbox dedup needs byte-stable props).
  • uuid5 id conventions (normative; email_worker/identity.py). Namespace = uuid.NAMESPACE_URL (6ba7b811-9dad-11d1-80b4-00c04fd430c8) for ALL of: PERSON uuid5(ns, "person-email:" + lowercase(address)) (the Entra spec §5 placeholder convention — **PR-062 MUST adopt this namespace
    • string format** or ALIAS_OF merging breaks); MESSAGE uuid5(ns, "email-msg:" + message_key) where message_key = Message-ID normalized (surrounding whitespace + one pair of angle brackets stripped, case preserved) or, when absent, "blake3:" + blake3(raw_message_bytes).hexdigest(); ATTACHMENT_REF uuid5(ns, f"email-att:{message_key}:{index}").
  • Self-healing dangling IN_REPLY_TO replaces pending-refs. The edge always targets the parent's deterministic MESSAGE id, even before the parent arrives; reads validate endpoints so the edge dangles invisibly and resolves the moment the parent node is created under that id — simpler than pending-ref properties and retroactive by construction (§11 satisfied). Stats report thread_edges: {resolved, pending_selfheal: 0 (by construction), heuristic: 0}.
  • Heuristic threading tier deferred. The worker sees one message per job and cannot query the graph, so subject+time-window heuristics are not implementable worker-side; needs graph read access or a core-side linkage pass (config key email.heuristic_thread_window_h reserved and parsed). subject_normalized (Re:/Fwd:/localized prefixes stripped, whitespace collapsed, lowercased) IS emitted as a MESSAGE property so a future pass can key on it.
  • PERSON display names live on edges, not the node. PERSON properties are {address} ONLY so the core upsert always matches (no version churn per message); each mention's display name rides the SENT_BY/TO/CC edge as displayName (TO/CC also carry ord). Deviates from §3 "display names accumulate as a property set" — accumulation needs read-modify-write core support; edge properties preserve every mention and stay queryable.
  • ATTACHMENT_REF carries child_source_name, not child_job_id. Job ids are assigned by core at enqueue, after the worker builds its result; linkage is by the deterministic child source name ({message_key}/{filename|attachment-<i>}), and child job options carry {parent: {messageId, attachmentRefId}}. An explicit ATTACHMENT_REF→DOCUMENT linkage step remains a recorded follow-up.
  • Quote exclusion mechanism (verified against core). Core embedding text assembles from NODE PROPERTIES (core-engine/src/vector/embed.rs::assemble_embed_text), so the MESSAGE body property carries FRESH text only (capped at 8192 bytes on a char boundary); quoted blocks exist solely as quote-kinded canonical spans (headers block = headers kind, fresh = body kind, blocks in original order). That property-level exclusion IS "quotes excluded from embedding by default".
  • Edge addressing. fromRef/toRef for nodes within the same submission (cheaper, span-linkage positions preserved); toId ONLY for cross-job targets (IN_REPLY_TO parents).
  • Attachment routing. Magic sniff first (%PDF-, OLE), then declared MIME, then extension: pdf/docx/pptx → docling; csv/xlsx → tabular; nested .eml/message/rfc822/.msg → email. text/html is skipped-with-stat (web is a fetch format with URL identity; docling does not poll html); unknown types are skipped-with-stat but still get ATTACHMENT_REF nodes (no child_source_name). Caps: max_attachments 50, max_message_mb 50 (ResourceLimit).
  • .msg parser = extract-msg (0.55). A realistic OLE .msg fixture proved impractical to construct offline: the .msg path is unit-tested by monkeypatching the extract-msg Message surface, plus a real-library corrupt-container test — the fixture gap is recorded here. Embedded .msg-in-.msg attachments (Message-object data, not bytes) are skipped in v1.
  • Failure taxonomy mapping. §6's unsupported_encoding maps to workers_common.failures.UnsupportedFeature (permanent, dead-letter); unparseable input → CorruptInput; size caps → ResourceLimit; everything else Transient.
  • 2026-07-02 — v0.1: initial draft for peer review.

15. Addendum — Temporal Ambiguity Detection (PR-061)

Detection rules for the ambiguities submission field (temporal-clarification spec §5 owns the wire shape, lifecycle, and binding policy; this addendum owns only the email envelope rules). Emission is gated on options.clarify == true in the job payload options. No severityHint in v1.

Scope. The MESSAGE node's valid time IS its sent time (§3), so the only temporal facts this worker can honestly question are envelope dates. Detection targets the MESSAGE node; spanRef = the headers span. Body-content phrases ("the new rate is effective next quarter") describe OTHER facts, not the message's validity — correcting MESSAGE valid time from them would be wrong. Body detection is deferred until a projection extracts fact nodes from bodies (recorded deviation, same reasoning as the v0.2 heuristic-threading deferral).

Transport evidence. Each Received: header's date is parsed from the text after its last ; (email.utils.parsedate_to_datetime, same µs convention as the Date header). The transport anchor is the EARLIEST parseable receipt (closest to origin; Received headers are prepended, so later relays sort above it).

Rules (first match wins; at most one ambiguity per message by construction):

  1. Date header parseable AND transport anchor exists AND |Date − anchor| > 48 h (constant DATE_SKEW_US; clock skew and mailbox-move relays sit well inside two days — beyond it one of the two is wrong) → conflicting_dates. Candidates: "<ISO> (Date header)" FIRST — the projection already applied the Date header as valid time, so the best guess and the applied interval stay consistent — then "<ISO> (earliest transport receipt)". detectorConfidence 0.8, appliedCandidateConfidence 0.6.
  2. Date header ABSENT, transport anchor exists → undated, one candidate "<ISO> (earliest transport receipt)" (core applies it, giving the MESSAGE the valid time the projection could not). detectorConfidence 0.85, appliedCandidateConfidence 0.7.
  3. Date header present but UNPARSEABLE, transport anchor exists → low_confidence_extraction, one candidate as rule 2. detectorConfidence 0.85, appliedCandidateConfidence 0.7.
  4. No parseable date evidence anywhere (Date absent OR unparseable, and no parseable Received) → undated, one interval-less diagnostic candidate "No date evidence in message headers", appliedCandidateConfidence 0.0, detectorConfidence 0.9. (The operative condition is no parseable evidence: an unparseable Date with no transport anchor lands here — rule 3 requires an anchor.)
  5. Date parseable and within skew of the anchor (or no Received headers at all — .msg extractions often lack them) → no ambiguity.

validFrom = the candidate date, validTo absent (MESSAGE validity is open-ended, §5). Stats: ambiguities (0 or 1). Determinism: only message bytes are consulted — never wall clock.