Spec: Docling Worker — Rich Document Projection¶

File: .ai/specs/core-engine/2026-07-02-docling-projection.md Status: Draft Owners: Workers lane Related: PR-034; ingestion-provenance spec (canonical text + spans); job-queue-leases spec

1. Overview¶

Defines how PDF/DOCX/PPTX become graph structure: the hierarchical node projection, span emission rules, table handling, page/bbox metadata, and the failure taxonomy for the docling-based worker.

2. Problem Statement¶

Rich documents are the highest-value, messiest ingest format. Without a pinned projection: two runs of the worker produce different graph shapes; tables collapse into text soup; provenance loses page anchoring (operators need "page 12" not just byte 48213); and failures (corrupt file vs OOM vs unsupported feature) are indistinguishable, defeating retry policy.

3. Proposed Solution¶

Docling parses to its document model; the worker projects deterministically: one DOCUMENT node; SECTION nodes per heading level; PARAGRAPH nodes per block; TABLE nodes with per-row TABLE_ROW children (cells as row properties keyed by header); CONTAINS edges parent→child in reading order (edge property ord). Canonical text = reading-order concatenation of all text blocks (NFC, LF), spans per block. Every node carries page and bbox metadata where docling provides them. Failures map to the typed taxonomy: corrupt_input (dead-letter immediately, attempts irrelevant), unsupported_feature (dead-letter with feature tag), resource_limit (retryable — may succeed on a larger worker), transient (retryable).

4. Architecture¶

PollJobs(kind=ingest:richdoc) → fetch raw from job payload reference → docling parse (subprocess with memory ceiling) → projection → SubmitIngestionResult(canonical text, spans, nodes, edges). Stateless; all state in the job.

5. Data Models¶

Node labels: DOCUMENT, SECTION, PARAGRAPH, TABLE, TABLE_ROW. Properties: title|text, level (sections), page, bbox [x0,y0,x1,y1], ord. Edges: CONTAINS with ord. Spans: one per PARAGRAPH/heading/cell-row, kind mapped per ingestion-provenance spec. Determinism rule: identical input bytes ⇒ identical projection (node count, ordering, spans) — projection uses only document content, never wall-clock or randomness.

6. API Contracts¶

Consumes/produces exactly the WorkerService messages (grpc-contracts spec). Feature support matrix v1: text, headings, lists (as paragraphs with list_level), tables, page metadata. Explicitly unsupported v1 (→ unsupported_feature only if they are the sole content, else skipped with a warning stat): embedded images OCR, forms, footnote resolution.

7. UI/UX¶

Not applicable beyond trace viewing (page/bbox let the app layer deep-link into rendered PDFs later). Worker exposes /health for orchestration.

8. Configuration¶

docling.max_pages (2000), docling.mem_ceiling_mb (4096, enforced via subprocess rlimit), docling.timeout_s (600).

9. Alternatives Considered¶

Marker as parser (deferred: docling's structured model maps cleaner to the projection; Fetcher-style swappability not needed here yet — revisit if docling quality gaps appear). Cells as individual nodes (rejected: node explosion for wide tables; row-granularity with keyed properties keeps queries sane — tabular-projection spec handles data-table semantics properly). OCR pipeline for scanned PDFs (deferred to a Phase-4 worker; v1 emits unsupported_feature for image-only pages).

10. Implementation Approach¶

PR-034: this spec → worker on workers_common harness → projection per §5 → failure taxonomy → golden corpus (5 fixtures with expected node/edge/span counts) + large-doc rlimit test.

11. Migration Path¶

Projection changes ⇒ new canonical output for the same input ⇒ handled naturally as a new source version on re-ingest (ingestion-provenance spec); no in-place rewrites ever.

12. Success Metrics¶

Golden corpus byte-stable across runs and platforms. Determinism test (same input twice, identical submission hash). rlimit test: oversized doc fails as resource_limit, retryable, without OOM-killing the harness. Span slice audit: every span resolves to the text of its block.

13. Open Questions¶

OQ-1: Should heading hierarchy infer levels when documents misuse styles (all Heading1)? Stance: v1 trusts document structure; a repair heuristic is a projection change requiring spec amendment.

14. Changelog¶

2026-07-02 — v0.1: initial draft for peer review.
2026-07-03 — v0.2 (PR-034 as built): (1) failure taxonomy carried on the wire as IngestionResult.permanent (proto field 5) + failure_reason prefixed with the kind; core dead-letters permanent kinds immediately. (2) Parse isolation = subprocess with taxonomy exit codes (0/10/20/30/other → ok/corrupt/unsupported/resource/transient); rlimit(RLIMIT_AS) applied on POSIX only — on Windows the timeout_s ceiling is the backstop. (3) Worker package depends on docling-core only (document model); the torch-backed parse stack is the parse extra — golden corpus is therefore five committed DoclingDocument fixtures (schema-valid via docling-core constructors, tests/gen_fixtures.py) rather than raw PDFs; real-parse determinism test is gated on docling availability. (4) Skipped-feature warnings (pictures/formulas/footnotes) flow via a new stats field on the submission, surfaced as a "stats" event on the completed job. (5) Table header rule: headerless columns get positional col_<i> names (aligned with the tabular spec).
2026-07-05 — v0.3: temporal ambiguity detection (§15 addendum, PR-061; temporal-clarification spec §5/§6 is the wire + lifecycle authority). Document-level detection targeting the DOCUMENT node: role-dated mention conflicts, anchored relative effectiveness phrases, undated documents. Emission gated on the options.clarify capability flag; no severity hints in v1 (core policy owns severity).

15. Addendum — Temporal Ambiguity Detection (PR-061)¶

Detection rules for the ambiguities submission field (temporal-clarification spec §5 owns the wire shape, lifecycle, and binding policy; this addendum owns only the docling heuristics). Emission is gated on the core-advertised capability flag (options.clarify == true in the job payload options); without it the worker emits nothing. The worker never assigns severity: no severityHint is emitted in v1 — kind→severity elevation is tenant policy ([clarify.severity]).

Scope. Detection is document-level and targets the DOCUMENT node (nodeRef 0). At most ONE ambiguity per submission (core enforces one per node in v1); when multiple rules fire, priority is conflicting_dates > relative_date > undated and the drop is counted in the ambiguities_suppressed stat. Detection runs over the same block texts the projection spans — every candidate's spanRef is the span of the block containing its evidence, so RBAC-gated question rendering shows exactly the sentence the date came from.

Date recognition (shared helper workers_common.temporal). Absolute dates only, deterministic, stdlib-only: ISO YYYY-MM-DD, and English textual forms D Month YYYY / Month D, YYYY (full and 3-letter month names, optional ordinal suffix, case-insensitive), all validated as real calendar dates and converted to µs-since-epoch at UTC midnight. Numeric slash/dash forms (1/2/2026) are deliberately NOT recognised here — they are inherently reading-ambiguous and belong to the tabular worker's cell rules (tabular spec §15).

Rule 1 — conflicting_dates. A role-dated mention is a date match preceded within 48 characters (same block) by a role keyword. Role lexicon with precedence (highest wins): effective/with effect from/takes effect/commencing/valid from (5), as of (4), executed (3), signed/signature date (3), dated (2), countersigned (1). If role-dated mentions carry ≥ 2 distinct dates, emit conflicting_dates: one candidate per distinct date labeled "YYYY-MM-DD (role)" (its highest-precedence role), ordered by precedence descending then date ascending, capped at 4 candidates. validFrom = date, validTo absent. detectorConfidence 0.7, appliedCandidateConfidence 0.55.

Rule 2 — relative_date. An effectiveness marker (effective|with effect|takes effect|starting|commencing|begins) within 32 characters before a relative unit phrase ((the )?(next|following) (month|quarter|year)). Anchor = the highest-precedence role-dated mention in the document (ties: first in reading order), else the first date mention in reading order. Anchored: candidates = [start of the next calendar unit strictly after the anchor ("YYYY-MM-DD (start of next quarter after YYYY-MM-DD)", quarters are calendar quarters Jan/Apr/Jul/Oct, UTC), the anchor itself ("YYYY-MM-DD (document date)")]; detectorConfidence 0.85, appliedCandidateConfidence 0.5. Unanchored (no absolute date anywhere): ONE interval-less diagnostic candidate labeled with the matched phrase + "(no anchor date in document)", appliedCandidateConfidence 0.0 (core applies no interval; the card offers only free-form).

Rule 3 — undated. No absolute date match anywhere in the canonical text AND canonical text ≥ 200 bytes (trivial fragments are not worth a human's tap): ONE interval-less diagnostic candidate "No date found in document", no spanRef, detectorConfidence 0.9, appliedCandidateConfidence 0.0.

Stats. ambiguities (count emitted, 0 or 1), ambiguities_suppressed (rules that fired but lost the priority pick). Determinism rule of §5 applies unchanged: detection uses only document content — never wall clock (relative phrases anchor to document dates, not to "today").