Skip to content

Spec: Web Worker — Fetcher Abstraction & Web/SPA Projection

File: .ai/specs/core-engine/2026-07-02-web-ingestion.md Status: Draft Owners: Workers lane Related: PR-035; F5 ratified (swappable Fetcher, Playwright fallback); ingestion-provenance spec

1. Overview

Defines the Fetcher abstraction that makes the web-rendering backend swappable (Firecrawl hosted API vs built-in Playwright), the crawl policy, recrawl semantics, and the Markdown→graph projection shared identically by both backends.

2. Problem Statement

F5 established the licensing/cost constraint: Firecrawl self-host is AGPL, hosted is paid. Beyond that: web content changes under our feet (recrawl semantics must be defined), crawling has legal etiquette (robots), and two rendering backends will drift into two different graphs unless projection is backend-independent by construction.

3. Proposed Solution

A Fetcher protocol — fetch(url, opts) -> {markdown, title, canonical_url, fetched_at, links[], meta} — with FirecrawlFetcher (API client) and PlaywrightFetcher (headless Chromium + readability extraction + html→markdown) implementations selected by config. Everything downstream of the Fetcher return value is one shared code path, so backend-equivalence is structural. Recrawl of the same URL = new source version (content-hash dedup applies per ingestion-provenance spec). v1 scope: single-URL jobs plus bounded same-origin crawl (depth ≤ 2, page cap 50) when requested; robots.txt honored (fetch-and-cache per origin, 1h TTL); no auth-walled content.

4. Architecture

PollJobs(kind=ingest:web) → robots check → Fetcher.fetch → shared projection → SubmitIngestionResult; discovered same-origin links within crawl bounds enqueue child jobs (child_jobs field) rather than looping in-worker (keeps jobs small, leases honest).

5. Data Models

Node labels: WEB_PAGE (title, canonical_url, fetched_at), SECTION per heading (level), PARAGRAPH per block. Edges: CONTAINS (ord); REFERENCES from WEB_PAGE/PARAGRAPH to target URL — targets within the same ingestion resolve to their WEB_PAGE node; external targets become lightweight URL_REF nodes (dedup per tenant by normalized URL). Canonical text = the markdown (NFC, LF); spans per block per ingestion-provenance rules. URL normalization: lowercase host, strip fragments, strip tracking params (utm_*, fbclid, gclid), preserve everything else.

6. API Contracts

Fetcher protocol as §3 (Python Protocol, both impls tested against one contract suite). Job payload: {url, crawl: {depth, max_pages} | null, fetcher_override?}. Failure taxonomy: fetch_denied (robots/4xx auth — dead-letter), fetch_transient (5xx/timeout — retryable), render_failed (retryable once, then dead), resource_limit (retryable).

7. UI/UX

None beyond standard job status. Worker /health includes per-backend reachability (Firecrawl API ping / Chromium launch check).

8. Configuration

web.fetcher (playwright default — zero external dependency out of the box; firecrawl opt-in with API key), web.crawl.max_depth (2), web.crawl.max_pages (50), web.robots_ttl (3600s), web.timeout_s (60), web.user_agent (identifies Telha, contact URL).

9. Alternatives Considered

Firecrawl self-host (rejected default: AGPL isolation is arguable over the network but the operational burden plus license review per F5 makes Playwright the safer shipped default; hosted API remains a config away). Raw HTTP + readability without headless browser (rejected as sole path: SPAs render empty; kept internally as PlaywrightFetcher's fast path when no JS detected). Unbounded crawling (rejected: scope creep into a crawler product; depth-2/50-page bounds cover "ingest these docs pages" without becoming Heritrix).

10. Implementation Approach

PR-035: this spec → Fetcher protocol + both impls → shared projection → fixture-server crawl test → backend-equivalence test: both fetchers against the same fixture site must produce identical projections (the F5 gate).

11. Migration Path

Greenfield. Adding a third backend = new Fetcher impl passing the contract + equivalence suites; no projection changes.

12. Success Metrics

Backend-equivalence test green. Robots test: disallowed path never fetched. Recrawl test: changed page ⇒ new source version; unchanged ⇒ dedup no-op. Link-graph test: internal REFERENCES resolve to WEB_PAGE nodes, external to deduped URL_REFs.

13. Open Questions

  • OQ-1: JS-heavy sites behind consent walls — auto-dismiss common consent frameworks or record fetch_denied? Stance: v1 records fetch_denied with reason=consent_wall; auto-dismissal is a legal/etiquette decision to take explicitly later.

14. Changelog

  • 2026-07-02 — v0.1: initial draft for peer review.
  • 2026-07-03 — v0.2 (PR-035 as built): (1) PlaywrightFetcher's fast path is plain HTTP + readability + markdownify; Chromium escalates only on an unrendered-SPA heuristic (short extraction + script tags). (2) Projection consumes ONLY markdown/title/canonical_url — fetcher links feed crawl discovery, never the graph — so backend equivalence is structural; the offline F5 gate runs both fetchers against a local fixture site (the Firecrawl side via an API emulator that shares the html→markdown helper). (3) URL_REF dedup is per-submission in v1, not per-tenant (workers cannot query; per-tenant dedup needs deterministic node identity/upsert — deferred). Likewise cross-page REFERENCES resolve to URL_REF, self-links to the WEB_PAGE node. (4) Recrawl semantics as specced via two core changes: fetch formats skip request-level dedup, and submissions carry an optional contentHash (hex BLAKE3 of fetched markdown) driving submit-time dedup against the latest version (same-job retries exempt so crash recovery re-applies). (5) Crawl fan-out: submission childJobs[{format,name,options}] enqueued by core; the page budget (maxPages) is split deterministically across same-origin links in document order, each child at depth-1. Job options flow REST options → job payload options → worker ({url?, crawl:{depth,maxPages}, fetcherOverride?}); for format=web the REST content field may be omitted (URL is the identity). (6) robots.txt: 401/403 on robots itself = deny-all (RFC 9309), other 4xx = allow, unreachable = allow; disallow ⇒ fetch_denied (permanent).