Spec: Code Worker — Tree-sitter AST Projection¶

File: .ai/specs/core-engine/2026-07-02-code-projection.md Status: Approved Owners: Workers lane Related: PR-054; ingestion-provenance spec; job-queue-leases spec

1. Overview¶

Defines how source code becomes graph structure via tree-sitter: node taxonomy, edge semantics (including the honesty limits of static call detection), the v1 language set, incremental re-ingest, and span rules.

2. Problem Statement¶

Code graphs oversell easily: "CALLS edges" implies whole-program analysis tree-sitter cannot do. Without pinned limits, consumers trust edges that are best-effort syntax matches; without incremental semantics, every repo push re-ingests everything; without deterministic projection, the same commit produces different graphs.

3. Proposed Solution¶

Per-file parsing (tree-sitter grammars: rust, python, javascript/typescript v1). Nodes: FILE (path, language, content_hash), MODULE (py/rs module semantics), FUNCTION, CLASS (incl. rs struct+impl grouping), METHOD. Edges: CONTAINS (file→decl, class→method, ord); IMPORTS (file→file when resolvable within the ingested set, else file→CODE_REF node carrying the raw import string); CALLS — explicitly best-effort: emitted only for same-file or same-ingestion resolvable simple identifiers, with edge property resolution: syntactic; dynamic dispatch, aliasing, and re-exports are out of scope and documented as such. Canonical text per file = the file itself (NFC, LF); spans = byte ranges of each declaration. Incremental: job payload carries per-file content hashes; unchanged files (hash match against latest source version) are skipped entirely.

4. Architecture¶

PollJobs(kind=ingest:code) → payload lists files (inline or archive ref) → per-file: hash check → parse → project → SubmitIngestionResult per batch of files (multi-submit like tabular).

5. Data Models¶

Node properties: name, qualified_name (best-effort per language), signature (raw text of the declaration line(s)), line_start/line_end (display convenience; spans remain byte-authoritative), visibility where the grammar exposes it. Each source file is its own source_id (path-keyed within a repo namespace from the job payload) so file history versions naturally.

6. API Contracts¶

Job payload: {repo_ns, files: [{path, hash, content? | archive_ref}], languages?}. Result stats: files parsed/skipped/failed, decls, unresolved_imports, calls_emitted. Failure taxonomy: parse_error per-file is a stat not a job failure (one broken file must not dead-letter a repo); job-level failures only for payload/resource problems.

7. UI/UX¶

None direct; downstream chat queries like "what changed in the parser module since March" ride standard compare + history on these nodes.

8. Configuration¶

code.max_file_kb (1024, larger files skipped-with-stat), code.max_files_per_job (5000), grammar version pins (grammar upgrades are projection changes — see §11).

9. Alternatives Considered¶

LSP/semantic indexers per language (rejected v1: heavy per-language infrastructure; tree-sitter's uniform CST covers structure and provenance; semantic resolution is a Phase-4 enrichment). Emitting unresolved CALLS to CODE_REF nodes (rejected: call-graph noise at scale; unresolved calls remain a per-decl property list instead). Repo-as-single-source (rejected: file-level sources give file-level version history, which is what code questions ask about).

10. Implementation Approach¶

PR-054 commits as planned: this spec → worker + grammar bindings → projection + resolution rules → incremental hash-skip → fixture repos per language with expected graph shapes + determinism tests.

11. Migration Path¶

Grammar version upgrades change projections ⇒ pinned versions in config; upgrading is a spec-amended, deliberate re-ingest event, never a silent drift.

12. Success Metrics¶

Fixture graphs exact-match per language. Determinism (same files ⇒ same submission hash). Incremental: unchanged-file jobs produce zero writes. Honesty check: no CALLS edge without resolution: syntactic property; unresolved imports land as CODE_REF, never guessed.

13. Open Questions¶

• OQ-1: Add go and java grammars in v1.1? Stance: demand-driven; the projection generalizes, each grammar adds fixture cost — decide per request.

14. Changelog¶

• 2026-07-02 — v0.1: initial draft for peer review.
• 2026-07-04 — v0.2: as-built (PR-054). Deviations from v0.1: (1) Manifest + child-job fan-out replaces §4's multi-submit. Core pins ONE source version per job, but §5 requires each file to be its own source; so format=code is a MANIFEST job that fans out one code-file child job per supported file (bytes inline via childJobs[].contentB64). Incremental hash-skip (§3) is therefore core-side for free: an unchanged file re-enqueues with the same (source name, content hash) and core no-ops it — the worker performs no hash check of its own. (2) Deterministic client node ids (uuid5; convention in §15 addendum) for FILE, CODE_REF, and — after weighing decl-churn — ALL declaration nodes: unchanged decls match (no new version), changed decls version at a stable logical id instead of orphaning stale-but-live copies. (3) CALLS scope actually implemented: same-file only. Callee must be a simple identifier resolving to exactly ONE same-file FUNCTION (ambiguity → unresolved); callers are FUNCTION/METHOD decls. Same-ingestion cross-file resolution was NOT implemented (it needs other files' contents, which child-per-file jobs do not carry) — recorded as a v1 limitation; unresolved simple-identifier calls land on the calling decl's calls_unresolved property (§9). (4) MODULE nodes are rust inline mod blocks only; a python file's module identity is its FILE node, and mod foo; is projected as a file reference (IMPORTS), not a decl. (5) CLASS also covers rs enum/union/trait and ts interface/enum; rust impl blocks group onto the type's CLASS node as extra spans (impl of an external type creates the CLASS from the impl block). CONTAINS generalizes to the immediate structural parent (FILE/MODULE/CLASS/FUNCTION), ord per parent. (6) visibility: rust pub → public/private; js/ts exported decls → public (omitted otherwise); ts accessibility modifiers on methods; python omitted (the grammar exposes nothing — convention guessing rejected). (7) Import resolution is lexical candidate matching only (§15) — a miss is a CODE_REF {name: raw, repo_ns} carrying the import statement's span, never a guessed edge; resolved edges use the target's deterministic file id and may dangle until that file's job lands (reads validate; self-heals). (8) Failure posture: oversize/unsupported-language → skipped-with-stat, syntax/decode errors → files_failed stat + parse_error reason; the FILE node (id + content_hash) still lands in every case and the job completes. Job-level failures only for corrupt manifests (corrupt_input) and the file-count ceiling (resource_limit). (9) Config env names TELHA_CODE_MAX_FILE_KB / TELHA_CODE_MAX_FILES; grammar pins are exact package versions in code_worker.languages.GRAMMAR_PINS, asserted before the first job. (10) Manifest jobs' canonical text is the deterministic path hash listing (one line per file, spanless); js/ts class fields holding arrow functions and ts interface members are not extracted (v1 limitation).

15. Addendum — As-Built Wire Shapes & Id Conventions (PR-054)¶

Job shapes. format=code content is JSON {"repoNs": str, "files": [{"path", "hash", "contentB64"}], "languages"?: [str]} (≤ max_files_per_job; paths repo-relative, /-separated, no ..). Each supported file yields a child job: format=code-file, name = "{repoNs}/{path}", content = the file bytes, options {"repoNs", "path", "repoFiles": [all manifest paths]} (the import-resolution set). The worker polls both formats.

Deterministic ids. Namespace CODE_NS = uuid5(NAMESPACE_URL, "telha:code-worker"); then FILE = uuid5(CODE_NS, "code-file:{repoNs}/{path}"), CODE_REF = uuid5(CODE_NS, "code-ref:{repoNs}:{raw_import}"), decl = uuid5(CODE_NS, "code-decl:{repoNs}/{path}:{qualified_name}") with a #<n> key suffix (n ≥ 2) for repeated qualified names within one file. Qualified names join with . (py/js/ts) or :: (rs).

Import resolution candidates (first match in repoFiles wins; purely lexical): python — a.b → a/b.py, a/b/__init__.py from the repo root then the importing file's directory (src-layouts); from- imports try X.name per name before X; relative imports climb from the file's package dir. rust — mod foo; → sibling foo.rs / foo/mod.rs (plus <stem>/foo.rs); use crate::… longest-prefix descent from the crate src root (self::/one-level super:: by directory approximation); external crates never resolve. js/ts — relative specifiers only, trying exact, .ts/.tsx/.js/.jsx/.mjs/.cjs, then index.*; require() and re-exports count as imports; package specifiers never resolve.

Stats (per code-file job): files_parsed/skipped/failed, decls, unresolved_imports, calls_emitted, plus skip_reason / parse_error when applicable. Per manifest job: files, children_enqueued, skipped_oversize/unsupported/filtered lists.