Spec: Evaluation Harness — Metrics, Datasets & Release Gates¶

File: .ai/specs/core-engine/2026-07-02-eval-harness.md Status: Approved Owners: Planner lane; reviewers: all lanes (gates apply to releases) Related: PR-057; memory-planner / claim-verification specs (measured systems); F6 doctrine (claims become gates)

1. Overview¶

Defines the end-to-end evaluation harness: metric definitions, dataset governance, runners, nightly trend tracking, and the release gates that make answer quality a build-blocking property rather than a hope.

2. Problem Statement¶

F6 turned performance claims into CI gates; answer quality claims need the same treatment or they regress silently: retrieval that misses the labeled evidence, verification that drifts permissive after a threshold tweak, cost-per-query creep. Without dataset governance, evals overfit to their own fixtures and rot.

3. Proposed Solution¶

An /eval crate + versioned datasets: seeded corpora (built from the ingestion fixture sets — docling/tabular/web/email golden corpora, temporally scripted) with labeled items: QA pairs, expected-evidence span sets, adversarial falsehoods, and temporal questions with coordinate-pinned answers. Runners execute the full pipeline (ingest → plan → generate → verify) against a live binary and score. Metrics (normative definitions): retrieval recall@plan = |expected spans ∩ selected spans| / |expected|; claim-support precision = supported-verdicts that are actually correct / all supported (against labels); falsehood rejection = adversarial claims landing unsupported; temporal accuracy = coordinate-pinned answers matching labels; plus latency p50/p95 and tokens-per-query cost. Nightly CI runs publish JSON trends; release gates (blocking): claim-support precision ≥ floor (initial 0.90, ratcheted only upward), falsehood rejection ≥ 0.95, retrieval recall ≥ baseline − 2pts.

4. Architecture¶

CI nightly → compose stack up → dataset ingest (idempotent, hash-checked) → runner sweeps → scorer → JSON report → trend store + gate evaluation; release pipeline consumes the latest gate verdict.

5. Data Models¶

Dataset manifest: {version, items: [{id, kind: qa|adversarial|temporal, input, labels, corpus_refs}], corpus_hashes} — items reference corpus content by hash so dataset and corpus version together. Report: per-metric values + per-item results (failures inspectable), pinned to git SHA + dataset version + model ids (LLM-dependent metrics record the model; gate comparisons only apply within same-model lineage to avoid blaming code for model drift).

6. API Contracts¶

eval run [--suite --dataset-version --report-out], eval gate --report (exit-code contract for CI). LLM usage in evals runs under a dedicated tenant with its own cost cap so eval spend is visible and bounded.

7. UI/UX¶

Trend dashboard from stored JSON (simple static render in CI artifacts v1). Per-item failure drill-down links trace ids — debugging an eval failure IS reading a trace.

8. Configuration¶

Gate floors (ratchet-only: lowering a floor requires a spec amendment with justification — the anti-erosion rule), suite selection, model pinning for gate lineage.

9. Alternatives Considered¶

Public benchmark suites (rejected as gates: they measure generic QA, not tritemporal provenance behavior; may run informationally). LLM-as-judge scoring (partially adopted: label-based scoring is primary; judge-scoring allowed for exploratory metrics, never for gates — nondeterminism and judge-drift disqualify it from blocking). Human eval loops (out of scope for CI; the labeled datasets ARE the crystallized human judgment, refreshed by dataset versioning).

10. Implementation Approach¶

PR-057 commits as planned: this spec → datasets v1 (from golden corpora + scripted temporal histories) → runners → scorer + report → nightly job + trend store → gate wiring into release builds.

11. Migration Path¶

Dataset evolution is versioned; gates compare within dataset-version lineage; new versions establish new baselines explicitly (never silently).

12. Success Metrics¶

The harness's own: gate evaluation deterministic given a report; dataset ingest idempotent; a seeded known-regression (deliberately broken verifier threshold in a test branch) is caught by the gate — the harness's smoke test is proving it can fail things.

13. Open Questions¶

• OQ-1: α/β/threshold auto-tuning from labeled data (confidence-decay OQ-2, claim-verification thresholds) — owned here or per-spec? Stance: tuning experiments run here (the data lives here); adopted values change via the owning spec's amendment. Phase 4.

14. Changelog¶

• 2026-07-02 — v0.1: initial draft for peer review.
• 2026-07-04 — v0.2: as-built (PR-057). Deviations and pinned details:
• Standalone crate at /eval (telha-eval, own target dir/build lock): a black-box REST driver with NO dependency on telha-core — the API surface is the contract under test, and the harness builds without the RocksDB toolchain.
• Deterministic mock provider (spec §4's "compose stack" realized in-process): one loopback server speaking the core's two provider protocols — OpenAI-compat /chat/completions (SSE) and /embeddings. Embeddings = 32-dim bag-of-words feature hashing (cosine tracks token overlap); decomposition = sentence split with exact byte spans + [S#] extraction; generation = echo of the first two evidence blocks (perfectly grounded by construction, so supported-precision drift is attributable to the verifier), with an ECHO: question prefix as the planted-falsehood channel for adversarial items. --url mode targets an operator-configured server (real models) instead; gate lineage separates the two.
• Run modes: run --spawn <telha-bin> provisions a fresh temp data dir + tenant per run (idempotency by construction; the §4 "idempotent, hash-checked" ingest is enforced by the dataset loader's corpus hashes + core dedup), registers the embed model and mints the api key via the CLI, then sweeps.
• Metrics as-built: recall@plan resolves selected span text LOCALLY — corpus docs ingest via the synchronous format=json path whose canonical NDJSON the harness reconstructs bit-exactly, so trace span offsets slice without a server round trip. Claim-support precision scores supported claims against labeled truths by token coverage ≥ 0.6. Falsehood rejection is per-claim. Temporal items are a small scripted-history DSL (create/update/ delete steps, $tx:<step> transaction-time substitution) probed via /v1/query or /v1/compare with coordinate-pinned labels.
• v1 scope cuts (recorded): structured recall leg only (the semantic leg needs a stable-vector embedding fixture — follow-up); corpus = json-bypass docs (docling/tabular/web/email golden corpora need a live worker fleet — follow-up wired through the same manifest); trend store = CI artifacts of the JSON reports (§7's static render deferred); LLM-dependent metrics record model lineage and gates only compare within it.
• Gates: gate --report [--baseline --floors], exit-code contract; floors ratchet enforced at load (below-default floors are refused — amend this spec instead); recall gate arms only with a same-lineage same-dataset baseline. §12's own smoke test is a unit test proving doctored reports FAIL.
• Nightly: .github/workflows/eval-nightly.yml (build release → run → gate → report artifact). Cost caps: the mock provider makes nightly runs token-free; --url runs against real providers use the target server's own tenant caps (§6's dedicated-tenant cap is the operator recipe, not harness code).