Eval harness¶

Architecture

F6 turned performance claims into CI gates; the eval harness does the same for answer quality. Without it, verification drifting permissive after a threshold tweak, retrieval quietly missing labeled evidence, or cost-per-query creep would all regress silently. The harness makes these build-blocking properties instead of hopes, by driving the real /v1/generate and /v1/query pipeline against labeled datasets and gating on the result.

Overview & purpose¶

eval/ is a standalone Rust crate, telha-eval, that deliberately does not depend on telha-core. It drives a live binary black-box over REST: the API surface is the contract under test, and the harness itself builds without pulling in the RocksDB toolchain core-engine needs. This separation matters operationally too: eval has its own Cargo.lock and target directory, so its build never contends with core-engine's build gates.

flowchart LR
    DS["Dataset<br/>(manifest.json + corpus/)"] --> RUN["telha-eval run"]
    RUN --> TARGET{spawn or url?}
    TARGET -->|--spawn| MOCK["Fresh telha + in-process<br/>mock provider + temp data dir"]
    TARGET -->|--url| EXT["Operator-configured server<br/>(real models)"]
    MOCK --> SWEEP["Sweep dataset items through<br/>ingest → plan → generate → verify"]
    EXT --> SWEEP
    SWEEP --> REPORT["JSON report<br/>(metrics + per-item results)"]
    REPORT --> GATE["telha-eval gate<br/>exit 0 pass / 1 fail"]

Design¶

Six metrics¶

All six are computed by eval/src/score.rs and eval/src/runner.rs against the labels in the dataset manifest, never by LLM-as-judge. The spec explicitly excludes judge-scoring from anything that gates, citing nondeterminism and judge-drift.

A supported claim is scored correct by claim_is_true iff some labeled truth string covers ≥ 0.6 of the claim's tokens (case-insensitive alphanumeric token intersection over union of claim tokens), generous enough to tolerate article/inflection drift, strict enough that an unrelated falsehood never coincidentally clears it.

pub fn claim_coverage(claim: &str, truth: &str) -> f64 {
    // |claim_tokens ∩ truth_tokens| / |claim_tokens|
}
pub fn claim_is_true(claim: &str, truths: &[String]) -> bool {
    truths.iter().any(|t| claim_coverage(claim, t) >= 0.6)
}

Dataset governance¶

{
  "version": "v1",
  "corpus": [{"name": "eval/project-alpha.json", "label": "PROJECT_NOTE", "file": "project-alpha.json"}],
  "corpusHashes": {"project-alpha.json": "c1382471f713c812"},
  "items": [{"id": "qa-project-alpha", "kind": "qa", "question": "...", "find": "PROJECT_NOTE", "expectedEvidence": [...], "truths": [...]}]
}

Every corpus file is hash-checked at load (Dataset::load in eval/src/dataset.rs): an xxh3_64 hex digest of each corpus file must match manifest.corpus_hashes, or loading fails loudly with a "bump the dataset version" message. This is what keeps dataset and corpus versioned together: a silent edit to a corpus file under an unchanged manifest version is treated as corruption, not a quietly-different eval run. telha-eval hash-corpus prints the hashes to paste into a manifest when deliberately bumping a version.

Item kinds:

• qa: a /v1/generate question plus expectedEvidence (recall labels) and truths (precision labels).
• adversarial: a question prefixed ECHO: whose remainder is a planted falsehood; the mock provider's generation path echoes it verbatim as the "perfectly grounded" answer would, so any resulting supported verdict is a verifier failure, not a generator one.
• temporal: a scripted history (setup: [{op: create|update|delete, save, target?, label?, properties, validTime?}]) followed by a probe (/v1/query or /v1/compare) and an expect block. Probe values of the form $tx:<step> (optionally $tx:<step>+N) are substituted with the transaction-time start captured from that setup step's response before the probe is sent. This is how a temporal item pins a query to "the transaction time recorded by step r1," without knowing that timestamp in advance.

The runner: spawn or external¶

eval run requires exactly one of --spawn <bin> (env TELHA_EVAL_BIN) or --url <server> --api-key <key>.

Spawn mode (eval/src/runner.rs::spawn_target) provisions a fresh temp data dir and a throwaway config per run, wiring [llm] and [embedding] both at an in-process mock server, then:

1. Starts the mock provider (mock::serve()).
2. Writes a config pointing [llm].endpoint / [embedding].endpoint at the mock's loopback address, with model_default = "eval-gen" and verify.embed_model = "eval-embed".
3. Runs the CLI once each to vector register-model (registering the 32-dim mock embedding model) and api-key create (minting a fresh tenant/org/key), both against the closed data directory before serve opens it.
4. Spawns the built binary as a child process and polls GET /readyz for up to 60 seconds (120 attempts, presumably ~0.5s apart) before handing back a Target.

A fresh tenant and temp data directory per run makes re-runs trivially idempotent by construction, satisfying the spec's "idempotent, hash-checked ingest" requirement without needing explicit dedup logic in the harness.

External mode (--url) targets an already-running server with a supplied API key: real models, real cost, gated by that server's own tenant caps rather than anything eval-side.

The deterministic mock provider¶

eval/src/mock.rs runs one loopback HTTP server speaking both provider protocols the core consumes:

• POST /chat/completions (OpenAI-compatible, SSE): what OpenAiCompatProvider parses.
• POST /embeddings: what the embedding provider parses.

Every behavior is a pure function of the request, making eval runs reproducible:

• Embeddings: 32-dim bag-of-words feature hashing. Each alphanumeric word lowercases, hashes via xxh3_64, and increments or decrements one of 32 buckets (sign from a hash bit); the result is L2-normalized. Cosine similarity between two such vectors tracks raw token overlap, which is all the claim matcher's semantic leg needs to behave sensibly against a synthetic corpus.
• Decomposition: detected by the core's fixed decompose system prompt; the mock sentence-splits the draft on . (extending over any trailing [S#] marker run) and returns exact UTF-8 byte spans plus extracted markers per sentence, i.e. the same constrained-JSON shape the real decomposer targets.
• Generation: echoes the first two evidence blocks from the assembled prompt, each cited, a "perfectly grounded" generator by construction, so any precision drift in a report is attributable to the verifier, not the generator. A question beginning ECHO: instead emits the remainder of the question verbatim, uncited: the planted-falsehood channel adversarial items use.

Data model¶

Report¶

pub struct Report {
    pub dataset_version: String,
    pub models: ModelLineage,       // {generate, embed}
    pub started_at_ms: u64,
    pub duration_ms: u64,
    pub metrics: Metrics,
    pub items: Vec<ItemResult>,
}

pub struct ItemResult {
    pub id: String,
    pub kind: String,
    pub pass: bool,
    pub trace_id: Option<String>,   // present when the item ran generation
    pub detail: String,
}

{
  "datasetVersion": "v1",
  "models": {"generate": "eval-gen", "embed": "eval-embed"},
  "startedAtMs": 1751500000000,
  "durationMs": 4200,
  "metrics": {
    "retrievalRecall": 1.0,
    "claimSupportPrecision": 1.0,
    "falsehoodRejection": 1.0,
    "temporalAccuracy": 1.0,
    "latencyMsP50": 42.0,
    "latencyMsP95": 88.0,
    "tokensPerQuery": 310.5
  },
  "items": [
    {"id": "qa-project-alpha", "kind": "qa", "pass": true, "traceId": "...", "detail": "..."}
  ]
}

The report is pinned to dataset_version and model lineage (ModelLineage{generate, embed}) precisely so gate comparisons only ever apply within the same lineage. The spec's explicit goal is to avoid blaming code for model drift.

Gate floors¶

pub struct Floors {
    pub claim_support_precision: f64,   // default 0.90
    pub falsehood_rejection: f64,       // default 0.95
    pub recall_regression_pts: f64,     // default 0.02
    pub temporal_accuracy: f64,         // default 1.0
}

Algorithms & invariants¶

Gate evaluation¶

flowchart TB
    REPORT["Report"] --> CSP{claimSupportPrecision<br/>≥ 0.90?}
    REPORT --> FR{falsehoodRejection<br/>≥ 0.95?}
    REPORT --> TA{temporalAccuracy<br/>≥ 1.0?}
    REPORT --> BASE{same-lineage<br/>baseline supplied?}
    BASE -->|yes, matching models + dataset| RECALL{retrievalRecall ≥<br/>baseline − 0.02?}
    BASE -->|no, or lineage differs| SKIP["recall gate skipped<br/>(informational only)"]
    CSP & FR & TA & RECALL --> RESULT{all pass?}
    RESULT -->|yes| EXIT0["exit 0"]
    RESULT -->|no| EXIT1["exit 1"]

claim_support_precision, falsehood_rejection, and temporal_accuracy gate unconditionally against their floors. The retrieval recall gate only arms with a same-lineage baseline: it requires a previous report whose models (both generate and embed) and dataset_version exactly match the current report; only then does recall ≥ baseline.recall − recall_regression_pts get checked. A different model lineage, or no baseline at all, skips the recall check entirely and prints a note explaining why. A new lineage establishes its own baseline explicitly rather than silently inheriting one.

CLI surface¶

telha-eval run --dataset <dir> (--spawn <bin> | --url <url> --api-key <key>) --report-out <path>
telha-eval hash-corpus --dataset <dir>
telha-eval gate --report <path> [--baseline <path>] [--floors <path>]

gate reads a report (and optional baseline and floors override), prints one PASS/FAIL line per check with its value and floor, then prints an overall gate: PASS or gate: FAIL and exits 0 or 1 accordingly; the exit code is the CI contract.

Configuration¶

Gate floor defaults: claimSupportPrecision ≥ 0.90, falsehoodRejection ≥ 0.95, temporalAccuracy ≥ 1.0, retrievalRecall ≥ baseline − 0.02 (lineage-gated only).

As-built notes¶

From the spec's §14 v0.2 changelog, reconciled against eval/:

• Standalone crate confirmed: eval/Cargo.toml states explicitly that the harness is "black-box over REST" and deliberately does not depend on telha-core, so it builds without the RocksDB toolchain.
• Deterministic mock provider realizes the spec's "compose stack" entirely in-process rather than via docker-compose: one loopback server speaking both provider protocols, as described above.
• v1 scope cuts, recorded as deliberate, not accidental gaps:
  • Only the structured recall leg (find) is exercised; the semantic recall leg needs a stable-vector embedding fixture and is a follow-up.
  • Corpus documents ingest via the synchronous format=json bypass path rather than through docling/tabular/web/email workers, which would need a live worker fleet; wiring the golden corpora through the same manifest shape is a follow-up.
  • The trend store is CI report artifacts (plain JSON files), not the dashboard render the original spec's §7 sketched.
  • LLM-dependent metrics record model lineage and gates only ever compare within it, per the lineage-binding design above.
• Recall resolves span text locally: because corpus docs ingest via the synchronous json-bypass path, the harness reconstructs the exact canonical NDJSON bit-for-bit locally (Dataset::load's canonical field: each record re-serialized with serde_json, joined by \n, matching the core's own canonical rendering since serde_json maps are key-sorted identically on both sides). This lets trace span offsets slice without a server round-trip.
• Nightly runs are token-free: the mock provider makes CI nightly runs cost nothing; --url runs against real providers instead spend under the target server's own tenant caps, which the spec treats as the operator's recipe for the "dedicated tenant with its own cost cap" requirement, not something the harness enforces itself.
• Build/test scripts: eval/run-eval-build.cmd runs cargo test in the eval/ crate on its own target dir/lock; eval/run-selftest.cmd spawns the freshly built telha.exe against the mock provider and requires every dataset item and every gate to pass (--test selftest).

Related¶

• Architecture › Claim verification
• Architecture › Memory planner
• Architecture › Generation orchestration
• Concepts › Verified answers