Spec: Generation Orchestration — LlmProvider, Prompt Assembly & Cost Caps¶

File: .ai/specs/core-engine/2026-07-02-generation-orchestration.md Status: Approved (security review of credential handling complete 2026-07-04 — conditions incorporated in v0.2; see Changelog) Owners: Planner lane; security review for credential handling Related: PR-051; F8 ratified (generation in core behind trait); memory-planner spec (input); claim-verification spec (consumer)

1. Overview¶

Defines the LlmProvider abstraction, credential handling inside the core binary, citation-marked prompt assembly with prefix-stable ordering, streaming, retries, and per-tenant cost governance for /v1/generate.

2. Problem Statement¶

F8 placed generation in the core: the binary now holds LLM credentials and makes egress calls — that demands explicit handling rules, not defaults. Additionally: prompts must carry span citations the verifier can resolve; unstable evidence ordering defeats Phase-4 prefix caching before it starts; and without per-tenant cost caps, one tenant's agent loop is everyone's bill.

3. Proposed Solution¶

trait LlmProvider: Send + Sync {
    fn generate(&self, req: GenRequest) -> BoxStream<Result<GenChunk>>;   // streaming-first
    fn count_tokens(&self, model: &str, text: &str) -> Option<u64>;      // None => planner heuristic mode
}

Implementations v1: Anthropic Messages API; OpenAI-compatible chat completions (covers local endpoints). Credentials: env/file only, loaded at startup, held in a zeroize-on-drop wrapper, never serialized, never logged; config validation refuses credentials in the main config file (must be env or a secrets file). Security conditions pinning the details: (a) redaction is structural, not pattern-based — the loaded secret value is registered with the tracing layer at startup and exact-match-scrubbed from every log line, the wrapper implements Debug/Display as [REDACTED], and provider error bodies are capped at 256 chars and scrubbed before entering any error chain (reqwest errors can echo request context); (b) secrets-file permissions are enforced at load on unix (refuse group/world-readable, i.e. anything broader than 0600); on Windows dev boxes this is a logged warning (production targets are linux per CI); (c) egress scheme validation — llm.endpoint must be https unless the host is loopback (local model servers); plaintext http to a remote host would put the Authorization header on the wire; (d) endpoint, provider, and model configuration are operator-only — no tenant-reachable API may influence where the binary sends credentials; per-tenant surface is limited to cost caps; (e) request.model is validated against an operator-configured allowlist (400 on miss) — arbitrary pass-through strings are an unvetted-model and billing surface. Prompt assembly: system frame + evidence blocks, each block headed [S{n}: source={source_id}@{version} span={start}-{end}], ordered by (source_id, spanStart) — deterministic and prefix-stable, so identical plans yield byte-identical prompt prefixes (Phase-4 cache groundwork). Instructions require citing [S{n}] markers; the verifier treats them as hints, not proof (claim-verification spec).

4. Architecture¶

/v1/generate → planner.plan → assemble(plan) → cost-cap gate → LlmProvider.generate (stream) → [SSE to client] and [buffer for verification] → verification → verdict-annotated response + trace persist.

5. Data Models¶

GenRequest {model, system, messages, max_tokens, temperature=0.2 default, stop}; GenChunk {delta, usage?}. Cost ledger (traces-adjacent CF rows, tenant-scoped): per-tenant rolling token counters (input/output) with daily and monthly windows, shared with embedding accounting (vector-storage spec).

6. API Contracts¶

POST /v1/generate {query parts, window, budget?, model?, stream?} → non-stream: full verified response; stream: SSE events chunk, verification, done{trace_id} (verification necessarily trails the stream — clients see draft text live, verdicts arrive at the end; documented client contract). Cap exceeded → 429 TENANT_BUDGET_EXCEEDED with reset time. The cap gate is a pre-flight reservation (security condition): estimated usage (assembled prompt tokens + max_tokens reserve) is checked and reserved BEFORE the provider call, then reconciled against actual usage after — a post-hoc-only check lets a tenant at 99.9% of cap fire one arbitrarily large request. Retries: transient provider errors retried ≤2 with jitter, but only before the first chunk has been delivered — a mid-stream retry would splice two different generations under one verification pass (condition); after first delivery, fail loudly. Token-usage from failed attempts still counts (honest accounting).

7. UI/UX¶

Chat surface streams chunks then paints verification badges (PR-042/056). Cost dashboards read the ledger (app layer).

8. Configuration¶

llm.provider (anthropic|openai_compat), llm.endpoint (https required for non-loopback), llm.api_key_env / llm.secrets_file, llm.model_default, llm.models_allowed (allowlist; requests naming other models → 400), llm.max_output_tokens (4096 — per-request ceiling independent of tenant budget, bounds single-request blast radius), llm.tenant_caps {daily_tokens, monthly_tokens} with per-tenant overrides via operator-scoped admin surface only (CLI/system scope in v1 — never tenant self-service), llm.timeout_s (120).

9. Alternatives Considered¶

Generation in app layer (the F8 alternative; deferred not rejected — the trait boundary makes later migration a module swap; core-side v1 keeps planner adjacency and one verification locus). Provider SDKs (rejected: reqwest + explicit request shapes keep the dependency surface auditable in a binary that holds credentials). Blocking non-stream only (rejected: operator chat UX requires streaming; buffering for verification is compatible). Prompt templates as config (rejected: templates ARE semantics — verifier depends on marker format; changes go through this spec).

10. Implementation Approach¶

PR-051 commits as planned: this spec → trait + two impls (mock-tested) → assembly golden files → endpoint + SSE → cost caps. Credential-handling section requires security review sign-off (recorded in Changelog like F7).

11. Migration Path¶

Greenfield. Marker format changes version the prompt model (prompt_model: v1 in traces). App-layer migration path: implement LlmProvider over an app-layer service URL — zero planner changes.

12. Success Metrics¶

Golden prompt files byte-stable given fixed plans. Redaction test: credentials absent from logs/traces under debug verbosity. Cap test: 429 at boundary, resets on window. Stream test: chunks precede verification event; done carries resolvable trace_id. Prefix-stability: two runs, identical plan → identical prompt prefix bytes.

13. Open Questions¶

• OQ-1: Resolved at security review (2026-07-04) — confirmed not v1, with the security argument added to the verification one: silent substitution changes answer characteristics under verification AND doubles the live credential surface (two providers' keys hot in memory, two egress destinations to reason about). Fail loudly; config chooses the provider.

14. Changelog¶

• 2026-07-04 — v0.3: as-built record (PR-051). Implemented as core-engine/src/llm/ (trait + Anthropic Messages + OpenAI-compat impls over a hand-rolled SSE parser in mod.rs, credential hygiene in secret.rs, prompt assembly in assemble.rs, cost ledger in costs.rs); [llm] config with egress validation at load; POST /v1/generate (JSON + SSE chunk/verification/done{traceId} — verification event wired by PR-052). Security-condition placements: structural redaction = process-wide exact-match scrub at the log sink (telemetry::ScrubWriter) + [REDACTED] Debug/Display + provider bodies capped 256 chars and scrubbed before entering any error (incl. reqwest transport errors); unix 0600 enforced in load_credential, Windows surfaced via config.warnings(); runtime fail-fasts at boot when generation is configured but the credential is missing. As-built deviations, recorded: (1) cost-ledger windows are FIXED-LENGTH v1 (daily = 24 h, monthly = 30 d epoch buckets), not calendar months — pure reset arithmetic; rows live in the traces CF keyed by deterministic UUIDs (xxh3_128("cost:{window}:{start}")), no key-shape change; (2) per-tenant cap OVERRIDES are deferred — v1 enforces the global default caps; the operator-scoped override surface (CLI/system) is a recorded follow-up; (3) the retry boundary is provider acceptance (HTTP 200 + headers): failures after the stream exists are mid-stream and never retried — a strictly conservative reading of "before the first delivered chunk"; (4) provider count_tokens/context_window return None v1 (planner heuristic mode; budget ceiling activates when a tokenizer-bearing provider lands); (5) client disconnect mid-SSE releases the in-memory reservation via Drop — durable reconcile happens only for streams that ran to completion (the spawned task keeps draining, so normal completion reconciles even if the client left). Golden prompt bytes pinned at tests/golden/prompt_v1.txt; cap-boundary, retry, redaction-in-error-path, and prefix-stability tests in tests/generate_e2e.rs + tests/prompt_golden.rs.
• 2026-07-04 — v0.2: Credential-handling security review complete — Approved with conditions, all incorporated (reviewer: Claude, acting security reviewer by J. Porter's delegation; jp@densops.com to countersign or veto). Conditions: (1) structural exact-match redaction + [REDACTED] Debug/Display + scrubbed/capped provider error bodies; (2) secrets-file permission enforcement on unix, warning on Windows dev; (3) https-or-loopback egress validation; (4) endpoint/provider/model config operator-only — tenants can never influence credential destinations; (5) model allowlist with 400 on miss; (6) per-request max_output_tokens ceiling; (7) cost-cap gate is pre-flight reservation + post-hoc reconciliation; (8) retries only before first delivered chunk; (9) cap overrides operator-scoped only in v1. OQ-1 resolved (no fallback provider — verification integrity + credential-surface minimization). §12 redaction test extended by (1): must assert absence under debug verbosity AND in provider-error paths.
• 2026-07-02 — v0.1: initial draft; credential-handling pending security review sign-off.