Generation orchestration¶

Architecture

F8 put generation inside the core binary rather than the app layer: the binary that holds tenant data now also holds LLM credentials and makes outbound calls. That single decision is why this subsystem reads the way it does, every design choice here traces back to "a process holding credentials must treat egress, secrets, and cost as first-class concerns, not defaults."

Overview & purpose¶

core-engine/src/llm/ implements a swappable generation backend behind one trait, two concrete providers (Anthropic Messages API and any OpenAI-compatible chat-completions endpoint), and the supporting machinery that makes holding credentials in-process safe: structural log redaction, egress scheme validation, a pre-flight cost gate, and a retry policy that never risks splicing two different generations together.

flowchart LR
    REQ["POST /v1/generate"] --> PLAN["planner.plan"]
    PLAN --> ASM["assemble(plan)<br/>citation-marked prompt"]
    ASM --> CAP["cost-cap gate<br/>(pre-flight reservation)"]
    CAP --> GEN["LlmProvider.generate<br/>(stream)"]
    GEN --> SSE["SSE to client"]
    GEN --> BUF["buffer for verification"]
    BUF --> VER["claim verification"]
    VER --> TRACE["verdict-annotated response<br/>+ trace persist"]

Design¶

The LlmProvider trait¶

#[axum::async_trait]
pub trait LlmProvider: Send + Sync {
    async fn generate(&self, request: &GenRequest) -> Result<GenStream, GenError>;
    fn count_tokens(&self, _model: &str, _text: &str) -> Option<u64> { None }
    fn context_window(&self, _model: &str) -> Option<u64> { None }
}

generate is streaming-first; the non-streaming REST shape simply collects the stream. Ok means the provider accepted the request (HTTP 200 and headers received): this acceptance point is the retry boundary, discussed below. count_tokens and context_window return None in v1 for both shipped providers, which puts the planner in heuristic budget mode until a tokenizer-bearing provider lands.

Providers are hand-rolled over reqwest rather than built on vendor SDKs, a deliberate spec decision (§9): explicit request shapes keep the dependency surface auditable in a binary that holds credentials.

Hand-rolled SSE parsing¶

Both providers stream Server-Sent Events, parsed by a small incremental parser in core-engine/src/llm/mod.rs (SseEvents, take_event, find_blank_line) rather than a crate dependency. It buffers raw bytes from the response body, finds the next blank-line-terminated event block (tolerating both \n\n and \r\n\r\n), and accumulates event:/data: fields, joining multi-line data: fields with \n.

AnthropicProvider posts to {endpoint}/v1/messages with header x-api-key and anthropic-version: 2023-06-01, body including "stream": true. It reads events by type: message_start seeds input_tokens, content_block_delta yields the text delta from /delta/text, message_delta updates output_tokens, message_stop emits the final chunk carrying total usage, and error maps to a permanent GenError.

OpenAiCompatProvider posts to {endpoint}/chat/completions with bearer auth, body including "stream": true and "stream_options": {"include_usage": true}. It reads delta.content per chunk, tracks usage from any chunk that carries it, and treats the [DONE] marker as the terminal chunk.

Credential hygiene¶

pub struct Secret(Vec<u8>);

Secret (core-engine/src/llm/secret.rs) is the only place a loaded API key exists in memory. Its Debug and Display impls both render [REDACTED]; it is never Serialize; expose() is the single call site that returns the raw string (grep-able by design, deliberately not Deref). On drop, every byte is overwritten via std::ptr::write_volatile followed by a compiler_fence(SeqCst), a manual best-effort zeroization the module doc explicitly notes is not the zeroize crate; this hand-rolled pair is the entire dependency surface for that guarantee.

Redaction elsewhere is structural, not pattern-based: at credential load, crate::telemetry::register_redaction(secret.expose()) registers the exact secret value with a process-wide ScrubWriter that exact-match-replaces it with [REDACTED] at the log sink, for every log line, regardless of which code path produced it. Provider error bodies get an additional, narrower treatment before they can enter any error chain: scrub_body(body, secret) caps the body at 256 characters and replaces the secret substring, because reqwest transport errors and non-2xx response bodies can otherwise echo request context that includes the credential.

Loading (load_credential) tries the env var named by llm.api_key_env first, then llm.secrets_file. On unix, a secrets file broader than 0600 is refused outright (CredentialError::Permissions); on non-unix (this repo's Windows dev environment) the same check is a no-op here and surfaces instead as a config warning, since production targets are linux per CI.

Egress validation¶

fn validate_llm_egress(endpoint: &str) -> Result<(), ConfigError> {
    match scheme {
        "https" => Ok(()),
        "http" => if loopback { Ok(()) } else { Err(...) },
        _ => Err(...),
    }
}

https is always accepted. http is accepted only when the host is loopback (localhost, ::1, or a parsed 127.0.0.0/8 address): the concern is concrete, plaintext HTTP to a remote host would put the Authorization/x-api-key header on the wire in the clear. This runs at config validation time, so a misconfigured endpoint fails at boot, not on the first request.

Boot fail-fast¶

if config.llm.provider.is_some() {
    crate::llm::provider_from_config(&config.llm)
        .map_err(|e| anyhow::anyhow!("generation credential unavailable at boot: {e}"))?;
    tracing::info!(endpoint = %config.llm.endpoint, "generation provider configured");
}

If llm.provider is configured, runtime::serve loads (and discards) the credential once at startup purely to fail loudly if it is missing or unreadable, rather than surfacing that failure on a tenant's first /v1/generate call.

Prompt assembly¶

core-engine/src/llm/assemble.rs builds the system frame plus evidence blocks from the planner's EvidencePlan. Each block is headed [S{n}: source={source_id}@{version} span={start}-{end}] and blocks are ordered by (source_id, spanStart), a deterministic ordering independent of recall order, so identical plans yield byte-identical prompt prefixes (the spec calls this Phase-4 prefix-cache groundwork). The model is instructed to cite [S{n}] markers; claim verification treats those markers as hints, never as proof on their own.

Retry policy¶

pub async fn generate_with_retry(
    provider: &dyn LlmProvider,
    request: &GenRequest,
    max_retries: u32,
) -> Result<GenStream, GenError> {
    // retries only Err(GenError::Transient) while attempt < max_retries
}

/v1/generate calls this with max_retries = 2. The retry boundary is provider acceptance: generate returning Ok(stream) means the provider has already committed to a generation (HTTP 200 and headers received). A failure discovered mid-stream is never retried, because retrying would splice two different generations under one verification pass. The spec treats this as a conservative reading of "retry only before the first delivered chunk," since acceptance happens strictly before any chunk is read from the stream.

Backoff is 250ms * 2^(n-1), jittered ±20% (n = attempt number, so 250ms then 500ms base, ±50/100ms jitter respectively) using a nanosecond-of-wall-clock source for the jitter draw.

Cost governance¶

pub fn reserve(&self, scope: &TenantScope, estimated: u64, now_us: Ts) -> Result<Reservation, CapError>;
pub fn settle(&self, scope: &TenantScope, reservation: Reservation, actual: TokenUsage, now_us: Ts) -> Result<(), StorageError>;

CostLedger (core-engine/src/llm/costs.rs) implements a pre-flight reservation, post-hoc reconciliation pattern, the spec's explicit answer to a specific attack: a post-hoc-only check would let a tenant sitting at 99.9% of their cap fire one arbitrarily large request before the ledger ever sees it. reserve holds the admission lock across check-and-insert so two concurrent requests cannot both pass the same headroom, checks used + outstanding + estimated against each non-zero cap, and on success adds estimated to an in-process Arc<Mutex<HashMap<Uuid, u64>>> of outstanding reservations. The returned Reservation is an RAII drop-guard: settle() releases the hold and durably records actual usage; a plain drop() (client disconnect, panic, early return) releases the hold with no durable write, so failed or abandoned requests never consume budget they didn't use.

Data model¶

GenRequest / GenChunk¶

pub struct GenRequest {
    pub model: String,
    pub system: String,
    pub messages: Vec<ChatMessage>,
    pub max_tokens: u64,
    pub temperature: f64,   // default 0.2
    pub stop: Vec<String>,
}

pub struct GenChunk {
    pub delta: String,
    pub usage: Option<TokenUsage>,   // present on the final chunk only
}

Cost ledger rows¶

Durable per-window usage rows live in cf::TRACES (traces-adjacent, no new key shape), keyed by Uuid::from_u128(xxh3_128("cost:{window_label}:{window_start}")). Windows are fixed-length, not calendar-aligned:

struct CostRow { input_tokens: u64, output_tokens: u64 }

POST /v1/generate¶

{
  "question": "string, required",
  "query": {"...": "optional query-DSL object, structured recall leg"},
  "semanticModel": "optional embedding model name, semantic recall leg",
  "k": 16,
  "expandDepth": 1,
  "atValidTime": "RFC3339 or unix-µs",
  "atTxTime": "RFC3339 or unix-µs",
  "budgetTokens": 4096,
  "model": "optional, defaults to llm.model_default",
  "maxTokens": 1024,
  "temperature": 0.2,
  "stream": false,
  "allowUngrounded": false
}

Non-streaming response shape:

{
  "draft": "...",
  "model": "...",
  "traceId": "uuid",
  "promptModel": "v1",
  "planModel": "v1",
  "planHash": "...",
  "promptHash": "...",
  "citations": ["S1", "S2"],
  "plan": {"budget": 4096, "used": 3120, "spans": ["..."], "stats": {"...": "..."}},
  "usage": {"inputTokens": 812, "outputTokens": 240},
  "verification": {"status": "verified", "claims": ["..."]}
}

Streaming (stream: true) sends text/event-stream frames instead:

sequenceDiagram
    participant C as Client
    participant S as /v1/generate (stream)
    C->>S: POST {stream: true}
    loop as tokens arrive
        S-->>C: event: chunk {delta}
    end
    S-->>C: event: verification {status, claims}
    S-->>C: event: done {traceId, usage, planHash, promptHash, citations}
    Note over S,C: A mid-stream provider failure instead emits<br/>event: error {message} and ends the stream.

Verification necessarily trails the stream: it requires the full draft, so clients see draft text live and verdicts arrive only in the final verification event, immediately before done.

Algorithms & invariants¶

flowchart TB
    R["reserve(scope, estimated, now)"] -->|cap=0| SKIP["window skipped: unlimited"]
    R -->|used+outstanding+estimated > cap| DENY["CapError::Exceeded"]
    R -->|else| HOLD["reservation held in-process"]
    HOLD --> DONE{stream completes?}
    DONE -->|yes| SETTLE["settle(): durable write + release"]
    DONE -->|client disconnects / panics| DROP["Drop: release only, no durable write"]

Configuration¶

As-built notes¶

From the spec's §14 v0.3 changelog, reconciled against core-engine/src/llm/:

• Cost-ledger windows are fixed-length, not calendar months, as documented above: pure reset arithmetic, deliberately simpler than calendar-aware bucketing.
• Per-tenant cap overrides are deferred. v1 enforces only the global default caps from llm.tenant_caps; an operator-scoped override surface (CLI/system) is a recorded follow-up, not yet built.
• The retry boundary is provider acceptance (HTTP 200 + headers), a strictly conservative reading of the spec's "before the first delivered chunk," acceptance happens before any chunk is read at all, so this reading never retries anything the looser wording would have forbidden.
• count_tokens / context_window return None for both shipped providers in v1. The planner runs in heuristic budget mode until a tokenizer-bearing provider is added; the budget ceiling activates automatically once one is.
• Client disconnect mid-SSE releases the reservation via Drop, but durable reconciliation only happens for streams that ran to completion: the spawned server-side task keeps draining the provider stream after a client leaves, so normal completion still reconciles usage even though the client never saw the tail.
• Golden prompt bytes are pinned at tests/golden/prompt_v1.txt; cap-boundary, retry, redaction-in-error-path, and prefix-stability tests live in tests/generate_e2e.rs and tests/prompt_golden.rs.

Related¶

• Architecture › Memory planner
• Architecture › Claim verification
• Architecture › Job queue & leases
• Concepts › Verified answers