Operator Handbook - Day-2 Operations¶

The deployment runbook is the document of record for installing, backing up, upgrading, and recovering a Telha instance. This handbook covers the recurring operator work on a running system: keys, retention, stewarding, cost caps, release gating, and the failure modes you will actually meet. It cross-links the runbook rather than duplicating it.

General CLI rule: most telha subcommands open the data dir directly (RocksDB exclusive lock) - stop the server first, or use a one-shot container on the same volume (runbook §3.2). The exceptions are pure token minting (api-key worker-token|mcp-token|clarify-token|admin-token|connector-token) and backup verify|restore, which never touch a live data dir.

Key inventory and rotation¶

The full inventory table lives in runbook §5. Rotation flows:

API keys (tenant credentials, x-api-key)¶

Hashes live in {data_dir}/api_keys.json; the server reads the file at boot.

telha api-key create --tenant <uuid> --org <uuid> --data-dir <dir>
# optionally bind a clarification-answering principal:
#   --person <PERSON uuid> --role viewer|editor|steward

Rotate: mint the new key (server stopped), distribute it, remove the old entry from api_keys.json, restart. Keys without --person cannot answer clarifications at all.

gRPC token key (grpc.token_key)¶

The 32-byte HMAC key behind every internal audience (app, worker, mcp, clarify, admin, connector). Generate with openssl rand -hex 32 and make sure the 64-hex value contains at least one letter: the env config layer parses an all-digit value as an integer and rejects it (runbook §2.3 gotcha). Rotation is coordinated: update TELHA_GRPC__TOKEN_KEY on core and on every token-minting service (app layer, clarify-bot, worker token-mint commands) and restart both sides. Tokens themselves are self-expiring (TTL 300 s) and re-minted on an interval, so there is nothing else to revoke.

Service tokens (minted, not stored)¶

telha api-key worker-token    --worker-id docling-1                 # audience "worker"
telha api-key mcp-token       --tenant <uuid> --org <uuid>          # audience "mcp"
telha api-key clarify-token   --worker-id clarify-bot               # audience "clarify" (poll)
telha api-key clarify-token   --tenant <uuid> --org <uuid>          # audience "clarify" (answer)
telha api-key admin-token     --tenant <uuid> --org <uuid>          # audience "admin"
telha api-key connector-token --connector sharepoint --tenant <uuid> --org <uuid>

All require grpc.token_key in the resolved config and expire in 300 s. Audiences are disjoint by construction: a worker token cannot answer clarifications, a connector token leases only its tenant's sync: jobs and deletes only its own namespace, and API keys never open /admin - the admin surface is operator-scoped by token audience, not by key privilege.

Field-crypto KEK and DEK rotation (app layer)¶

Envelope encryption for classified fields lives entirely in the app layer (app-layer/packages/enterprise); core stores opaque {"$bytes": ...} envelopes and is oblivious.

• KEK: FIELD_CRYPTO_KEK env (32 bytes, base64 or hex) on the app-layer service only - never on the core host. Wrapped DEKs live in the app-layer PostgreSQL (which is why two-surface backups matter, runbook §6).
• DEK rotation runs through the enterprise rotation queue (BullMQ, one job per tenant): rotate(tenant) mints a new active DEK and marks the prior one retiring (decrypt-only); the sweep then re-encrypts FORWARD by writing new record versions under the new key id. Historical versions keep their old ciphertext and stay readable until an explicit destruction event. The sweep result reports remainingOldKeyRefs; it should be 0 - non-zero means a page raced a concurrent writer, so re-run the sweep before considering old-key destruction.

LLM and embedding keys¶

Env vars named by llm.api_key_env (default TELHA_LLM_API_KEY) and embedding.api_key_env (default TELHA_EMBEDDING_API_KEY), or llm.secrets_file (0600 enforced on unix; warning-only on Windows). Update the env and restart; generation credentials are checked at boot and fail fast when llm.provider is set without one.

Backups and the restore drill¶

Schedule, stamp anatomy, partial-stamp semantics, and the quarterly restore drill are in runbook §6. The operator's checklist:

• [backup] enabled = true with dir set; the first tick fires at boot; stamps are verified before anything is pruned.
• Watch for partial: true stamps - no DATABASE_URL at backup time means wrapped DEKs / audit / role maps were NOT captured.
• Run the §6.2 restore drill quarterly, not during an incident. A present erasure ledger makes restore refuse by design (until PR-081 lands ledger replay); do not delete the ledger to force it.
• After a restore, rebuild skipped open vector buckets: telha vector rebuild --tenant <uuid> --org <uuid> --model <name>.

Retention, legal holds, dry runs¶

Nothing ever expires without an explicit policy: the default for every selector is keep_forever, so an enabled ticker on a policy-less tenant is a no-op. Enable the in-process ticker with [retention] enabled = true (requires retention.archive_dir or backup.dir to default under; interval_hours default 24, max_entities_per_run default 10000 - runs are bounded and resume next interval).

Doctrine, deliberately not configurable: hold beats policy, erasure beats hold, archive is verified before any delete, every delete is ledgered, whole-entity granularity.

# policies are per selector: exactly one of --label / --source-kind
telha retention policy set --tenant <uuid> --org <uuid> \
  --label CONTRACT_DOCUMENT --action archive_then_delete \
  --archive-days 365 --delete-days 730 --data-dir <dir>
# actions: keep_forever | archive_after | delete_after | archive_then_delete
telha retention policy ls  --tenant <uuid> --org <uuid> --data-dir <dir>
telha retention policy rm  --tenant <uuid> --org <uuid> --label CONTRACT_DOCUMENT --data-dir <dir>

# ALWAYS preview first: the normative dry-run; never writes.
# Per selector: eligible entities, hold-blocked (with hold refs),
# estimated archive bytes, delete count, remaining backlog.
telha retention preview --tenant <uuid> --org <uuid> --data-dir <dir>

# one bounded pass: archive -> verify (hard gate) -> hold-rechecked
# deletes -> orphan sweep; prints the JSON report + ledger path
telha retention run --tenant <uuid> --org <uuid> --data-dir <dir>

# legal holds suspend archive AND delete for matching entities
telha retention hold set  --tenant <uuid> --org <uuid> \
  --label CONTRACT_DOCUMENT --reason "Smith v. ACME discovery" --data-dir <dir>
# scope: --label | --source-kind | --id <logical uuid> | none = whole tenant
telha retention hold ls   --tenant <uuid> --org <uuid> --data-dir <dir>
telha retention hold lift <hold-uuid> --tenant <uuid> --org <uuid> --data-dir <dir>
# lifting warns: matching entities become actionable on the next run

# archive slice artifacts
telha retention archives ls --data-dir <dir>
telha retention archives inspect  --from <stamp-dir>
telha retention archives sideload --from <stamp-dir> --data-dir <FRESH empty dir>
# side-load is inspection-only; never merge back into the live tenant

Every delete lands in the erasure ledger ({data_dir}/erasure-ledger.jsonl), which then gates backup restores.

The /admin/retention/* REST mirror¶

A live server exposes the same surface without a stop, behind the "admin"-audience bearer token (telha api-key admin-token --tenant --org; the operator names the tenant being administered at mint time).

ADMIN=$(telha api-key admin-token --tenant <uuid> --org <uuid> --config <toml>)
H="Authorization: Bearer $ADMIN"
BASE=http://127.0.0.1:7625

curl -fsS $BASE/admin/retention/policies -H "$H"                       # list
curl -fsS -X POST $BASE/admin/retention/policies -H "$H" \
  -H "Content-Type: application/json" \
  -d '{"label":"CONTRACT_DOCUMENT","action":"archive_then_delete","archiveDays":365,"deleteDays":730,"actor":"jp"}'
curl -fsS -X POST $BASE/admin/retention/policies/remove -H "$H" \
  -d '{"label":"CONTRACT_DOCUMENT"}'
curl -fsS $BASE/admin/retention/holds -H "$H"                          # list
curl -fsS -X POST $BASE/admin/retention/holds -H "$H" \
  -d '{"label":"CONTRACT_DOCUMENT","reason":"Smith v. ACME","actor":"jp"}'
curl -fsS -X POST $BASE/admin/retention/holds/<hold-uuid>/lift -H "$H"
curl -fsS $BASE/admin/retention/preview -H "$H"                        # the dry-run

Bodies are camelCase (sourceKind, logicalId, archiveDays, deleteDays). Errors use the standard envelope with code RETENTION_INVALID. There is no run endpoint on the REST mirror; the pass itself runs via CLI or the ticker.

Clarification steward duties¶

Clarifications that need a human referee surface on the CLI (opens the data dir directly - stop the server, or work a restored copy):

# the review queue: open/partially-confirmed at high/critical severity,
# conflicted, expired, dead-lettered; most urgent first
telha clarify queue --tenant <uuid> --org <uuid> --data-dir <dir>

# everything, optionally by state:
# open|attested|partially_confirmed|answered|conflicted|expired|dead_lettered
telha clarify ls --tenant <uuid> --org <uuid> --state conflicted --data-dir <dir>

# one clarification: candidates, responses, quorum, events, plus the
# delivery job trail (attempt N -> ranked candidate N; final = steward)
telha clarify show <id> --tenant <uuid> --org <uuid> --data-dir <dir>

# record an answer as a declared person (channel recorded as "cli")
telha clarify answer <id> --tenant <uuid> --org <uuid> \
  --responder <PERSON uuid> --index 0 --data-dir <dir>
# free-form instead of a candidate: --valid-from <µs|RFC3339> [--valid-to ...] [--text "..."]

# steward override on a conflicted clarification (explicit, never implicit)
telha clarify review <id> --tenant <uuid> --org <uuid> \
  --responder <steward PERSON uuid> --index 1 --data-dir <dir>

# GDPR: tombstone a person's attribution (asked entries, responder ids,
# message refs, channel metadata) across all clarifications in scope;
# confirmation evidence and corrected facts survive
telha clarify erase-attribution --person <uuid> --tenant <uuid> --org <uuid> --data-dir <dir>

The CLI is the local-operator trust boundary: --responder is a declaration, recorded as channel "cli". Remote surfaces (REST, chat) have the responder-assertion rule instead. Delivery-side duties (who got asked, why someone was skipped) are in the clarify-bot runbook.

LLM cost caps and the reservation model¶

Per-tenant token caps live in core config (operator-only; never tenant self-service):

[llm.tenant_caps]
daily_tokens = 2000000    # input+output per daily window; 0 = unlimited
monthly_tokens = 40000000

How enforcement works (worth understanding before sizing caps): every /v1/generate request reserves its worst case up front (estimated prompt tokens + maxTokens) against the window, then settles to the provider-reported actual usage after the stream ends - including failed streams (honest accounting). A request that cannot reserve gets 429 TENANT_BUDGET_EXCEEDED with the window reset time in µs. Consequence: many concurrent requests with large maxTokens can exhaust a cap on reservations alone; keep llm.max_output_tokens (per-request ceiling, default 4096) sane. The model allowlist (llm.models_allowed plus llm.model_default) bounds which models spend at all.

Eval gate as a release check¶

The eval/ crate (binary telha-eval) sweeps a pinned dataset through a candidate binary against a deterministic mock provider and gates the report; the exit-code contract is CI-ready (0 pass / 1 fail).

cd eval
cargo run --release -- run --dataset datasets/v1 \
  --spawn ../core-engine/target/release/telha.exe --report-out eval-report.json
# (or --url http://host:7625 --api-key ... against a running server)

cargo run --release -- gate --report eval-report.json \
  --baseline last-release-report.json     # arms the recall-regression gate
# gate: PASS   (exit 0)  /  gate: FAIL  (exit 1)

Floors are ratchet-only (--floors overrides below the defaults are refused). Run the gate against every release candidate; the build doc's Phase 3c exit criteria make it the release gate. On this Windows dev box use eval/run-eval-build.cmd (the MSYS2 toolchain wrapper).

Troubleshooting¶

The runbook's §9 table is the first stop. The classics, plus operator-level notes:

See also¶

• Deployment runbook - install, backup, upgrade, recover
• Quickstart
• Connectors · Clarify bot
• User guide: Administration