Key & secret management¶

API keys · token keys · the field-encryption KEK · provider credentials

Telha's trust boundary rests on five distinct kinds of secret, each with its own storage location, consumer, and rotation flow. This page is the full inventory: what each secret is, where it lives, how to rotate it, and what breaks if it is lost.

The inventory at a glance¶

API keys¶

API keys are the REST authentication credential (x-api-key header), scoped to a tenant UUID + organization UUID pair that you choose at creation time and that then scopes every row the key can touch.

telha api-key create --tenant <uuid> --org <uuid> --data-dir <dir>
# optionally bind a clarification-answering principal:
telha api-key create --tenant <uuid> --org <uuid> \
  --person <PERSON-uuid> --role viewer|editor|steward --data-dir <dir>

Only a sha256 hash of the key is written to {data_dir}/api_keys.json; the plaintext is printed to stdout exactly once and is not recoverable afterward. api-key create opens the data dir directly, so stop the server first (or mint keys before first start). The server reads api_keys.json only at boot, so any change requires a restart to take effect.

Rotation:

1. Mint the new key (server stopped): telha api-key create --tenant ... --org ....
2. Distribute the new plaintext to the client out of band.
3. Remove the old entry from {data_dir}/api_keys.json.
4. Restart the server.

Without --person, a key can authenticate REST calls but cannot answer clarifications; telha clarify answer and the REST answer surface both require a bound PERSON.

If lost: the plaintext cannot be recovered (only its hash is stored). Create a replacement key and revoke the old entry per the rotation flow above; there is no "reveal" operation.

gRPC/MCP token key¶

grpc.token_key is a single 32-byte HMAC key, hex-encoded to 64 characters, that backs every internal service-token audience: worker, mcp, clarify, admin, and connector. gRPC stays disabled until it is set; MCP requires it too, because MCP sessions authenticate with the same signed-token scheme (audience mcp).

Generate it with:

openssl rand -hex 32

Set it in the environment, never in the TOML file:

export TELHA_GRPC__TOKEN_KEY=<64-hex-with-a-letter>

Minting service tokens¶

Every internal audience is minted on demand from grpc.token_key and expires after TOKEN_TTL_SECS (300 seconds). Nothing is stored; there is no separate revocation step because tokens simply expire.

telha api-key worker-token    --worker-id docling-1                 # audience "worker"
telha api-key mcp-token       --tenant <uuid> --org <uuid>          # audience "mcp"
telha api-key clarify-token   --worker-id clarify-bot               # audience "clarify" (poll)
telha api-key clarify-token   --tenant <uuid> --org <uuid>          # audience "clarify" (answer)
telha api-key admin-token     --tenant <uuid> --org <uuid>          # audience "admin"
telha api-key connector-token --connector sharepoint --tenant <uuid> --org <uuid>  # audience "connector"

Audiences are disjoint by construction: a worker token cannot answer clarifications; a connector token leases only its own tenant's sync: jobs and deletes only records in its own namespace; and API keys never open /admin regardless of the principal's role. telha api-key worker-token, mcp-token, clarify-token, and admin-token are pure token-minting operations and do not open the data dir directly, so they work against a live server.

Rotation: coordinated across every service that mints or verifies tokens signed by this key: update TELHA_GRPC__TOKEN_KEY on the core host and on every token-minting service (app layer, clarify-bot, worker token-mint commands), then restart both sides together. Because tokens self-expire in 300 seconds, there is no separate revocation list to maintain; the last old-key tokens age out within five minutes of the restart.

If lost: anything already holding a live (unexpired) token keeps working for at most 300 seconds. Once the key is gone, no new tokens can be minted or verified, set a new grpc.token_key, and re-run the coordinated rotation flow above (this is effectively a forced rotation).

Field-encryption KEK and DEK¶

Field-level encryption (XChaCha20-Poly1305 envelope encryption of classified record fields) lives entirely in the app layer (app-layer/packages/enterprise). Core stores only opaque {"$bytes": ...} envelopes and never sees plaintext or key material, this is the opacity contract the vector-storage exclusion logic and index paths are tested against.

Key hierarchy:

• KEK (key-encryption key): a 32-byte key from the FIELD_CRYPTO_KEK environment variable on the app-layer service only, it must never be set on the core host. The shipped EnvKekProvider wraps/unwraps DEKs with it; pointing at a real KMS is a deployment configuration change to the KmsProvider interface, not a code change.
• DEK (data-encryption key): one active DEK per tenant, wrapped by the KEK, stored as a row in the app-layer PostgreSQL database ({key_id, tenant, wrapped_dek, created, state}). Decrypted DEKs are cached in-memory with a TTL (crypto.dek_cache_ttl, default 300s) in zeroize-on-drop wrappers.
• Every envelope carries key_id, so historical ciphertext under a retiring key remains readable until an explicit destruction event, even after the active key rotates.

Rotation flow (BullMQ queue field-crypto-rotation, one job per tenant):

1. rotate(tenant) mints a new active DEK and marks the prior one retiring (decrypt-only from that point).
2. The rotation sweep re-encrypts forward only: it reads existing records, decrypts locally, and writes new Telha versions under the new key_id through the normal encrypting write path. Append-only history means historical versions keep their old ciphertext and stay readable under the retiring key.
3. The sweep reports remainingOldKeyRefs. This should reach 0; a non-zero count means a concurrent writer raced the sweep, so re-run it before considering destruction of the old key.
4. destroy(keyId, order) is the explicit crypto-shred order and requires zero remaining references (or an explicit destruction directive tied to GDPR erasure).

Rotating the KEK itself is a KMS-side operation (re-wrap every tenant's DEK under the new KEK); the DEK values and envelopes never change.

LLM and embedding API keys¶

Generation and embedding credentials are named by config, never stored in it:

llm.secrets_file holds the entire (trimmed) file content as the key. On Unix it is refused unless the file's permissions are 0600 or tighter; on Windows this same check is a logged warning only, since Windows ACLs are not POSIX mode bits (see TelhaConfig::warnings).

Both credentials are checked at boot, not at first request: if llm.provider is set but no credential source resolves, the server fails fast with a clear error rather than accepting the config and later returning 501 on every generation call. Rotate either by updating the named environment variable (or secrets file) and restarting the server. Config validation additionally enforces that llm.endpoint uses HTTPS unless the host is loopback, so the Authorization header carrying this key never crosses the network in the clear.

If lost: /v1/generate (or the embed runner) fails at the next restart's boot check. Obtain a new key from the provider, update the named env var or secrets file, and restart.

What breaks if each is lost¶

Related¶

• Configuration reference, where grpc.token_key, llm.api_key_env, and embedding.api_key_env sit in the config surface.
• Backup & restore, why the app-layer PostgreSQL database (wrapped DEKs) is a mandatory second backup surface.
• Architecture › Field encryption, the envelope format, AAD binding, and opacity contract in depth.
• Deployment runbook, the key inventory table and first-boot key setup.
• Operator handbook, day-2 rotation flows and the LLM cost-cap reservation model.