Skip to content

Troubleshooting

Symptom · cause · fix

Fast lookup tables for the failures operators actually hit, synthesized from the deployment runbook and the operator handbook. Start here; each row links back to the page with the full procedure.

Startup and configuration failures

Symptom Likely cause Fix Where to look
error: invalid configuration: grpc.token_key ..., or gRPC/MCP silently disabled grpc.token_key is not 64 hex characters, or is all decimal digits, the env layer parses an all-digit value as an integer and rejects it as the expected string type Regenerate with openssl rand -hex 32 until the value contains at least one letter (a-f) Configuration reference, Key & secret management
error: config file not found: ... --config (or TELHA_CONFIG) points at a path that does not exist Fix the path, or omit --config to use built-in defaults + env only Configuration reference
error: invalid configuration: rest.addr and grpc.addr must differ Both listeners resolved to the same non-zero port Change one of rest.addr / grpc.addr / mcp.addr, or set the colliding port to 0 for an ephemeral bind Configuration reference
error: invalid configuration: log.level ... is not a valid filter log.level is not a valid tracing_subscriber::EnvFilter directive Use a directive like info or telha_core=debug,info Configuration reference
error: invalid configuration: llm.endpoint ... uses plaintext http to a non-loopback host [llm] endpoint is http:// to a remote host Use https://, or point at a loopback host (localhost, 127.0.0.1, ::1) for local dev only Configuration reference
error: invalid configuration: backup.enabled requires backup.dir [backup] enabled = true with no dir set Set backup.dir, or leave enabled = false Configuration reference
error: invalid configuration: retention.enabled requires retention.archive_dir (or backup.dir to default under) [retention] enabled = true with neither archive_dir nor backup.dir resolvable Set retention.archive_dir, or set backup.dir for it to default under Configuration reference
check-config exits non-zero with a validation message not listed here Any of the range/positivity checks in TelhaConfig::validate Run telha check-config --config <path> locally and read the specific message, it names the exact key and expected range Configuration reference

Windows DLL exit-127

Symptom Likely cause Fix Where to look
telha.exe exits instantly, exit code 127 (or STATUS_DLL_NOT_FOUND / 0xC0000135) The GNU-toolchain build needs MinGW runtime DLLs (libstdc++-6.dll, libgcc_s_seh-1.dll, libwinpthread-1.dll) that are not next to the exe or on PATH Use the packaged Windows dev/eval zip, which ships the DLL closure alongside the exe (the Windows loader prefers the application directory), or add C:\msys64\mingw64\bin to PATH Deployment runbook §4
The zip's staged exe still fails after extraction An unrelated PATH-dependent DLL is missing, or the zip was built without -Verify Rebuild with powershell -File tooling\packaging\make-windows-zip.ps1 -Verify, which runs the staged exe with PATH stripped to C:\Windows\System32;C:\Windows to prove the closure is complete Deployment runbook §4

Windows is dev/eval only

This whole failure class exists because the Windows zip is explicitly not a production target. Production deploys are Linux + systemd or Docker; see the deployment runbook.

Authentication and authorization failures

Symptom Likely cause Fix Where to look
Every /v1/* request rejected; server logs no API keys registered No API key has been minted into {data_dir}/api_keys.json yet telha api-key create --tenant <uuid> --org <uuid> --data-dir <dir>, then restart the server (it only reads the file at boot) Key & secret management
A newly created API key doesn't work The server was not restarted after api-key create Restart, api_keys.json is loaded once at boot Key & secret management
CLI: failed to open data dir ... lock The server holds the RocksDB exclusive lock; most telha subcommands open the data dir directly Stop the server first, or run a one-shot container against the same volume (Docker) Deployment runbook §3.2
telha api-key worker-token \| mcp-token \| clarify-token \| admin-token \| connector-token fails: grpc.token_key is not configured grpc.token_key is unset in the resolved config Set TELHA_GRPC__TOKEN_KEY (see the all-digit gotcha above) Key & secret management
A worker/connector token is rejected mid-operation Tokens expire after 300 seconds by design; the token simply aged out Re-mint the token; long-running services should re-mint on an interval well under 300s Key & secret management
A worker token can't answer clarifications, or a connector token touches another tenant's jobs Audiences are disjoint by construction, this is expected behaviour, not a bug Mint the correct audience for the operation (clarify-token, admin-token, etc.) Key & secret management
MCP listener not reachable on port 7627 MCP requires both mcp.enabled = true (the default) and grpc.token_key set, MCP sessions authenticate with mcp-audience tokens signed by that key Set grpc.token_key; confirm mcp.enabled was not explicitly set to false Configuration reference

Workers not picking up jobs

Symptom Likely cause Fix Where to look
Ingest job stuck in pending No worker process is leasing that job kind Check the worker process is running and pointed at the right TELHA_GRPC_TARGET; confirm its token audience matches the kind (worker for ingest:*, connector for sync:*) Key & secret management
telha jobs ls --state dead shows accumulating dead jobs The job exceeded jobs.max_attempts (default 5) Investigate the failure reason via telha jobs show <id> (includes the event log), then telha jobs retry <id> once the underlying issue is fixed Configuration reference
Connector seed refused: a sync:{source} chain step is already pending or leased Only one live chain per source is allowed, by design Run telha connector status <source> and either let the current step drain, or telha jobs retry the dead step Retention & archival (job-chain mechanics generally)
A job kind never seems to back off correctly after failures Custom per-kind backoff isn't configurable via env; jobs.backoff_base_secs_by_kind is populated internally (e.g. clarify:temporal) Check jobs.backoff_base_secs, jobs.backoff_factor, and jobs.backoff_cap_secs for the general defaults Configuration reference

Generation failures (/v1/generate)

Symptom Likely cause Fix Where to look
501 llm.provider is unset, or the credential env/secrets file was missing at boot, the endpoint disables itself and the boot log records why Set llm.provider + llm.endpoint + llm.model_default, and ensure llm.api_key_env (or llm.secrets_file) resolves, then restart Configuration reference, Key & secret management
422 The request named a model outside models_allowed ∪ {model_default}, or the request otherwise failed shape validation Use an allowed model name, or add it to llm.models_allowed (operator-only config change) Configuration reference
429 TENANT_BUDGET_EXCEEDED The reservation model: every request reserves estimated prompt tokens + maxTokens up front against the tenant's daily/monthly window, then settles to actual usage after the stream ends (including failed streams). Many concurrent requests with a large maxTokens can exhaust the cap on reservations alone Lower per-request maxTokens / llm.max_output_tokens, or raise llm.tenant_caps.daily_tokens / monthly_tokens Configuration reference
Boot fails outright with an egress-scheme error llm.endpoint is http:// to a non-loopback host Switch to https://, or use a loopback host for local dev Configuration reference

Vector search returning nothing

Symptom Likely cause Fix Where to look
Semantic search returns zero results right after ingestion Embedding fan-out isn't active, embedding.endpoint and/or embedding.model are unset, or the model was never registered Set both config keys, then telha vector register-model --name <model> --dims <n> before ingesting Configuration reference
Vector search returns nothing right after a restore Open (unsealed) partition buckets are deliberately excluded from backups by design telha vector rebuild --tenant <uuid> --org <uuid> --model <name> Backup & restore
Vector search returns nothing for a specific time window Vector partitions are time-bucketed (vector.bucket); a query outside any sealed/loaded bucket range may miss recent writes still in an open bucket within seal_grace_secs Wait out vector.seal_grace_secs (default 48h) for late-arriving data to seal, or reduce vector.bucket granularity if this recurs Configuration reference
An entity that should be searchable never shows up The entity was expired by a retention policy (whole-entity removal), or is under a legal hold and blocked from actioning, check the other direction: was it deleted, not merely un-indexed telha retention preview to see if it was actioned; check telha retention hold ls if you expected a hold Retention & archival

Disk pressure

Symptom Likely cause Fix Where to look
data_dir growing without bound Append-only history by design, with no retention policy configured (default is keep_forever for every selector) Set retention policies for high-volume labels/source-kinds; always telha retention preview before enabling a policy on live data Retention & archival
Backup directory (backup.dir) filling the disk backup.keep is set too high for the checkpoint size, or pruning never runs because backups keep failing Lower backup.keep; investigate why creates are failing (pruning only runs after a successful verified create) Backup & restore
A single checkpoint takes a very long time or is unexpectedly large Approaching the spec's incremental-backup tripwire (a checkpoint regularly exceeding 30 minutes, or an artifact exceeding 50GB) This is a known, documented threshold at which incremental backup design gets reopened, flag it rather than just tuning around it Backup & restore
Orphaned provisional source versions accumulating Crash-orphaned streamed ingestion (canonical_text_hash = None) whose owning job lost its lease The retention pass's orphan sweep removes these automatically once retention.orphan_ttl_days (default 14) has elapsed, enable [retention] if it is not already Retention & archival