Telha Core, Deployment Runbook¶

Status: Draft, pending CI verification (PR-084). Everything in the Windows section and the Docker image build/healthcheck was verified locally on 2026-07-06; the Linux binary, systemd, and CI sections are written defensively but have not yet executed on a real Linux host or GitHub runner (this repository has no git remote/CI yet). The verification checklist at the end tracks exactly what remains unproven.

This runbook is the operator's document of record for deploying, backing up, upgrading, and recovering a Telha Core instance. It assumes no prior contact with the codebase.

Related: docs index · quickstart (first grounded answer in ~10 minutes) · operator handbook (day-2 operations: key rotation, retention, stewarding, cost caps).

1. What ships¶

One binary contains everything: the server (telha serve) and the operator CLI (telha api-key|backup|restore|retention|jobs|vector| clarify|connector|debug|index). There are no other runtime components on the core side. The optional app layer (Next.js + PostgreSQL + Redis) and Python ingestion workers are separate deployables outside this runbook's core scope, but their state figures in backups (§6).

Ports (defaults): 7625 REST, 7626 gRPC, 7627 MCP. gRPC and MCP stay disabled until grpc.token_key is configured.

Probes: GET /healthz (liveness, never touches storage), GET /readyz (readiness: engine open and snapshot-able). Both are unauthenticated.

2. Clean-host deploy (Linux + systemd)¶

Target: a fresh Debian/Ubuntu-family host with systemd. Everything runs as the dedicated telha system user; no root at runtime.

2.1 Install the binary¶

# From a CI artifact (or an SCP'd build):
sudo install -m 0755 telha /usr/local/bin/telha
telha version    # → telha-core 0.1.0 (x86_64, rust ...)

2.2 Create the service user and directories¶

sudo useradd --system --home-dir /var/lib/telha --shell /usr/sbin/nologin telha
sudo mkdir -p /etc/telha /var/backups/telha
sudo chown telha:telha /var/backups/telha
# /var/lib/telha is created by systemd (StateDirectory) on first start.

2.3 Configuration¶

sudo cp tooling/packaging/telha.sample.toml /etc/telha/telha.toml
sudo cp deploy/telha.env.example /etc/telha/telha.env
sudo chown root:telha /etc/telha/telha.toml && sudo chmod 0640 /etc/telha/telha.toml
sudo chown root:root  /etc/telha/telha.env  && sudo chmod 0600 /etc/telha/telha.env

Edit /etc/telha/telha.toml:

• data_dir = "/var/lib/telha" (matches the unit's StateDirectory)
• [rest] addr, keep 127.0.0.1:7625 behind a reverse proxy, or 0.0.0.0:7625 on a firewalled host.
• [backup] enabled = true, dir = "/var/backups/telha" (§6).

Edit /etc/telha/telha.env and set TELHA_GRPC__TOKEN_KEY (see §5 key inventory; generate with openssl rand -hex 32). The 64-hex value must contain at least one letter, the config env layer parses an all-digit value as an integer and rejects it.

Validate before first start (prints the fully resolved config):

sudo -u telha env $(sudo grep -v '^#' /etc/telha/telha.env | xargs) \
  telha check-config --config /etc/telha/telha.toml

2.4 Install and start the service¶

sudo cp deploy/telha.service /etc/systemd/system/telha.service
sudo systemctl daemon-reload
sudo systemctl enable --now telha
systemctl status telha
curl -fsS http://127.0.0.1:7625/healthz   # → ok
curl -fsS http://127.0.0.1:7625/readyz    # → ok

Logs are JSON on stdout → journalctl -u telha -f.

2.5 Mint the first API key¶

API keys are per tenant scope (tenant UUID + organization UUID, you choose these; they scope every row in storage). The CLI writes the hash into {data_dir}/api_keys.json; the plaintext is printed once.

sudo systemctl stop telha    # key file is loaded at boot; CLI opens the data dir directly
sudo -u telha telha api-key create \
  --tenant 11111111-1111-4111-8111-111111111111 \
  --org    22222222-2222-4222-8222-222222222222 \
  --data-dir /var/lib/telha
sudo systemctl start telha

Note: api-key create only touches api_keys.json, but the server reads that file at startup, restart (or deploy keys before first start) for new keys to take effect.

2.6 First-request smoke¶

KEY=<plaintext from 2.5>
curl -fsS -X POST http://127.0.0.1:7625/v1/records \
  -H "x-api-key: $KEY" -H "Content-Type: application/json" \
  -d '{"records":[{"labels":["NOTE"],"properties":{"text":"hello"}}]}'
curl -fsS -X POST http://127.0.0.1:7625/v1/query \
  -H "x-api-key: $KEY" -H "Content-Type: application/json" \
  -d '{"find":"NOTE","limit":10}'

Alternatively run the full scripted probe against a spare port: bash tooling/packaging/smoke-probe.sh /usr/local/bin/telha 17625 (uses its own temp data dir; does not touch the service's state).

2.7 Optional subsystems¶

Enable only what the deployment needs; each is a config flip plus a secret from §5:

3. Docker deploy¶

3.1 Image¶

Built from the repo root (the Dockerfile COPYs proto/ and core-engine/):

docker build -f core-engine/Dockerfile -t telha-core .

Image facts: Debian bookworm-slim, non-root user telha (uid 10001), state volume /var/lib/telha, EXPOSE 7625 7626 7627, built-in HEALTHCHECK on /readyz, ENTRYPOINT telha, CMD serve. Container env defaults already bind REST/gRPC on 0.0.0.0.

3.2 Run¶

docker run -d --name telha \
  -p 7625:7625 -p 7626:7626 \
  -v telha-data:/var/lib/telha \
  -e TELHA_GRPC__TOKEN_KEY=<64-hex-with-a-letter> \
  telha-core

# wait for health, then:
curl -fsS http://127.0.0.1:7625/readyz    # → ok
docker inspect --format '{{.State.Health.Status}}' telha   # → healthy

Configuration is environment-first (TELHA_* with __ nesting for whitelisted keys). For TOML-only sections ([embedding], [llm] details, [vector] tuning) mount a config file:

docker run -d ... -v /srv/telha/telha.toml:/etc/telha/telha.toml:ro \
  telha-core serve --config /etc/telha/telha.toml

Operator CLI against a running container's volume: stop the container first for commands that open the data dir directly (api-key create, backup create, retention, jobs...), or exec while stopped is impossible, so use a one-shot container on the same volume:

docker stop telha
docker run --rm -v telha-data:/var/lib/telha telha-core \
  api-key create --tenant <uuid> --org <uuid> --data-dir /var/lib/telha
docker start telha

3.3 Local dev stack¶

docker-compose.yml at the repo root brings up core + PostgreSQL + Redis (the app layer's stores): docker compose up. The compose file carries a dev-only token key and a /readyz healthcheck.

3.4 Backups in Docker¶

Prefer the in-process ticker: TELHA_BACKUP__ENABLED=true, TELHA_BACKUP__DIR=/var/lib/telha/backups (inside the volume) or a second mounted volume. Copy stamps off-host with docker cp / volume-level tooling. pg_dump is not in the image, for two-surface capture (§6.2) run backups where PostgreSQL client tools exist, or accept partial: true core-only stamps and dump PG separately.

4. Windows dev/eval zip¶

Dev/eval only, not a production target. The zip exists because the GNU-toolchain telha.exe needs MinGW runtime DLLs (libstdc++-6.dll etc.) and dies with STATUS_DLL_NOT_FOUND (exit 127 under bash) when C:\msys64\mingw64\bin is not on PATH. The zip ships the DLL closure next to the exe; the Windows loader prefers the application directory, so the zip runs on a clean machine with nothing installed.

Usage: extract anywhere, follow README.txt inside the zip (version → api-key create → serve → curl probes).

Building it (dev box or CI mirrors this exactly):

# needs a release build first: core-engine\run-release-build.cmd
# (works under Windows PowerShell 5.1 and pwsh; CI uses pwsh)
powershell -File tooling\packaging\make-windows-zip.ps1 -Verify

-Verify runs the staged exe with PATH stripped to C:\Windows\System32;C:\Windows, success proves the closure. manifest.json in the zip records the exact DLLs bundled.

Static-link alternative (considered, documented, not taken): the DLLs could be eliminated with -C link-arg=-static-libgcc -C link-arg=-static-libstdc++ (or +crt-static for a fully static CRT) in RUSTFLAGS. Rejected for now because (a) changing RUSTFLAGS invalidates every cargo fingerprint on the dev box, a ~40-minute RocksDB C++ rebuild per profile, and would fork packaging builds from the test/bench configuration; (b) crt-static on windows-gnu is the least-exercised corner of the toolchain and has a history of link oddities with cxx/usearch-style C++ deps; (c) the zip-with-DLLs layout is behaviorally identical for the eval use case. Revisit if the zip ever graduates beyond dev/eval.

5. Key & secret inventory¶

Baseline: /etc/telha/telha.env is 0600 root:root; the TOML config holds no secrets; JSON logs scrub registered LLM secrets at the sink, but treat journal access as privileged anyway.

6. Backups¶

6.1 Schedule (online ticker)¶

In /etc/telha/telha.toml:

[backup]
enabled = true
dir = "/var/backups/telha"
interval_hours = 24   # first tick fires immediately at boot
keep = 7              # verified-create prunes older stamps
pg_database_url_env = "DATABASE_URL"

Each run produces a stamp directory telha-backup-<µs> containing a RocksDB checkpoint (hardlink-based, writers never stall), sealed vector partition files, api_keys.json, a pg_dump --format=custom of the app-layer database when DATABASE_URL is set, and a blake3-hashed manifest. Stamps are verified before retention prunes anything.

Partial stamps: without a reachable DATABASE_URL the manifest is marked partial: true, wrapped field-crypto DEKs, audit rows, and role mappings live in PG and are not in a partial stamp. Production with an app layer must run two-surface backups.

Manual/cold backup (server stopped, exclusive open):

sudo systemctl stop telha
sudo -u telha telha backup create --out /var/backups/telha --data-dir /var/lib/telha
sudo systemctl start telha

Ship stamps off-host (rsync/object storage), keep only manages local disk. telha backup verify --from <stamp> re-hashes a copied stamp at its destination.

6.2 Restore drill (run it quarterly, not during an incident)¶

# 1. verify the stamp
telha backup verify --from /var/backups/telha/telha-backup-<µs>

# 2. restore into an EMPTY dir (refuses non-empty; runs the mandatory
#    per-tenant index consistency gate; failure removes the target)
telha backup restore --from /var/backups/telha/telha-backup-<µs> \
  --data-dir /var/lib/telha-restore \
  --erasure-ledger /var/lib/telha/erasure-ledger.jsonl

# 3. if the stamp carried a PG dump, the CLI prints the exact command:
#    pg_restore --clean --if-exists --dbname=<DATABASE_URL> <dump>

# 4. point the service at it and verify
#    (swap data dirs or update data_dir), then:
curl -fsS http://127.0.0.1:7625/readyz
sudo -u telha telha debug index-check --tenant <uuid> --org <uuid> \
  --data-dir /var/lib/telha-restore

Erasure-ledger seam: a present erasure ledger makes restore refuse outright until PR-081 lands ledger replay, by design, so a restore can never resurrect erased personal data. Absent ledger = recorded no-op.

Vector partitions: open (unsealed) buckets are intentionally skipped by backup (derived, rebuildable). After a restore, telha vector rebuild --tenant <uuid> --org <uuid> --model <name> recreates anything missing.

7. Retention & archival¶

Off by default; nothing ever expires without an explicit policy, so enabling the ticker on a policy-less host is a no-op.

[retention]
enabled = true
interval_hours = 24

# policy: archive CONTRACT_DOCUMENT rows older than 1y, delete after 2y
sudo -u telha telha retention policy set --tenant <uuid> --org <uuid> \
  --label CONTRACT_DOCUMENT --action archive_then_delete \
  --archive-days 365 --delete-days 730 --data-dir /var/lib/telha

# ALWAYS preview first — the normative dry-run (never writes):
sudo -u telha telha retention preview --tenant <uuid> --org <uuid> \
  --data-dir /var/lib/telha

# legal hold suspends archive+delete for matching entities:
sudo -u telha telha retention hold set --tenant <uuid> --org <uuid> \
  --label CONTRACT_DOCUMENT --reason "Smith v. ACME discovery" \
  --data-dir /var/lib/telha

Archive slices land under retention.archive_dir (default {backup.dir}/archives), are verified before any delete happens, and side-load into fresh dirs for inspection (telha retention archives sideload). Every delete is recorded in the erasure ledger ({data_dir}/erasure-ledger.jsonl), which then gates backup restores (§6.2). A live server exposes the same surface over /admin/retention/* with an admin token (telha api-key admin-token).

8. Upgrade procedure¶

Rolling upgrades are not applicable (single process, exclusive data dir). Downtime = one stop/start.

# 1. stage the new binary
install -m 0755 telha-new /usr/local/bin/telha-next
/usr/local/bin/telha-next version

# 2. stop
sudo systemctl stop telha

# 3. backup (cold; §6.1) — the rollback point
sudo -u telha telha backup create --out /var/backups/telha --data-dir /var/lib/telha

# 4. swap
sudo mv /usr/local/bin/telha /usr/local/bin/telha-prev
sudo mv /usr/local/bin/telha-next /usr/local/bin/telha

# 5. start + verify
sudo systemctl start telha
curl -fsS http://127.0.0.1:7625/readyz
journalctl -u telha -n 50 --no-pager     # no errors/warnings you can't explain

# 6. integrity gate per active tenant (opens the data dir directly —
#    run against a restored copy, or accept the brief stop):
sudo systemctl stop telha
sudo -u telha telha debug index-check --tenant <uuid> --org <uuid> --data-dir /var/lib/telha
sudo systemctl start telha

Rollback: stop → restore step-3 stamp into a fresh dir (§6.2) → swap telha-prev back → point data_dir at the restored dir → start. Do not run an older binary against a data dir a newer binary has written to unless the release notes say the change was storage-compatible (record model is additive-msgpack by convention, but the stamp is the guarantee).

Docker variant: docker pull/build new tag → docker stop telha → backup the volume (one-shot container, §3.2) → docker rm telha → docker run the new tag with the same volume → /readyz.

9. Troubleshooting¶

Appendix A, out-of-scope state on the same host¶

An app-layer deployment (the Telha app / Next.js) adds: PostgreSQL (first-class state: wrapped DEKs, audit, sessions, role maps, back it up, §6), Redis (cache/queues, rebuildable), TELHA_AUTH_CONFIG (SSO/OIDC per-tenant JSON), FIELD_CRYPTO_KEK, CLARIFY_BOT_CONFIG. Python workers need only a gRPC address + worker tokens. Those components' deploy docs live with their packages.

Appendix B, PR-084 verification checklist¶

Local proofs completed on the dev box (2026-07-06):

• Docker image builds (docker build -f core-engine/Dockerfile .)
• docker run reaches healthy (5 s); /readyz returns ok from the host; container runs as uid 10001 (non-root)
• Runbook §3.2 recipe proven live: one-shot api-key create on the named volume → restart → authenticated POST /v1/records + POST /v1/query roundtrip; docker stop (SIGTERM) → "shutdown complete", exit 0 in under a second
• Windows zip staged from the release exe; DLL closure = libgcc_s_seh-1 / libstdc++-6 / libwinpthread-1; with PATH stripped to C:\Windows\System32;C:\Windows the staged exe ran version, --help, api-key create, AND a full serve → /readyz → authenticated records+query roundtrip
• Workflow YAML parses; scripts lint/execute locally

Remaining, run when the repo gets a git remote + Actions (none exists today; .github/workflows/ has never executed):

• release-build / build-linux produces both target binaries
• smoke-linux passes (first-ever boot of telha on Linux)
• docker job: build + healthcheck green on a runner
• windows-zip job: msys2 action mirrors the local build, -Verify passes
• rust.yml test job: first-ever unix run of the full test suite (covers cfg(unix) paths: secrets 0600 enforcement, SIGTERM)
• UDS leg: blocked, serve-side UDS was never implemented (PR-026 deferred the whole leg). Needs core work, then flip if: false on the uds-test-leg job (details in .github/workflows/release-build.yml)
• systemd unit on a real host (unit is authored, unexercised)
• Restore drill §6.2 on a Linux host incl. pg_restore