Skip to content

Backup & restore

Consistent snapshots · verified restore · the erasure-ledger seam

Telha backups are atomic, online, point-in-time captures of the RocksDB store plus every piece of sidecar state a restore needs to actually answer queries again. This page covers the backup procedure, the restore procedure, and the drill that proves both work before you need them for real.

Normative spec

backup-restore (PR-080) is the normative source for everything on this page: the manifest format, the mandatory consistency gate, the two-surface (core + PostgreSQL) capture, and the erasure-ledger replay seam.

What a backup stamp contains

A single telha backup create run produces one stamp directory, <out>/telha-backup-<createdAtUs>, containing:

  • A RocksDB checkpoint (rocksdb/), taken via the engine's native Checkpoint mechanism, hardlink-based, consistent, and non-blocking. Live writers never stall during a backup.
  • Sealed vector partition files only (vectors/). Open (unsealed, mid-mutation) buckets are deliberately excluded: derived indexes are rebuildable by doctrine, and rebuilding is cheaper and safer than backing up mutable mmap state.
  • api_keys.json (the API key hash file).
  • A pg_dump --format=custom of the app-layer PostgreSQL database, when a connection string is available (see Two-surface capture below).
  • MANIFEST.json: a BLAKE3 hash, size, and relative path for every file in the stamp, the column-family inventory, engine and format versions, creation time, and the erasure watermark (the creation instant, used to decide which ledger entries must replay on a later restore).

Consistent snapshot, not a live copy

Copying a live RocksDB directory with cp or rsync tears SST/WAL files mid-write and produces an unopenable backup. telha backup create never does this, it always goes through the engine's checkpoint mechanism, which is why it is safe to run against a live server.

Backup procedure

Scheduled (online ticker)

/etc/telha/telha.toml
[backup]
enabled = true
dir = "/var/backups/telha"
interval_hours = 24   # first tick fires immediately at boot
keep = 7              # verified-create prunes older stamps
pg_database_url_env = "DATABASE_URL"

The ticker runs inside the serving process. Every run is verified immediately after creation; retention pruning only ever runs after a successful, verified create, a failed backup never causes pruning of older, good stamps. keep = 0 disables pruning entirely.

Manual / cold backup

For an exclusive, stop-the-world capture (equivalent consistency to the online path, just simpler to reason about):

sudo systemctl stop telha
sudo -u telha telha backup create --out /var/backups/telha --data-dir /var/lib/telha
sudo systemctl start telha

Two-surface capture (core + PostgreSQL)

An app-layer deployment stores wrapped field-encryption DEKs, the audit log, sessions, and role mappings in PostgreSQL, not in Telha's data dir. A core-only backup with a lost PostgreSQL database is data loss with extra steps: classified fields become permanently unreadable.

telha backup create --out /var/backups/telha \
  --pg-database-url "postgres://..." \
  --data-dir /var/lib/telha

If --pg-database-url is omitted, the CLI falls back to the environment variable named by backup.pg_database_url_env (default DATABASE_URL).

Partial stamps

Without a reachable PostgreSQL connection string at backup time, the manifest is marked partial: true and both create and a later restore print a loud warning. A partial: true stamp is missing wrapped DEKs, audit rows, and role mappings, production deployments running the app layer must run two-surface backups, not core-only ones.

Shipping stamps off-host

backup.keep only manages local disk. Copy stamps to another host or object storage yourself (rsync, cloud CLI, or your platform's tooling); this is a runbook-owned step, not a built-in feature. Always copy erasure-ledger.jsonl, the erasure-programs/ directory, and erasure-secret.json offsite alongside every stamp, restore-with-replay needs all three (see Erasure-ledger replay below). After copying a stamp to its destination, re-verify it there:

telha backup verify --from /path/to/copied/telha-backup-<µs>

Restore procedure

# 1. verify the stamp
telha backup verify --from /var/backups/telha/telha-backup-<µs>

# 2. restore into an EMPTY data dir
telha backup restore --from /var/backups/telha/telha-backup-<µs> \
  --data-dir /var/lib/telha-restore \
  --erasure-ledger /var/lib/telha/erasure-ledger.jsonl

# 3. if the stamp carried a PostgreSQL dump, the CLI prints the exact command:
#    pg_restore --clean --if-exists --dbname=<DATABASE_URL> <dump>

# 4. point the service at the restored dir (swap data_dir or the volume), then:
curl -fsS http://127.0.0.1:7625/readyz
telha debug index-check --tenant <uuid> --org <uuid> --data-dir /var/lib/telha-restore

Restore targets an EMPTY directory only

Restoring over a live or non-empty data dir is refused, always, there is no in-place restore. This is not configurable. Always restore into a fresh directory and then point the service at it.

Restore runs, in order:

  1. Verify (mandatory, no --fast opt-out). Every file's BLAKE3 hash is re-checked and the manifest's completeness is confirmed. This is the same check telha backup verify performs on its own.
  2. Erasure-ledger check. See Erasure-ledger replay.
  3. Copy rocksdb/, vectors/, and api_keys.json into the target directory.
  4. Mandatory consistency gate. The restore opens the copied store and cross-verifies every tenant's covering temporal indexes against the primary column families. This has no bypass flag: hash verification proves the files survived the trip; the consistency gate proves the database is coherent. Any missing or orphaned index entry fails the restore and removes the target directory, an incoherent store can never be served by accident.

The restore report includes tenants_checked and, once ledger replay ran, erasure_entries_replayed.

Erasure-ledger replay

A restore's relationship to GDPR erasure is what stops a backup from resurrecting legally erased data:

Ledger state Restore behaviour
No --erasure-ledger given / ledger absent Proceeds; recorded as a no-op in the report.
Ledger path given but unreadable Refused outright. By design, a restore can never silently skip replay it cannot perform.
Ledger present and readable Entries with at_us newer than the manifest's erasure_watermark_us replay, in file order, through their content-free, idempotent erasure programs (expected at {ledger_parent}/erasure-programs/) before the restored directory may serve traffic.

After replay, the pass-1 structural verification from the GDPR erasure spec re-runs against every replayed program. A missing program, a program-hash mismatch, or any execution failure during replay is a failed restore, the target directory is removed, consistent with the same fail-closed posture as the consistency gate.

Replay also rebuilds the erased-content blocklist rows from the ledger (they are keyed with hmac_blake3_v1, so replaying a pre-erasure backup first re-seeds the per-tenant erasure secret from the offsite {data_dir}/erasure-secret.json mirror) so that post-restore ingestion keeps refusing already-erased content.

The byte-level scan is a separate, manual step

Ledger replay proves the erasure programs re-ran; it does not by itself re-run the full byte-level residual scan, which needs the plaintext descriptor of what was erased. After a restore-with-replay, run telha erase verify per affected erasure request and record any residuals in that request's completion report.

Vector partitions after restore

Open (unsealed) vector buckets are intentionally excluded from every backup. After a restore, rebuild anything the backup didn't carry:

telha vector rebuild --tenant <uuid> --org <uuid> --model <name>

The restore drill

Run this quarterly, not during an incident

The first time you run a restore should never be during a real outage. The deployment runbook schedules this drill; treat it as a standing operational task.

  1. Pick a recent verified stamp (or take a fresh one).
  2. telha backup verify --from <stamp>, confirms every file hashes clean.
  3. telha backup restore --from <stamp> --data-dir <fresh empty dir> --erasure-ledger <path>.
  4. If the stamp carries a PostgreSQL dump, run the printed pg_restore command against a scratch database, not production.
  5. Point a scratch instance of telha serve at the restored dir and confirm /readyz returns ok.
  6. Run telha debug index-check --tenant <uuid> --org <uuid> --data-dir <restored dir> per active tenant.
  7. Rebuild any open vector buckets with telha vector rebuild.
  8. Record the drill outcome (stamp used, elapsed time, any manual steps) somewhere durable, this is your evidence that recovery actually works.

Failure modes and what they mean

Symptom Cause What to do
backup create leaves a <stamp>.partial directory Create failed partway through Safe to delete; never counted by retention pruning.
Manifest says partial: true No reachable DATABASE_URL at backup time Fine for a core-only deployment; for an app-layer deployment, fix connectivity and re-run with --pg-database-url or the named env var.
backup verify reports a hash mismatch A file was corrupted or tampered with after backup Do not use this stamp; fall back to an earlier verified one.
backup restore refuses a non-empty target Restoring into an existing/live data dir Always target a fresh, empty directory.
backup restore refuses: erasure ledger unreadable Ledger path given but the file cannot be read Fix file permissions/path; do not delete the ledger to force the restore through.
Restore fails at the consistency gate An index/primary-CF mismatch in the copied store The target directory is auto-removed; investigate the source stamp rather than retrying blindly.
Vector search returns nothing right after a restore Open buckets were skipped by backup, as designed Run telha vector rebuild --tenant ... --org ... --model ....