Spec: Backup & Restore — Online Point-in-Time Durability¶

File: .ai/specs/core-engine/2026-07-05-backup-restore.md Status: Approved with conditions (review 2026-07-05 by J. Porter — conditions incorporated in v0.2; PR-080 as-built owes two follow-ups before pilot data: the mandatory restore consistency gate and the PostgreSQL backup surface) Owners: Storage lane Related: gdpr-erasure spec (restore-hook: backups pre-dating erasures must replay the ledger; its §8 backup_horizon_days consumes this spec's manifests), hnsw-partitioning spec (sealed partition files + rebuildability), composite-key-layout spec (CF inventory), PR-080 (build doc Phase 3c — URGENT, precedes any real-data pilot).

1. Overview¶

Defines Telha's first durability path: online point-in-time backups of the primary RocksDB store plus the derived vector files and key material, a verifiable manifest, an attestable restore procedure, and the erasure-replay seam that keeps restores from resurrecting legally erased data. Scope is a single-node data dir (the current deployment shape); replication and offsite transport are follow-ups, not this spec.

2. Problem Statement¶

Until now the plan had NO durability story: a data_dir loss is total data loss, which is disqualifying for any real-data pilot. Naive mitigations all fail in specific ways: (a) copying a live RocksDB directory tears SST/WAL files and produces unopenable backups; (b) backing up RocksDB alone silently drops the HNSW partition files, api_keys.json, and the blob-addressed canonical text living in the same store — a "successful" restore that cannot answer queries; (c) a backup taken before a GDPR erasure and restored after it resurrects erased data unless restores replay the erasure ledger (gdpr-erasure spec §8); (d) without content hashes a bit-rotted backup is discovered at restore time, which is the worst possible moment.

3. Proposed Solution¶

Normative decisions:

1. A backup is an atomic, online, point-in-time capture: RocksDB via the engine's native Checkpoint (hardlink-based, consistent, non-blocking — writers never stall), plus SEALED vector partition files (open/dirty buckets are deliberately excluded: derived indexes are rebuildable by doctrine F4 and are rebuilt on demand after restore), plus api_keys.json, plus a manifest.
2. The manifest records a BLAKE3 hash, size, and relative path for EVERY file in the backup, the CF inventory, engine and format versions, creation time, and the erasure watermark (creation time in µs; PR-081's ledger entries after this instant must replay on restore).
3. verify is independent of restore: hash-check every file and confirm the manifest is internally complete. Restore MUST verify first and refuse on any mismatch.
4. Restore targets an EMPTY directory only. Restoring over a live or non-empty data dir is refused, always — no in-place restore exists.
5. Erasure-replay seam (PR-081 as built; see §3.10): restore checks for an erasure ledger. Ledger absent → proceed (nothing to replay, recorded in output). Ledger present but unreadable → REFUSE. Ledger present and readable → entries newer than the manifest's erasure watermark REPLAY before the restored dir may serve traffic. The PR-083 interim rule (refuse whenever a ledger is present) is LIFTED.
6. Retention: backup.keep prunes the oldest backups only AFTER a new backup completes and verifies — pruning can never run on a failed backup path.
7. Backups are operator-boundary artifacts (local directory; the CLI is the surface). No REST/gRPC exposure — local-dir IS the v1 contract; remote transport is the operator's rsync/object-store job, shown as an explicit runbook step (review decision 2026-07-05).
8. Restore runs the consistency checker before the restored dir may serve — mandatory, no --fast opt-out (review condition, 2026-07-05). Hash verification proves the files survived the trip; it does not prove the database is coherent. Restore executes the index consistency check (and bitmap/HNSW rebuild where derived state is absent) as part of the restore command; a failure is a failed restore. Restore is rare and serious; integrity bypasses get used exactly once, on the worst possible day.
9. The v1 backup boundary has TWO surfaces (review condition, 2026-07-05): core data_dir AND app-layer PostgreSQL. PG holds the wrapped DEK records (field-encryption), the audit log, sessions/role mappings (sso-auth), and the future subject index — a perfect core backup with a lost PG means classified fields are permanently unreadable: data loss with extra steps. The backup procedure captures a pg_dump alongside the core artifact; the manifest records its presence and hash; the restore drill exercises BOTH together. A core-only backup is recorded as partial: true in its manifest and the create output warns.

3.10 Erasure ledger replay (addendum, PR-081)¶

restore runs the mandatory consistency gate (§3.8) and THEN, when a readable erasure ledger is supplied, replays it (gdpr-erasure spec §3.6): GDPR ledger entries with at_us greater than the manifest's erasure_watermark_us are replayed in file order through their content-free erasure programs (expected at {ledger_parent}/erasure-programs/), which are idempotent by construction (deleting an absent key, masking an already-masked range, and rewriting a blocklist row are no-ops). After replay, pass-1 structural verification (gdpr-erasure §3.8) re-runs against every replayed program before the dir may serve. A missing program, a program-hash mismatch, or any execution failure is a FAILED restore — the target dir is removed (the PR-080 consistency-gate posture). The full pass-2 byte scan needs the plaintext descriptor and therefore runs as an operator step with the erasure request record (telha erase verify); the restore report says replay ran and re-verified, and the runbook names the byte-scan step.

Blocklist rebuild + secret re-seed. Replay rebuilds the erased-content blocklist rows from the ledger so post-restore ingestion keeps refusing erased content. Because the blocklist digests are KEYED (hmac_blake3_v1), replay of a PRE-erasure backup re-seeds the per-tenant erasure secret from the offsite {data_dir}/erasure-secret.json mirror before rebuilding rows, so dedup checks against the rebuilt rows resolve. Lost-ledger reality: a total data-dir loss loses ledger entries written after the last backup; the erasure requests survive app-side (their PG backup surface) and the operator re-executes any uncovered request before serving (app leg). Runbook (offsite copy + post-restore scan). Every backup transport step copies erasure-ledger.jsonl, erasure-programs/, and erasure-secret.json offsite alongside the backup; after a restore-with-replay, the operator runs the pass-2 byte scan per erasure request and records residuals in the completion report.

4. Architecture¶

telha backup create --out <dir>          (server may be live)
  ├─ rocksdb Checkpoint::create_checkpoint → <dir>/<stamp>/rocksdb/
  ├─ copy sealed vector files             → <dir>/<stamp>/vectors/…
  ├─ copy api_keys.json                   → <dir>/<stamp>/api_keys.json
  ├─ hash everything (BLAKE3)             → <dir>/<stamp>/MANIFEST.json
  └─ retention: prune oldest beyond backup.keep

telha backup verify  --from <dir>/<stamp>   (hashes + completeness)
telha backup restore --from <dir>/<stamp> --data-dir <EMPTY target>
  ├─ verify (mandatory)
  ├─ erasure-ledger check (absent → proceed; unreadable → refuse)
  ├─ copy rocksdb/ + vectors/ + api_keys.json into target
  └─ open engine read-only, CF inventory sanity check, report
runtime ticker (optional): [backup] enabled → create every interval_hours

5. Data Models¶

MANIFEST.json: {v: 1, createdAtUs, telhaVersion, engine: "rocksdb", cfs: [names…], erasureWatermarkUs, files: [{path, bytes, blake3}]} — files covers every byte in the backup including the RocksDB checkpoint contents; path is stamp-relative with forward slashes. Backup directory naming: <out>/telha-backup-<createdAtUs> (µs timestamp = collision-free ordering key for retention). Vector selection rule: files under data_dir/vectors/** whose bucket has a .manifest.json sidecar (sealed); open buckets are skipped and noted in the create output — the partition manager rebuilds them from the vectors CF on first search after restore (existing corruption/rebuild path).

6. API Contracts¶

CLI only (operator boundary): telha backup create --out <dir> [--data-dir …] (exit 0 = created AND verified; prints stamp path + counts), telha backup verify --from <stamp-dir> (exit 0 = every hash matches), telha backup restore --from <stamp-dir> --data-dir <target> (exit ≠ 0 on: verify failure, non-empty target, unreadable erasure ledger). Deliberately NOT configurable: verify-before-restore, the empty-target rule, manifest hashing, the ledger-unreadable refusal. Failure taxonomy: create failures leave a partial <stamp>.partial directory that is never counted by retention and is safe to delete.

7. UI/UX¶

Operator CLI output is the artifact trail: create prints the stamp, file/byte counts, skipped-open-bucket count, and prune actions; verify/restore print per-file failures. Each command emits structured audit log lines (backup_created, backup_verified, backup_restore_*). The pilot runbook schedules create via the platform scheduler (cron/Task Scheduler) until the runtime ticker is enabled.

8. Configuration¶

[backup]: enabled (false — the runtime ticker), dir (backup target), interval_hours (24), keep (7). The CLI works regardless of enabled. keep = 0 disables pruning.

9. Alternatives Considered¶

• Copying the live data dir. Rejected: torn SSTs/WAL; unopenable backups discovered at restore time.
• RocksDB BackupEngine (incremental). Deferred: incremental state adds a second stateful system to operate and it cannot cover the non-RocksDB files anyway; Checkpoint + manifest is one mechanism for everything. Revisit when data volume makes full checkpoints expensive (OQ-2).
• Logical export (scan → NDJSON). Rejected as the primary path: slow, loses transaction-time internals needed for bitemporal fidelity, and reimports are not byte-faithful. May later serve cross-version migration.
• Backing up open vector buckets. Rejected: they are mid-mutation mmap state; sealed files + rebuildability (F4) make them dead weight with corruption risk.
• Built-in offsite transport (S3). Deferred (OQ-1): operators already own transport tooling; scope creep before the local path is proven.

10. Implementation Approach¶

One PR (PR-080): StorageEngine::checkpoint(&self, dir) trait method (RocksDB = native Checkpoint; MemoryEngine = serialize CFs to files, keeping the conformance suite meaningful) → src/backup/mod.rs (create/verify/restore/prune + manifest) → CLI telha backup … → runtime ticker behind [backup].enabled → tests per §12. The erasure seam is a function with a documented contract and a no-op body plus the refusal path; PR-081 fills the body.

11. Migration Path¶

Greenfield operational surface; no data-format changes. Manifest v is additive. Backups taken by older builds are readable as long as v and the engine string match; restore refuses unknown v loudly.

12. Success Metrics¶

• backup_roundtrip_equality: write fixtures → backup → keep writing → restore into a fresh dir → queries at pre-backup coordinates return identical results from both dirs; post-backup writes exist only in the original.
• backup_online_consistency: backup taken while a writer thread is active verifies and opens cleanly.
• backup_tamper_detected: flipping one byte in any backed-up file fails verify AND blocks restore.
• backup_vector_restore: vector search works after restore (sealed files load; an excluded open bucket rebuilds via the existing rebuild path).
• restore_refuses_nonempty_target: restore into a non-empty dir fails without writing.
• restore_erasure_seam: ledger absent → proceeds with a recorded no-op; ledger path present but unreadable → refused.
• backup_retention_prunes: keep=2 retains exactly the two newest stamps; a failed create prunes nothing.

13. Open Questions¶

• OQ-1: Offsite transport (object storage) built in? Resolved 2026-07-05 (J. Porter): v1 is local-dir + runbook-owned offsite copy; native S3/GCS/Azure before the first real pilot is premature platform polish. Revisit when a customer is expected to run backup operations themselves.
• OQ-2: Incremental backups via BackupEngine once checkpoint sizes hurt? Resolved 2026-07-05 (J. Porter): deferred WITH a written tripwire — if a checkpoint regularly exceeds 30 minutes or the artifact exceeds 50GB, incremental design is automatically reopened (tripwire lives in the runbook; deferrals without tripwires become quiet architecture debt).

14. Changelog¶

• 2026-07-06 — v0.4: erasure ledger replay landed (§3.10 addendum, PR-081). The PR-083 interim refuse-when-ledger-present rule is LIFTED: restore now replays GDPR ledger entries past the manifest watermark through their content-free idempotent programs, re-runs pass-1 structural verification, and removes the target dir on any replay failure; an unreadable ledger still refuses. Replay rebuilds the keyed-digest blocklist and re-seeds the per-tenant erasure secret from the offsite mirror. RestoreReport gains erasure_entries_replayed; the runbook gains offsite-copy of ledger/programs/secret and a post-restore pass-2 byte-scan step. The pass-2 byte scan stays an operator step (it needs the plaintext descriptor).
• 2026-07-05 — v0.3: review conditions (1) and (2) implemented same day — the as-built no longer owes them. (1) Restore consistency gate: restore opens the copied store and cross-verifies every tenant's covering temporal indexes (consistency::check_temporal_indexes over scopes enumerated from the primary CFs — a system-maintenance whole-store scan, the reaper's key-spec §5.3 exception class). Any missing/orphaned index entry fails the restore AND removes the target dir, so an incoherent store can never be served by accident; no flag disables it. Gate results ride the report/CLI output (tenants_checked). HNSW rebuild stays the existing lazy load-failure path; bitmap rebuild applies only where [bitmap] is enabled (default off) via the existing CLI. Tested by an in-crate sabotage test (hash-perfect files, one deleted index row → ConsistencyGate error, target removed). (2) Two-surface capture: create gains pg_database_url (CLI --pg-database-url, else the env var named by new config backup.pg_database_url_env, default DATABASE_URL; ticker reads the same). Present → pg_dump --format=custom into the stamp (hash- manifested like everything else; a FAILED requested capture fails the backup — never a silent degrade). Absent → manifest partial: true + loud warnings at create AND restore. Restore surfaces the dump with the pg_restore runbook command (core never writes into a live PG). Real-pg_dump round-trip is exercised in the pilot restore drill, not CI (no PG toolchain on the dev box; the failure and partial paths are unit-tested).
• 2026-07-05 — v0.2: Approved with conditions (review by J. Porter). Conditions incorporated: (1) restore consistency gate mandatory, no opt-out (§3.8) — provable state is the brand, restores must prove coherence not just file integrity; (2) PostgreSQL named as the second v1 backup surface (§3.9) — DEK records/audit/subject-index loss is data loss with extra steps; manifest + restore drill cover both; (3) local-dir confirmed as the v1 contract with runbook-owned offsite step; (4) incremental deferral gains a concrete tripwire (>30min checkpoint or >50GB artifact reopens the design). PR-080 as-built must add (1) and (2) as follow-up work before any pilot holds real data. Cross-spec: restore is ledger-aware per the shared recovery contract (gdpr-erasure + retention-archival ledger entries replay/refuse per their specs).
• 2026-07-05 — v0.1: initial draft (PR-080; build doc Phase 3c "URGENT — precedes real-data pilots"). Written spec-first in-session; engineering review by Claude under the standing delegation, J. Porter countersign/veto pending.