Skip to content

Spec: Retention Policies & Archival — Lifecycle for an Append-Only Store

File: .ai/specs/core-engine/2026-07-05-retention-archival.md Status: Approved (review 2026-07-05 by J. Porter — conditions incorporated in v0.2: normative dry-run, double hold check, archive-encryption wording corrected; verify-before-delete confirmed as a hard gate) Owners: Storage lane; reviewers: app-layer lane (policy surface), legal-adjacent review recommended (retention schedules are compliance artifacts) Related: gdpr-erasure spec (shared executor and ledger — this spec resolves its OQ-2; erasure ALWAYS outranks retention), backup-restore spec (archive artifacts follow its manifest/verify conventions; instance backups are NOT archives), ingestion-provenance spec (source versions, orphaned streamed versions), hnsw-partitioning spec (vector cleanup on expiry), job-queue-leases spec (retention runs as system jobs).

1. Overview

Defines how tenants control the lifecycle of their data: per-label/source-kind retention policies (keep forever, archive after N days, delete after N days), legal holds that override everything, the archival slice format, the retention executor (shared with GDPR erasure), and the cleanup of operational debris (orphaned streamed source versions). Append-only is the doctrine for history within retained data; retention is the governed boundary of how long "forever" is.

2. Problem Statement

An append-only world-state layer grows monotonically by design. Without lifecycle controls: (a) storage grows without bound and the 27GB-target-dir problem becomes the customer's data problem; (b) enterprises with mandated retention schedules (HR data 7 years, complaints 6 years, chat exports 90 days) cannot adopt at all — indefinite retention is a compliance violation, not a feature, for regulated data classes; (c) "delete after N days" implemented ad hoc would bypass the erasure ledger and verification machinery that makes deletion defensible; (d) abandoned streamed source versions (crash-orphaned, hash=None — a recorded PR-032 debt) accumulate as unreferenced junk with no owner; (e) archive-then-delete without a restorable artifact turns a retention policy into silent data loss.

3. Proposed Solution

Normative decisions:

  1. Policies are per (tenant, selector) where selector = a node label or a source kind (connector namespace, ingest format). Policy ∈ keep_forever (default for everything) | archive_after {days} | delete_after {days} | archive_then_delete {archive_days, delete_days}. Nothing expires unless a policy explicitly says so.
  2. Legal hold outranks policy; erasure outranks legal hold. A hold (per tenant, per selector, or per logical id) suspends archive AND delete actions for matching entities until lifted; holds are append-only records with actor and reason. GDPR erasure (PR-081) executes regardless of holds — Article 17 is not suspendable by an internal flag — and the ledger records both when they collide. Holds are checked TWICE (review condition, 2026-07-05): during run planning AND again immediately before each entity's delete stage executes — a hold set between planning and execution must win; the skipped entity is ledgered with the hold ref.
  3. The clock is transaction time of the entity's newest version. An entity (all its versions, as one unit) is eligible when now - max(tx_time.start) > policy window — i.e. time since the entity last changed. Valid-time-based expiry ("expired contracts") is a query concern, not a retention concern; retention reasons about knowledge age, not business validity.
  4. v1 expires whole entities only. All versions, spans, adjacency, index entries, vectors, and bitmap postings for the logical id go together, via the same executor machinery as subject-level erasure (gdpr-erasure OQ-2 = resolved: same executor, different trigger, same ledger with method: retention). History-trimming (dropping old versions while keeping the current one) is explicitly deferred: partial-history deletion inside a bitemporal chain has correction-semantics hazards that need their own review.
  5. Archival precedes any policy delete. archive_after exports eligible entities into tenant archive slices; archive_then_delete deletes only entities whose covering archive artifact has verified. A failed or unverified archive blocks the delete stage for its entities — retention can never destroy the only copy.
  6. Archive slice format: length-prefixed msgpack frames {cf, key, value} for every row belonging to the archived entities (all CFs, including span reverse-index and vector rows), plus a manifest per the backup-restore spec's conventions (BLAKE3 per file, format version, entity list, policy + ledger refs). Slices are read-only artifacts; v1 restore of an archive is side-loading into a FRESH data dir for inspection (same tooling as backup restore), not merge-back into the live tenant.
  7. Dedup-aware blob handling: canonical-text chunks are content-addressed and may be shared across source versions; a chunk is deleted only when its last referencing source version expires (reference check at execution time, recorded in the ledger entry).
  8. Orphan sweep is part of the retention job: provisional streamed source versions with canonical_text_hash = None whose lease generation is dead and whose age exceeds orphan_ttl_days are deleted (with ledger entries) — retiring the PR-032 recorded debt. Orphans are operational debris, not tenant data; no policy opt-in required, only the TTL.
  9. Every retention action (archive, delete, orphan sweep, hold collision) writes erasure-ledger entries with method: retention | retention_archive | orphan_sweep; completion proofs follow the gdpr-erasure conventions (row counts, artifact refs). Retention shares the ledger because "prove what you deleted and why" is the same problem whether the trigger is law or policy.

3.10 GDPR erasure interaction (addendum, PR-081)

PR-081 lands the GDPR erasure leg over this spec's executor and ledger. As-built consequences:

  • erasure_pending() filled + archive supersession. The interim erasure_pending() stub now consults the erasure request store ({data_dir}/erasure-requests/): open (non-completed, non-failed) requests pin their person_ids and whole_sources, and retention skips those entities (erasure outranks everything and owns the deletion). When erasure masks or deletes rows that a retention archive slice already captured, PR-081 rewrites the affected slice (erased-row frames dropped; masked-blob chunk frames replaced with the live masked bytes), re-hashes and re-manifests it with a supersedes: <original manifest blake3> field, verifies the rewrite, and only THEN destroys the original (rewrite-verify-destroy — the verify-before-delete doctrine applied in reverse). RetentionOps::remap_archive_stamp keeps the per-tenant archive-coverage index pointing at the rewritten slice. Side-load refuses a slice whose manifest is recorded superseded in the erasure ledger (telha retention archives sideload --erasure-ledger). Blocklist rows are exempt from retention expiry by construction — they live outside entity rows in the reserved sources-CF slot (§3.7 of composite-key-layout).
  • Ledger method enum gains gdpr_* values. The shared LedgerEntry gains gdpr_tenant | gdpr_subject | gdpr_subject_pg | gdpr_residual_review methods plus additive fields (request_ref, request_digest, program_ref + program_hash, retired_digests (KEYED, hmac_blake3_v1), superseded_manifests, pg_counts, residual). Retention entries are untouched; replay only touches gdpr_* methods. The archive manifest gains the optional supersedes field.

4. Architecture

[retention] policies (per tenant, per selector)     legal holds (append-only)
        └──────────────┬───────────────────────────────────┘
              system:retention job (interval, system namespace)
  scan: selector → candidate entities (newest-version tx age > window)
        ├─ hold match? ──▶ skip + ledger note (hold ref)
        ├─ archive stage: export slices ─▶ verify (blake3) ─▶ manifest
        └─ delete stage: only archive-verified (or delete-only policy)
             └─ shared erasure executor: versions, spans, adjacency,
                indexes, vectors (+partition rebuild), bitmap postings,
                last-reference blob chunks ─▶ ledger completion proof
  orphan sweep: dead provisional source versions > orphan_ttl_days
telha retention policy ls|set|rm · hold ls|set|lift · archives ls|inspect

5. Data Models

Policy rows (schema CF, system-keyed per tenant): {selector: {label} | {source_kind}, action, windows, created_by, created_at} — policy changes are new versions (append-only, auditable). Hold rows likewise: {scope: tenant|selector|logical_id, reason, actor, created_at, lifted_at?}. Archive manifest: backup-restore manifest shape plus {entities: [logical_id], policy_ref, ledger_refs}. Ledger entries extend the gdpr-erasure record with method and archive_ref? fields (additive; that spec's engineering review co-owns the final shape).

6. API Contracts

Operator-scoped only (no tenant self-service in v1, no MCP exposure — same posture as erasure): CLI telha retention policy set --tenant --label|--source-kind --action --days, policy ls|rm, hold set|lift|ls, archives ls|inspect. REST admin mirror under /admin/retention/* behind the same admin auth boundary as /admin/erasure (dual approval NOT required for policy set — policies act after a window, giving review time; hold lift on held-for-legal selectors SHOULD warn). Failure taxonomy: archive verify failure → delete stage blocked for those entities + operator alert; executor failures follow erasure-job semantics (staged, resumable, ledgered).

7. UI/UX

CLI-first (PR-086 console adds read views later). Dry-run is normative, not a courtesy (review condition, 2026-07-05): operator-only does not mean blind execution. telha retention preview MUST report, per selector: eligible entities, entities blocked by holds (with hold refs), estimated archive size, delete count, and remaining backlog beyond max_entities_per_run — so operators see blast radius before any window fires. Audit events for policy set/rm, hold set/lift, each run's counts, and every hold collision. The runbook (PR-085) ships retention-schedule templates for common regimes rather than encoding jurisdictions in the product.

8. Configuration

[retention]: enabled (false), interval_hours (24), orphan_ttl_days (14), archive_dir (defaults under backup target), max_entities_per_run (10k — bounded runs; the scan resumes next interval). Deliberately NOT configurable: hold-beats-policy, erasure-beats-hold, archive-verify-before-delete, ledger emission, whole-entity granularity.

9. Alternatives Considered

  • RocksDB TTL / compaction-filter expiry. Rejected: silent, unprovable, ignores derived state (vectors, bitmaps, blobs), and bypasses the ledger; the same reason gdpr-erasure rejected lazy deletion as primary.
  • Valid-time-based expiry. Rejected for the clock: "the contract ended 7 years ago" and "we last touched this record 7 years ago" are different regimes; the latter is what retention schedules regulate. Valid-time cleanup queries remain available to operators via normal query + explicit erasure if truly wanted.
  • History trimming in v1. Deferred (§3.4): dropping superseded versions while keeping current interacts with correction semantics (a trimmed chain can no longer prove what was believed when) — needs its own review with legal input.
  • Merge-restore of archives into the live tenant. Deferred: same key-collision and ledger-consistency problems the backup spec deferred for tenant slices; side-load-and-inspect covers the audit/discovery use case that actually drives archive reads.
  • Per-entity retention properties (TTL on the record). Rejected: policy scattered across millions of rows is unauditable; selector-level policy rows are the reviewable artifact.
  • A separate retention ledger. Rejected: one ledger with a method field keeps "what left the store and why" answerable in one place.

10. Implementation Approach

One PR (PR-083), after PR-080 (manifest/verify conventions) and coordinated with PR-081's ledger schema (the executor and ledger are shared; whichever PR lands first carries the executor skeleton, the other extends it — build doc sequences 081 first only for the legal gate, so PR-083 SHOULD land the executor skeleton if it goes first): policy/hold models + CLI/admin API → scanner + eligibility (+ preview) → archive writer over backup-restore conventions → delete stage via shared executor → orphan sweep → tests per §12.

11. Migration Path

Greenfield; default keep_forever means enabling the feature changes nothing until a policy is written. Archive format versioned in its manifest. Pre-existing orphaned streamed versions are swept by the first enabled run (counts surfaced in the run report). If PR-081 lands first, its ledger gains the additive method field here; if PR-083 lands first, PR-081 adopts the shared executor.

12. Success Metrics

  • retention_expiry_boundary: entity with newest-version tx age just under/over the window is retained/actioned respectively; the clock uses tx time, not valid time (fixture proves the distinction).
  • retention_legal_hold: held entity survives an otherwise-eligible run with a ledger note; lifting the hold makes the next run action it; erasure of a held subject still executes (collision ledgered).
  • retention_archive_then_delete: entities are deleted only after their archive verifies; a corrupted archive blocks deletion for exactly its entities and alerts.
  • retention_archive_sideload: an archive slice side-loads into a fresh data dir and its entities are queryable at historical coordinates.
  • retention_shared_blob_refcount: two source versions sharing a canonical chunk — expiring one keeps the chunk; expiring both removes it; ledger records the reference decision.
  • retention_orphan_sweep: dead provisional versions older than TTL are removed with ledger entries; live streams and fresh provisionals are untouched.
  • retention_whole_entity: after deletion, zero rows for the logical id across ALL CFs (versions, spans, adjacency, both temporal indexes, vectors, bitmap postings), partitions rebuilt, verification scan green.
  • retention_preview_accuracy: preview counts equal what the next run actions given no interim writes; preview reports all normative dry-run fields (eligible, hold-blocked with refs, archive size estimate, delete count, remaining backlog).
  • retention_hold_recheck: a hold set AFTER run planning but BEFORE the delete stage still blocks that entity's deletion; the skip is ledgered with the hold ref (the double-check of §3.2).

13. Open Questions

  • OQ-1: Tenant-admin self-service policies in v1? Resolved 2026-07-05 (J. Porter): operator-only. Retention is data destruction on a timer; self-service arrives only with the console's preview/confirm/impact-summary UX and probably second approval for destructive policy changes — and cautiously even then.
  • OQ-2: Archive storage encryption? Resolved 2026-07-05 (J. Porter, wording corrected): archives inherit the platform's SINGLE shared artifact-encryption stance for backup-family artifacts (currently: no built-in artifact encryption — classified fields are already ciphertext, operators layer disk/transport encryption; if the backup spec ever introduces artifact encryption, archives adopt it in the same change). One stance, one config surface, one runbook paragraph — the original draft's pointer to "backup OQ-2" was wrong (that OQ is incremental backups).
  • OQ-3: First-enable backlog drain? Resolved 2026-07-05 (J. Porter): bounded runs confirmed; a multi-day drain is acceptable because retention windows are measured in months or years — urgency here would be manufactured. Visible remaining counts are the operational contract.

14. Changelog

  • 2026-07-06 — v0.4: GDPR erasure interaction (§3.10 addendum, PR-081). erasure_pending() filled from the erasure request store; archive supersession (rewrite-verify-destroy + supersedes manifest field + side-load refusal of superseded slices, with remap_archive_stamp keeping the coverage index coherent); blocklist rows exempt from expiry by construction; ledger method enum gains gdpr_tenant | gdpr_subject | gdpr_subject_pg | gdpr_residual_review plus the additive PR-081 fields. Retention entries and the executor mechanism are unchanged; PR-081 consumed execute_erasure_at, EntityRows coordinate accessors, and post_delete_maintenance without redesign.
  • 2026-07-06 — v0.3: As built (PR-083). core-engine/src/retention/ = policy/hold/archive-index documents (append-only versioned rows in the schema CF under namespaced label hashes retention/…, mirroring the vector_model/… precedent), planner (plan_at IS the normative dry-run with the pinned §7 fields), executor (executor.rs — the SHARED erasure-executor skeleton per §10: mechanism-only row collection + DeleteRange/batch removal; PR-081 consumes RetentionOps::execute_erasure_at, which executes over holds and ledgers the collision), archive slices (archive.rs — length-prefixed msgpack {cf, key, value} frames + backup-convention BLAKE3 manifest with entities/policyRef/ledgerRefs, .partial staging, verify-before-delete hard gate, side-load into fresh dirs only), shared ledger (ledger.rs — JSONL erasure-ledger.jsonl in the data dir; method + archive_ref fields per §5; a present ledger keeps backup restores refusing until PR-081 replay). StorageEngine::delete_range added (native RocksDB DeleteRange; scan+delete default). system:retention job runs in-process via a [retention]-gated ticker reusing the jobs CF + reaper semantics. Operator surface: telha retention policy|hold|preview|run|archives CLI + /admin/retention/* REST behind a new "admin"-audience signed token (telha api-key admin-token) — /admin/erasure does not exist yet (PR-081 held), so this PR establishes the admin auth boundary. Orphan sweep (§3.8) deletes provisional streamed versions (canonical_text_hash = None, explicit provisional blob id) past orphan_ttl_days whose owning job no longer holds a lease — retiring the PR-032 debt (ingestion-provenance §16). §12 tests land in tests/retention_e2e.rs (incl. retention_hold_recheck). Deviations: archive-size preview is a version-row byte floor, not a full-row estimate; incident edge entities cascade with an expired node (an edge to a deleted endpoint is undiscoverable junk); archive_after idempotence is tracked in a per-tenant archive-index document.
  • 2026-07-05 — v0.2: Approved (review by J. Porter). Conditions incorporated: (1) dry-run made normative with pinned report fields (§7); (2) legal holds checked twice — at planning and immediately before delete (§3.2) — with test retention_hold_recheck; (3) OQ-2 wording corrected to the shared artifact-encryption stance (the draft's "backup OQ-2" pointer was to incremental backups); (4) archive verify-before-delete confirmed as a hard gate. Clock (tx-time age), whole-entity-only expiry, and precedence chain (erasure > hold > policy) confirmed as drafted. PR-083 unblocked; coordinate executor/ledger schema with PR-081 per §10.
  • 2026-07-05 — v0.1: initial draft for review (PR-083). Resolves gdpr-erasure OQ-2 (shared executor/ledger, method field); defines the archive slice format the backup spec deferred; retires the orphaned-streamed-version debt via the sweep.