Skip to content

Retention & archival

Policy · legal holds · archival slices

An append-only world-state layer grows monotonically by design. Retention is the governed boundary on how long "forever" actually is: per-selector policies that archive and/or delete whole entities once they age past a window, always subordinate to legal holds and to GDPR erasure.

Normative spec

retention-archival (PR-083) is the normative source: the policy model, the precedence chain, the archive slice format, and the shared executor/ledger it reuses from GDPR erasure.

The policy model

Policies are set per (tenant, selector), where a selector is either a node label or a source kind (a connector namespace or ingest format), never both at once.

Action Effect
keep_forever The default for every selector not explicitly configured. Nothing happens.
archive_after {archive_days} Export eligible entities into a tenant archive slice after archive_days.
delete_after {delete_days} Delete eligible entities after delete_days, without archiving.
archive_then_delete {archive_days, delete_days} Archive first; delete only after the covering archive artifact has verified.

Nothing expires unless a policy explicitly says so, enabling the [retention] ticker on a tenant with no policies is a complete no-op.

telha retention policy set --tenant <uuid> --org <uuid> \
  --label CONTRACT_DOCUMENT --action archive_then_delete \
  --archive-days 365 --delete-days 730 --data-dir <dir>

telha retention policy ls --tenant <uuid> --org <uuid> --data-dir <dir>
telha retention policy rm --tenant <uuid> --org <uuid> --label CONTRACT_DOCUMENT --data-dir <dir>

--source-kind is the alternative selector to --label (they are mutually exclusive on every policy command). Policy commands, like most telha subcommands, open the data dir directly, stop the server first, or use the /admin/retention/* REST mirror against a live one.

The clock: transaction time, not valid time

Eligibility is computed from transaction time of the entity's newest version: an entity becomes eligible when now - max(tx_time.start) > policy window, time since Telha last recorded a change to it. This is a deliberate choice: "the contract expired 7 years ago" (valid time) and "we haven't touched this record in 7 years" (transaction time) are different regimes, and retention schedules regulate the latter, how long you may retain knowledge, not how long a fact was true in the world. If you want valid-time-based cleanup ("delete contracts that lapsed years ago"), that is a query-plus-explicit-erasure operation, not a retention policy.

Whole entities only

v1 expires whole entities: all versions, spans, adjacency, both temporal indexes, vectors, and bitmap postings for a logical id are removed together, through the same executor machinery used for GDPR subject-level erasure. Partial-history trimming (dropping superseded versions while keeping the current one) is explicitly deferred, it interacts with correction semantics in ways that need their own review.

Always preview first

telha retention preview is the normative dry-run, it is not a courtesy, and it never writes anything. Run it before every policy change and before every scheduled window fires for the first time on a tenant.

telha retention preview --tenant <uuid> --org <uuid> --data-dir <dir>

Per selector, the preview reports:

  • Eligible entity count.
  • Entities blocked by a legal hold, with the hold's reference.
  • Estimated archive size (a version-row byte floor in the current implementation, not a full-row estimate).
  • Delete count.
  • Remaining backlog beyond retention.max_entities_per_run.

Legal hold outranks policy. GDPR erasure outranks legal hold. A hold suspends both archive and delete actions for matching entities until it is explicitly lifted. Holds are append-only records (an actor and a reason are required); lifting a hold leaves the original record intact with lifted_at set.

telha retention hold set --tenant <uuid> --org <uuid> \
  --label CONTRACT_DOCUMENT --reason "Smith v. ACME discovery" --data-dir <dir>
# scope: --label | --source-kind | --id <logical uuid> | omit all three for a tenant-wide hold

telha retention hold ls   --tenant <uuid> --org <uuid> --data-dir <dir>
telha retention hold lift <hold-uuid> --tenant <uuid> --org <uuid> --data-dir <dir>

Lifting a hold makes matching entities actionable starting on the next retention run, not immediately.

Holds are checked twice

A hold is checked once during run planning and again immediately before each entity's delete stage executes. A hold set in the gap between planning and execution still wins, the entity is skipped and the skip is ledgered with the hold's reference. This is deliberate: an in-flight retention run must never race a hold that arrives mid-run.

GDPR erasure of a held subject still executes regardless, Article 17 is not suspendable by an internal flag, and the ledger records the collision between the hold and the erasure. See Architecture › GDPR erasure.

Running a pass

telha retention run --tenant <uuid> --org <uuid> --data-dir <dir>

One bounded pass executes, in order: archive → verify (a hard gate) → hold-rechecked deletes → orphan sweep. It prints a JSON report and the ledger path. Each pass is bounded by retention.max_entities_per_run (default 10000); a backlog beyond that ceiling simply resumes on the next scheduled interval, retention windows are measured in months or years, so a multi-day drain on first enable is expected and acceptable, not an incident.

The same pass runs automatically on the [retention] ticker's interval when retention.enabled = true in config (see Configuration reference).

Archival vs. tombstoning: do not confuse them

These are two different mechanisms and this is the most common point of operator confusion:

Tombstone (deletion, ordinary write path) Archival (retention)
What triggers it A normal delete through the graph/record API A retention policy's archive_after or archive_then_delete action
What happens to the data A new tombstone version is written; valid time opens at the delete instant. History stays intact and reconstructable at any past (validTime, txTime). Rows for the entity are exported into an archive slice, then (for archive_then_delete) the entity is fully removed from the live store via the shared erasure executor.
Is it queryable afterward? Yes, the tombstoned entity reads as absent from the delete instant forward, but all prior versions remain queryable via atTxTime. No, a deleted-after-archive entity is gone from the live tenant. The archive slice itself is queryable only by side-loading it into a fresh, separate data dir.
Reversible? N/A, nothing was removed; it is append-only history. Only by side-loading the archive slice for inspection; there is no merge-back into the live tenant in v1.

See The tritemporal model for how tombstones fit into append-only history. Archival is the separate, policy-governed answer to "how long do we keep whole entities around at all," layered on top of that append-only foundation.

Archive slice format and lifecycle

Archive slices use length-prefixed msgpack frames ({cf, key, value}) for every row belonging to the archived entities, every column family, including the span reverse-index and vector rows, plus a manifest that follows the backup-restore spec's conventions: BLAKE3 hash per file, format version, the entity list, and policy/ledger references. Archive slices are read-only artifacts; a archive_then_delete policy only proceeds to the delete stage for entities whose archive has verified. A failed or unverified archive blocks the delete stage for exactly those entities, retention can never destroy the only remaining copy of something.

telha retention archives ls --data-dir <dir>
telha retention archives inspect --from <stamp-dir>
telha retention archives sideload --from <stamp-dir> --data-dir <FRESH empty dir>

Side-loading is inspection-only: it loads a verified slice into a fresh data dir so an operator (or auditor) can query the archived entities at their historical coordinates. It is never merged back into the live tenant.

Superseded archives refuse to side-load

If a GDPR erasure later touches rows that an archive slice already captured, the erasure process rewrites the affected slice (dropping erased-row frames, replacing masked-blob chunks with the live masked bytes), re-hashes it, and records it as supersedes: <original manifest blake3>. telha retention archives sideload refuses a slice that the erasure ledger records as superseded, pass --erasure-ledger to have the CLI check.

Dedup-aware blob handling

Canonical-text chunks are content-addressed and can be shared across multiple source versions. A chunk is deleted only when the last source version referencing it expires; the reference check happens at execution time and the decision is recorded in the ledger entry. Expiring one of two versions sharing a chunk keeps the chunk; expiring both removes it.

Orphan sweep

Every retention run also sweeps operational debris: provisional streamed source versions (canonical_text_hash = None) whose owning job no longer holds a lease and whose age exceeds retention.orphan_ttl_days (default 14) are deleted, with ledger entries. This needs no policy opt-in, it is cleanup of crash-orphaned state, not tenant data, only the TTL applies.

Every action is ledgered

Every retention action (archive, delete, orphan sweep, hold collision) writes an entry to the same erasure ledger GDPR erasure uses ({data_dir}/erasure-ledger.jsonl), tagged with method: retention | retention_archive | orphan_sweep. This ledger is what later backup restores check for replay before serving traffic, retention and GDPR erasure share one ledger because "what left the store and why" should be answerable in one place regardless of which process triggered the removal.

The /admin/retention/* REST mirror

A live server exposes the same policy/hold/preview surface without a stop, behind an "admin"-audience bearer token:

ADMIN=$(telha api-key admin-token --tenant <uuid> --org <uuid> --config <toml>)
H="Authorization: Bearer $ADMIN"
BASE=http://127.0.0.1:7625

curl -fsS $BASE/admin/retention/policies -H "$H"
curl -fsS -X POST $BASE/admin/retention/policies -H "$H" -H "Content-Type: application/json" \
  -d '{"label":"CONTRACT_DOCUMENT","action":"archive_then_delete","archiveDays":365,"deleteDays":730,"actor":"jp"}'
curl -fsS -X POST $BASE/admin/retention/policies/remove -H "$H" -d '{"label":"CONTRACT_DOCUMENT"}'
curl -fsS $BASE/admin/retention/holds -H "$H"
curl -fsS -X POST $BASE/admin/retention/holds -H "$H" \
  -d '{"label":"CONTRACT_DOCUMENT","reason":"Smith v. ACME","actor":"jp"}'
curl -fsS -X POST $BASE/admin/retention/holds/<hold-uuid>/lift -H "$H"
curl -fsS $BASE/admin/retention/preview -H "$H"

Bodies are camelCase (sourceKind, logicalId, archiveDays, deleteDays). Errors use the standard envelope with code RETENTION_INVALID. There is no run endpoint on the REST mirror, a retention pass executes only via CLI or the in-process ticker.