GDPR erasure¶

Architecture

Telha's history is append-only by design, but Article 17 is not optional: this page explains the two mechanisms Telha uses to physically destroy personal data on request, and how that coexists with a store that otherwise never deletes anything.

Overview¶

GDPR erasure defines two classes of request:

Both classes run through the same staged orchestrator (ErasureOps), write to the same JSONL erasure ledger, and are the one sanctioned violation of the append-only doctrine: bounded, ledgered, and legally compelled, not a silent exception.

Design¶

Two mechanisms, and when each applies¶

Crypto-shredding (Tier 1, immediate). Classified fields are already encrypted per-tenant at the application layer before core ever sees them (see Field encryption and Tenancy & security). Subject-level erasure extends this with subject-scoped DEKs: an HKDF derivation from the tenant DEK plus the subject id. Destroying the derivation record for a subject renders every classified field keyed to that subject permanently unreadable, instantly, including in backups. This is Tier 1 because it requires no row-by-row physical work: the ciphertext survives everywhere, but nothing can ever decrypt it again. Core's erasure stage machinery records the ordering contract (Tier 1 must run before any physical stage); the actual field_crypto_deks destruction is the app-layer leg's action.

Tombstoning and masking (Tier 2, asynchronous physical deletion). Everything crypto-shredding cannot reach, unclassified plaintext fields, canonical source text, spans, vectors, traces, job payloads, is handled by the staged executor pipeline described below: subject nodes are physically purged down to a sanitized terminal tombstone; canonical text is masked in place; derived data (vectors, bitmap postings, HNSW partitions) is deleted and rebuilt.

Reconciling "never delete history" with "must erase on request"¶

The append-only doctrine and Article 17 are reconciled by treating erasure as a distinct, narrow, audited exception rather than bending the temporal model to accommodate it:

• Erasure runs through its own orchestrator, not the ordinary node/edge write path. A tombstoned subject node keeps its pseudonymous logical id (so a future re-ingestion upsert against that id is skipped, per the ingestion-provenance spec's dedup rule) but every prior version and every property value is physically gone; the sole survivor is a sanitized terminal tombstone with empty properties.
• Canonical text blobs are masked, not spliced: every byte in a redaction range is overwritten with the mask byte 0x2A, same length, so every other entity's span byte-offsets stay valid. The blob keeps its original blob_id as an addressing key; a blob redaction record becomes the new integrity anchor (post_redaction_hash) from that point forward.
• The erasure ledger lives outside the erased scope entirely ({data_dir}/erasure-ledger.jsonl), and contains no subject plaintext, only coordinates and digests. What proves erasure happened is not the ledger's prose, it's the two-pass verification scan.
• Retention policy and erasure share the same executor and ledger (EntityEraser / execute_erasure_at, built for the retention-archival spec and extended, never redesigned, for erasure). The ledger's method field is the discriminator between a retention-driven deletion and a GDPR one. Precedence is: erasure outranks legal hold; legal hold outranks retention policy.

flowchart TB
    subgraph history["Append-only history (everywhere else)"]
        V1["Version 1"] --> V2["Version 2 (correction)"]
        V2 --> V3["Tombstone (ordinary delete)"]
    end
    subgraph erasure["Erasure (this page)"]
        SUBJ["Subject node: ALL versions"] --> PURGE["Physically purged"]
        PURGE --> TOMB["Sanitized terminal tombstone<br/>(pseudonymous id kept, properties empty)"]
        BLOB["Canonical blob"] --> MASK["Bytes masked in place (0x2A)<br/>same address, new post_redaction_hash"]
    end
    LEDGER["Erasure ledger (outside erased scope)<br/>coordinates + digests only, no plaintext"]
    erasure --> LEDGER

The operator surface¶

Erasure is operator/admin-only: it lives behind a signed bearer token with audience "admin" (the same admin-token boundary as /admin/retention/*), entirely separate from tenant API keys, and it is never exposed over MCP. There is no tenant self-service erasure endpoint.

Both a REST surface and a CLI mirror the same eight-stage lifecycle:

Creating a request:

POST /admin/erasure
{
  "kind": "subject",
  "descriptor": {
    "personIds": ["b3f5c1a0-..."],
    "identifiers": [
      { "value": "alice@example.com", "strength": "strong" },
      { "value": "Alice Smith", "strength": "weak" }
    ],
    "wholeSources": []
  },
  "legalRef": "DSR-2026-0142",
  "requestedBy": "dpo@tenant.example",
  "deletionFallback": false
}

The response never echoes the descriptor plaintext back; it returns state and a digest:

{
  "id": "6f0e...",
  "kind": "subject",
  "state": "requested",
  "legalRef": "DSR-2026-0142",
  "requestedBy": "dpo@tenant.example",
  "approvals": [{ "principal": "dpo@tenant.example", "atUs": 1751500000000000 }],
  "descriptorDigest": "9a3c...",
  "stages": [],
  "residualReviews": 0,
  "createdAt": 1751500000000000
}

Guarantees the operator gets back: a stages list recording every completed stage with its ledger entry ids, a completion report (GET /admin/erasure/:id/report) once the request reaches Completed, and every stage transition durably ledgered so a crashed job resumes at its stage boundary rather than restarting or silently losing progress.

Limits and async behavior: execution is staged and resumable, not a single synchronous call. A deployment with an app-layer PostgreSQL worker parks in AwaitingPg after the core stages finish, until that worker posts /admin/erasure/:id/stage/pg; a core-only deployment records the PG stage as not_applicable with a loud warning instead. The verification loop caps at MAX_VERIFY_PASSES (5); if residual hits keep reappearing past that, the request returns ERASURE_VERIFICATION_OVERFLOW (HTTP 409) and pages the operator instead of looping silently.

Data model¶

The request record (ErasureRequest, core-only mirror of the app-layer PG request row; holds descriptor plaintext outside every column family):

pub struct ErasureRequest {
    pub v: u32,
    pub id: Uuid,
    pub kind: ErasureKind,           // Tenant | Subject
    pub tenant_id: Uuid,
    pub organization_id: Uuid,
    pub descriptor: SubjectDescriptor,
    pub legal_ref: String,
    pub requested_by: String,
    pub approvals: Vec<Approval>,
    pub state: RequestState,
    pub deletion_fallback: bool,     // OQ-1 executor policy flag
    pub created_at: Ts,
    pub stages: Vec<StageRecord>,
    pub pg_report: Option<PgStageReport>,
    pub residual_reviews: Vec<ResidualReview>,
    pub scan_report: Option<ScanReport>,
    pub completion: Option<CompletionReport>,
}

The subject descriptor is exactly what identifies who is being erased:

pub struct SubjectDescriptor {
    pub person_ids: Vec<Uuid>,           // PERSON logical ids, ALIAS_OF included
    pub identifiers: Vec<IdentifierString>,
    pub whole_sources: Vec<Uuid>,        // deletion path, not masking
}

pub struct IdentifierString {
    pub value: String,
    pub strength: IdentifierStrength,    // Strong | Weak
}

Identifier strength is a closure gate, not a label. Strong covers emails, phone numbers, government/employee/customer ids, AAD object ids, account numbers, and erased-content digests: closure is absolute zero, with no waiver mechanism at all. Weak covers common names and name fragments: a residual hit on a weak identifier may be closed by a dual-approved review, recorded as its own ledger entry.

Request lifecycle states (RequestState): Requested, Approved, Executing, AwaitingPg, Verifying, Completed, or Failed { stage, error } (resumable at the failed stage boundary).

The ledger record (adopts the retention-archival spec's as-built shape, additive fields for erasure):

// Ledger entry `method` values this spec adds:
// gdpr_tenant | gdpr_subject | gdpr_subject_pg
// | gdpr_blocklist_release | gdpr_waiver | gdpr_residual_review

Every GDPR ledger entry carries request_ref, request_digest (BLAKE3 of the descriptor; the plaintext itself never leaves the app-side request record), program_ref + program_hash, and retired_digests. The ledger and the programs it references are content-free by construction: the verification scan sweeps them as one of its own haystacks to prove it.

The blob redaction record (sources CF, keyed at the reserved chunk index u64::MAX):

pub struct RedactionRecord {
    pub v: u32,
    pub ledger_entry_ids: Vec<Uuid>,
    pub ranges: Vec<(u64, u64)>,        // masked byte ranges, canonical
    pub post_redaction_hash: String,    // BLAKE3 hex of the masked text
    pub original_hash_retired: bool,
    pub deleted: bool,                  // whole-blob deletion tombstone
}

The blocklist row (sources CF, tenant-scoped, keyed by a keyed digest, never the raw content hash):

pub struct BlocklistRow {
    pub v: u32,
    pub blocklist_id: Uuid,
    pub digest_scheme: String,   // "hmac_blake3_v1"
    pub digest: String,          // hex; blake3::keyed_hash(tenant_secret, raw_digest)
    pub kind: DigestKind,        // Content | Canonical
    pub created_by_erasure_id: Uuid,
    pub created_at: Ts,
    pub purpose: String,         // "prevent_reingest_resurrection"
    pub release_supported: bool, // always false in v1
}

The completion report (CompletionReport) is the DPO's artifact: it aggregates rows_deleted per column family, blobs_masked / blobs_deleted, blocklist_rows, api_keys_revoked, clarify_rewritten / clarify_prior_versions_purged, partitions_rebuilt, superseded_manifests, the pg stage outcome, scan_passes, strong_hits_open (must be zero), weak_hits_reviewed, residual_reviews, a backup_horizon_us, a log_attestation block, and a free-form warnings list (e.g. the running-server key-reload caveat).

The erasure program (JSONL, one op per line, at {data_dir}/erasure-programs/{entry_id}.jsonl) is the coordinate-only replay format shared by execute-time application and restore-time replay:

#[serde(tag = "op", rename_all = "snake_case")]
pub enum ProgramOp {
    DeleteRange { cf: String, start: String, end: String },
    Delete { cf: String, key: String },
    MaskValue { cf: String, key: String, ranges: Vec<(u64, u64)> },
    MaskBlob { blob_id: Uuid, ranges: Vec<(u64, u64)> },
    DeleteBlob { blob_id: Uuid },
    Blocklist { keyed_digest: String, kind: DigestKind },
    TombstoneNode { node_id: Uuid, labels: Vec<String>, valid_start: Ts, tx_start: Ts },
    ScrubSpanLinks { key: String, remove: Vec<Uuid> },
    JobDelete { job_id: Uuid },
    JobScrub { job_id: Uuid, payload_ranges: Vec<(u64, u64)>, nil_ranking: Vec<usize>,
               event_masks: Vec<(usize, Vec<(u64, u64)>)> },
    RevokeApiKey { key_hash: String },
    DeleteVectorTree { tenant_id: Uuid },
}

Every key and range in a program is hex-encoded specifically so the program files stay scannable and content-free: the verification scan's pass 2 sweeps the program directory itself as a haystack.

Personal-data surface register¶

The spec's §5 register is the normative, exhaustive list of every place subject data can live; new personal-data surfaces must extend it in the same change that introduces them. A representative slice:

Algorithms & invariants¶

The eight-stage pipeline¶

sequenceDiagram
    autonumber
    participant Op as Operator
    participant API as /admin/erasure
    participant Ex as ErasureOps
    participant Ledger as Erasure ledger

    Op->>API: POST /admin/erasure (descriptor, legal_ref)
    API->>Ex: request_subject_at / request_tenant_at
    Ex->>Ledger: (no write yet, Requested state)
    Op->>API: POST /admin/erasure/:id/approve (2nd principal)
    API->>Ex: approve_at
    Note over Ex: state to Approved (>= 2 distinct approvers)
    Op->>API: GET /admin/erasure/:id/preview
    Ex-->>Op: per-identifier match counts per surface (no writes)
    Op->>API: POST /admin/erasure/:id/execute
    Ex->>Ledger: stage 1 shred (Tier-1 DEK ordering contract)
    Ex->>Ledger: stage 2 freeze (cancel in-flight jobs)
    Ex->>Ledger: stage 3 enumerate to program (ledgered before any row touched)
    Ex->>Ex: stage 4 execute (shared executor applies the program)
    Ex->>Ledger: stage 5 archives (rewrite-verify-destroy superseded slices)
    alt app-layer PG expected
        Ex-->>Op: parks in AwaitingPg
        Op->>API: POST /admin/erasure/:id/stage/pg (worker report)
    else core-only
        Ex->>Ledger: stage 6 pg (not_applicable, loud warning)
    end
    Ex->>Ex: stage 7 verify (pass 1 structural + pass 2 byte scan, loop)
    Ex->>Ledger: stage 8 close (completion report)
    Op->>API: GET /admin/erasure/:id/report

Verification: proof, not assumption¶

Completion is a two-pass scan, not a claim:

• Pass 1 (structural): every coordinate in every program this request wrote verifies against current state, keys are absent, ranges are masked, blob redaction records exist with matching post_redaction_hash, blocklist rows exist, api-key bindings are gone. Tenant erasure additionally requires every column family's tenant prefix to scan empty and the vector tree to be gone.
• Pass 2 (byte scan): one streaming Aho-Corasick sweep of identifier_strings (NFC-normalized, case-folded) across all 13 column families within the tenant prefix, HNSW .ids sidecars and manifests, api_keys.json, and the erasure programs and ledger themselves. Every hit becomes a masking work item and the scan re-runs, up to MAX_VERIFY_PASSES (5) before it pages the operator instead of looping.

Invariants enforced¶

Shared blobs (the common case for a mailbox attachment referenced by many messages) are masked once, for every referrer, at the same address: redacting a shared blob is a deliberate consequence of content addressing, not an accident, because forking the blob per-referrer would preserve the plaintext under the fork and defeat erasure entirely. All referrer consequences stay tenant-bounded, since content addressing itself is tenant-scoped.

Configuration¶

ErasureConfig (nested under the engine config as erasure):

A deployment whose configured log_retention_days_max exceeds 30 sets manual_log_erasure_required: true on every completion report, and the runbook is expected to back that with a manual per-subject log-erasure procedure.

As-built notes¶

Per the spec's §14 changelog, the current build (v0.6, PR-081 CORE leg) landed:

• Subject descriptor, request/approval records, the full /admin/erasure/* family (behind the admin token audience), and the telha erase CLI family.
• Canonical-blob same-length 0x2A masking (span-derived ranges unioned with Aho-Corasick needle matches, NFC-normalized with case variants, codepoint-widened) with blob redaction records at the reserved chunk index u64::MAX.
• The keyed-digest blocklist (hmac_blake3_v1), wired into all three dedup paths (request-level, submit-time canonical, child-enqueue), returning INGEST_ERASED_CONTENT.
• Content-free erasure programs plus idempotent restore replay: backup restore now lifts the earlier PR-083 rule that refused any restore whenever a ledger was present at all.
• Archive rewrite-verify-destroy supersession, with side-load refusal for any manifest recorded as superseded.
• Both PR-064 pinned gaps closed: clarification prior-version purge, and job-payload ranking / content_b64 scrubbing.
• Trace masking (draft and claim text) with a new, distinct evidence_redacted claim-verification verdict.
• api_keys.json person-binding revocation.
• The two-pass verification scan with the residual-review closure model (strength classes, absolute zero for strong identifiers).
• The completion report, including the OQ-6 log-retention attestation block.

Deferred to the app leg (not yet built, but its core-facing contract is defined in this spec): the BullMQ PostgreSQL erasure worker, subject- scoped DEK shredding, and the subject index. Their surfaces, the stage/pg report-back endpoint, the Tier-1 ordering requirement, and the per-table register, are already normative here so the app leg has a fixed contract to build against.

Several open questions carry pinned engineering positions with counsel confirmation still pending, and the build proceeds on the conservative default in each case: no residual-review waivers are exercised by default (OQ-4), the blocklist has no release action in v1 (OQ-5), the 30-day log bound is documented as the GA gate rather than enforced retroactively (OQ-6), and PostgreSQL rows redact-by-default rather than delete (OQ-7). Legal sign-off remains the outstanding gate for GA; it does not block the build itself.

Related¶

• Field encryption - the envelope format and the subject-scoped DEK derivation that makes Tier-1 crypto-shredding possible.
• Tombstone cascade - ordinary, non-erasure deletion semantics that the append-only model already supports.
• Snapshot & compare - how temporal queries treat tombstones versus a physically erased entity.
• Concepts › The tritemporal model - why history is append-only everywhere except here.
• Concepts › Tenancy & security - the tenant-prefixed key layout that keeps every erasure program bounded to one tenant.
• Operators › Retention - the shared executor and ledger this spec consumes, and the policy/hold precedence erasure sits above.