Spec: GDPR Erasure — Physical Deletion in an Append-Only Store (pre-GA)¶

File: .ai/specs/core-engine/2026-07-02-gdpr-erasure.md Status: Approved for build (engineering re-review 2026-07-06, delegated to Claude by J. Porter in-session per the F7/security delegation precedent; user may countersign or veto — record either in §14). LEGAL SIGN-OFF REMAINS OUTSTANDING and is the pre-GA gate (Phase-3c exit criterion); OQ-4..7 stay open for counsel/ops and the build uses the conservative defaults (no waivers, blocklist retained, 30-day log bound documented, PG rows redact-by-default). v0.3 redaction design (2026-07-06) addressed all 2026-07-05 review blockers. Owners: Storage lane + app-layer lane (joint); legal sign-off required Related: KEY-OQ2/CAS-OQ3 register entries; composite-key-layout spec (contiguous prefixes preserved for this); field-encryption spec (crypto-shredding; subject-scoped DEK addendum required, see §10); retention-archival spec v0.3 (AS-BUILT shared executor + ledger this spec consumes); backup-restore spec v0.3 (restore seam, erasure watermark, PG backup surface); ingestion-provenance spec v0.5 (canonical blobs, spans, streamed chains §16, shared attachments §18); temporal-clarification spec (attribution tombstoning and its recorded PR-064 gaps); hnsw-partitioning spec (derived files)

1. Overview¶

Defines the two erasure classes — tenant-level offboarding and subject-level Right-to-be-Forgotten — and how physical deletion coexists with the append-only doctrine: erasure is an explicitly extra-temporal, legally mandated operation, executed through defined procedures and recorded in an erasure ledger that lives outside the erased scope. v0.3 adds the full canonical-blob redaction design (in-place same-length masking under the original blob address), the normative personal-data surface register, the normative app-layer PostgreSQL stage, and the normative restore/ledger replay contract, consuming the PR-083 executor and ledger as built.

2. Problem Statement¶

Append-only immutability and Article 17 collide by design. Unmanaged, the collision surfaces as: tenant data lingering in HNSW files, job payloads, canonical-text blobs, and backups after a "delete"; subject data smeared across versions, spans, embeddings, and traces with no enumerable footprint; and no defensible record that erasure actually completed. The 2026-07-05 review sharpened the hardest case: the subject's raw words survive in content-addressed canonical-text chunks even after version and span rewrites, so any design that stops at versions is erasure theater. A second structural problem: content addressing means a blob is shared (dedup, and deliberately so for repeated email attachments per ingestion-provenance §18), so removing the subject's bytes from one document's blob is removing them from every document that references the same bytes; the design must make that a feature with defined consequences, not an accident.

3. Proposed Solution¶

Tenant-level: the contiguous 32-byte prefix (preserved for exactly this — composite-key-layout §9) makes it a DeleteRange(scope, scope+1) across every CF, plus: HNSW partition file tree removal (data_dir/vectors/{tenant}/), in-flight job cancellation + payload purge, DEK destruction (crypto-shredding any copies that escape enumeration), compaction trigger to reclaim tombstoned SSTs, and backup policy interaction (§8). Subject-level: two-tier. Tier 1 (primary, immediate): crypto-shred — classified fields were encrypted per-tenant; subject-level shredding requires per-subject DEK derivation, so field-encryption gains subject-scoped DEKs for subject-linked classified fields (HKDF from tenant DEK + subject_id; destroying the derivation record renders those fields permanently unreadable, satisfying erasure for classified content instantly, including in backups). Tier 2 (physical, asynchronous): the staged executor pipeline of §4, built on the retention-archival spec's AS-BUILT EntityEraser/execute_erasure_at machinery, extended with the subsections below. This is the one sanctioned violation of append-only — bounded, ledgered, legally compelled. Erasure ledger: the PR-083 as-built shared ledger (JSONL at {data_dir}/erasure-ledger.jsonl; record shape in §5), retained per legal policy; contains no erased content.

3.1 Erasure subject model (what identifies a subject)¶

An erasure request carries a subject descriptor, assembled at request time and reviewed by both dual-approval principals:

• person_ids: the subject's PERSON logical id(s), including ALIAS_OF placeholder nodes. Deterministic ids derived from identifiers (the email worker's uuid5 person-email:{addr} convention, ingestion-provenance §17) are retained as tombstone keys after erasure; they are pseudonymous and not reversible to the identifier, and they are required so the §17 upsert skip ("latest version tombstoned → skipped") can prevent re-ingestion from resurrecting the entity. This retention is recorded in the ledger entry.
• identifier_strings: the exact strings that constitute the subject's identifiers: email addresses, UPN, display name(s), AAD object id, chat user ids. These drive the byte-level masking and the verification scan. High-precision strings only (a bare first name is a false-positive machine); the preview report (§7) shows per-string match counts before approval so the principals see blast radius.
• whole_sources (optional): source ids whose entire content is the subject's personal data (their CV, their exported mailbox). These take the deletion path (whole-source erasure via the executor's existing Source kind), not the masking path.

Coverage contract (normative, honest): v1 guarantees erasure of (a) every structure-linked row: rows reachable from person_ids via the graph, spans (linked_nodes), provenance, attribution, vectors, key bindings, and the app-layer subject index; and (b) every exact occurrence of each identifier_strings entry, NFC-normalized and simple-case-folded, in canonical blobs, node/edge property string values, trace text, job payload names and event details, and registered PG columns. v1 does NOT guarantee removal of paraphrases, nicknames, misspellings, inflected forms, or contextual descriptions that do not literally contain a registered identifier: that is NLP territory and claiming it would be dishonest. The DPO may extend identifier_strings and re-run; the verification scan report (§3.8) is the evidence of what was and was not found. Free-text mentions of the subject inside other people's documents are covered exactly to the extent they are span-linked to a subject node or literally contain a registered identifier string.

3.2 Canonical-blob redaction (resolves OQ-3): in-place same-length masking¶

Canonical text lives as chunks in the sources CF keyed (scope, blob_id, chunk_index); blob_id is derived from canonical_text_hash for single-shot ingests and is an explicit provisional id for streamed chains (ingestion-provenance §16). Spans everywhere (span rows, Provenance.spans, trace span refs) are byte offsets into this text. The redaction mechanism:

1. Compute redaction ranges for each affected blob: the union of (a) byte ranges of span rows whose linked_nodes consist ONLY of subject nodes (the mention IS the content: From-header person spans, name-cell spans), and (b) exact case-folded matches of identifier_strings found by scanning the blob (this also covers occurrences inside spans shared with other entities, without destroying the rest of the shared span). Ranges are widened outward to UTF-8 codepoint boundaries, never narrowed.
2. Mask, do not splice: every byte in a redaction range is overwritten with the mask byte 0x2A (*). Same-length ASCII substitution keeps the blob valid UTF-8, keeps every other entity's span offsets valid, keeps the total length identical, and keeps every chunk's size identical, so streamed multi-chunk blobs (per-partial chunk sizes, NOT uniform) need no boundary recomputation: only chunks intersecting a range are rewritten, under their existing keys.
3. Keep the address, retire the identity: the blob keeps its blob_id. After masking, content hash no longer equals stored content, so a blob redaction record is written adjacent to the chunks (sources CF, same blob id, reserved chunk index u64::MAX; composite-key-layout addendum required): {ledger_entry_ids, ranges: [(start,end)], post_redaction_hash: blake3 of the masked text, original_hash_retired: true}. From then on, blob_id/canonical_text_hash is an addressing key only; every integrity check (consistency checker, claim verification, restore drills) MUST verify a blob with a redaction record against post_redaction_hash instead of its address. Repeated erasures against the same blob extend the record (ranges union, hash recomputed) idempotently.
4. Referrers are untouched rows: source-version rows keep their canonical_text_hash and are NOT re-pointed. There is no blob→source reverse index and none is needed under this design; every referrer sees the masked content through the shared address, which is the correct semantics: the subject's words are the subject's personal data at every address that resolves to them.
5. Read behavior: the text-slice API (GET /v1/sources/:id/versions/:v/text) and telha debug span return the masked bytes and set redacted: true (with ledger refs) whenever the requested range intersects the redaction record. Exports behave identically. Claim verification classifies a span overlapping a redacted range as evidence_redacted, a distinct verdict from mismatch or corruption, so other documents' verification degrades explicably instead of failing mysteriously.
6. Deletion fallback (OQ-1 posture): the executor also supports whole-blob deletion (chunks removed, redaction record written as a full-range tombstone, all spans over the blob deleted, referrer versions marked content-erased with ledger refs) for whole_sources entries and for any case where counsel requires deletion over redaction.

3.3 Shared blobs and the effect on other documents¶

Redaction is by content, so redacting a shared blob (the §18 attachment case: one attachment:{blake3} source shared by many messages across a tenant's mailboxes) redacts it for every referrer at once. This is intended: erasure of the subject's words cannot be scoped to one referrer of the same bytes without either forking the blob (which preserves the plaintext under the fork, defeating erasure) or lying about completeness. Consequences, all bounded to the tenant (content addressing is tenant-scoped; keys carry the 32-byte scope, so there is no cross-tenant blast radius): other messages' ATTACHMENT_REF nodes, source versions, and span rows are untouched; their span slices over redacted ranges render mask bytes; their claim verifications over those ranges return evidence_redacted; their unaffected ranges are byte-identical. The ledger entry records the blob id, the range list, and the count of referring source versions found at execution time (a sources-CF scan within the tenant, done once, for the ledger's completeness proof, not for re-pointing).

3.4 Content-hash identity, dedup, and the erased-content blocklist¶

A future re-ingest of the same original bytes would compute the PRE-redaction hashes and match dedup, silently linking new documents to masked content, or worse, a deletion-path blob could be recreated in full. Therefore every GDPR ledger entry that redacts or deletes content carries the retired digests (content_hash, canonical_text_hash of the pre-redaction content), and these are materialized as tenant-scoped blocklist rows in the sources CF (rebuilt from the ledger on restore replay). Every dedup path MUST check the blocklist before matching: request-level (name, content_hash) dedup, submit-time canonical-hash dedup, §17 child-enqueue dedup (which includes the code worker's incremental skip and the §18 shared-attachment collapse). A hit refuses ingestion with INGEST_ERASED_CONTENT and the ledger ref. Releasing a blocklist entry requires a new dual-approved admin action, itself ledgered (whether release should be permitted at all is OQ-5). The digests themselves are retained deliberately: a hash cannot reproduce the content, and it exists precisely to prevent resurrection; this retention is flagged to counsel in OQ-5.

3.5 Derived and secondary data (vectors, traces, jobs, clarifications, bitmap, key bindings)¶

• Vectors: all vectors CF rows for subject nodes are deleted (every model id, every version) and affected HNSW partitions are rebuilt from the vectors CF (rebuild also refreshes .ids sidecars and manifests, so no subject node id survives in vector disk artifacts). Non-subject entities whose property values were masked get re-embedded through the normal embed fan-out if their assembled embed-text hash changed (the temporal-clarification §10 mechanism); their stale vectors are deleted at masking time so nothing derived from pre-masking text remains searchable.
• Traces: per the OQ-1 pinned position, redact and preserve the skeleton. Query traces and GenerationTraces have identifier occurrences masked in request question text, draft, and claims[].text; trace span refs are left intact (offsets remain valid) and re-verification of a redacted span returns evidence_redacted. Cost rows carry token counts only (verified by the scan, not rewritten). The executor also supports whole-trace deletion as the OQ-1 fallback.
• Jobs (closes the PR-064 pinned gap): job rows persist indefinitely, and ingest payloads carry name and possibly content_b64 (full raw document bytes), which makes the jobs CF a first-class personal-data surface, not plumbing. Erasure: terminal-state (completed/failed/dead) ingest job rows whose payload content_hash is blocklisted or whose source was erased are deleted outright; retained rows get identifier masking in payload name and event detail strings; clarify:temporal payloads have the subject's id replaced with the nil UUID in ranking (position preserved so attempt indexing stays coherent); in-flight jobs touching erased content are cancelled first (§4).
• Clarification history (closes the second PR-064 gap): erase_person_attribution rewrites only the clarification node's CURRENT version. The executor purges the PRIOR versions of every clarification node whose asked_json/responses_json/events_json reference the subject, leaving the current attribution-tombstoned version as the sole survivor (answer values, response tx times, rights-at-answer-time, binding statuses, quorum state, and the corrected fact version all survive, exactly the clarify_gdpr_tombstone contract). Free-form answer_custom.source_text given BY the subject is treated as subject content and masked.
• Bitmap index: postings for deleted or rewritten versions are removed; eq-bucket keys are derived from property values, so buckets emptied of subject postings are deleted; the CF is rebuildable, and verification uses targeted emptiness checks plus rebuild where enabled.
• api_keys.json: a KeyRecord whose person binding matches a subject id has the key revoked and the binding removed (a credential that can answer clarifications as the erased person must not survive them), ledgered.

3.6 Restore and ledger replay (normative; adopts PR-080/PR-083 as-builts)¶

Backups predating an erasure contain the erased data by construction; the restore path is where that is reconciled. Adopted as built: the ledger is JSONL at {data_dir}/erasure-ledger.jsonl; backup manifests carry erasureWatermarkUs; restore refuses when a ledger is present but unreadable; the interim PR-083 rule (restore refuses whenever a ledger is present) is lifted by PR-081 landing the replay body and the erasure_pending() implementation. Replay semantics owned here:

1. Machine-replayable entries. Every GDPR ledger entry additively carries program_ref: a content-free erasure program file at {data_dir}/erasure-programs/{entry_id}.jsonl whose BLAKE3 is recorded in the entry. The program lists coordinates only: per-CF exact keys and delete ranges, (blob_id, ranges) masking instructions, blocklist digests, api-key revocations, and the PG stage reference. No plaintext identifier ever enters the ledger or a program; the descriptor's plaintext lives in the app-layer request record (its digest is in the ledger as request_digest).
2. Replay procedure. On restore, entries with at_us greater than the manifest watermark are replayed in file order. Replay executes each program's coordinates; every operation is idempotent (deleting an absent key, masking an already-masked range, and rewriting a blocklist row are no-ops), so partial pre-crash replays re-run safely. A program hash mismatch, a missing program, or any execution failure is a FAILED restore: the target dir is removed (the PR-080 consistency-gate posture). After replay, the structural verification of §3.8 pass 1 re-runs against every replayed program before the dir may serve. The full byte scan (pass 2) needs the plaintext descriptor and therefore runs as an operator step with the request record; the restore report says whether it ran.
3. Lost-ledger reality. A total data-dir loss loses ledger entries written after the last backup. The requests themselves survive in app-layer PG (which has its own backup surface per backup-restore §3.9): restore MUST cross-check open/completed erasure requests against the restored ledger and RE-EXECUTE any request not covered before serving. The runbook copies erasure-ledger.jsonl + erasure-programs/ offsite alongside each backup transport step.
4. Coverage argument. Program coordinates were enumerated against the full pre-erasure state; any backup is a subset of some pre-erasure state plus already-erased state, so replaying coordinates over a restored dir covers every row the backup can contain; rows the backup lacks are idempotent no-ops. Blocklist rows are rebuilt from the ledger so post-restore ingestion keeps refusing erased content.

3.7 Archive slices (retention-archival interaction)¶

Retention archive slices are pre-erasure copies on disk with verified manifests; unlike backups they have no replay-on-restore hook (side-load is read-only inspection), so erasure must rewrite the artifacts themselves. The archives stage consults the per-tenant archive index; every slice containing frames for erased rows or redacted blob chunks is rewritten (erased-row frames dropped; chunk frames masked with the same ranges), re-hashed, and re-manifested with supersedes: <original manifest blake3> plus ledger refs; the rewrite verifies before the original is destroyed (the retention spec's verify-before-delete doctrine, applied in reverse). Side-load tooling refuses a slice whose manifest is recorded as superseded in the ledger. Backup stamps are NOT rewritten: they are covered by the replay contract plus backup_horizon_days aging (§8); archives are kept indefinitely by intent, which is why they get the rewrite treatment and backups do not.

3.8 Verification scan (the completion gate)¶

Completion is proven, not assumed, by a two-pass scan whose report is the DPO's artifact:

• Pass 1, structural: every coordinate in every program verifies (keys absent, ranges masked, redaction records present with matching post_redaction_hash, blocklist rows present, api-key bindings gone); per-CF counts match the ledger's rows_deleted; the app-layer subject index has zero entries for the subject; tenant-level erasure verifies the prefix is empty in every CF and the vector tree is gone.
• Pass 2, byte scan: the needle set is identifier_strings, NFC-normalized with Unicode simple case folding, matched via one streaming Aho-Corasick pass per CF within the tenant prefix (cost is O(tenant bytes), bounded memory, run as a system job). Haystacks: all 13 CFs' values (scanned both as raw bytes and as decoded string properties), canonical blob chunks, HNSW .ids sidecars and manifests, api_keys.json, and the erasure programs themselves (which must contain no plaintext by construction; scanning them proves it). Field-encryption envelopes (magic 0x7E 0x1A) are excluded as ciphertext: they are unreadable after the Tier-1 shred and scanning ciphertext for plaintext needles is meaningless; the exclusion rule is itself tested. The PG stage runs the equivalent per-table scan app-side (§3.9).
• Hits loop back: every pass-2 hit becomes a masking work item and the scan re-runs. Closure is NOT binary (OQ-4 review, 2026-07-06): identifier strings carry a strength class (see §3.10). The stage closes when a pass finds zero STRONG-identifier hits AND every WEAK-identifier hit is either zero or closed by a dual-approved residual review. The build default exercises no reviews. A residual hit that is not a store-CF row (a file-store hit: sidecar, program, ledger) cannot be masked in place and pages the operator instead of looping.
• Logs stance (normative): log scrubbing covers secrets, not subject identifiers (core's telemetry::ScrubWriter sink scrubs registered secrets only, per the generation-orchestration spec); ingestion and clarification logs can echo source names and person ids. v1 does not scrub logs per-subject. Instead, deployments accepting subject data MUST bound log retention to at most 30 days (inside the Article 12(3) response window); the completion report records the date after which no log predating the erasure survives. If an operator cannot bound retention, logs become a manual erasure surface in the runbook. This is a defensible storage-limitation posture, not a gap swept under the rug; OQ-6 carries the open ops question.

3.9 App-layer PostgreSQL stage (normative)¶

PG holds subject data outside the core executor's reach; the register in §5 makes it checkable. Execution model: when the core erasure job reaches the pg stage, the app-layer erasure worker (a BullMQ job in the enterprise package, same pattern as key rotation) executes the per-table actions and reports per-table before/after counts to POST /admin/erasure/:id/stage/pg (the PR-083 admin-token boundary); core writes the ledger entry from that report and will not start the verification stage until the PG stage reports. A deployment with no app layer records the stage not_applicable with a loud warning in the completion report (the backup spec's partial: true precedent). Per-table actions are in the §5 register; three deserve stating here: auth_sessions rows for the subject are deleted outright (which also revokes the sessions); ai_tool_invocations follows the OQ-1 redact-not-delete posture (user_id replaced with an erasure tombstone marker, args_json identifier-masked, skeleton and counts preserved for other users' audit continuity), pending the OQ-7 legal read; field_crypto_deks subject-scoped rows (once the field-encryption addendum lands) are destroyed FIRST, as Tier 1, before any physical stage runs.

3.10 As-built addenda (PR-081; OQ-4..7 review responses incorporated 2026-07-06)¶

The build lands the design above with these normative refinements from J. Porter's OQ-4..7 review; counsel confirmations remain pending where marked. The core report and stage/pg protocol carry the per-table decision shape the review implies; the per-table PG actions themselves are the app leg's implementation.

Identifier strength (OQ-4). Each identifier_strings entry is {value, strength: strong|weak}. STRONG = emails, phone numbers, government/employee/customer ids, AAD object ids, exact account numbers, subject-specific aliases, erased-content digests. WEAK = common full names, first/surname fragments, ambiguous person/organisation overlaps, shared household/company contact names. Residual closure is absolute-zero for strong identifiers; weak-identifier residuals MAY be closed by dual-approved review. The scan report partitions hits into strongHits and weakHits.

Residual-review ledger model (OQ-4). A residual review is a ledger entry method=gdpr_residual_review carrying {residual_id, scan_id, subject_id, matched_identifier, identifier_strength, store, object_ref, decision (erased|retained_redacted|waived_non_subject|retained_article_17_3), reason_code (fixed list: COMMON_NAME_FALSE_POSITIVE, DIFFERENT_PERSON_CONFIRMED, LEGAL_HOLD, LEGAL_CLAIMS_RETENTION, COMMERCIAL_RECORD_RETENTION, AUDIT_CONTINUITY_RETENTION, SYSTEM_IDENTIFIER_NOT_PERSONAL_AFTER_REDACTION), reason_text, approved_by[2] (two DISTINCT principals), approved_at, evidence_refs, expires_or_review_at}. Recording a review for a STRONG identifier is refused structurally. Residual waivers MUST NOT retain confirmed subject personal data absent a counsel-approved Article 17(3) basis.

Keyed blocklist digests (OQ-5). Blocklist rows store a KEYED digest, never the raw BLAKE3: scheme hmac_blake3_v1 = blake3::keyed_hash(tenant_erasure_secret, raw_digest). Row shape: {blocklist_id, digest_scheme, digest, kind (content|canonical), created_by_erasure_id, created_at, purpose: prevent_reingest_resurrection, release_supported: false}. Entries are append-only; v1 provides NO release action and no operator delete. Secret management: the per-tenant tenant_erasure_secret is 32 random bytes, minted lazily on first erasure, stored as a schema-CF document (erasure/secret) AND mirrored to {data_dir}/erasure-secret.json. The schema-CF document rides engine checkpoints (so it is in any backup taken after the first erasure); the mirror file lets restore replay of a PRE-erasure backup re-seed the secret so the rebuilt blocklist stays reachable by dedup checks. The secret is never derived from erased content and shares the ledger/programs custody boundary (the runbook copies it offsite alongside them). Whether the retained keyed digest is personal, pseudonymous, or compliance metadata is a pending counsel item.

Log attestation (OQ-6). The completion report carries {operational_log_retention_days, log_targets_checked[], per_subject_log_scrub_supported, manual_log_erasure_required, attested_by, attested_at}. A bound of ≤ 30 days is the GA gate; a deployment above it sets manual_log_erasure_required: true, which the report discloses and the runbook backs with a manual per-subject log-erasure procedure.

Per-table PG register (OQ-7). Default v1 behaviour, all app-leg-implemented; the core report accepts {table, row_state, retention_basis, before, after} per table:

Other as-built notes. Claim verification gains an evidence_redacted verdict (distinct from mismatch/corruption) when a truth span overlaps a redaction record. INGEST_ERASED_CONTENT is a 409 REST code carrying the ledger ref. api_keys.json revocations require a running server to reload/restart before the in-memory table drops the key (recorded limitation; the completion report warns). The core-side subject index (register row 24) remains the app leg's to build; core enumerates subject rows from person_ids + span links + identifier scan.

4. Architecture¶

POST /admin/erasure (dual approval, subject descriptor / tenant scope)
  └─ erasure job (system namespace, own kind, non-poolable by workers)
       stage 0 preview      descriptor match counts (per string, per surface) → approvers
       stage 1 shred        destroy subject-scoped DEKs (Tier 1; PG, app layer)
       stage 2 freeze       cancel/purge in-flight jobs touching subject content
       stage 3 enumerate    subject index + graph + span links + string scan
                            → erasure program (coordinates only) → ledger entry (program_ref)
       stage 4 execute      shared executor (PR-083 EntityEraser / execute_erasure_at):
                            versions, spans, adjacency, indexes, vectors (+rebuilds),
                            bitmap postings, blob masking + redaction records,
                            blocklist rows, jobs/traces/clarify-history, api-key bindings
       stage 5 archives     rewrite + supersede affected retention slices
       stage 6 pg           app-layer erasure worker → per-table counts → ledger
       stage 7 verify       pass 1 structural + pass 2 byte scan (loop until zero/waived)
       stage 8 close        ledger close entry + completion report (DPO artifact)
restore path: verify → ledger replay (programs, idempotent) → structural re-verify → serve

Every stage writes its own ledger entries; a crashed job resumes at its stage boundary (staged, resumable, ledgered: the retention executor's failure semantics).

5. Data Models¶

Ledger record (adopts PR-083 as built, additive fields). As-built shape: {v, entry_id, at_us, tenant_id, organization_id, method, subject_kind, subject_id, policy_ref?, hold_ref?, archive_ref?, rows_deleted: {cf: count}, note?}. PR-081 additions (additive, per the retention spec's §5 co-ownership): method gains gdpr_tenant | gdpr_subject | gdpr_subject_pg | gdpr_blocklist_release | gdpr_waiver (the v0.2 method names are superseded by the as-built naming); new optional fields request_ref (erasure request id), request_digest (BLAKE3 of the descriptor; plaintext stays app-side), program_ref + program_hash, retired_digests: [{content_hash?, canonical_text_hash?}], superseded_manifests: [blake3], pg_counts: {table: {before, after}}. The ledger and programs contain NO subject plaintext; the verification scan proves it.

Blob redaction record (sources CF, (scope, blob_id, u64::MAX)): {v, ledger_entry_ids, ranges: [(start,end)], post_redaction_hash, original_hash_retired}. Blocklist row (sources CF, tenant-scoped, keyed by retired digest): {ledger_entry_id, at_us}. Both key shapes require a composite-key-layout addendum before build.

Erasure request (app-layer PG): {id, kind: tenant|subject, descriptor (plaintext, encrypted at rest per classification), legal_ref, requested_by, approvals: [principal, at], state, completion_report_ref}. The subject index (app-layer PG, as previously specified): {tenant, subject_id, logicalIds[], updated}, maintained at write time via classification-adjacent subject-linkage declarations; still to be built (§10).

Personal-data surface register (normative, exhaustive; any new personal-data surface MUST extend this table in the same change)¶

6. API Contracts¶

POST /admin/erasure {kind: tenant|subject, descriptor | target, legal_ref} behind the PR-083 admin-token boundary; a second approver confirms via POST /admin/erasure/:id/approve before any stage past preview runs; GET /admin/erasure/:id returns staged status; GET /admin/erasure/:id/report returns the completion report; POST /admin/erasure/:id/stage/pg is the app-worker report-back (per-table counts). CLI mirror: telha erasure request|approve|preview|status|report, plus telha erasure release-blocklist (dual-approved, ledgered; existence pending OQ-5). Hard rules: erasure APIs live behind admin auth entirely separate from tenant API keys; no MCP exposure ever; INGEST_ERASED_CONTENT is the ingestion-side refusal for blocklisted content. Failure taxonomy: stage failures are resumable at stage boundaries; verification-loop overflow (hits that keep reappearing) pages the operator rather than looping silently.

7. UI/UX¶

Enterprise admin screen: request, dual approvals, staged progress, ledger export (the DPO's artifact). Preview is normative (the retention spec's dry-run doctrine): before the second approval, the preview report shows per-identifier-string match counts per surface, affected blob count with shared-referrer counts, affected archive slices, PG per-table counts, and the in-flight jobs that will be cancelled, so the approvers see blast radius, including the shared-attachment fan-out, before anything is destroyed. The completion report records: ledger entry ids, per-surface counts, waivers with reasons, the backup-horizon date, the log-retention attestation, and the not_applicable PG marker where relevant. Nothing tenant-user-facing beyond confirmation of completion.

8. Configuration¶

erasure.backup_horizon_days — backups older than restore-policy horizon age out naturally; the ledger records the date after which no backup contains the data; restores within the horizon MUST replay pending erasures from the ledger before serving (restore-hook, tested). erasure.log_retention_days_max (30) — attested in the completion report per §3.8. Dual-approval principal set. Deliberately NOT configurable: the masking semantics (same-length, codepoint-widened), the blocklist check on dedup paths, verification-scan-before-close, ledger and program emission, dual approval, the replay-before-serve rule, and the archive supersession rule.

9. Alternatives Considered¶

Pure crypto-shredding for everything (rejected: unclassified plaintext fields exist by design; shredding only covers what was encrypted). Compaction-filter lazy deletion for tenant offboarding (rejected as primary: completion is unprovable on a deadline; DeleteRange + verification scan is attestable; compaction filters remain the SST-reclaim mechanism after). Treating traces as exempt "logs" (rejected: traces quote span text; they are personal data when the span is). Refusing subject-level erasure as incompatible with the product (rejected: Article 17 is not optional; the two-tier design is the honest reconciliation).

v0.3 blob-redaction alternatives, all rejected in favor of §3.2 in-place masking:

• Rewrite under a new content hash + re-point every referrer. Rejected: there is no blob→source reverse index, so re-pointing requires a full tenant sources-CF scan per erasure just to find referrers; it breaks the dedup identity chain for every future ingest of sibling documents; it does nothing for spans (offsets are the hard part, and they survive unchanged under masking anyway); and for shared attachments it forces a choice between forking (per-referrer copies that preserve the plaintext, defeating erasure) and the same global rewrite masking already achieves without the bookkeeping.
• Compact splice (remove bytes, shift offsets). Rejected: every span row, every version's Provenance.spans, and every trace span ref over the blob, across ALL referrers, would need rewriting in lockstep; streamed chains would need chunk re-boundarying (per-partial sizes are not uniform); historical traces cannot be coherently re-offset at all. Unbounded blast radius for zero privacy gain over masking.
• Tombstone marker without byte removal. Rejected outright: the review's whole point is that the bytes survive to be served, restored, exported, or re-indexed. Markers alone are erasure theater.
• Whole-blob deletion as the only mode. Rejected as default: destroys other data subjects' and other referrers' evidence disproportionately (one shared PDF, ten messages, one erased subject). Retained as the OQ-1-aligned fallback and the whole_sources path.
• NLP mention detection for free-text coverage. Rejected for v1: unverifiable recall makes the completion claim dishonest; the exact-string guarantee plus scan report plus descriptor extension is the defensible contract (§3.1).
• Storing identifier plaintext in the ledger for replayable scans. Rejected: the ledger must outlive the erasure; programs carry coordinates and digests only; plaintext stays in the app-side request record under its own lifecycle.

10. Implementation Approach¶

Gates first: this v0.3 → engineering re-review (J. Porter) → counsel sign-off (with the OQ-1 recommendation and OQ-4/5/6/7 questions packaged) → PR-081 build. Build order, consuming PR-083 as built (EntityEraser, EntityRows, execute_erasure_at, the JSONL ledger, the admin-token boundary; the executor is extended, never redesigned): (1) subject descriptor + request/approval records + admin API + preview; (2) field-encryption addendum: subject-scoped DEK derivation + field_crypto_deks.subject_id + subject-linkage declarations in classification.ts; (3) subject index + backfill tool (scan existing data via the declarations); (4) blob masking primitive + redaction record + blocklist rows + dedup-path checks; (5) executor extension: subject mode over EntityEraser (multi-entity, value masking, jobs/traces/clarify-history, api-key bindings) + program emission; (6) verification scan (both passes) + report; (7) PG stage worker + report-back endpoint; (8) restore replay body (fills erasure_pending(), lifts the PR-083 interim restore refusal) + archive rewrite; (9) tenant-erasure procedure over the same stages; (10) completion-proof e2e for both classes. OQ-1's deletion fallback is an executor policy flag carried on the request.

11. Migration Path¶

Pre-GA deployments: subject-index backfill tool scans existing data using classification declarations; tenants ingested before subject-DEK derivation get Tier-2-only coverage for historical classified fields (documented limitation in the ledger) — one more reason this spec is pre-GA blocking. Existing ledgers (PR-083 retention entries) need no migration: PR-081 fields are additive and replay only touches gdpr_* methods. Existing backups have no erasureWatermarkUs gap (the field shipped with PR-080). Blocklist rows are derived state: rebuildable from the ledger at any time, exempt from retention expiry by construction (they live outside entity rows).

12. Success Metrics¶

Tenant erasure e2e: post-verification scan finds zero rows/files/jobs/keys for the scope; ledger complete. Subject erasure e2e: shred renders classified fields unreadable instantly; physical stage leaves zero subject-index entries; redaction markers resolve to ledger; rebuilt partitions contain no subject vectors. Named tests, CI-gateable:

• erasure_blob_mask_offsets: shared blob, two referrers; subject ranges masked; the other document's spans slice byte-identical text outside the ranges; blob length and chunk sizes unchanged; redaction record hash verifies.
• erasure_blob_streamed_chunks: multi-chunk streamed blob with non-uniform chunk sizes, mask range crossing a chunk boundary; only intersecting chunks rewritten; concatenation-order read reproduces the masked text.
• erasure_shared_attachment: §18 shared attachment across two mailboxes; redaction for one subject affects both referrers; the other message's ATTACHMENT_REF and source version untouched; claim verification over the masked range returns evidence_redacted.
• erasure_dedup_blocklist: re-ingesting the pre-redaction bytes is refused with INGEST_ERASED_CONTENT on the request path, the child-enqueue path, and the incremental-skip path; blocklist survives restore replay.
• erasure_restore_replay: backup taken pre-erasure; erase; restore replays the program; structural verify green; unreadable ledger still refuses; failed replay removes the target dir.
• erasure_archive_rewrite: a retention slice containing subject frames is superseded and re-verified; side-load of the original manifest is refused; unaffected slices untouched.
• erasure_job_payload_purge: clarify ranking id nil'd (the PR-064 pinned gap closed); terminal ingest job carrying blocklisted content_b64 deleted; retained payload names masked.
• erasure_clarify_history_purge: prior clarification node versions referencing the subject purged; the current tombstoned version and the corrected fact survive; clarify_gdpr_tombstone stays green.
• erasure_scan_zero_residual: pass 2 finds zero identifier occurrences across all CFs, sidecars, api_keys.json, and programs; an envelope-encrypted field containing the (shredded) identifier ciphertext does not count as a hit.
• erasure_pg_stage: app worker erases/redacts every registered table, reports counts, core ledgers them; a core-only deployment records not_applicable loudly.
• erasure_resurrection_guard: post-erasure re-ingest of an email naming the subject upserts against the tombstoned deterministic id and is skipped.
• Restore-hook test (retained from v0.1): mid-horizon backup restore replays the erasure before serving traffic.

13. Open Questions¶

• OQ-1: Trace redaction vs deletion. Position pinned at review 2026-07-05 (J. Porter), pending counsel: redact personal data, preserve the trace skeleton, mark the ledger — audit continuity is part of the product; whole-trace deletion punches holes in the evidence chain for other legitimate users. Taken to counsel as a concrete recommendation, not an open question; the executor supports the full-deletion policy too in case counsel requires it for specific cases. v0.3 extends the same posture to ai_tool_invocations (§3.9).
• OQ-2: Retention-policy erasure — same machinery? Resolved 2026-07-05: yes — the retention-archival spec (2026-07-05) shares this executor and ledger; the ledger method field is the discriminator (as-built naming per §5). Erasure outranks legal hold; hold outranks retention policy.
• OQ-3: Redacted-blob design (opened at review 2026-07-05 — approval blocker). Resolved by the v0.3 design pass (2026-07-06), pending re-review: in-place same-length masking under the original blob address, blob redaction record with post_redaction_hash, no referrer re-pointing, shared-blob semantics, deletion fallback, dedup blocklist, restore replay via content-free programs, archive supersession (§3.2 through §3.7; rejected alternatives in §9). Engineering may not start Tier-2 work before this text reaches Approved.
• OQ-4: Engineering position pinned (J. Porter, 2026-07-06), counsel confirmation pending. The gate is absolute-zero for STRONG subject identifiers and erased-content digests; WEAK/ambiguous identifiers (names, fragments) MAY be closed by dual-approved residual review recorded in the ledger with matched identifier, object ref, reason code, approvers, and evidence (§3.10). Build default: no reviews exercised; the mechanism exists and is tested (a weak residual closed by dual approval; a strong residual can never be waived). Residual waivers must not retain confirmed subject data absent an Article 17(3) basis.
• OQ-5: Engineering position pinned (J. Porter, 2026-07-06), counsel confirmation pending. Blocklist retained as restricted compliance state, no release action in v1. Digests are KEYED (hmac_blake3_v1), never raw BLAKE3, to defeat offline guessing and cross-tenant correlation; secret management is in §3.10. Whether the retained keyed digest is personal, pseudonymous, or compliance metadata under the controller's Article 17 posture is the pending counsel question; any future release would be a new controlled ledger event, not built in v1.
• OQ-6: Engineering position pinned (J. Porter, 2026-07-06), ops readiness gate at GA. The ≤ 30-day operational log bound is a GA requirement; pilots above it ship the manual per-subject log-erasure runbook from day one, and the completion report attests the bound, the covered targets, and any manual actions (§3.10 log attestation).
• OQ-7: Engineering position pinned (J. Porter, 2026-07-06), per-table counsel confirmation pending. Redact-by-default for ai_tool_invocations and orders (both marked pending counsel); subject_index/sessions delete, oidc_links tombstone, dek_records retain only where non-erased data requires it. The per-table register is in §3.10; core's report/stage-pg protocol carries the {table, row_state, retention_basis, counts} shape; the actions are the app leg's.

14. Changelog¶

• 2026-07-06 — v0.6: as-built (PR-081 CORE leg). Landed: subject descriptor + request/approval records + /admin/erasure/* (behind the PR-083 admin audience) + telha erase CLI family; canonical-blob same-length 0x2A masking (span-derived + Aho-Corasick needle matches, NFC + case variants, codepoint-widened) with blob redaction records at chunk index u64::MAX; keyed-digest blocklist (hmac_blake3_v1) wired into all three dedup paths with INGEST_ERASED_CONTENT; content-free erasure programs + idempotent restore replay (backup restore lifts the PR-083 refuse-when-ledger-present rule); archive rewrite-verify-destroy supersession with side-load refusal; clarify prior-version purge + job-payload ranking/content_b64 scrubbing (both PR-064 pinned gaps closed); traces draft/claim masking with the evidence_redacted verdict; api_keys person-binding revocation; two-pass verification scan with the residual-review closure model; completion report with the OQ-6 log attestation. Fills erasure_pending(). App leg deferred: BullMQ PG erasure worker, subject-scoped DEK shred, subject index — its surfaces (stage/pg report-back, Tier-1 ordering, per-table register) are defined here. Gates: full cargo test -j 1 green, clippy -D warnings clean, fmt clean, spec-lint 0 failing. Legal sign-off still gates GA.
• 2026-07-06 — v0.5: OQ-4..7 review responses (J. Porter, 2026-07-06) incorporated — residual-review ledger model with identifier strength classes, keyed blocklist digests (hmac_blake3_v1) with secret management, log-attestation completion report, per-table PG register (§3.10); §13 OQ-4..7 updated with the pinned engineering positions. Counsel items remain flagged (residual-waiver legal acceptability, keyed-digest data classification, per-table Article 17(3) basis).
• 2026-07-06 — v0.4: Approved for build (engineering re-review delegated to Claude by J. Porter in-session, F7/security delegation precedent; countersign/veto slot open). Review notes: masking mechanism, blocklist, surface register, PG stage, and restore semantics accepted as designed; the candid v1 limits (exact-string mask precision, lost-ledger dependence on PG request records, referrer-count scan cost) are acceptable as recorded limitations, not blockers. One correction applied at review: the logs stance now pins core's telemetry::ScrubWriter as the secrets-only scrubbing component. Build (PR-081) proceeds with conservative defaults for the open counsel/ops questions (OQ-4 no waivers exercised, OQ-5 blocklist retained with no release action, OQ-6 30-day bound documented in the runbook, OQ-7 PG rows redact-by-default). Legal sign-off remains outstanding and gates GA, not the build.
• 2026-07-06 — v0.3: redaction design pass; addresses the 2026-07-05 review blockers; ready for re-review. OQ-3 resolved by design: in-place same-length masking under the original blob address (codepoint-widened ranges, mask byte 0x2A), blob redaction record at reserved chunk index u64::MAX with post_redaction_hash (content-address downgraded to addressing-only), no referrer re-pointing, tenant-bounded shared-blob semantics with evidence_redacted verdicts, whole-blob deletion fallback (OQ-1 aligned), erased-content blocklist wired into every dedup path (INGEST_ERASED_CONTENT). Subject model made explicit (descriptor: person ids + identifier strings + whole_sources; honest coverage contract for free-text mentions). Surface enumeration is now the normative 25-row register (§5) including jobs-CF payloads (content_b64, clarify rankings: the PR-064 pinned gaps closed), traces (draft, claim text), clarification prior versions, api_keys person bindings, HNSW sidecars, archives, ledger/programs, six PG tables incl. PR-082 auth sessions, and the logs stance (bounded retention). PG handling normative (§3.9: app-layer erasure worker + report-back endpoint + not_applicable path). Restore/ledger interaction normative (§3.6: content-free erasure programs, idempotent replay, failure removes target, lost-ledger re-execution from PG request records); archive slices rewritten + superseded (§3.7). Verification scan designed (§3.8: structural pass + case-folded Aho-Corasick byte pass, envelope exclusion, waiver path). §10 rebuilt to consume the PR-083 executor/ledger as built. New OQ-4..7 (waivers, blocklist permanence, log bound, Art. 17(3) tables) packaged for counsel alongside the OQ-1 recommendation. Status: In Review; approval is J. Porter's, then legal.
• 2026-07-05 — v0.2: engineering review (J. Porter) — approval HELD. Recorded as normative: canonical-blob redaction requirement (OQ-3 opened as the design blocker), exhaustive personal-data surface enumeration (incl. PERSON/aliases, clarification attribution, PG audit + subject index, archives), mandatory restore/ledger interaction (shared recovery contract with backup-restore + retention-archival), ledger method field shared with retention (OQ-2 resolved). OQ-1 position pinned (redact, recommended to counsel; executor supports deletion policy as fallback). Next: redacted-blob design pass → re-review → legal sign-off → PR-081.
• 2026-07-02 — v0.1: initial draft; requires legal review sign-off in addition to engineering review before GA.