Spec: Partitioned HNSW — Bucket Lifecycle, Persistence & Multi-Bucket Search¶

File: .ai/specs/core-engine/2026-07-02-hnsw-partitioning.md Status: Draft Owners: Storage lane Related: PR-039/040; F4 ratified; vector-storage spec (source of truth); confidence-decay spec (sim feeds fusion)

1. Overview¶

Defines the derived-index half of the vector subsystem: time-bucket partitioning, open/sealed lifecycle, usearch persistence layout, startup/rebuild procedures, memory management, and the multi-bucket query merge with temporal post-validation.

2. Problem Statement¶

A single HNSW over all time forces post-filtering that destroys recall guarantees at high temporal selectivity ("valid in May 2025" against a 3-year index returns k neighbors mostly outside May). Per-bucket indexes fix that but create lifecycle problems: unbounded open partitions, crash-corrupted save files, memory blowup at (tenants × models × buckets), and cross-bucket result merging semantics.

3. Proposed Solution¶

Partition key = (tenant, model, time_bucket(validTimeStart)); bucket size monthly by default. Partitions are open (in-memory usearch, accepting inserts) while their bucket window is current or recently written, then sealed: serialized to data_dir/vectors/{tenant}/{model}/{bucket}.usearch plus a manifest row (vector count, CF checksum basis, usearch params). Sealed partitions load lazily on first query and evict LRU under a memory budget. All partitions are rebuildable from the vectors CF (F4): missing/corrupt file, checksum mismatch, or bucket-size reconfiguration triggers rebuild (foreground for the queried bucket, background for the rest).

4. Architecture¶

vector write ─▶ PartitionManager.route(bucket) ─▶ open partition insert (+CF write already durable)
query(window, k) ─▶ overlapping buckets ─▶ per-bucket search(k) ─▶ merge by score
              ─▶ temporal end-boundary validation of survivors ─▶ top-k out
seal ticker ─▶ open partitions past grace ─▶ serialize + manifest + demote to sealed

5. Data Models¶

Manifest (schema CF, system-adjacent keying under tenant scope): {tenant, model, bucket, count, hnsw: {M:16, ef_construction:200}, file_hash, built_from_cf_watermark}. usearch metric: cosine via dot on normalized vectors (vector-storage spec). Bucket id = floor(validTimeStart / bucket_len) — pure function, so reconfiguration deterministically remaps and rebuild regenerates everything.

6. API Contracts¶

fn search(scope, model, query: &[f32], window: (vT_lo, vT_hi), tx_t, k, ef_search) -> Vec<(logicalId, sim)>;

Per-bucket k is the full k (cheap insurance for skewed windows); merge keeps global top-k after validation. Validation: each hit's vector version end-boundaries checked at (vT, tx_t) against the CF — a sealed bucket may contain vectors whose validity was later capped; validation-via-primary (same doctrine as global-temporal-scan spec) makes that harmless. telha vector rebuild [--tenant --model --bucket] admin command.

7. UI/UX¶

Operator metrics: partitions open/sealed/loaded, memory in use vs budget, rebuilds triggered, per-query buckets touched. Nothing user-facing.

8. Configuration¶

vector.bucket (month | week | quarter), vector.seal_grace (48h past bucket end), vector.memory_budget_mb (default 4096), vector.ef_search (64, per-query override bounded), hnsw build params per registry model override.

9. Alternatives Considered¶

Single global index + temporal post-filter (rejected: recall collapses at high selectivity — the core reason for partitioning). Per-(tenant,model) index with time as metadata filter (rejected: usearch filtered search still traverses the full graph; partition pruning is the win). Buckets by txTime (rejected: queries are dominated by validTime windows; txTime validation is cheap at merge). Immediately sealing on bucket end (rejected: late-arriving backfill writes with past validTime are common in ingestion; grace period + reopen-on-write handles them, with reopen counting as a rebuild trigger for the sealed file).

10. Implementation Approach¶

PR-039 commits as planned: this spec → PartitionManager → persistence + startup/rebuild → search + merge + validation → admin command → recall/rebuild-equivalence/cross-bucket tests. PR-040 wires sim into fusion.

11. Migration Path¶

None needed ever for index files (derived, rebuildable — that is the point). Bucket-size change: config flip + background rebuild; queries stay correct throughout because rebuild-from-CF is authoritative.

12. Success Metrics¶

Recall ≥0.95@10 vs brute force (100k vectors, realistic windows). Rebuild-equivalence: rebuilt partition returns identical results to original. Cross-bucket window merge correctness fixtures. Memory budget honored under 1000-partition load test (LRU evicts, queries still succeed via reload). Late-backfill test: write into sealed bucket reopens and reconciles.

13. Open Questions¶

• OQ-1: Open-partition durability between CF write and seal — on crash, open partitions rebuild from CF at startup; is startup rebuild latency acceptable for hot buckets? Stance: measure in PR-029 soak; if material, add periodic open-partition snapshots (pure optimization, no semantics change).

14. Changelog¶

• 2026-07-03 — v0.2 (PR-039 as built): (1) Bucket lengths are FIXED (week 7d / month 30d / quarter 90d) so bucket = floor(validStart/len) stays a pure function — "monthly" means 30-day windows, not calendar months. (2) Deviation: manifests are .manifest.json files next to the save-files (blake3 hashes of index + ids sidecar), not schema-CF rows — everything under data_dir/vectors is wholly derived and disposable; a corrupt manifest simply triggers rebuild like any other corruption. (3) The ids sidecar (.ids, msgpack [(nodeId, validStart)]) maps usearch's dense u64 keys back to graph identity. (4) Reopen-on-write loads the sealed file, appends, and re-seals on the next sweep (no separate rebuild of the old file — the re-seal IS the rebuild). (5) Startup is fully lazy: partitions load on first query; there is no boot-time manifest sweep (spec §3's "startup loads manifests" satisfied on-demand). (6) Search selects buckets overlapping the window by validTimeStart routing; vectors whose validity STARTED before the window are in earlier buckets and are not selected — PR-040 passes (0, valid_t] for point queries to cover still-valid history; recall-driven look-back tuning is an open item for narrow windows. (7) Shutdown seals all open partitions (seal ticker's shutdown hook).
• 2026-07-02 — v0.1: initial draft for peer review.