Spec: Vector Storage — Embeddings, Model Registry & Embed Jobs¶

File: .ai/specs/core-engine/2026-07-02-vector-storage.md Status: Draft Owners: Storage lane Related: PR-038; F4 ratified (vectors CF canonical); hnsw-partitioning spec (derived indexes); field-encryption spec (classified-content exclusion)

1. Overview¶

Defines the durable half of the vector subsystem: the vectors CF as source of truth, the embedding-model registry, the EmbeddingProvider trait, embed-job fan-out, temporal versioning of embeddings, and the classified-content exclusion guarantee.

2. Problem Statement¶

Without a registry, dimension mismatches corrupt indexes silently; without temporal versioning, "what did we think this meant in March" is unanswerable and re-embedding destroys history; without a pinned exclusion rule, encrypted PII leaks into embedding-provider API calls — a compliance breach through the side door.

3. Proposed Solution¶

Vectors CF (key per composite-key-layout spec, value = raw little-endian f32) is canonical per F4. A model registry in the schema CF maps embeddingModelId (8B) → {name, provider, dims, normalize: bool}; writes validate dims against the registry or hard-fail. Content changes produce new temporal vector versions (append-only, like everything else). Embedding text is assembled from node properties/spans after classified-field filtering: any property flagged classified in ingest metadata, and any property whose value carries the ciphertext envelope magic (field-encryption spec), is excluded before the provider call — enforced in one assembly function with tests, not scattered checks.

4. Architecture¶

Ingestion completes → core enqueues embed jobs (per new/changed node or span batch) → embed job: assemble text (exclusion applied) → EmbeddingProvider.embed(model, texts) → write vector versions → notify PartitionManager (hnsw-partitioning spec) of new vectors.

5. Data Models¶

Key: [scope][nodeLogicalId][embeddingModelId][inv(validTimeStart)][inv(txTimeStart)]; validTimeStart inherits the embedded node version's validTimeStart (the vector represents that version's content). Value: [f32; dims] LE, optionally L2-normalized per registry flag (normalized at write so search is dot-product). Registry record includes provider_ref for audit but never credentials.

6. API Contracts¶

trait EmbeddingProvider { fn embed(&self, model: &ModelRef, texts: &[String]) -> Result<Vec<Vec<f32>>>; }

HTTP impl: OpenAI-compatible /embeddings + configurable local endpoint; batching (max 96 texts/call), retry/backoff, per-tenant token accounting shared with LlmProvider machinery (generation-orchestration spec). Query-time text embedding (PR-040's vector.text) reuses the same trait and exclusion rules trivially (query text is user-supplied, no filtering needed, but rate-capped).

7. UI/UX¶

telha vector models lists the registry; telha vector stats <tenant> shows counts per model/bucket. No end-user surface.

8. Configuration¶

embedding.endpoint, embedding.api_key_env, embedding.batch_size (96), embedding.max_text_bytes (32KB, longer content spans chunk-embedded per span), registry entries via CLI (telha vector register-model).

9. Alternatives Considered¶

Storing vectors only inside usearch files (rejected: F4 — derived structures cannot be canonical; CF-as-truth buys rebuildability and bucket re-sharding). Overwriting vectors on content change (rejected: destroys temporal semantics; "similar to X as of March" requires March's vectors). Embedding in workers (rejected: exclusion enforcement must sit in core where classification metadata is authoritative; workers never hold embedding credentials). f16 storage (deferred: halves space, costs precision; measure recall impact in Phase 4 before adopting).

10. Implementation Approach¶

PR-038: this spec → registry + write path with dims validation → EmbeddingProvider + HTTP impl → embed-job kind + fan-out → exclusion assembly function with adversarial tests (planted envelope-magic values must never reach the provider mock).

11. Migration Path¶

Greenfield. New models register additively; deprecating a model stops new writes but historical vectors remain queryable (their partitions persist until GDPR/retention processes say otherwise).

12. Success Metrics¶

Dims-mismatch hard-fail test. Versioning test: content change ⇒ new vector version, old version still retrievable at its coordinate. Exclusion test: classified/enveloped content never appears in provider-mock request logs across randomized nodes. Accounting test: per-tenant embed token counts match provider-mock tallies.

13. Open Questions¶

• OQ-1: Span-level vs node-level embedding granularity default? Stance: v1 embeds node-level (title+text assembly) AND span-level for PARAGRAPH-kind spans (verification needs span vectors, claim-verification spec); measure storage cost, revisit selectivity.

14. Changelog¶

• 2026-07-02 — v0.1: initial draft for peer review.
• 2026-07-03 — v0.2 (PR-038 as built): (1) Model registry = system-scoped, tx-versioned rows in the schema CF ([nil 32][modelId 8][inv tx 8]; modelId = xxh3("vector_model/"+name) per the hash-namespace rule); re-registration versions the metadata, but a dims change for an existing name is REFUSED — that is a new model. (2) Embed jobs are processed by an in-core runner (poll loop over the job queue, kind embed), never by external workers — credentials and the exclusion function stay in core (§9). Fan-out fires per applied submission (including per streamed partial; idempotent replays excluded) and from the JSON bypass, gated on embedding.model being configured. (3) Deviations from OQ-1: v1 embeds node-level only; span-level vectors need a span discriminator the key layout lacks — deferred to the claim-verification lane. (4) Exclusion detects the envelope magic in raw bytes AND base64-encoded strings (envelopes crossing the JSON wire arrive base64); the ingest-metadata classification flag is not yet modeled in core — it plugs into the same assembly function when the app-layer classification policy lands. (5) Token accounting is the in-process TokenLedger (per-tenant requests/tokens) that the generation-orchestration cost ledger will consume; durable rows land there. (6) embedding.api_key_env names the env var; keys never appear in config files. Retry: ≤2 attempts, jittered exponential backoff, 429/5xx/timeouts transient, 4xx permanent.