Vector storage¶

Architecture

Every embedding Telha stores is a temporal-versioned fact, not a cache entry: the vectors column family is canonical per the F4 doctrine, and HNSW indexes are rebuildable structures derived from it, never the other way around.

Overview & purpose¶

Three problems motivate treating vector storage as its own durable layer rather than an implementation detail of the HNSW index:

• Without a registry, dimension mismatches corrupt indexes silently. A vector written under the wrong model's dimensionality would poison every downstream search. The registry pins (name, dims, normalize) per model and every write validates against it, hard-failing on mismatch.
• Without temporal versioning, "what did we think this meant in March" is unanswerable. Content changes append new vectors CF versions, exactly like node and edge versions; re-embedding never overwrites history.
• Without a pinned exclusion rule, encrypted PII leaks into embedding provider API calls. Classified content must never reach an external HTTP call. Exclusion is enforced in one assembly function (assemble_embed_text), not scattered checks, and is covered by adversarial tests that plant envelope-magic values in every shape they can arrive in.

Design¶

flowchart LR
    ING["Ingestion completes"] --> FANOUT["embed_fanout()<br/>(4 call sites)"]
    FANOUT -->|"embedding.model set"| ENQ["enqueue_embed_jobs()<br/>kind = embed"]
    ENQ --> QUEUE["Job queue (jobs CF)"]
    QUEUE --> RUNNER["EmbedRunner.process_next()<br/>lease_next_any(['embed'])"]
    RUNNER --> ASSEMBLE["assemble_embed_text()<br/>exclusion applied"]
    ASSEMBLE --> PROVIDER["EmbeddingProvider.embed()"]
    PROVIDER --> WRITE["VectorOps.write_embedding()<br/>dims check + normalize"]
    WRITE --> CF["vectors CF (canonical)"]
    WRITE -.->|"if partitions attached"| PART["PartitionManager.insert()"]

Ingestion (or a clarification answer that changes assembled text) triggers fan-out; the in-core EmbedRunner leases embed jobs one at a time from the shared job queue, assembles exclusion-filtered text, calls the provider, and writes versioned vectors. Embedding never happens in external workers: provider credentials and the exclusion function must stay in core, where classification metadata is authoritative (spec §9).

Why the vectors CF is canonical, not the HNSW files¶

The spec's Alternatives Considered (§9) rejects storing vectors only inside usearch save files: derived structures cannot be canonical (F4); CF-as-truth buys rebuildability and bucket re-sharding for free. It also rejects overwriting vectors on content change (destroys temporal semantics: "similar to X as of March" needs March's vectors to still exist) and embedding inside external workers (exclusion enforcement must sit where classification metadata lives).

Data model¶

vectors CF key (72 bytes)¶

Read from core-engine/src/storage/keys.rs (documented alongside every other CF in Storage engine & key layout):

[ scope 32 ][ node_logical_id 16 ][ embedding_model_id 8 ][ inv(valid_time_start) 8 ][ inv(tx_time_start) 8 ]
                                                                                        total: 72 bytes

valid_time_start inherits the embedded node version's validTimeStart (the vector represents that version's content); tx_time_start is the embedding write time. Both are inverted (u64::MAX - ts), so iteration is newest-first: the same latest-first discipline every other temporal CF uses.

The value is a raw little-endian f32 array with no version byte (f32s_to_le / le_to_f32s in core-engine/src/vector/mod.rs), L2-normalized in f64 precision at write time when the model's normalize flag is set:

pub(crate) fn l2_normalize(vector: &mut [f32]) {
    let norm = vector.iter().map(|v| (*v as f64) * (*v as f64)).sum::<f64>().sqrt();
    if norm > 0.0 {
        for v in vector.iter_mut() {
            *v = (*v as f64 / norm) as f32;
        }
    }
}

Normalizing in f64 and casting back to f32 keeps the unit-norm guarantee tight even for high-dimensional vectors, where a naive f32 sum-of-squares can lose enough precision to matter.

Model registry row (48 bytes, system-scoped)¶

The registry lives as system-scoped, tx-versioned rows in the schema CF (spec v0.2, PR-038 as built), reusing the same 48-byte shape schema_history uses elsewhere:

[ nil 32 ][ model_id 8 ][ inv(tx_time_start) 8 ]
                          total: 48 bytes

model_id = xxh3_64("vector_model/" + name), namespaced so registry rows can never collide with label-hash rows sharing the schema CF (the same hash-namespace rule the bitmap and schema-inference subsystems use). The all-zero 32-byte prefix places every registry row under the reserved system scope described in Storage engine & key layout: unreachable from any tenant scan, unforgeable by any external request path.

The value is MessagePack plus a leading version byte (encode_versioned / decode_versioned, the same envelope every other versioned value in the engine uses):

Re-registration versions the metadata (a new tx-time row, history preserved), but changing dims for an existing name is refused:

if let Some(existing) = self.model(id)? {
    if existing.dims != record.dims {
        return Err(VectorError::Invalid(format!(
            "model {:?} is registered with dims {}; register a new name instead of \
             changing dimensionality", record.name, existing.dims
        )));
    }
}

A dims change is a new model, not an update: historical vectors would otherwise become unreadable against the registry.

Embed job payload¶

#[serde(rename_all = "camelCase")]
pub struct EmbedJobPayload {
    pub v: u32,             // payload schema version
    pub model: String,      // registered model to embed under
    pub node_ids: Vec<Uuid>, // one batch ≤ the provider batch size
}

Job kind = "embed" (EMBED_JOB_KIND), leased exclusively by the in-core EmbedRunner via lease_next_any(["embed"]) in a drain-then-sleep loop (spawn_embed_runner): drain everything queued, then sleep embedding.poll_interval_secs and repeat, until shutdown.

Algorithms & invariants¶

Text assembly and the exclusion guarantee¶

assemble_embed_text (core-engine/src/vector/embed.rs) is the single function the spec requires: one place where classified content is filtered, not scattered checks across call sites.

flowchart TD
    START["NodeVersion"] --> LABELS["labels.join(' ') → line 1"]
    LABELS --> LEAD["LEAD_KEYS: title, name, text<br/>(in that order, if present)"]
    LEAD --> REST["remaining properties<br/>(BTreeMap order)"]
    REST --> CHECK{"contains_envelope(value)?"}
    CHECK -->|yes| DROP["dropped (never reaches provider)"]
    CHECK -->|no| EMIT["'key: rendered value' line"]
    EMIT --> JOIN["lines.join('\\n')"]
    JOIN --> TRUNC["truncate to max_text_bytes<br/>on a char boundary"]

Assembly order: node labels first, then the three LEAD_KEYS (title, name, text) if present, then every other property in map order. Labels alone (no properties) return None: "labels are not content."

contains_envelope recurses through Value::Array and Value::Object and checks two shapes at every leaf:

• Raw bytes: Value::Bytes(b) starting with the field-encryption ciphertext envelope magic [0x7E, 0x1A].
• Base64-encoded strings: Value::Str(s) whose decoded first 4 bytes start with the same magic (envelopes crossing the JSON wire arrive base64-encoded).

The ingest-metadata classification flag (a separate, coarser signal than the envelope-magic check) is not yet modeled in core as built; it is designed to plug into this same function once the app-layer classification policy lands (spec §14 v0.2 note 4).

Dimension validation and write path¶

VectorOps::write_embedding looks up the model by name, hard-fails with VectorError::DimsMismatch on any length mismatch, normalizes if the registry says so, and appends one new key at (node_id, model_id, valid_time_start, clock.now()). There is no read-modify-write: every embedding write is a pure append, so a concurrent reader never observes a partially-written vector.

Provider retry and error classification¶

HttpEmbeddingProvider::call_once classifies every failure as Transient or Permanent:

embed retries transient failures at most twice, with jittered exponential backoff:

attempt 1: base = 250ms,  jitter = ±20% (50ms)  → 200-300ms
attempt 2: base = 500ms,  jitter = ±20% (100ms) → 400-600ms

A Permanent failure, an unknown model, or a dims mismatch dead-letters the job immediately (EmbedError::retryable() returns false for all three); every other failure is retried by the job queue's normal lease mechanics.

Fan-out: 4 call sites, 1 gate¶

Every fan-out path routes through one helper, AppState::embed_fanout (core-engine/src/api/mod.rs), so "no-op unless embedding.model is configured" is enforced in exactly one place rather than duplicated per call site:

pub fn embed_fanout(&self, scope: &TenantScope, node_ids: &[Uuid]) {
    if self.embedding.model.is_empty() || node_ids.is_empty() {
        return;
    }
    if let Err(e) = crate::vector::embed::enqueue_embed_jobs(
        &self.jobs, scope, &self.embedding.model, node_ids, self.embedding.batch_size,
    ) {
        tracing::error!(error = %e, nodes = node_ids.len(), "embed fan-out enqueue failed");
    }
}

Enqueue failures log and are swallowed everywhere: embedding is derived data, so an enqueue hiccup must never fail the ingestion or clarification write that produced the underlying nodes.

Node-level only (span-level deferred)¶

The spec's OQ-1 asked whether v1 should embed node-level, span-level, or both. As built, only node-level embeddings exist. The vectors CF key has no span discriminator, so a per-span vector has nowhere to live without a key-layout change. Span-level embedding (needed for claim-verification) is deferred to that lane (spec §14 v0.2 note 3).

Configuration¶

As-built notes¶

From spec §14 (Changelog v0.2, PR-038 as built), reconciled against core-engine/src/vector/:

• Model registry placement and refusal rule match the changelog exactly: system-scoped schema-CF rows, `modelId = xxh3("vector_model/"
• name)`, re-registration versions metadata, dims changes refused.
• Embed jobs run only in the in-core runner, never external workers, as the changelog specifies. EMBED_JOB_KIND = "embed" is leased via lease_next_any(&["embed"]), distinct from the ingest:* kinds external workers poll.
• Fan-out is 4 sites, not enumerated in the spec prose itself (the spec says "fires per applied submission... and from the JSON bypass"); the code shows exactly 4, all routed through one gate (embed_fanout), enumerated above.
• OQ-1 deviation confirmed in code: node-level only. The vectors key (node_logical_id, no span field) structurally cannot address a span-level vector.
• Exclusion covers raw bytes and base64-prefixed strings, matching changelog note 4; the ingest-metadata classification flag itself is not yet modeled in core, also as noted.
• Retry policy matches exactly: ≤2 attempts, jittered exponential backoff, 429/5xx/timeouts transient, other 4xx permanent.
• Token accounting is the in-process TokenLedger keyed by tenant Uuid, recording (requests, tokens); durable cost-ledger rows are future work for generation-orchestration to consume, per the changelog.
• embedding.* absent from the env-override whitelist is not mentioned in the spec at all; it is a code-level observation from reading config.rs directly (see Configuration above).
• Not directly verifiable from this code alone: the spec's success metric "per-tenant embed token counts match provider-mock tallies" is exercised by the embed_job_round_trip_writes_versioned_vectors test (core-engine/src/vector/embed.rs), which does assert the ledger matches the mock's tally for one call; broader property-style coverage across randomized batch shapes was not inspected beyond that test module.

Related¶

• HNSW partitioning, the derived index layer that consumes these vectors.
• Confidence decay, how similarity from a vector query fuses with graph decay.
• Storage engine & key layout, the vectors and schema CF shapes and the reserved system namespace.
• Field encryption, the ciphertext envelope format the exclusion check detects.