Schema inference¶

Architecture

Telha accepts any JSON-shaped node or edge write with no schema declared up front. This page documents how the engine still answers "what does a RISK look like, and what did it look like in March": a type lattice merges every observed property type per (tenant, label), and a new schema version is written only when that merge actually changes something.

Overview & purpose¶

core-engine/src/schema/inference.rs states its own scope precisely: every node or edge write observes its property types; observations merge into the current per-(tenant, label) schema through a fixed lattice; a new schema version is written in the same WriteBatch as the data write and only when the merge changed something. Inference never rejects data, only records it; validation is deliberately left to the application layer (spec §3).

Three problems motivate treating this as its own subsystem rather than an ad-hoc side effect of the write path:

• Zero-schema ingest needs a deterministic merge rule. Without one, conflicting observations (severity: 8 then severity: "high") would produce arbitrary answers depending on write order.
• Naive versioning churns the schema CF. If every write appended a schema version regardless of content, steady-state ingestion of identical shapes would flood the schema column family. The trigger is "merge changed something," not "a write happened."
• Introspection needs to time-travel too. Schema records are themselves tx-time-versioned, so schema_as_of(tenant, label, T) is answerable the same way any other tritemporal fact is: a prefix scan bounded by transaction time.

Design¶

Write path integration¶

Both NodeOps (core-engine/src/temporal/node_ops.rs) and EdgeOps (core-engine/src/graph/edge_ops.rs) hold their own SchemaInference instance, constructed over the same shared StorageEngine as the rest of the write path:

pub struct NodeOps {
    engine: Arc<dyn StorageEngine>,
    clock: Arc<TxClock>,
    schema: crate::schema::SchemaInference,
    bitmaps: Option<crate::storage::bitmap::BitmapMaintenance>,
    write_lock: Mutex<()>,
}

NodeOps::schema_ops calls observe() once per label on the version being written, collecting the resulting WriteOps and SchemaRecords. Those ops are appended to the same batch that carries the node or edge version write, the adjacency rows, and (if enabled) the bitmap posting updates, and the whole batch goes through one engine.batch_write(batch) call. Only after that batch succeeds does the caller invoke schema.confirm(scope, record) for each observed record, updating the in-process cache. This ordering is deliberate: the cache must never reflect a schema version that did not actually make it to storage.

sequenceDiagram
    autonumber
    participant W as Write path (NodeOps/EdgeOps)
    participant S as SchemaInference
    participant B as WriteBatch
    participant E as StorageEngine

    W->>S: observe(scope, label, properties, tx)
    S->>S: load_current (cache or latest stored version)
    S->>S: merge_observation (lattice join)
    alt merge changed something
        S-->>W: Some((WriteOp::Put schema row, SchemaRecord))
        W->>B: push schema op alongside data ops
    else churn-free
        S-->>W: None
    end
    W->>E: batch_write(B)
    E-->>W: Ok
    W->>S: confirm(scope, record)

Edge properties observed under a namespaced label¶

Edges have no "label" of their own, but the same lattice and versioning machinery covers them: EdgeOps observes edge properties under a synthetic label edge_type/<type>, built by:

fn edge_schema_label(edge_type: &str) -> String {
    format!("edge_type/{edge_type}")
}

An edge of type OWNS therefore has its own schema history addressable as label edge_type/OWNS, disjoint from any node label by construction (a real node label can happen to be the literal string edge_type/OWNS, but the two route through independent label_hash inputs since edge-type observation always carries the edge_type/ prefix before hashing).

Same-batch schema puts, confirm-after-commit cache¶

The spec's rationale for keeping schema writes in the triggering batch rather than a separate store (§9, Alternatives Considered: "Schema in a separate store, rejected") is that schema history must be exactly as durable as the data that caused it. As built, SchemaInference::observe only returns a WriteOp; it never writes it itself. The caller (NodeOps/EdgeOps) is responsible for folding it into the version's write-set before calling batch_write. confirm() is a separate, explicit step called only after that batch returns Ok, so a crash or storage error between observe() and commit leaves the cache untouched and the next observe() call re-reads from storage.

Reading the current schema: cache or latest stored version¶

load_current checks the in-process HashMap<(tenant_id, organization_id, label), SchemaRecord> cache first; on a miss it calls version_at(scope, label, Ts::MAX), which is a bounded prefix scan over the label's schema history that returns the first row whose tx_time_start <= Ts::MAX (i.e. the newest version), and populates the cache with what it finds. There is no explicit tx-watermark comparison against a stored high-water mark; watermark invalidation is implicit: the cache is only ever refreshed from storage on miss or after a confirm() following a successful write, and a full cache clear happens if CACHE_ENTRIES (10_000 per process, hardcoded) is reached, rather than an LRU eviction (the code's own comment: "simple full-evict; refine with LRU if it matters").

Data model¶

The type lattice¶

SchemaType is the lattice's carrier (spec §5):

pub enum SchemaType {
    Unknown,              // bottom: nothing observed yet
    Bool,
    Int,
    Float,                // Int ⊔ Float = Float
    Str,
    Datetime,              // only Value::Datetime counts; no string sniffing
    Bytes,
    Array(Box<Shape>),     // element shape is the join of all elements
    Object(BTreeMap<String, Shape>),  // per-key shapes; missing key ⇒ optional
    Any,                   // top: conflicting branches join here
}

pub struct Shape {
    pub ty: SchemaType,
    pub optional: bool,    // Null ⊔ T = Optional<T> lives in this flag
}

                       Any
   ┌───┬────┬───┴────┬────────┬───────┐
 Bool  Int Float    Str    Datetime  Bytes    Array<T>   Object{k:T}
        └───⊔───┘
       (Int ⊔ Float = Float)

join(a, b) computes the least upper bound of two shapes: optional is the logical OR of both sides, and join_ty dispatches on the type pair. Unknown is absorbed by anything; Any absorbs everything; Int/Float join to Float; matching Array/Array and Object/Object recurse (an Object join walks every key present in either map, joining shared keys and marking a key optional when only one side has it); identical types join to themselves; every other cross-branch pair (e.g. Datetime and Int, or Bool and Bytes) joins to Any. Notably, Datetime has no numeric bridge: a property that is sometimes an Int and sometimes a Datetime widens straight to Any, the same as any other unrelated-type conflict.

Records and the field entry¶

pub struct FieldSchema {
    pub shape: Shape,
    pub first_seen: Ts,     // tx-time of the first observation
    pub last_widened: Ts,   // tx-time of the last widening change (0 = never)
}

pub struct SchemaRecord {
    pub label: String,
    pub version: u32,        // monotonic, 1-based
    pub properties: BTreeMap<String, FieldSchema>,
}

merge_observation is the pure function that decides whether a new version is warranted. Given the current record (or None for a brand-new label) and one write's (property, Value) pairs, it:

1. Joins each observed property's shape_of(value) into the existing FieldSchema, or inserts a new one (marked optional if the schema already existed, since prior versions lacked the property).
2. Marks every known property absent from this write as optional if it was not already.
3. Returns None if nothing changed (churn-free steady state); otherwise increments version and returns Some(merged).

first_seen is set once and never overwritten; last_widened updates only when a property's type actually widens (not merely when optional flips).

Storage key and the edge-type registry's disjoint namespace¶

Schema rows share the 48-byte schema column family layout documented in Storage engine & key layout:

[ scope 32 ][ label_hash 8 ][ inv(tx_time_start) 8 ]
                               total: 48 bytes

label_hash is not xxh3_64(label) directly; it is xxh3_64("label/" + label):

const LABEL_NAMESPACE: &str = "label/";
fn label_hash(label: &str) -> LabelHash {
    LabelHash::of(&format!("{LABEL_NAMESPACE}{label}"))
}

The module doc explains why: the edge-type collision registry (PR-017) keys its rows by the raw edge-type hash in the same schema CF, so namespacing schema-record rows under "label/" keeps the two row families disjoint (up to an xxh3 collision, the same accepted risk class as the edge-type registry itself). schema_as_of(scope, None, tx_t), which lists every label, has to tolerate this: it decodes every row in the tenant's schema CF prefix and silently skips any that fail to decode as a SchemaRecord (the registry rows are raw UTF-8 strings, not the versioned msgpack schema encoding), rather than filtering by label-hash namespace membership.

Encoding¶

encode_record/decode_record use the same versioned envelope as other model values: one VALUE_VERSION byte followed by rmp_serde (MessagePack, named fields). Decoding rejects any version byte other than the current VALUE_VERSION with DecodeError::UnknownVersionByte.

Algorithms & invariants¶

observe, confirm, schema_as_of¶

fn observe(scope, label, props) -> Option<SchemaVersionWrite>;
fn schema_as_of(scope, label: Option<String>, tx_t) -> SchemaSnapshot;  // None = all labels

As built, the public surface is:

pub fn observe(&self, scope, label, properties, tx)
    -> Result<Option<(WriteOp, SchemaRecord)>, SchemaError>;
pub fn confirm(&self, scope, record: SchemaRecord);
pub fn schema_as_of(&self, scope, label: Option<&str>, tx_t)
    -> Result<Vec<SchemaRecord>, SchemaError>;

schema_as_of with label: Some(_) calls version_at, a prefix scan over ScanRange::schema_history(scope, label_hash) (latest-first, per the inv(tx_time_start) key inversion) that returns the first row whose tx_time_start <= tx_t. With label: None, it walks the tenant's entire schema CF range, decodes what it can, and keeps only the newest version per label observed at or before tx_t.

schema_as_of time travel¶

sequenceDiagram
    autonumber
    participant C as Caller
    participant SI as SchemaInference
    participant CF as schema CF (RocksDB/Memory)

    C->>SI: schema_as_of(scope, "DOC", tx_t=150)
    SI->>CF: scan_prefix(schema_history(scope, hash("label/DOC")))
    Note over CF: rows sorted newest tx_time_start first
    CF-->>SI: v2 (tx=200), v1 (tx=100), ...
    SI->>SI: skip v2 (tx_time_start 200 > 150)
    SI->>SI: return v1 (tx_time_start 100 <= 150)
    SI-->>C: SchemaRecord { version: 1, ... }

This is exactly the same "prefix scan, first row at or before T" pattern used elsewhere in the tritemporal engine (see The tritemporal model); schema history is a first-class tritemporal citizen of the store, not a bolt-on side table.

Configuration¶

As-built notes¶

From spec §14 (Changelog: only a v0.1 initial-draft entry exists; no amendments recorded), reconciled against core-engine/src/schema/:

• schema.cache_entries is not a runtime-configurable key. The spec's §8 Configuration section lists it as a config key with a "default 10k per process" framing. The code has no SchemaConfig struct and no schema.cache_entries entry in TelhaConfig or its ENV_KEYS allowlist (core-engine/src/config.rs); it is simply the const CACHE_ENTRIES: usize = 10_000 in inference.rs, changing it requires a code change and rebuild, not a config edit.
• Cache eviction is full-clear, not LRU. confirm() clears the entire cache once CACHE_ENTRIES is reached rather than evicting the least recently used entry; the code comments this explicitly as a deliberate v1 simplification ("refine with LRU if it matters").
• The edge-type collision registry's storage is only partially in scope here. This spec (§1) says it "also owns the edge-type collision registry's storage," and the schema CF does host both row families side-by-side (disjoint via the "label/" hash namespace). The registry's own hash-collision detection and write-time collision check live in EdgeOps (core-engine/src/graph/edge_ops.rs, check_type_registry), not in core-engine/src/schema/; this page covers only how schema_as_of's all-labels listing tolerates registry rows sharing the CF.
• OQ-1 (presence-ratio tracking for Optional) is unresolved as specified: the spec's stance is "not in v1; presence bool only," and the code matches exactly, Shape.optional is a bool, no ratio or count is tracked.
• REST surface GET /v1/schema?at=T (spec §6, "PR-024") and the telha debug schema <tenant> <label> --history CLI command (spec §7) were not inspected as part of this page; they are consumers of schema_as_of living outside core-engine/src/schema/.

Related¶

• Storage engine & key layout: the schema column family's byte layout and the reserved system namespace it also hosts.
• Concepts › The tritemporal model: the winner-rule pattern version_at reuses for schema history.
• Bitmap predicate index: another derived index maintained in the same write batch as the triggering version write.
• The version record model: the Value enum whose variants shape_of maps onto the lattice.