Skip to content

Spec: Schema Inference — Type Lattice & Versioning

File: .ai/specs/core-engine/2026-07-02-schema-inference.md Status: Draft Owners: Storage lane Related: PR-020; version-record-model spec (Value enum); composite-key-layout spec (schema CF)

1. Overview

Defines zero-schema ingest semantics: the type lattice used to merge observed property types per (tenant, label), when a new schema version is written, and time-travel introspection (schema_as_of). Also owns the edge-type collision registry's storage.

2. Problem Statement

"Push any JSON" is only useful if the system can later answer "what does a RISK look like, and what did it look like in March?" Without defined merge rules, conflicting types (severity: 8 then severity: "high") produce arbitrary answers; without versioning triggers, every write churns the schema CF; without a lattice, type widening is ad-hoc per call site.

3. Proposed Solution

On every node/edge write, observed property types merge into the current schema for (tenant, label) through a fixed lattice; a new schema version is written only when the merge changes something (new property, widened type, new label). Schema records are themselves tx-time-versioned (schema CF key: [scope][labelHash][inv(txTimeStart)]) so introspection at T is a prefix scan. Inference never rejects data — it only records; validation is an app-layer concern.

4. Architecture

Write path → extract (property, ObservedType) pairs → load current schema (cached per scope+label with tx-watermark invalidation) → lattice merge → if changed: append new schema version in the same WriteBatch as the data write (schema history is exactly as durable as the data that caused it).

5. Data Models

Lattice (⊔ = least upper bound):

            Any
   ┌───┬───┬─┴─┬───────┬──────┐
 Bool Int Float Str Datetime Bytes   Array<T>   Object{k:T}
        └──⊔──┘
        (Int ⊔ Float = Float)        Array<A> ⊔ Array<B> = Array<A⊔B>
                                     Object merges per-key; missing key ⊔ T = Optional<T>
 Null ⊔ T = Optional<T>              any cross-branch ⊔ = Any

Schema record: { label, version, properties: Map<name, {ty, optional, first_seen, last_widened}>, edge_types_seen }. The edge-type collision registry (hash → canonical string) lives under a reserved labelHash in this CF; a second string mapping to an existing hash fails the write loudly (composite-key-layout spec).

6. API Contracts

fn observe(scope, label, props) -> Option<SchemaVersionWrite>;  // called by write paths, same batch
fn schema_as_of(scope, label: Option<String>, tx_t) -> SchemaSnapshot;  // None = all labels

REST surface: GET /v1/schema?at=T (PR-024). Datetime detection: only values already typed Datetime in the Value enum count — no string sniffing (a string that looks like a date is a Str; see Alternatives).

7. UI/UX

Schema introspection feeds the MCP getSchema tool ("what entity types exist?") — output shaping there, not here. telha debug schema <tenant> <label> --history prints the version chain.

8. Configuration

schema.cache_entries (default 10k per process). Lattice and trigger rules are not configurable.

9. Alternatives Considered

String date-sniffing (rejected: locale/format ambiguity makes inference nondeterministic; ingestion workers emit typed Datetime when they parse dates, which is the honest boundary). Per-property value histograms (rejected for v1: useful for planners eventually, but heavy; selectivity telemetry from global-temporal-scan spec covers planner needs first). Rejecting type conflicts (rejected: violates zero-schema promise; widening to Any with last_widened recorded gives auditability instead). Schema in a separate store (rejected: same-batch durability with the triggering write is the correctness property).

10. Implementation Approach

PR-020 commits: this spec → lattice + merge (pure functions, exhaustively property-tested) → observe() wired into write paths → schema_as_of → cache with watermark invalidation → conflict fixture suite (documented lattice outcomes for every branch pair).

11. Migration Path

Greenfield. Lattice extensions (e.g., Decimal per version-record-model OQ-1) are additive: old schema versions remain valid; new type joins the lattice with defined ⊔ against all existing types via spec amendment.

12. Success Metrics

Property test: ⊔ is commutative, associative, idempotent (lattice laws) across randomized type trees. Fixture suite: every branch-pair conflict produces the documented result. Time-travel test: schema_as_of(T) matches replay of writes up to T. Churn test: steady-state identical writes produce zero new schema versions.

13. Open Questions

  • OQ-1: Should Optional be tracked per property presence-ratio (e.g., "present in 94% of versions") for planner/UX value? Stance: not in v1; presence bool only; revisit with histogram work.

14. Changelog

  • 2026-07-02 — v0.1: initial draft for peer review.