Spec: Source Connector Framework (SharePoint, Exchange, Slack, Salesforce, …)¶

File: .ai/specs/integration/2026-07-04-source-connector-framework.md Status: Approved (review 2026-07-04 by J. Porter — namespace-scoped delete condition incorporated in v0.3; OQ stances ratified) Owners: integration lane; reviewers: workers lane (connector runtime, projections), storage lane (source identity, tombstones), app-layer lane (ACL consumers) Related: ingestion-provenance spec (source identity, dedup, spans), job-queue-leases spec (sync job chains), entra-identity-connector spec (person resolution for authorship), temporal-clarification spec (authority routing consumes the ownership edges defined here), tombstone-cascade spec (source deletions), data-split-routing spec (what may leave the source system).

1. Overview¶

Defines the common contract every pull-based content connector implements: how a connector discovers and syncs items from a source system (SharePoint, Exchange, Slack content, Salesforce, Google Drive, …) into the existing ingestion pipeline, and, critically, which metadata it MUST carry so the rest of Telha works: authorship and ownership (powering clarification authority routing), source ACLs (powering the RBAC gate's source-side check), timestamps (powering temporal grounding), and stable external identity (powering dedup and deletion). Individual connectors get thin per-source sections here (§10) and may grow their own specs when their projection is non-trivial; the contract in §3–§6 is normative for all of them.

2. Problem Statement¶

Today ingestion is push-based: something must call POST /v1/ingest. Without a connector framework: (a) every pilot improvises export scripts, so "connects to the tools you already run" is a promise without a mechanism; (b) each ad-hoc integration invents its own metadata shape, so authorship/ownership arrives inconsistently or not at all, and clarification routing has nothing to rank; (c) source ACLs are dropped at the door, so the source-side check in the clarification RBAC gate cannot exist; (d) deletions in the source silently persist in Telha; (e) re-syncs duplicate content because item identity is not stable; (f) one failing item aborts a whole sync run.

3. Proposed Solution¶

Normative contract for every content connector:

Connectors are read-only against the source. They never write, move, tag, or annotate source content. (This is load-bearing for the "no rip-and-replace" positioning.)
Connectors run as sync:{source} job chains (audience "connector", same fence as the Entra connector): each run leases a job whose payload carries the delta cursor, processes the window, and enqueues the successor with the advanced cursor. The job chain IS the checkpoint; there is no separate connector-state store.
Every discovered item is submitted through the EXISTING ingest path (POST /v1/ingest with the appropriate format: docling for files, tabular for sheets, json for records) with a source_meta envelope (§5). Connectors fetch and route; parsing stays in the format workers.
source_meta MUST carry, when the source exposes them: author, last-modified-by, owner (as principal references), created/modified timestamps, container path, source deep link, ACL principal list, and the source's stable external_id. Omitting an available field is a connector bug.
Source identity is deterministic: source_id = uuidv5("{connector}:{tenant}:{external_id}"). Re-syncing an unchanged item is a no-op via ingestion-provenance content-hash dedup; a changed item creates a new source version.
Principals resolve through the identity model: author/owner references become PERSON refs (canonical when an alias matches, placeholder-by-email otherwise, per the Entra connector spec), written as AUTHORED_BY / OWNED_BY / LAST_MODIFIED_BY edges from the source's DOCUMENT/root node. These edges are the authority substrate the clarification loop ranks on.
Mirrored ACLs are advisory metadata, floor-only: they may be used to NARROW who is shown content (the clarification gate's source-side check) and never to widen access beyond Telha workspace RBAC. Telha RBAC remains the enforcement floor.
Deletions propagate: a delete observed in the delta feed tombstones the source-derived nodes via the standard cascade; history remains reconstructable at prior transaction times. Delete authority is namespace-scoped (approval condition, 2026-07-04): a connector-audience token may tombstone only records whose source identity (source_connector_id + external_id) belongs to the authenticated connector's namespace. Core MUST reject delete attempts against records outside that namespace with 403 CONNECTOR_NAMESPACE_MISMATCH and emit a security audit event. A compromised SharePoint connector can therefore never tombstone Slack-synced, Exchange-synced, uploaded, or manually created records — connector tokens are not tenant-wide delete credentials.
Item-level fault isolation: a failing item is recorded (audit + per-run error list) and skipped; it MUST NOT abort the run. Runs report processed/skipped/failed counts as job stats.

4. Architecture¶

source system (delta API)                    e.g. Graph drive delta, Slack cursor,
        │  read-only, per-tenant credential        Salesforce updated/deleted
        ▼
{source} connector (python, workers_common, audience "connector")
  sync:{source} job chain: lease ─▶ walk delta window
        │  per item: fetch bytes/record + collect source_meta
        ▼
POST /v1/ingest (format worker parses; provenance spec applies)
        │  core: source version + spans + nodes
        ├─ source_meta → DOCUMENT node properties + AUTHORED_BY/OWNED_BY/
        │                LAST_MODIFIED_BY edges (persons via alias resolution)
        ├─ acl_json stored on the source version (advisory, floor-only)
        └─ deletes → tombstone cascade
        ▼
consumers: queries (temporal grounding), clarification routing (authority),
           RBAC gate source-side check (ACLs), trace (deep links)

5. Data Models¶

source_meta envelope (additive field on IngestRequest.options). {external_id, container_path, source_url, created_at, modified_at, author?: principal, last_modified_by?: principal, owner?: principal, acl?: [principal], extra?: {}} where principal = {kind: aad_oid|email|slack_user|sf_user, value} (µs timestamps). Core maps principals → PERSON logical ids per the resolution rules (Entra spec §5) at apply time, so late identity sync upgrades placeholders without connector involvement.

Graph shape. The format worker's projection is unchanged (docling/tabular/json specs). Core attaches: source_url, container_path, external_id as properties on the projection root; AUTHORED_BY / LAST_MODIFIED_BY / OWNED_BY edges root → PERSON; acl_json on the source version row (not on nodes: ACLs describe the container/file, and the version pins what was true at sync time).

Job payloads. sync:{source} payload: {connection_id, cursor(s), window_started_at, stats?}. Cursor semantics are per source (§10); an invalidated cursor (source-side expiry) triggers full resync, same rule as the Entra connector.

6. API Contracts¶

Connector-side (consumed). Each connector declares its source API surface and scopes in §10; all are least-privilege read-only. Throttling: honor source Retry-After; page natively; never parallel-fetch beyond the source's published guidance.

Core-side (produced). POST /v1/ingest gains options.source_meta (additive JSON; ignored by cores predating this spec only in the sense that they reject unknown option keys loudly per deny_unknown_fields, so deploy order is core before connectors, matching the clarification capability pattern). GET /v1/ingest/:jobId unchanged. Deletion: connectors call DELETE /v1/records/:id on the projection root resolved via external_id lookup (GET /v1/records?externalId= added as an indexed filter), cascading per the tombstone spec; core enforces the §3.8 namespace scope on every connector-token delete (403 CONNECTOR_NAMESPACE_MISMATCH + security audit event on violation — the externalId index makes the ownership check a point lookup). Failure taxonomy: source auth failure → permanent, dead-letter + CONNECTOR_AUTH_FAILED audit; item fetch/parse failure → item-level skip with audit; cursor invalidation → in-place full resync, not an error.

7. UI/UX¶

Operator CLI: telha connector ls|status|run <name> (shared with the Entra connector): last run, cursor age, processed/skipped/failed, top skip reasons. Audit events per run and per item-level failure. End-user visibility comes through provenance: every answer's evidence links back to source_url, so "open it in SharePoint" is one click and lands behind the source's own login.

8. Configuration¶

Per connection (a tenant may have several): [connectors.{source}.{connection_id}]: enabled (false), credentials by env-var name only, interval_minutes (30), scope (site/mailbox/channel/object allowlist: sync nothing by default, sync what is listed), max_item_mb (respects ingest.max_upload). Deliberately NOT configurable: read-only posture, ACL floor-only rule, deterministic source identity, item-level fault isolation, deletion propagation.

9. Alternatives Considered¶

Push/webhook-first integration. Rejected as the primary mechanism: webhook coverage and reliability vary wildly per source; delta pull is universal and self-healing. Webhooks may later ACCELERATE a connector (trigger an early run) but never replace the cursor chain.
Connectors bypassing the ingest pipeline (writing nodes directly). Rejected for content: it would fork provenance (no source versions, no spans, no dedup). Structured-record sources (Salesforce objects) still go through format=json ingest for the same reason. (The Entra connector's direct records writes are the deliberate exception: identity has no document to span.)
Enforcing source ACLs as Telha authorization. Rejected: mirrored ACLs go stale between syncs and mix models; floor-only advisory use keeps Telha RBAC the single enforcement point while still preventing over-sharing in clarification delivery.
One spec per connector from day one. Rejected: four near-identical specs would drift; the contract is the design. Per-source specs are added only when a projection is genuinely novel (already true of docling/web/tabular formats).
Storing sync state in a dedicated CF/table. Rejected: the job-chain checkpoint pattern already exists, survives crashes, and needs no new storage surface.

10. Implementation Approach¶

Core PR first (PR-065, build doc Phase 3b): options.source_meta handling, principal resolution + authorship/ownership edges, acl_json on source versions, externalId lookup, sync: audience fence extension. Then per-connector PRs in commercial priority order (PR-066..069; Google Drive is unnumbered backlog), each a thin package over workers_common:

SharePoint / OneDrive (Graph sites/{id}/drive/root/delta; scopes Sites.Read.All, Files.Read.All; files → docling/tabular by extension; ACL via permissions; author/owner from createdBy/lastModifiedBy aad_oid).
Exchange / Outlook (Graph messages/delta per mailbox allowlist; MIME → email projection per the email-projection spec; sender/recipients as principals; attachments fan out as child items).
Slack content (conversations.history cursors per allowlisted channel; thread-aware batching into conversation documents; files via files.list; authors by slack_user principal, resolved through verified aliases).
Salesforce (REST updated/deleted endpoints per allowlisted object; records via format=json with the object describe as shape hint; OwnerId → sf_user principal via verified-email match; Notes/Files through docling).
Google Drive (changes API), thereafter by demand.

Fixtures: recorded API responses per connector; no live tenants in CI. The pilot runbook's "export scripts" path is retired connector by connector.

11. Migration Path¶

Greenfield; additive API surface only (deploy core before connectors). Content previously ingested manually coexists: if a connector later syncs the same bytes under a new source identity, content-hash dedup prevents node duplication at the version level, and the manual source can be tombstoned at the operator's pace. Forward compatibility: new source_meta fields are additive; new principal kinds extend the enum.

12. Success Metrics¶

connector_meta_completeness: for a fixture item exposing author/owner/ACL/timestamps, the resulting graph has all three edges, acl_json, and correct temporal properties (per-connector parametrised test).
connector_authority_routing: a clarification raised on a synced document ranks the document's LAST_MODIFIED_BY person first (end-to-end with the clarification spec's fixture).
connector_acl_floor: a person present in Telha RBAC but absent from the item ACL is skipped by the clarification gate's source-side check; the reverse (in ACL, not in RBAC) is also skipped: floor-only in both directions.
connector_dedup_stability: two full syncs of an unchanged corpus create zero new source versions; a modified item creates exactly one.
connector_deletion: a delta-reported delete tombstones the projection; as-of queries before the delete still resolve it.
connector_fault_isolation: a poisoned item in a 100-item window yields 99 processed, 1 audited failure, cursor advanced.
connector_readonly: adapter test asserts no non-GET verbs reach any source API (recorded-traffic assertion).
connector_delete_namespace_scope: a SharePoint connector token tombstones its own synced records; the same token attempting to tombstone Slack-synced, Exchange-synced, uploaded, or manually created records gets 403 CONNECTOR_NAMESPACE_MISMATCH; the failed attempt emits a security audit event.

13. Open Questions¶

~~OQ-1: Expand group ACL entries at sync or resolve at gate time?~~ Ratified 2026-07-04: store the group ref, resolve via MEMBER_OF at gate time (the Entra connector already syncs membership).
~~OQ-2: Slack batching — rolling conversation documents or per-thread items?~~ Ratified 2026-07-04: per-thread with a daily rollup ceiling; revisit with volume data.
~~OQ-3: Shared DOCUMENT node for repeated Exchange attachments?~~ Ratified 2026-07-04: yes, share by content hash; small ingestion-provenance addendum lands with the Exchange connector (PR-067).
~~OQ-4: Webhook acceleration over the cursor chain?~~ Ratified 2026-07-04: later; correctness must never depend on webhook delivery.

14. Changelog¶

2026-07-05 — v0.8: as-built record for PR-068 (Slack content connector). workers/slack_connector/ per §10.3 + OQ-2 (per-thread documents, daily rollup ceiling): payload {v, cursors: {channelId: {latest}}, connectionId} (per-channel cursor = newest fully processed history ts; oldest= exclusive paging). PINNED ROUTE: one thread (parent + replies via conversations.replies; standalone = thread of one) → ONE format="json" ingest through the core json bypass with options.label = "SLACK_THREAD", options.nested = "child" and record {channel, threadTs, messages: [{author, authorEmail?, ts, text}]} — the messages array projects as HAS_MESSAGES children with ord. externalId channel:thread_ts (files channel:file:id, rollups channel:rollup:YYYY-MM-DD); a thread gaining replies re-ingests whole (content-hash dedup handles unchanged threads). Author = email principal ONLY when users.info exposes a profile email (placeholder authorship evidence — never a slack alias, never binding; slack_user principals are never sent); acl OMITTED v1 (channel membership churns; recorded stance); NO delete propagation v1 (polling surfaces no delete delta; Events-API webhooks per OQ-4 later). Files from message files facets via url_private (thread-scoped; no files.list sweep), routed by extension w/ oversize/unsupported skips. Rollup ceiling is per-run per UTC day (payload carries no day counters; recorded refinement). Bot token rides the Authorization header — no token POST exists, so the read-only gate holds with no exemption. 30 tests; worker suite 365 green. COORDINATION NOTE: a concurrent session produced a competing PR-068 variant (delete propagation via a lookback window, different record shape); THIS record describes the landed spec-pinned implementation — reconcile at the operator level if the variant is preferred.
2026-07-05 — v0.7: as-built record for PR-069 (Salesforce connector). workers/salesforce_connector/ per §10.4: sync:salesforce chains with payload {v, windows: {sobject: {since}}, connectionId} — the cursor is a per-object WINDOW END (the updated/deleted REST feeds take start/end; each run replays [since, now] and core dedups the overlap by hash). Seed OR a cursor beyond the feeds' 30-day reach = invalidated → in-place FULL RESYNC via paged SOQL id enumeration (SELECT Id FROM {obj}), which doubles as initial backfill; recorded limitation: a full resync carries no delete feed, so deletions during a stale-cursor gap are not tombstoned. Records ride format=json: describe-shaped scalar content in describe field order, date/datetime values → µs ints per the describe types, base64/compound (address/location) fields excluded, system noise (audit stamps + principal-id fields) held OUT of the content so audit-only record touches re-submit identical bytes (§3.5 dedup no-ops); options.label = UPPER_SNAKE object API name, nesting left at core default (flat content). Identity: externalId = the 18-char org-unique record Id, containerPath = object API name, sourceUrl = {instance}/lightning/r/{obj}/{id}/view. Principals: OwnerId/ CreatedById/LastModifiedById resolve to {kind: "email"} via SELECT Email FROM User (per-run cache, one SOQL per distinct id) — NO raw sf_user principals ride (core skips them, v0.4 item 4); unresolved slots (queue owners, email-less Users) are omitted + counted (principals_unresolved), raw ids always ride sourceMeta.extra. ACL: NONE in v1 — record-level Salesforce sharing is not cheaply exportable; Telha RBAC is the only gate (floor-only posture unaffected). Notes/Files: opt-in by allowlisting ContentDocument in SF_OBJECTS (refinement over the ContentVersion/ContentDocumentLink sketch: the ContentDocument id is the stable identity AND its deleted feed emits that same id, so deletion propagates; ContentVersion ids churn per upload); bytes from the latest published version's VersionData, routed pdf/docx/pptx → docling, csv/xlsx → tabular, others skip+stat, oversize skip+stat before download; no linked-entity scoping in v1 (recorded limitation). Auth: OAuth client-credentials against {instance}/services/oauth2/token (connected app run-as integration user); invalid_grant-class → CONNECTOR_AUTH_FAILED permanent dead-letter. Throttle: 429 AND Salesforce's 403 REQUEST_LIMIT_EXCEEDED honored via Retry-After (client in-place + engine budget) then retryable. Deletes via lookup_record + delete_record (unknown → skip+stat); item-level fault isolation incl. blown User lookups; read-only gate asserted over recorded traffic (token POST exempt). REST API version pinned v62.0. 31 connector tests.
2026-07-05 — v0.6: as-built record for PR-067 (Exchange/Outlook connector). workers/exchange_connector/ per §10.2: Graph /users/{upn}/messages/delta chains per mailbox allowlist (EXCHANGE_MAILBOXES; payload {v, deltaLink: {mailbox: link}, connectionId} — a per-MAILBOX cursor dict, the same multi-stream refinement as v0.5's per-site dict). Every message ingests as its raw MIME via /messages/{id}/$value → format="email" (the PR-055 email projection parses; attachments fan out projection-side — the connector never fetches attachments separately). Prefer: IdType="ImmutableId" on every Graph GET so externalId survives folder moves (default Exchange ids do not). sourceMeta: externalId = immutable message id; containerPath = {upn}/{subject-or-id} (messages/delta exposes only the opaque parentFolderId — folder display names are not cheaply available; recorded v1 refinement); sourceUrl = webLink; createdAt = receivedDateTime (createdDateTime fallback for drafts) and modifiedAt = lastModifiedDateTime as µs; author = from SMTP address as an email principal (sender fallback for delegate/send-as mail); owner/lastModifiedBy omitted (not exposed by messages); acl = the mailbox-owner email principal (the mailbox IS the container; recipients arrive as TO/CC edges via the projection and are never duplicated into acl); options.mailboxNs rides to the email worker (stat-only breadcrumb). Oversize is enforced POST-fetch (messages expose no size facet) as skip+stat; @removed deletes via lookup_record + delete_record (unknown → skip+stat); 410 → in-run full resync; Retry-After honored; CONNECTOR_AUTH_FAILED permanent dead-letter; read-only gate asserted over recorded traffic AND at the HTTP layer (token POST sole exemption, immutable-id Prefer header pinned). OQ-3 landed with this PR: the email worker's attachment child names are now content-hash-derived (attachment:{blake3_hex}), so core's (name, content-hash) child dedup shares ONE DOCUMENT per distinct attachment bytes across messages and mailboxes — ingestion-provenance spec v0.5 (§18 addendum) + email-projection v0.4 record the mechanics; pinned by test_exchange_attachment_shared_document.py. 25 connector tests; email worker 57 green.
2026-07-05 — v0.5: as-built record for PR-066 (SharePoint/OneDrive connector). workers/sharepoint_connector/ per §10.1: Graph drive delta chains (payload {v, deltaLink: {siteId: link}, connectionId} — the cursor is a PER-SITE dict, a multi-site refinement over the single-link sketch), format routing by extension (pdf/docx/pptx → docling, csv/xlsx → tabular, json → bypass; others skip+stat), full sourceMeta envelopes (author/lastModifiedBy as aad_oid principals, owner from the shared.owner facet when exposed, ACL from permissions — group grants stored as the group aad_oid per OQ-1, email grants as email principals), deletes via the SDK's new lookup_record(externalId, connector) + delete_record (unknown item → skip+stat), item-level fault isolation, 410 → in-run full resync, Retry-After honored, CONNECTOR_AUTH_FAILED permanent dead-letter, read-only gate asserted over recorded traffic (token POST exempt). Python SDK additive: ingest_text/ingest_bytes gained options, lookup_record added. Refinements: oversize items skip+stat (skipped_oversize) per §8 caps rather than counting as §3.9 failures; granular skip stats (folders/unsupported/oversize/ missing) feed §7's "top skip reasons"; SP_SITE_IDS comma allowlist cannot carry Graph composite ids (recorded v1 limitation). 24 connector tests; worker suite 278 green.
2026-07-05 — v0.4: as-built record for PR-065 (core framework; status stays Approved). Deviations and refinements: (1) DTO casing: the envelope rides options.sourceMeta with camelCase inner keys (externalId, containerPath, sourceUrl, createdAt, modifiedAt, lastModifiedBy; §5's snake_case is conceptual — same convention note as temporal-clarification v0.7). The envelope also carries a REQUIRED connector field: the {connector} leg of the §3.5 identity cannot come from the API-key-authenticated ingest call. (2) Connector tokens are a THIRD token shape ({v, tid, oid, wid, iat, aud}, audience "connector"): tenant-scoped AND named — job leasing is tenant-scoped (lease_next, not the cross-tenant worker scan) and the name is the delete namespace. Worker-shaped tokens fail its decode (fail-safe); audience disjointness rejects it everywhere else. Minted via telha api-key connector-token --connector --tenant --org. (3) Chain wire: sync completion rides the EXISTING SubmitIngestionResult RPC (no proto change) with a discriminated sync-result JSON {v: 1, sync: true, nextPayload?, delaySecs?, stats?}; core completes the window and enqueues the successor (same kind, not_before = now + delaySecs). Kind↔audience fence enforced both directions (sync results need connector audience; ingest submissions now require worker audience). Chains are seeded/inspected via telha connector run|ls|status (one live chain step per source enforced at seed). (4) Principal resolution at apply time covers aad_oid (canonical id uuid5(NAMESPACE_URL, "person:"+oid), stub props {aad_oid}) and email (placeholder id per the PR-055 pin, props {address} ONLY so upserts against email-worker persons never churn); slack_user/ sf_user principals are skipped with a log line — their connectors resolve them to emails first (Slack via the clarify-bot verified-match path). Principal edges + root properties land on the FIRST submission (root = its node 0) and BEFORE clarification opening, so authority ranking sees them; the same envelope on later stream partials is ignored. (5) Source version row gained additive fields source_connector, external_id, acl_json, root_node_id (msgpack skip-if-none); the row is re-landed with the root id once nodes exist (streamed chains carry it in the continuation). GET /v1/records?externalId= resolves connector→source→root as a point lookup (namespace inferred from a connector token; explicit &connector= for API keys); the DELETE namespace check reads the root node's source_connector property — one get_current. (6) REST records/ingest/relationships routes accept connector bearer tokens alongside API keys (new require_api_key_or_connector_token middleware); API keys remain unrestricted tenant-admin credentials. RecordInput gained upsert: true (with id) exposing the PR-053 upsert seam over REST — connector identity syncs replay delta windows idempotently (create / identical→no-write / different→new version / tombstoned→409, never resurrected). (7) Per-connection [connectors.*] config lives with the CONNECTOR process (env), not in core config — core's surface is generic (seed CLI + chain semantics); §8's table is the connector-side contract. Tests: connector_meta_completeness, connector_authority_routing, connector_dedup_stability, connector_delete_namespace_scope, connector_external_id_lookup (tests/connector_e2e.rs) + connector_fence_and_sync_chain (worker_lifecycle). Deferred to the connector PRs: connector_acl_floor (gate-side check is PR-063's delivery leg), connector_deletion delta-feed leg, connector_readonly
connector_fault_isolation (per-connector adapter tests; the Entra connector ships its own).
2026-07-04 — v0.3: Approved (cross-spec review by J. Porter). Condition incorporated: connector delete authority namespace-scoped (403 CONNECTOR_NAMESPACE_MISMATCH + security audit; connector tokens are not tenant-wide delete credentials); test connector_delete_namespace_scope added. OQ-1..4 stances ratified.
2026-07-04 — v0.2: PR numbers filled in from build doc v2.1 Phase 3b (PR-065 core, PR-066..069 connectors); no design changes.
2026-07-04 — v0.1: initial draft for peer review.