Connector framework¶

Architecture

Every connector, Entra, SharePoint, Exchange, Slack, Salesforce, pulls from a source system on a delta cursor and lands what it finds through the same ingest path everything else uses, carrying enough metadata that authorship, ownership, ACLs, and deletions all keep working without a bespoke integration per source.

Overview¶

A connector's job is narrow by design: discover, fetch, and describe. Parsing, projection, dedup, and graph writes all stay in the code paths that already exist for manually ingested content. The source-connector-framework spec states the normative contract every content connector implements (spec §3):

1. Connectors are read-only against the source. They never write, move, tag, or annotate source content.
2. Each runs as a sync:{source} job chain: a run leases one job whose payload carries the delta cursor, processes a window, and enqueues the successor with the advanced cursor. The job chain is the connector's state; there is no separate registry.
3. Every discovered item goes through the existing ingest path (POST /v1/ingest, format worker parses) carrying a sourceMeta envelope.
4. Principals (author, owner, last-modified-by, ACL entries) resolve through the identity model, never invented locally.
5. Mirrored ACLs are advisory and floor-only.
6. Deletions observed in a source delta feed propagate as tombstones.
7. A failing item is skipped and audited; it never aborts the run.

The Entra identity connector is architecturally distinct from the content connectors even though it shares the same runtime harness and job-chain pattern: it writes PERSON/GROUP identity directly through the SDK records API rather than through the ingest/parsing path, because identity has no document to span (framework spec §9). Its own spec, 2026-07-04-entra-identity-connector.md, is what makes Entra the canonical identity source the other connectors depend on (see Entra as the identity substrate below).

For deploying and configuring a connector, day-2 operator commands, and the per-connector app-registration steps, see the operator how-to. This page covers the shared abstraction, the wire shapes, and the invariants underneath it.

Design¶

The connector abstraction¶

There is no Connector abstract base class in workers_common. The shared abstraction is the harness contract that workers_common.run and WorkerConsumer (workers/workers_common/workers_common/consumer.py, run.py) impose on any worker package, connectors included:

• A module <package>.worker exposing FORMATS: list[str] and a callable handle(job: Job) -> bytes | None (or a generator of submission dicts, for streaming workers; connectors are single-shot and return bytes).
• Job (workers_common.consumer.Job) is the leased-job dataclass every handler receives: job_id, tenant_id, organization_id, kind, payload (the dict cursor), lease_deadline.
• python -m workers_common.run <package> resolves that module, starts a /healthz endpoint, and runs WorkerConsumer (poll → dispatch → heartbeat → submit) until SIGINT/SIGTERM.

Every connector package (entra_connector, sharepoint_connector, exchange_connector, slack_connector, salesforce_connector) implements this contract identically:

# entra_connector/worker.py
FORMATS = ["entra"]

def handle(job: Job) -> bytes:
    """Run one sync window and return the sync-result JSON bytes."""
    config = ConnectorConfig.from_env()
    graph, writer, close = _runtime(config)
    try:
        result = run_sync(job.payload, graph, writer, config)
    finally:
        close()
    return json.dumps(result, ensure_ascii=False).encode("utf-8")

Inside that shared shape, each package supplies its own four building blocks: a config.py (ConnectorConfig.from_env() / .require_runtime(), fails loudly at startup if a required env var is missing), a source-API client (graph.py for the three Graph-backed connectors, a REST/OAuth client for Slack and Salesforce), a sync.py (run_sync(payload, client, writer, config), the per-window delta walk and item processing), and, for content connectors, a meta.py that builds the sourceMeta envelope from the source's native item shape. sync.py is where the framework's real contract lives: the delta-walk loop, throttle/backoff, item-level fault isolation, and the pinned sync-result JSON are structurally the same function across every connector, reimplemented per source rather than inherited from a base class.

The sync-result contract¶

A connector's handle does not call any "complete the job" RPC directly. It returns a discriminated JSON result over the existing SubmitIngestionResult RPC (no proto change, framework spec v0.4 item 3):

{
  "v": 1,
  "sync": true,
  "nextPayload": { "...": "connector-owned cursor shape" },
  "delaySecs": 1800,
  "stats": { "ingested": 42, "deleted": 1, "failed": 0 }
}

Core recognizes sync: true, completes the current job, and enqueues the successor of the same kind with not_before = now + delaySecs and payload nextPayload. This is the entirety of "connector state": there is no row anywhere that says "SharePoint site X is at delta token Y" outside of that next job's payload.

Entra as the identity substrate¶

The Entra connector is not just one connector among five, it is a dependency the other four content connectors have on person identity, and the docs and specs both call this out structurally rather than as an afterthought.

SharePoint's ACL principals, Exchange's mailbox owner, and Slack's message authors all arrive as references (an aad_oid, an email address, a Slack user id), not as resolved people. Something has to be the canonical place those references become an actual PERSON with a name, a manager, and group memberships that RBAC and clarification routing can reason about. The Entra identity connector spec (§3.1) states this plainly: "Entra is canonical. A PERSON's binding identity comes from the corporate IdP; source systems contribute authorship/ownership facts; chat surfaces contribute delivery aliases only." Concretely:

• A SharePoint file's permissions facet carries group grants as the group's AAD object id (spec OQ-1, ratified): the group itself, and who is a member of it, is Entra's GROUP node and MEMBER_OF edges, not something SharePoint's connector can resolve on its own.
• Content connectors that only see an author's email create a placeholder PERSON (uuidv5(NAMESPACE_URL, "person-email:" + lowercase(email))). When Entra later syncs the real person, the placeholder is merged via an ALIAS_OF edge, not overwritten, append-only, and existing AUTHORED_BY/OWNED_BY references keep resolving through it (Entra spec §3.6, §5).
• Slack identities are weaker still: a Slack user id is never sent as a principal at all (only an email, when the Slack profile exposes one), and even that email never creates binding authority on its own. Turning a Slack identity into something that can be trusted for clarification routing requires a separate, audited verified-match step (POST /v1/person-aliases/verified-match, clarify-audience only, exact-one-match against Entra-verified aliases) performed lazily by the clarify bot, never by the Entra connector reaching out to Slack.

This is why the operator runbook says to run the Entra connector before the other connectors and before the clarify bot: it is the authority substrate clarification routing and the RBAC-adjacent ACL check both depend on. A content connector can run first and will not fail, but its authorship edges resolve to placeholders until Entra sync catches up.

flowchart TB
    ENTRA["Entra connector<br/>PERSON / GROUP · MEMBER_OF · REPORTS_TO"]
    SP["SharePoint connector"]
    EX["Exchange connector"]
    SL["Slack connector"]
    SF["Salesforce connector"]
    CLAR["Clarification routing<br/>+ RBAC-adjacent ACL gate"]

    SP -->|"aad_oid principals resolve against"| ENTRA
    EX -->|"aad_oid / email principals resolve against"| ENTRA
    SL -->|"email principals; Slack user id needs<br/>verified-match to attach"| ENTRA
    SF -->|"email principals (OwnerId → email)"| ENTRA
    ENTRA -->|"canonical PERSON + MEMBER_OF"| CLAR
    SP -.->|"placeholder person until merged"| CLAR

Data model¶

Connectors covered¶

The sourceMeta envelope¶

Every synced item carries options.sourceMeta on the ingest call. The spec's prose is snake_case (§5); the as-built DTO rides camelCase, pinned by the framework spec's v0.4 changelog entry, and is implemented in core-engine/src/ingest/source_meta.rs:

{
  "connector": "sharepoint",
  "externalId": "01ABCDEF...",
  "containerPath": "Shared Documents/Contracts/msa.pdf",
  "sourceUrl": "https://contoso.sharepoint.com/...",
  "createdAt": 1751328000000000,
  "modifiedAt": 1751414400000000,
  "author": { "kind": "aad_oid", "value": "11111111-2222-3333-4444-555555555555" },
  "lastModifiedBy": { "kind": "aad_oid", "value": "22222222-3333-4444-5555-666666666666" },
  "owner": { "kind": "email", "value": "finance-team@contoso.com" },
  "acl": [
    { "kind": "aad_oid", "value": "33333333-4444-5555-6666-777777777777" }
  ],
  "extra": {}
}

connector and externalId are required; every other field is emitted only when the source exposes it (omitting an available field is treated as a connector bug, spec §3.4). principal = {kind: aad_oid|email|slack_user|sf_user, value}; timestamps are µs since epoch. At apply time core:

• Derives the deterministic source id uuidv5(NAMESPACE_URL, "{connector}:{tenant}:{external_id}") (connector_source_id in source_meta.rs), so re-syncing an unchanged item is a no-op via content-hash dedup and a changed item creates a new source version.
• Attaches external_id, source_connector, source_url, container_path, source_created_at, source_modified_at as properties on the projection root (SourceMeta::root_properties).
• Resolves author/lastModifiedBy/owner into PERSON logical ids (aad_oid → canonical, email → placeholder; slack_user/sf_user are skipped at apply time with a log line, since those connectors resolve them to emails before the envelope is ever sent) and writes AUTHORED_BY / LAST_MODIFIED_BY / OWNED_BY edges from the projection root.
• Serializes acl to acl_json and stores it on the source version row, not on the nodes: the version pins what the ACL was at sync time, the same way every other temporally-scoped fact is pinned.

Algorithms & invariants¶

Sync loop¶

Every connector's sync.py follows the same shape, illustrated here against the SharePoint connector's per-site drive delta (sharepoint_connector/sync.py):

flowchart TB
    START["lease sync:{source} job<br/>payload = saved cursor(s)"] --> FETCH["fetch one delta page<br/>(honor Retry-After on 429)"]
    FETCH -->|"410 Gone"| RESET{"already reset<br/>this run?"}
    RESET -->|no| FULL["drop cursor, restart<br/>from a full listing"]
    RESET -->|yes| FAIL["raise Transient<br/>(retryable)"]
    FULL --> FETCH
    FETCH -->|page ok| ITEM["for each item in page"]
    ITEM --> KIND{"item kind"}
    KIND -->|"deleted facet"| DEL["lookup_record(externalId, connector)<br/>found → delete_record(rootId)<br/>not found → skip+stat"]
    KIND -->|"unsupported/oversize/folder"| SKIP["skip + stat<br/>(never an error)"]
    KIND -->|content item| ROUTE["fetch bytes + ACL,<br/>build sourceMeta,<br/>POST /v1/ingest"]
    ROUTE -->|raises| ITEMFAIL["catch, stats.failed += 1,<br/>audit log, continue"]
    DEL --> NEXT{"more pages?"}
    SKIP --> NEXT
    ROUTE --> NEXT
    ITEMFAIL --> NEXT
    NEXT -->|"@odata.nextLink"| FETCH
    NEXT -->|done| CURSOR["new delta link"]
    CURSOR --> RESULT["return {sync:true, nextPayload,<br/>delaySecs, stats}"]
    RESULT --> SUBMIT["SubmitIngestionResult"]
    SUBMIT --> ENQUEUE["core enqueues successor<br/>not_before = now + delaySecs"]

Cursor shapes are connector-owned and per-source, but all follow the same "the job chain is the checkpoint" rule (framework spec §3.2):

A 410/expired cursor triggers an in-place full resync of that resource, never treated as an error (Entra spec §5; framework spec §5). Salesforce's variant is a stale-cursor case rather than an expiry: a window older than the feeds' 30-day reach invalidates and falls back to paged SOQL id enumeration, which doubles as the initial backfill.

ACL and deletion propagation¶

ACLs are floor-only, never enforcement. Mirrored source ACLs (sourceMeta.acl, stored as acl_json on the source version) may only narrow who is shown content in the clarification gate's source-side check; they can never widen access beyond Telha workspace RBAC. Both directions hold: a person present in Telha RBAC but absent from the item ACL is skipped for that item, and a person present in the ACL but absent from RBAC is also skipped. Telha RBAC remains the single enforcement point; ACLs go stale between syncs by design, which is the tradeoff the framework spec (§9) accepts explicitly rather than trying to keep source ACLs live. See Tenancy & security for the RBAC layer this sits on top of.

Deletions propagate through the standard tombstone cascade, not a connector-specific mechanism. When a source delta feed reports an item gone (SharePoint's deleted facet, Exchange's @removed, Salesforce's deleted feed), the connector resolves the item's projection root via GET /v1/records?externalId=&connector= (lookup_record in the Python SDK) and calls DELETE /v1/records/:id, which tombstones the root node (and cascades to its live edges) through the same tombstone cascade every other delete in Telha uses: a new version whose valid time opens at the delete instant, history reconstructable at prior transaction times, nothing physically erased. A 404 on the lookup (never synced, or already gone) is a skip+stat, not an error.

Not every connector propagates deletes, and the framework spec (§14, PR-068 changelog) records this as a deliberate per-source stance rather than a bug:

Delete authority is namespace-scoped. A connector-audience token may tombstone only records whose source_connector property matches its own connector name. Core checks this on every connector-token DELETE /v1/records/:id (core-engine/src/api/v1.rs) and rejects a mismatch with 403 CONNECTOR_NAMESPACE_MISMATCH plus a security_audit log event. A compromised SharePoint connector can therefore never tombstone Slack-synced, Exchange-synced, uploaded, or manually created records. API keys are unrestricted tenant-admin credentials and are not narrowed by this check. See Tenancy & security for how this fits the broader audience-fencing model.

Invariants¶

Configuration¶

Per-connection configuration is connector-process environment, not core TOML: core's surface stays generic (the seed CLI plus chain semantics), and [connectors.*] scope, credentials, and intervals live with whichever process hosts the connector (framework spec §8, item 7). Deliberately not configurable, per connector: read-only posture, the ACL floor-only rule, deterministic source identity, item-level fault isolation, deletion propagation.

Common harness environment (workers_common):

Per-connector credentials and scope (exact env var names, from each package's config.py):

Every oversized-item cap (SHAREPOINT_MAX_ITEM_MB, EXCHANGE_MAX_ITEM_MB, SLACK_MAX_ITEM_MB, SALESFORCE_MAX_ITEM_MB) defaults to 50 and is enforced as a skip+stat, not a failure. Full field-by-field detail, plus the Azure app-registration and Salesforce connected-app steps each connector needs, is in the operator how-to.

As-built notes¶

Both specs stay Approved; their §14 changelogs record deviations found while building, not design reversals.

Source-connector-framework (.ai/specs/integration/2026-07-04-source-connector-framework.md, v0.4-v0.8):

• The envelope rides options.sourceMeta with camelCase inner keys; the spec's snake_case (§5) is conceptual only.
• Connector tokens are a third token shape (audience "connector"), tenant-scoped and named; job leasing uses lease_next, not the cross-tenant worker scan.
• The sync-result contract reuses the existing SubmitIngestionResult RPC rather than a new proto message.
• slack_user/sf_user principals never ride the wire at all: those connectors resolve them to emails before submission, so core's principal resolution only ever needs to handle aad_oid and email.
• Slack (PR-068) landed with no ACL mirroring (channel membership churns) and no delete propagation (polling surfaces no delete delta) as recorded v1 stances, plus a documented coordination note: a concurrent session produced a competing PR-068 variant with delete propagation via a lookback window; the landed, spec-pinned implementation is the one described here.
• Salesforce (PR-069) excludes base64/compound fields from ingested content and holds audit-stamp/principal-id fields out of content so that audit-only touches resubmit identical bytes (dedup no-ops); ACL mirroring is out of scope for v1 (record-level Salesforce sharing is not cheaply exportable).
• Exchange (PR-067) uses Prefer: IdType="ImmutableId" so externalId survives folder moves, and shares one DOCUMENT per distinct attachment content hash across messages and mailboxes (OQ-3, landed with this PR).
• SharePoint (PR-066) cannot carry Graph's composite site-id form (hostname,siteCollectionGuid,siteGuid) through the comma-separated SP_SITE_IDS allowlist, since the id itself contains commas; recorded as a v1 limitation requiring one Graph-addressable site id per entry.

Entra identity connector (.ai/specs/integration/2026-07-04-entra-identity-connector.md, v0.5):

• Placeholder person shape is pinned to {address} only, matching the PR-055 email worker's convention, so upserts against email-worker persons never version-churn.
• proxyAddresses entries under non-SMTP schemes (sip:, x500:) are dropped outright rather than mangled into a garbage placeholder probe.
• Group @removed membership entries are audit-logged, not written: edge tombstoning needs an edge id the connector does not hold across runs (recorded v1 limitation).
• The Slack verified-match endpoint landed clarify-audience only, with exact-one-match tenant scanning and re-verification of evidence against the person's own Entra-verified aliases.

Related¶

• The operator how-to for connectors: deploying and configuring connectors, app registrations, and day-2 CLI operations.
• Tombstone cascade: the mechanism source deletions propagate through.
• Tenancy & security: the RBAC floor ACLs sit on top of, and the audience-bound token model connector credentials use.
• SSO & authentication: identity provider integration and the Entra connector's role as the canonical identity source.
• GDPR erasure: physical erasure versus the soft tombstones described here.
• Field encryption: how classified fields in synced content are protected before core ever sees them.
• Clarifications: the authority routing that consumes AUTHORED_BY / LAST_MODIFIED_BY / OWNED_BY edges.