Multi-user MCP control plane. Six phases (A–F) and migrations 0011→0025 turn fluidbox from a single-admin control plane into one that can host many organizations, many users, and many separately-owned credentials without ever letting a model pick an identity. Every hosted capability is opt-in behind a flag, and the default single-admin Docker deployment is byte-for-byte the same product — FLUIDBOX_REQUIRE_SSO unset means today's behavior, unchanged.
The organizing idea: connector definition ≠ credential-bearing connection ≠ agent connection requirement ≠ per-run resource binding. An agent declares what it requires, never whose credential satisfies it. Run creation resolves each requirement to an explicit, frozen authority source before any model spend. The model picks tools; it can never pick an identity.
Highlights:
- Per-organization, IdP-agnostic identity —
FLUIDBOX_REQUIRE_SSO=1 confines the admin token to /v1/admin/* as break-glass and introduces three principals: Operator (admin token), User (__Host-fbx_web session cookie), and Pat (fbx_pat_ bearer). Any conformant OIDC issuer is configured per org (issuer + client + sealed secret + claim mappings); logins are two-phase and browser-bound; sessions are server-side with idle/absolute/re-auth windows. No IdP configured ⇒ single-admin mode. - Tenant isolation with a database floor — every tenant-owned
fluidbox-db method now takes a TenantScope that carries its id into a tenant_id = $n predicate, so isolation is a signature requirement rather than a remember-to-filter convention. Migration 0018 adds the floor underneath: 37 tables ENABLE+FORCE row-level security keyed on a transaction-local fluidbox.tenant_id GUC, with FLUIDBOX_RUNTIME_ROLE=fluidbox_runtime splitting the pool onto a non-owner role holding enumerated per-table grants. Cross-tenant access exists only through a short, named, grep-able set of audited bypasses. - Connection ownership and per-run resource bindings — brokered MCP tools moved off capability bundles onto four objects: catalog connector definition → connection (owns the credential, plus append-only tool snapshots) → agent-revision **
connection_requirements → per-run run_resource_bindings** (migration 0013), resolved to a tagged authority (connection | subscription_secret | none) across typed slots (mcp | workspace_fetch | result_publish) before provisioning. Connections gained personal vs. organization ownership; a personal-connection approval is decidable only by its owner-who-invoked — no role, admin, or operator override. - Versioned envelope sealing with a real key-retirement path — migration
0014 makes every sealed column carry a _key_version companion: 1 is the legacy FLUIDBOX_CREDENTIAL_KEY format, 2 is a per-tenant DEK wrapped by a KEK (FLUIDBOX_KMS_MODE=off|static|aws) with AAD binding fbx:v2:{tenant}:{table.column} so a blob is untransplantable across tenants or columns. Thirteen sealed families; a resumable, CAS-guarded POST /v1/admin/reseal migrates v1→v2; two boot gates fail closed in both directions. Runbook: docs/hosted/kms-operations.md. - One hardened egress boundary for all control-plane traffic — two filtering-resolver clients plus a pure
admit_url pre-flight that blocks private/loopback/link-local/multicast/reserved and cloud-metadata address classes at every dial site (reqwest dials an IP literal without consulting a resolver, so the pre-flight is what actually stops 169.254.169.254). Broker, delivery callbacks, and both connector-OAuth token legs ride a client that refuses redirects outright. Git gets its own out-of-process policy. FLUIDBOX_EGRESS_ALLOW_CIDRS opts specific CIDRs back in; FLUIDBOX_EGRESS_PROXY re-points everything, including the git subprocess, through one waypoint. - **MCP
2025-11-25 conformance, with version drift denying the call** — upstream MCP is now a per-run session (initialize + notifications/initialized before every call, MCP-Protocol-Version on every request, credential re-resolved live on the terminal DELETE). A run's negotiated version must match its frozen surface exactly or the call is denied with a message naming the refresh endpoint. SSE is a real incremental assembler with per-event and total ceilings; outputSchema/structuredContent are preserved. - Frozen tool schemas enforced server-side — arguments are validated against the schema photographed at freeze time, with the JSON Schema dialect chosen by the snapshot's protocol version (
2025-11-25 ⇒ 2020-12 per SEP-1613, otherwise draft-07). The schema is untrusted input, so it is pre-guarded (size, depth, local-$ref-only) before compilation; a violation makes the tool un-callable rather than being silently ignored. This inserts exactly one new stage into the permission gate and moves nothing else. - At-most-once brokered dispatch — migration
0019 wraps every brokered call in a durable four-state execution claim keyed (session, tool_call_id, input_digest). failed_before_send requires positive proof nothing was written and is the only re-claimable state; a definitive upstream response is terminal; timeouts and mid-stream failures are recorded as ambiguous rather than retried. Decision idempotency and execution idempotency are now distinct properties. - Audience-scoped sandbox credentials — migration
0020 splits the sandbox's single bearer into four tokens (llm | tool | control | workspace), each checked as the first statement of its handler. Kubernetes ships one Secret with four keys routed per container, so the workspace init container never sees the others. - Replica coordination primitives — migration
0021: approval emission rides the deciding CAS inside one transaction (only the winner emits) with cross-replica pg_notify wakeups and the poll floor kept as a missed-notify backstop; sessions carry an orchestrator lease + epoch so a fenced-out driver cannot mutate lifecycle while a user's cancel stays deliberately unfenced; deliveries claim rows FOR UPDATE SKIP LOCKED, and the GitHub double-post window closes by reconcile-before-create on both comments and checks. - Durable LLM budget admission — migration
0022 replaces best-effort budget checks with a request-keyed reservation whose primary key is the usage entry's external id, which is what makes a 401 replay and a late drain idempotent. Booking uses a deliberately-high upper bound; release happens only on positively-proven non-dispatch; charging requires a durable usage write before the CAS. - Operations — a bounded-cardinality metrics registry at admin-gated
GET /v1/admin/metrics (plus optional unauthenticated FLUIDBOX_METRICS_BIND), durable cross-replica egress governance and capacity ceilings (0023), cross-replica MCP session teardown (0024), workload identity (0025), S3-compatible archive storage alongside the filesystem backend, and a guarded load harness (fluidbox-loadgen) with its own manual scale CI job.
Validation: five hermetic acceptance suites green against CI-identical throwaway databases — identity 87/0, bindings 104/0, secrets 128/0, hardening 274/0, scale 18/0 = 611/0 — plus live Docker-provider tiers (demo A, Codex) and a second live EKS acceptance on arm64/Graviton with the runtime-role RLS split active and an AWS-audited zero-orphan teardown (docs/reviews/2026-07-22-eks-acceptance-phase-f.md).
Still deferred: the gated 60/150/300-seat load campaign and the final two rollout gates (owner approval + cost estimate) remain open on #34 — real spend, tracked separately from code. The hosted OAuth Connect flow also carries one documented residual: a deliberately-shared start URL can still route a victim's grant into the initiating connection, closed only by moving the browser-facing leg onto the dashboard origin (full write-up in docs/hosted/threat-model.md).
- Identity and access — per-org OIDC login (
/v1/auth/*), logout, /v1/auth/me, PAT mint/list/revoke, org + IdP-config lifecycle and membership roles (/v1/admin/orgs*), break-glass owner arming, and staged issuer migration. All three token shapes (fbx_sess_, fbx_web_, fbx_pat_) are sha256-only at rest and scrubbed by the ledger redactor. - Dashboard SSO mode —
FLUIDBOX_WEB_MODE=admin|sso (static per deployment). In sso the proxy carries no admin token and forwards the session cookie plus a CSRF header on same-origin non-GETs; apps/web/proxy.ts redirects sessionless browsers to /login?next=… before first paint while authority stays in the control plane. - Per-tenant LLM keys —
FLUIDBOX_LLM_KEY_MODE=tenant (migration 0017) mints a per-tenant LiteLLM virtual key and confines the master key to provisioning; POST /v1/admin/orgs/{slug}/llm-key/rotate. Requires a LiteLLM backed by its own Postgres, so local deployments stay on shared. - Connection tool snapshots — a forced-
initialize photograph per connection (GET /v1/connections/{id}/tools, POST /v1/connections/{id}/tools/refresh) recording the negotiated protocol version, with cursor caps fail-closed. - Hosted operator documentation —
docs/hosted/: product compatibility matrix, threat model, network architecture, connector admission policy, rollout gates, and KMS operations runbook. Plus docs/guides/kubernetes.md, a zero-to-certified-cluster guide with real cloud acceptance costs and gotchas.
- Brokered tools no longer ride capability bundles.
capability_bundles survives for sandbox stdio tools only; registering a class:brokered server is refused with a cutover error. Migration 0013 appends converted agent revisions and repoints pinned subscriptions; a revision still pinning a brokered bundle is refused at run creation. Mixed brokered+sandbox bundles drop whole, with a raise-notice. - Custom connector-catalog entries are tenant-scoped. Curated and imported entries stay deployment-global; a tenant's custom entry shadows a same-slug global one. Migration
0013 backfills the single boot tenant, otherwise disabling the row. - Connector OAuth is a one-time, browser-bound flow.
oauth/start returns only a go_url; navigating it sets a __Host- flow cookie whose hash sits inside the atomic single-use claim, so a leaked authorization URL can neither complete nor burn a flow. Endpoints, resolved client, resource, sealed PKCE verifier, and expected generation are frozen at start and the callback exchanges against that row. Client identities are shared per (issuer, redirect_uri), DCR singleflighted by advisory lock. The stateless seal_state/open_state helpers are gone. - **
authorization_generation bumps on reconnect of ever-activated OAuth connections**, so stale-generation bindings refuse mid-run. Rotation within a generation is unaffected, and GitHub App lifecycle never bumps. - Multi-user boot now refuses a pool role that bypasses RLS (
SUPERUSER/BYPASSRLS, e.g. Neon's default owner) unless FLUIDBOX_ALLOW_RLS_BYPASS=1; single-user only warns. just doctor inspects the role the server will actually run as and fails on every unbootable combination.
- Prompts still never reach the ledger, and the redactor now also scrubs every session, web-session, and PAT token shape.
- The permission gate grew exactly one stage (frozen-schema argument validation) and reordered nothing: budget → frozen-set availability → schema → trust tier → policy → approvals.
- Before any brokered secret access, binding status,
authorization_generation, and — for personal connections — owner-membership-active are re-verified fail-closed. - Both connector-OAuth token legs ride the no-redirect client on purpose: a 307/308 replays the request body, which would forward an authorization code plus PKCE verifier, or a refresh token, to the redirect target. A source-grep test pins this.
Migration 0018 (RLS enforcement) is stop the old binary, migrate, then deploy — not a rolling upgrade. A pre-0018 binary sets no tenant GUC and would therefore see zero rows, and it holds transactions across outbound HTTP that would block the migration's ACCESS EXCLUSIVE locks.
Do not drop FLUIDBOX_CREDENTIAL_KEY when enabling FLUIDBOX_KMS_MODE: run POST /v1/admin/reseal and let boot prove zero remaining v1 rows first. From the moment any v2 row exists, the KEK is the root of custody and losing it is unrecoverable — back it up before enabling.