Skip to main content

Task Breakdown: Tenant-Scoped Job-Runtime Overlay

Feature ID: tenant-job-runtime-overlay Status: In progress — P0 + P1.0 + P1 + P2.0 + P2.1 + P3 (T20+T23+T24 resolver) + P3.1 (T21 cache + T22 stamper helper + T22 per-dispatcher wiring across all 10 in-platform dispatchers, with worker-host TenantRuntimeBindingResolverService consumption) + P3.2 (resolver bindToTenant wiring + SecretStoreResolver contract + InProcessSecretStoreResolver default + Trigger.dev bindToTenant minimal impl + non-inline: SecretStoreResolver plugin packages: Vault / K8s / Infisical / Doppler / AWS-SM / GCP-SM / Azure-KV + the 4 non-Trigger IJobRuntimeProvider plugin packages (Temporal / BullMQ / pg-boss / Inngest) with real bindToTenant memoisation AND operator-pluggable real dispatcher + worker-host factories (#1455 BullMQ, #1456 pg-boss, #1457 Temporal, #1458 Inngest)) + P4 (T31 contract — tenantId on JobEnqueueOptions) + P5 + P6 (T37 real-infra matrix complete across all 4 providers — BullMQ+pg-boss in #1515, Temporal+Inngest in #1522; cost-gated to direct main pushes in #1529; cross-provider tenant isolation resolver-level closeout in #1531) + P7 (T41+T42+T44) landed on develop (cascades through develop→stage→main where applicable, develop-only for the late-2026 batches). Platform-side Trigger.dev webhook receiver with per-tenant HMAC verify shipped in #1533 (T25 platform-side prep, receiver-side ready). Dependabot HIGH alerts patched in #1526; 28 of 33 MODERATE patched in #1532. P2.2 (schema-driven form + e2e), P4 (per-provider worker hosts T25–T30 + T31 per-provider stamping + T32 isolation tests), P5.1 (per-tenant whitelist), and P7 (docs T43) remain. T25 reframed (2026-06-21): the original "one Trigger.dev project per tenant" framing was rejected after the 10-project-per-org cap + Trigger.dev's own multi-tenant guidance surfaced. Full per-tenant Trigger.dev isolation is now achieved via the runtime-scoping pattern (single project + concurrencyKey: tenantId + externalId: tenantId + per-tenant credential resolution via the existing bindToTenant + TenantCredentialCache plumbing) across all 3 modes (inherit / byo / override); see T25 below and ./providers.md § Trigger.dev. Operator wiring docs for the 4 plugin packages live in ./providers.md § Operator wiring. Last updated: 2026-06-21 Spec: ./spec.md · Plan: ./plan.md · Providers: ./providers.md · Epic: EW-742 · Story: EW-743 · ADR: ADR-017

Ordered, granular tasks with explicit paths. Phases map to plan.md §10. Each phase is a candidate sub-PR. Mark [x] as landed. Suggested Jira child-issue grouping noted per phase (e.g. [EW-742 P1]). Builds on EW-683's instance-level job-runtime contract; nothing here re-opens ADR-017 Q1–Q5 (Temporal namespace-per-tenant, pg-boss schema-per-tenant, Inngest inherit/BYO uniform with Trigger.dev, credential rotation via graceful drain + separate force-invalidate, hybrid operator-gated platform-default shared/per-tenant/tiered).


Phase 0 — Spec-Kit (EW-743) · [EW-743 P0] ✅ Done in #1332 (cascaded #1333, #1334)

  • T1. Write ADR-017 at docs/specs/decisions/017-tenant-scoped-job-runtime-overlay.md capturing Q1–Q5 locked decisions and the inherit/BYO overlay rationale on top of EW-683.
  • T2. Write docs/specs/features/tenant-job-runtime-overlay/spec.md — behaviour-first user stories (tenant picks provider, operator gates allow-list, graceful drain on rotation, force-invalidate path) with no implementation detail.
  • T3. Write docs/specs/features/tenant-job-runtime-overlay/plan.md — architecture, data model, API surface, phased rollout, constitution reconciliation; mirrors structure of job-runtime-providers/plan.md.
  • T4. Write docs/specs/features/tenant-job-runtime-overlay/tasks.md (this file) — numbered T1..TN per phase with file paths.
  • T5. Write docs/specs/features/tenant-job-runtime-overlay/providers.md — per-provider tenant-isolation matrix (Temporal namespace, pg-boss schema, BullMQ prefix, Trigger.dev project, Inngest app) and inherit-vs-BYO knob per provider.
  • T6. Cross-link from docs/specs/architecture/job-runtime-providers.md to the tenant-overlay docs by adding a "Tenant-scoped overlay" see-also section pointing at spec.md, plan.md, and ADR-017.
  • T7. Verify constitution gates against .specify/memory/constitution.md — confirm Principle IV ("via the configured job-runtime provider") already covers tenant overlay; no new amendment required, document the verification in plan.md §Constitution Reconciliation.

Phase 1 — Data model + migration · [EW-742 P1] ✅ Done in #1338 (cascaded #1339, #1340). P1.0 (x-scope: tenant) shipped in #1335 (cascaded #1336, #1337).

  • T8. Audit docs/specs/architecture/settings-system.md for x-scope: tenant; if missing, add it to the scope taxonomy alongside x-scope: global (verify in worktree before writing — this is a prerequisite for tenant-scoped credentials).
  • T9. Define TypeORM entity TenantJobRuntimeConfig in packages/agent/src/entities/tenant-job-runtime-config.entity.ts — columns: tenantId (PK), providerId, mode (inherit | byo | override), credentialsSecretRef, credentialVersion, enabled, createdBy, createdAt, updatedAt. (Entity moved to @ever-works/agent/entities per EW-654/655 cycle-avoidance — no @ManyToOne back-refs, FK enforced only at the DB layer.)
  • T10. Write TypeORM migration apps/api/src/migrations/1780900000000-AddTenantJobRuntimeConfig.ts for the new table plus partial index on (providerId) WHERE enabled (PK already covers tenantId); forward-only + idempotent hasTable guards per docs/specs/architecture/database-migrations.md.
  • T11. Implement CredentialVersionService for graceful drain (Q4) in packages/agent/src/tasks/credential-version.service.ts — issues monotonic per-tenant version ids and resolves a snapshot for a given (tenantId, version) tuple so in-flight runs keep their original credentials. Full snapshot history table shipped in #1514: TenantCredentialSnapshot entity + migration + captureSnapshot() on every version bump + resolveSnapshot(tenantId, version) reads the row (FIFO history, idempotent on conflict). Bag re-issued as inline:<base64> so the existing secret-store resolver chain dereferences without a new scheme. 12 vitest history cases; 6091/6091 packages/agent green.
  • T12. Add audit-log table + entity for tenant runtime config changes in packages/agent/src/entities/tenant-job-runtime-audit.entity.ts plus matching migration apps/api/src/migrations/1781000000000-AddTenantJobRuntimeAudit.ts; records (tenantId, actorUserId, action, before, after, credentialVersion, occurredAt).
  • T13. Repository + unit tests under apps/api/src/works/__tests__/tenant-job-runtime-config.repository.spec.ts covering insert, update, version bump, audit-row emission, and tenant isolation (one tenant cannot read another's row).

Phase 2 — Admin UI · [EW-742 P2]

P2.0 (REST API) ✅ Done in #1341 (cascaded #1342, #1343). P2.1 (UI) ✅ Done in #1347 (cascaded #1348, #1349). P2.2 (schema-driven form + e2e) remains.

  • T14. API endpoints GET /api/account/job-runtime/config and PUT /api/account/job-runtime/config in apps/api/src/account/tenant-job-runtime/tenant-job-runtime.controller.ts — guarded by requireTenant() (1 User : 1 Tenant model, no :tenantId path param), returns redacted credentials on GET, accepts { providerId, mode, credentials } on PUT and writes through the service + audit log. Also added: POST rotate, POST force-invalidate, DELETE config (revert to inherit).
  • T15. Tenant settings page in apps/web/src/app/[locale]/(dashboard)/settings/job-runtime/page.tsx — server component that loads current config and renders the picker + credentials form.
  • T16. Provider picker component in apps/web/src/components/settings/JobRuntimeSettings.tsxSelect of operator-enabled providers (P5 wired via availableProviders prop) with inherit vs BYO vs override mode toggle. (Filed under components/settings/ not components/job-runtime/ per the dashboard's existing settings layout.)
  • T17. ✅ Done in #1486. Ships apps/web/src/components/settings/JobRuntimeCredentialsForm.tsx (alongside the settings sibling pattern, not under components/job-runtime/ — matches the existing JobRuntimeSettings.tsx location). The form reads from job-runtime-schemas.ts (a UI projection of each plugin's settingsSchema); each field renders with lock icon on secrets, reveal/hide eye toggle, env-var hint chip (BULLMQ_REDIS_URL etc.), helper text, multiline support for PEM-encoded mTLS cert/key pairs, required-fields validation, and a per-provider info banner for Trigger.dev (no tenant-form credentials). Plus a side fix to plugin-category-icons.ts covering 'job-runtime' (Cog) + 'secret-store-resolver' (KeyRound) so apps/web type-check is green at HEAD.
  • T18. Validation + error surfacing — "provider disabled by operator" surfaces via the P5 allow-list BadRequestException + warning banner; "credentials rejected by provider reachability probe" deferred to P4 (the probe itself doesn't exist yet); "force-invalidate in progress" surfaces via the controller's rate-limit response + button-disabled state.
  • T19. ✅ Done in #1486. Ships apps/web/e2e/tenant-job-runtime.spec.ts covering: page mounts without 5xx + provider picker visible; mode toggle to byo reveals creds, inherit hides them; switching provider changes the field set (proves schema-driven rendering, not opaque); secret fields default to type="password" (no plaintext DOM leak). End-to-end submit-then-verify-audit-row is layered on once a deterministic seed fixture lands.

Phase 3 — Dispatcher routing · [EW-742 P3]

T20 + T23 + T24 ✅ Done in #1380 (EW-747 — minimal viable resolver, byo/override modes return instance default with Logger.debug deferral note until per-provider credential-binding API lands). T21 (cache) ✅ shipped in #1381. T22 (stamper helper) ✅ shipped in this PR; per-dispatcher wiring still deferred to incremental follow-ups.

  • T20. Tenant-aware resolver at packages/agent/src/tasks/tenant-aware-runtime.resolver.ts (TenantAwareRuntimeResolver) wraps the EW-685 P0 T4 binding-factory registry: resolve(tenantId) returns the instance default for null / no-row / inherit / disabled; for byo + override + enabled it logs the deferral and still returns the instance default until EW-686 P2 lands the per-provider credential-binding hook. getEffectiveBinding(tenantId) returns the metadata T22 will stamp onto the run record.

  • T21. Credential cache with 15–60s TTL in packages/agent/src/tasks/tenant-credential.cache.ts — keyed by (tenantId, providerId, credentialVersion), in-process LRU with explicit invalidate on version bump or force-invalidate. Standalone @Injectable() class with no DI dependencies (defaults: maxEntries=1024, ttlMs=30_000); insertion-order LRU (no promotion-on-read so an in-flight runs snapshot does not get kept alive past its rotation window — see ADR-017 §3 / Q4). Provided + exported via TenantJobRuntimeModule. PR: #1381.

  • T22 (helper). RuntimeBindingStamperService at packages/agent/src/tasks/runtime-binding-stamper.service.ts — standalone @Injectable() whose stamp(tenantId) returns { providerId, credentialVersion } for byo/override+enabled tenants and { null, null } for inherit/no-row/disabled/no-tenant (fail-open on DB hiccup with Logger.warn). Provided + exported via TenantJobRuntimeModule. The dispatcher call sites adopt this helper incrementally — each enqueue payload type adds a credentialVersion?: number field on its own PR, no big-bang bus stop. The actual per-call-site wiring (the await stamper.stamp(tenantId) inside each of the twelve dispatchers + the run-record schema column it writes into) remains open below.

  • T22 (per-dispatcher wiring). Walk the dispatcher call sites in packages/agent/src/tasks/_tasks-symbols.ts (plus KB_REEMBED_WORK_DISPATCHER wired in apps/api/src/works/works.module.ts and WEBHOOK_DELIVERY_DISPATCHER in apps/api/src/webhooks/), call await stamper.stamp(tenantId) inside each enqueue path, and persist (providerId, credentialVersion) on the payload so the P4 worker host can later resolve the pinned snapshot via CredentialVersionService.resolveSnapshot. Each dispatcher migration shipped as its own PR — no coordinated bus stop.

    10 stamped + consumed dispatchers (producer ↔ task):

    #DispatcherResolver helperProducer PRWorker PR
    1KB_EMBED_DOCUMENT_DISPATCHERresolveForWork#1427 (PoC)#1435
    2KB_MIRROR_DOCUMENT_DISPATCHERresolveForWork#1430 (Tier 1)#1439
    3KB_NORMALIZE_MEDIA_DISPATCHER (video+audio)resolveForWork#1430 (Tier 1)#1439
    4KB_TRANSCRIBE_DISPATCHERresolveForWork#1436#1439
    5WORK_GENERATION_DISPATCHERresolveForWork#1431 (Tier 2)#1439
    6WORK_IMPORT_DISPATCHERresolveForWork#1431 (Tier 2)#1439
    7KB_ORG_OVERLAY_FANOUT_DISPATCHERresolveForOrganization#1430 (Tier 1)#1442
    8TEMPLATE_CUSTOMIZATION_DISPATCHERresolveForCustomization#1431 (Tier 2)#1442
    9WEBHOOK_DELIVERY_DISPATCHER (apps/api)resolveForSubscription#1445#1445
    10KB_REEMBED_WORK_DISPATCHER (pgvector→apps/api)resolveForWork#1448#1448

    Worker-host infrastructure shipped in #1432: TenantRuntimeBindingResolverService at packages/tasks/src/trigger/worker/services/tenant-runtime-binding-resolver.service.ts exposes resolve(payload, tenantId) + 4 convenience wrappers (resolveForWork/Organization/Customization/Subscription) returning a 4-state status (no-binding / resolved / drained / error, fail-open per FR-5). CredentialVersionService proxied through TriggerInternalController.remoteMap so worker tasks can call resolveSnapshot via the existing RPC channel. The webhook-delivery task uses direct providers in TriggerWebhookDeliveryModule (avoids a pointless RPC hop since that module already has direct DB access).

    Intentionally out-of-scope dispatchers (do NOT need T22 stamping):

    • KB_BACKFILL_SKELETON_DISPATCHER — fleet-wide operator bootstrap script. Runs as the instance-default credentials by design (no tenant scope; the caller enumerates workIds that may span tenants). Adding T22 stamping here would be semantically incorrect.

    Per-tenant credential application (deferred to a future PR with real BYO infra): the resolved snapshot.credentials is logged + threaded through the worker but the Trigger.dev SDK is still configured globally at boot. Per-tenant Trigger.dev BYO project switching (calling configure({ accessToken: snapshot.credentials.accessToken }) per dispatch) needs an actual operator with per-tenant Trigger.dev projects to validate; building speculatively risks getting the design wrong. Today the 'resolved' status is observability-only — the worker still runs against the instance default credentials.

  • T23. Fallback path to instance-global default when a tenant has no overlay row — resolver returns the EW-683 instance binding unchanged; the getActive() = null semantic propagates as null (in-process dev fallback preserved). Tests in T24 prove the zero-overhead path for tenants that never opt in.

  • T24. Unit tests for the resolver under packages/agent/src/tasks/__tests__/tenant-aware-runtime.resolver.spec.ts — 12 cases covering inherit fallback, no-row fallback, kill switch, byo/override stopgap + log emission, getEffectiveBinding metadata, and the getActive() = null propagation.

Phase 3.1 — Enqueue capture · [EW-742 P3.1] ✅ DONE

  • T22 (above) — per-dispatcher wiring shipped across PRs #1427 / #1430 / #1431 / #1436 / #1442 / #1445 / #1448 (producer side) and #1432 / #1435 / #1439 / #1442 / #1445 / #1448 (worker side). The T21 cache is in place to serve the snapshot lookup at run time; today the worker-host resolver consumes (providerId, credentialVersion) from the payload and classifies as no-binding / resolved / drained / error (skip-and-ack on drained, fall-through otherwise).

Phase 3.2 — Resolver bindToTenant wiring · [EW-742 P3.2]

✅ Done in this PR. The resolver's byo/override branch now actively binds the active provider to per-tenant credentials:

  • T20.1. SecretStoreResolver contract at packages/agent/src/tasks/secret-store-resolver.interface.ts — operator-pluggable credential-pointer resolution. SECRET_STORE_RESOLVER DI token + resolve(pointer): Promise<Record<string, unknown> | null> interface. Contract is fail-open: implementations MUST NOT throw on missing / malformed pointers — return null + log at warn instead.
  • T20.2. InProcessSecretStoreResolver default at packages/agent/src/tasks/in-process-secret-store-resolver.service.ts — supports only inline:<base64-of-json-object> for dev/test. Every other scheme (vault:, k8s:, op:) returns null + Logger.warn telling the operator to wire a real implementation for that scheme.
  • T20.3. TenantAwareRuntimeResolver.resolve() byo/override branch now does the full bind path: (a) check TenantCredentialCache keyed by (tenantId, providerId, credentialVersion); (b) on miss, resolve credentials via SecretStoreResolver; (c) call provider.bindToTenant(snapshot) (EW-686 P2 contract); (d) cache + return the bound provider. Any null/undefined/throw anywhere in the chain falls back to the instance default with a Logger.warn carrying enough tenant context for operators to diagnose. 20+ tests cover happy path + 5 fallback paths + cache-hit behaviour.
  • T20.4a. VaultSecretStoreResolver at packages/agent/src/tasks/vault-secret-store-resolver.service.ts — HTTP-based HashiCorp Vault implementation. Supports vault:<path> pointers, KV v1 + v2 envelope auto-detection, reads VAULT_ADDR + VAULT_TOKEN from env. Operators opt in by overriding the SECRET_STORE_RESOLVER DI binding. Default deployment unchanged.
  • T20.4b. K8sSecretStoreResolver at packages/agent/src/tasks/k8s-secret-store-resolver.service.ts — in-cluster API. Reads bearer token + CA cert + default namespace from /var/run/secrets/kubernetes.io/serviceaccount/{token,ca.crt,namespace}. Supports k8s:<name> (default namespace) and k8s:<ns>/<name> (explicit). Base64-decodes each .data[key] to UTF-8. Opt-in via DI binding override.
  • T20.4c. env: scheme added to InProcessSecretStoreResolver — reads JSON-encoded credential bag from process.env[<VAR_NAME>]. Self-hoster path for operators who already inject credentials via the standard 12-factor env-var pattern. Zero-dep, ships in the default resolver alongside inline:. (Replaces the dropped 1Password / op:// plan — that vendor lock-in was a poor fit for an OSS project.)
  • T20.4d. InfisicalSecretStoreResolver at packages/agent/src/tasks/infisical-secret-store-resolver.service.ts — HTTP-based against Infisical REST API. Supports infisical:<workspaceId>/<env>/<secretPath>; fetches a folder of secrets and returns them as a flat credential bag. Reads INFISICAL_TOKEN + optional INFISICAL_HOST (defaults to app.infisical.com) from env at every resolve call (rotation-friendly). Opt-in via DI binding override.
  • T20.4e. DopplerSecretStoreResolver at packages/agent/src/tasks/doppler-secret-store-resolver.service.ts — HTTP-based against Doppler REST API (/v3/configs/config/secrets). Supports doppler:<project>/<config>; fetches all secrets in the config and returns them as a flat bag (prefers .raw, falls back to .computed). Reads DOPPLER_TOKEN from env at every resolve call (rotation-friendly). Opt-in via DI binding override.
  • T20.5 (foundation). secret-store-resolver plugin category + ISecretStoreProvider capability contract at packages/plugin/src/contracts/capabilities/secret-store.interface.ts. Adds the category to PLUGIN_CATEGORIES, the plugin-shaped ISecretStoreProvider extends IPlugin interface, and the SECRET_STORE_CAPABILITIES.RESOLVE capability string. Gates the migration of Vault / K8s / Infisical / Doppler from @Injectable() services in packages/agent/src/tasks/ into true packages/plugins/secret-store-<vendor>/ plugin packages (T20.6+).
  • T20.6. VaultSecretStorePlugin at packages/plugins/secret-store-vault/ — true plugin package with everworks.plugin manifest (category: secret-store-resolver, capabilities: [secret-store-resolve]), tsup build, vitest tests (13/13 pass). Implements IPlugin + ISecretStoreProvider. Old @Injectable() class at packages/agent/src/tasks/vault-secret-store-resolver.service.ts deleted (no consumer was registering it).
  • T20.7. K8sSecretStorePlugin at packages/plugins/secret-store-k8s/ — true plugin package; old @Injectable() class deleted from packages/agent/src/tasks/. 13/13 plugin vitest pass.
  • T20.8. InfisicalSecretStorePlugin at packages/plugins/secret-store-infisical/ — true plugin package; old @Injectable() class deleted. 12/12 plugin vitest pass.
  • T20.9. DopplerSecretStorePlugin at packages/plugins/secret-store-doppler/ — true plugin package; old @Injectable() class deleted. 13/13 plugin vitest pass.
  • T20.10. New cloud-vendor plugin packages: secret-store-aws-sm, secret-store-gcp-sm, secret-store-azure-kv — all shipped as plugin packages and now run the shared ISecretStoreProvider conformance suite (#1471).

Phase 4 — Worker host · [EW-742 P4]

  • T25 (reframed 2026-06-21). Per-tenant Trigger.dev isolation — per-tenant Trigger.dev project routing is NOT pursued. Trigger.dev hard-caps 10 projects per organization across all tiers, and Trigger.dev's own multi-tenant guidance explicitly calls the per-tenant-project model an anti-pattern. The original "platform auto-provisions one Trigger.dev project per tenant via the management API" framing was therefore wrong at the premise level — replacing it with the vendor-recommended runtime-scoping pattern:

    • Single project per Trigger.dev account (the operator's for inherit, the tenant's for byo/override).
    • Per-call routing via concurrencyKey: tenantId + externalId: tenantId on every tasks.trigger(...) so each tenant gets its own queue + per-tenant concurrency budget inside the shared project. For dynamic schedules, the same externalId: tenantId pattern applies per Trigger.dev's per-tenant schedules guide.
    • Per-tenant credential resolution through the existing P3.2 plumbing: TenantAwareRuntimeResolverSecretStoreResolverprovider.bindToTenant(snapshot)TenantCredentialCache. The Trigger.dev plugin's bindToTenant already swaps accessToken / secretKey / projectRef / apiUrl per the resolved snapshot (EW-742 P3.2 T21.1) so byo/override mode wires up without any per-tenant-project ceremony.

    Implementation is split across follow-up PRs:

    • Runtime stamping — emit concurrencyKey: tenantId + externalId: tenantId on every tasks.trigger(...) / tasks.batchTrigger(...) call site. (Tracked under the session/1516-trigger-tenant-runtime-stamping branch.)
    • BYO credentials end-to-end — wire accessToken + secretKey + projectRef + optional apiUrl from the tenant settings form through the secret store into TriggerJobRuntimePlugin.bindToTenant. (Tracked under the session/1516-trigger-tenant-byo-credentials branch.)

    The platform-side Trigger.dev webhook receiver with per-tenant HMAC verify (lookup via TenantJobRuntimeConfig.credentials.webhookSecret) shipped in #1533 and remains the receive-side surface. See ./providers.md § Trigger.dev for the full mode matrix.

  • T26. ✅ Done in #1492. Ships tenantAwareInngestFunctionHandler HOF wrapper in packages/plugins/job-runtime-inngest/src/inngest-tenant-aware-handler.ts — reads event.data._ew.tenantId (T31 namespace), resolves snapshot via optional resolveSnapshot (or synthetic fallback), binds via plugin.bindToTenant, hands the binding as 2nd arg to the operator handler. Fail-open per FR-5: missing tenantId → handler receives plugin itself as binding. 7 tenant-isolation tests.

  • T27. ✅ Done in #1492. Ships TenantAwareTemporalWorkerHostFactory in packages/plugins/job-runtime-temporal/src/temporal-tenant-aware-worker-host-factory.ts as a registry mapper — one Worker per tenant. Operator's build(binding) callback receives the bound IJobRuntimeProvider view so it can configure Worker.create({ connection, namespace }) against the right namespace per ADR-017 Q1. start() materialises all workers + runs concurrently; stop() shuts down all + awaits run promises. 6 tenant-isolation tests.

  • T28. ✅ Done in #1492. Ships TenantAwareBullMqWorkerHostFactory in packages/plugins/job-runtime-bullmq/src/bullmq-tenant-aware-worker-host-factory.ts — multiplexing worker that reads job.opts.tenantId (the T31 stamping field, with job.data._ew.tenantId fallback), binds via plugin.bindToTenant, invokes handler with the binding. Per-tenant Redis prefix isolation is operator-side at dispatchersBuilder time (one BullMqDispatcherFactory per prefix). 6 tenant-isolation tests.

  • T29. ✅ Done in #1492. Ships TenantAwarePgBossWorkerHostFactory in packages/plugins/job-runtime-pgboss/src/pgboss-tenant-aware-worker-host-factory.ts — reads job.data._ew.tenantId (T31 namespace) per job (batched workers iterate per-job and route each independently), binds via plugin.bindToTenant, invokes handler with the binding. Per-tenant Postgres schema isolation (ADR-017 Q2) is operator-side at dispatchersBuilder time (one PgBoss instance per schema). 6 tenant-isolation tests.

  • T30. ✅ Done in #1492 — the multiplexing mode is the default behaviour of each TenantAware*WorkerHostFactory: one worker process per provider routes all tenants via the per-job _ew.tenantId metadata (BullMQ/pg-boss/Inngest) or one-Worker-per-tenant registry (Temporal, where per-namespace connection is required). The EVER_WORKS_JOB_RUNTIME_HOSTING={per-tenant|shared|tiered} env-var selector remains operator-side wiring — operators pick shared (multiplexing via TenantAware*) vs per-tenant (separate worker process per tenant) when launching their worker apps.

  • T31 (contract). JobEnqueueOptions.tenantId?: string added in packages/plugin/src/contracts/capabilities/job-runtime.interface.ts with per-provider routing semantic in the JSDoc (Trigger metadata, Inngest data, Temporal searchAttributes, BullMQ opts.tenantId, pg-boss payload field). 11/11 type tests pass. Per-provider stamping is still TODO — each provider's dispatcher implementation maps the field onto its native carrier on its own PR (incremental adoption, same seam pattern as T22).

  • T31 (per-provider stamping). ✅ Done across 4 PRs (#1474 BullMQ, #1475 pg-boss, #1476 Temporal, #1477 Inngest). Each provider's factory adds a mapEnqueueOptions(opts: JobEnqueueOptions) translator + .enqueue(name, payload/args, enqueueOpts, extraOpts?) convenience method that maps platform fields onto the native carrier per providers.md: idempotencyKeyjobId/singletonKey/workflowId/event.id; tenantId→native field or _ew namespace; concurrencyKey/tags/maxDurationSeconds/machineHint per provider. Trigger.dev already does metadata.tenantId via the bindToTenant impl shipped in EW-742 P3.2 T21.1.

  • T32. ✅ Done in #1492 for the mocks-only surface. Each of the 4 non-Trigger plugins ships __tests__/<provider>-tenant-isolation.spec.ts (6-7 cases each) verifying no cross-tenant binding leakage, route-A-distinct-from-route-B, concurrent A+B no cross-contamination, missing-tenantId fallback to platform default, and resolveSnapshot/bindToTenant memoisation. Real-infra integration tests (live Redis/Postgres/Temporal/Inngest) remain gated on CI infra availability.

Phase 5 — Plugin gating · [EW-742 P5] ✅ Done in #1350 (pending cascade)

  • T33. Instance plugin allow-list config in apps/api/src/config/constants.ts (under config.tenantJobRuntime.getAllowedProviders()) — operator-controlled list of bundled job-runtime provider ids exposed to tenants via the EVER_WORKS_TENANT_RUNTIME_ALLOWED_PROVIDERS env var (comma-separated). Empty/unset = all 5 bundled providers (fail-open). Q5 platform-default mode (shared/per-tenant/tiered) is a P4 worker-host concern; this gate only controls picker visibility.
  • T34. Picker filter in tenant admin UI — apps/web/src/components/settings/JobRuntimeSettings.tsx only renders providers that appear in the operator allow-list resolved from the new GET /api/account/job-runtime/available-providers endpoint. Warning banner shows when the tenant's currently-saved providerId is no longer in the list.
  • T35. Audit decoration on every overlay mutation — emitAudit decorates before / after snapshots with operatorAllowedProviders: string[] captured at write time so future audit-log reads can correlate a mutation with the active allow-list. (Per-tenant action='operator_allowlist_change' startup row is a P5.1 follow-up.)

Phase 5.1 — Per-tenant whitelist (deferred)

  • T35a. ✅ Done in #1501. Ships TenantRuntimeProviderAllowlist entity (composite PK (tenantId, providerId) + createdBy + createdAt) + AddTenantRuntimeProviderAllowlist migration (FKs to tenants/users) + OperatorTenantRuntimeAllowlistController (GET/PUT/DELETE under /api/operator/tenants/:tenantId/runtime-allowlist, gated by IsPlatformAdminGuard) + tenantJobRuntime.isPerTenantGatingEnabled() config knob (default OFF — when off the table is ignored and behaviour is byte-identical to today). Service: listTenantAllowlist / replaceTenantAllowlist (atomic txn) / deleteTenantAllowlistEntry (each mutation emits operator_allowlist_change audit row) + getAvailableProvidersForTenant(tenantId) — 4-way resolver: flag OFF → global; flag ON + zero rows → global (inherit); flag ON + rows ⊆ global → intersection in global's order; flag ON + rows ⊄ global → intersection silently drops unknown (global = upper bound).
  • T35b. ✅ Done in #1501. Ships TenantJobRuntimeBootAuditService (OnApplicationBootstrap) that hashes the effective allow-list + gating flag, dedups by hash, writes one operator_allowlist_boot row per real change (five pods restarting on same config write ONE row, not five). RelaxTenantJobRuntimeAuditTenantNullable migration relaxes tenantId to nullable so boot rows write tenantId=NULL. Bootstrap failures swallowed + logged so a transient DB hiccup at boot doesn't fail liveness; next pod restart retries.
  • T35c. ✅ Done in #1516. Operator UI for the per-tenant allow-list shipped at /admin/tenants/<tenantId>/runtime-allowlist: server component (apps/web/src/app/[locale]/(dashboard)/admin/tenants/[tenantId]/runtime-allowlist/page.tsx) gated by authAPI.getProfile()notFound() when !isPlatformAdmin (defense-in-depth on top of the API guard, same pattern as admin/usage/page.tsx); client component TenantRuntimeAllowlistManager.tsx with checkbox picker over the 5 runtime providers, status banner reflecting the config knob (isPerTenantGatingEnabled), delete chips per row; apps/web/src/lib/api/operator-tenant-runtime-allowlist.ts fetch client + apps/web/src/app/actions/admin/tenant-runtime-allowlist.ts server actions that revalidatePath on every mutation. i18n: new dashboard.adminTenantRuntimeAllowlist namespace (camelCase leaves, next-intl-safe).

Phase 6 — Conformance · [EW-742 P6]

  • T36. ✅ Done in #1480. runJobRuntimeTenantContractSuite published via @ever-works/plugin/contracts-conformance subpath; parameterised by (tenantA, tenantB); layers cross-tenant isolation + memoisation invariants on top of the base contract. Wired into all 4 non-Trigger plugins via 3-line spec files.
  • T37. ✅ Done across 3 PRs. Mocks-only matrix — two CI jobs in .github/workflows/ci.yml: job-runtime-plugin-matrix (4 cells: bullmq/pgboss/temporal/inngest) + secret-store-plugin-matrix (7 cells: vault/k8s/infisical/doppler/aws-sm/gcp-sm/azure-kv). Each cell runs pnpm --filter @ever-works/<plugin> test after building @ever-works/plugin first (needed for the contracts-conformance subpath). fail-fast: false so one provider's failure doesn't mask the others. Real-infra portion complete across all 4 providers: BullMQ + pg-boss in #1515 (Redis 7-alpine + Postgres 16-alpine service containers + 6 smoke cases each, gated by EW_TEST_REAL_REDIS_URL / EW_TEST_REAL_PGBOSS_URL); Temporal + Inngest in #1522 (temporalio/auto-setup + Inngest dev-server containers with the longer health-check windows, gated by EW_TEST_REAL_TEMPORAL_ADDRESS / EW_TEST_REAL_INNGEST_URL); skip-when-env-unset for local devs. Cost gate in #1529 restricts the job-runtime-real-infra job to direct pushes to main only (PRs + cascade branches skip it — saves CI minutes on every cascade hop). Cross-provider tenant isolation at the resolver contract surface closed out in #1531.
  • T38. ✅ Done in #1480 (as part of the tenant conformance harness). Verifies v=N+1 returns a DIFFERENT view, and the OLD view's dispatchers reference stays callable (in-flight runs holding the old view aren't broken by rotation). Real-infra rotation-while-running test is T32 (gated on CI infra).
  • T39. ✅ Done in #1480 (contract-surface portion). Verifies the cache-eviction semantic: re-binding (tenantId, v=N) after v=N+1 has been bound returns a FRESH view (cache stays bounded by tenant count, not version count). The full admin endpoint + "in-flight marked FAILED" behaviour is wired in the existing API force-invalidate route from P2.0 (POST /api/account/job-runtime/force-invalidate); the new conformance test pins the provider-side contract that backs it.
  • T40. ✅ Done in #1480 (contract-surface portion). Verifies cross-tenant isolation at the cache level: tenantA bind does NOT affect tenantB cache; view.bindToTenant(otherTenant) routes through the base rather than back to self. End-to-end cross-provider isolation (tenant A on pgboss + tenant B on temporal in the same process) is real-infra T32 territory.

Phase 7 — Docs · [EW-742 P7]

T41 + T42 ✅ Done in #1352 (cascaded). T44 ✅ Done in this PR (migration guide writeable now that P3 resolver + P3.1 cache + EW-686 P2 bindToTenant are on main — the drain procedure is a real thing to document). T43 still deferred until per-provider plugin packages exist.

  • T41. Tenant admin runbook in docs/runbooks/TENANT_JOB_RUNTIME.md — how to pick a provider, enter BYO credentials, rotate, force-invalidate, revert to inherit, and read the audit log; cross-link spec.md and providers.md. (Filed under docs/runbooks/ UPPERCASE per the existing EVER_WORKS_ZERO_FRICTION_FLOW.md convention.)
  • T42. Operator runbook in docs/runbooks/OPERATOR_JOB_RUNTIME_OVERLAY.md — covers allow-list gating, force-invalidate procedure with on-call checklist, operator-driven rollback to inherit, removing a provider from the bundled set, pre-deploy checklist. (Q5 hosting modes deferred to a follow-up edit once P4 worker-host wiring is in place; doc calls this out explicitly so operators don't expect the env var to do anything yet.)
  • T43. Per-provider tenant-config docs — added a "Tenant overlay (EW-742)" section to each of the 4 new provider plugin READMEs (packages/plugins/job-runtime-{bullmq,pgboss,temporal,inngest}/README.md, all newly authored in this batch) and to the Trigger.dev arm (packages/tasks/README.md). Each section documents inherit/byo/override modes, the tenant credential bag shape, per-tenant routing pattern (where applicable), and cross-links to ./providers.md + ADR-017 + the conformance suite spec.
  • T44. Migration guide for existing tenants in docs/runbooks/TENANT_JOB_RUNTIME_MIGRATION.md — four scenarios (inherit → BYO opt-in, BYO rotate with graceful drain, BYO → inherit rollback, force-invalidate emergency), failure-mode table, verification queries (curl + SQL). Cross-links the canonical tenant + operator runbooks + ADR-017 Q4. (Filed under docs/runbooks/ UPPERCASE per convention; original deferral note assumed the docs needed P4 worker host to be real — P3 resolver + P3.1 cache + EW-686 P2 are enough for the drain procedure to be documented honestly.)

Dependency notes

  • T1–T7 (spec-kit) block everything else; this PR ships only Phase 0.
  • T8 (x-scope: tenant) blocks T9–T13 (data model needs the scope token).
  • T9–T13 (data model) block T14–T19 (admin UI writes through the repository) and T20–T24 (resolver reads from the repository).
  • T20–T24 (dispatcher) block T25–T32 (worker host needs the tenant-aware resolver to know which credentials a run belongs to).
  • T31 (tenantId in JobEnqueueOptions) blocks T25–T30 (all per-tenant routing depends on the metadata field).
  • T33–T35 (gating) can land in parallel with Phase 4 but must precede T19 going green (e2e validates the operator-gated picker).
  • T36–T40 (conformance) trail each provider's worker-host task (T27/T28/T29 for self-host, T25/T26 for SaaS).
  • T41–T44 (docs) trail each phase as it lands.