Task Breakdown: Tenant-Scoped Job-Runtime Overlay
Feature ID: tenant-job-runtime-overlay
Status: In progress — P0 + P1.0 + P1 + P2.0 + P2.1 + P3 (T20+T23+T24 resolver) + P3.1 (T21 cache + T22 stamper helper + T22 per-dispatcher wiring across all 10 in-platform dispatchers, with worker-host TenantRuntimeBindingResolverService consumption) + P3.2 (resolver bindToTenant wiring + SecretStoreResolver contract + InProcessSecretStoreResolver default + Trigger.dev bindToTenant minimal impl + non-inline: SecretStoreResolver plugin packages: Vault / K8s / Infisical / Doppler / AWS-SM / GCP-SM / Azure-KV + the 4 non-Trigger IJobRuntimeProvider plugin packages (Temporal / BullMQ / pg-boss / Inngest) with real bindToTenant memoisation AND operator-pluggable real dispatcher + worker-host factories (#1455 BullMQ, #1456 pg-boss, #1457 Temporal, #1458 Inngest)) + P4 (T31 contract — tenantId on JobEnqueueOptions) + P5 + P6 (T37 real-infra matrix complete across all 4 providers — BullMQ+pg-boss in #1515, Temporal+Inngest in #1522; cost-gated to direct main pushes in #1529; cross-provider tenant isolation resolver-level closeout in #1531) + P7 (T41+T42+T44) landed on develop (cascades through develop→stage→main where applicable, develop-only for the late-2026 batches). Platform-side Trigger.dev webhook receiver with per-tenant HMAC verify shipped in #1533 (T25 platform-side prep, receiver-side ready). Dependabot HIGH alerts patched in #1526; 28 of 33 MODERATE patched in #1532. P2.2 (schema-driven form + e2e), P4 (per-provider worker hosts T25–T30 + T31 per-provider stamping + T32 isolation tests), P5.1 (per-tenant whitelist), and P7 (docs T43) remain. T25 reframed (2026-06-21): the original "one Trigger.dev project per tenant" framing was rejected after the 10-project-per-org cap + Trigger.dev's own multi-tenant guidance surfaced. Full per-tenant Trigger.dev isolation is now achieved via the runtime-scoping pattern (single project + concurrencyKey: tenantId + externalId: tenantId + per-tenant credential resolution via the existing bindToTenant + TenantCredentialCache plumbing) across all 3 modes (inherit / byo / override); see T25 below and ./providers.md § Trigger.dev. Operator wiring docs for the 4 plugin packages live in ./providers.md § Operator wiring.
Last updated: 2026-06-21
Spec: ./spec.md · Plan: ./plan.md · Providers: ./providers.md · Epic: EW-742 · Story: EW-743 · ADR: ADR-017
Ordered, granular tasks with explicit paths. Phases map to
plan.md§10. Each phase is a candidate sub-PR. Mark[x]as landed. Suggested Jira child-issue grouping noted per phase (e.g.[EW-742 P1]). Builds on EW-683's instance-level job-runtime contract; nothing here re-opens ADR-017 Q1–Q5 (Temporal namespace-per-tenant, pg-boss schema-per-tenant, Inngest inherit/BYO uniform with Trigger.dev, credential rotation via graceful drain + separateforce-invalidate, hybrid operator-gated platform-defaultshared/per-tenant/tiered).
Phase 0 — Spec-Kit (EW-743) · [EW-743 P0] ✅ Done in #1332 (cascaded #1333, #1334)
- T1. Write ADR-017 at
docs/specs/decisions/017-tenant-scoped-job-runtime-overlay.mdcapturing Q1–Q5 locked decisions and the inherit/BYO overlay rationale on top of EW-683. - T2. Write
docs/specs/features/tenant-job-runtime-overlay/spec.md— behaviour-first user stories (tenant picks provider, operator gates allow-list, graceful drain on rotation, force-invalidate path) with no implementation detail. - T3. Write
docs/specs/features/tenant-job-runtime-overlay/plan.md— architecture, data model, API surface, phased rollout, constitution reconciliation; mirrors structure ofjob-runtime-providers/plan.md. - T4. Write
docs/specs/features/tenant-job-runtime-overlay/tasks.md(this file) — numbered T1..TN per phase with file paths. - T5. Write
docs/specs/features/tenant-job-runtime-overlay/providers.md— per-provider tenant-isolation matrix (Temporal namespace, pg-boss schema, BullMQ prefix, Trigger.dev project, Inngest app) and inherit-vs-BYO knob per provider. - T6. Cross-link from
docs/specs/architecture/job-runtime-providers.mdto the tenant-overlay docs by adding a "Tenant-scoped overlay" see-also section pointing atspec.md,plan.md, and ADR-017. - T7. Verify constitution gates against
.specify/memory/constitution.md— confirm Principle IV ("via the configured job-runtime provider") already covers tenant overlay; no new amendment required, document the verification inplan.md§Constitution Reconciliation.
Phase 1 — Data model + migration · [EW-742 P1] ✅ Done in #1338 (cascaded #1339, #1340). P1.0 (x-scope: tenant) shipped in #1335 (cascaded #1336, #1337).
- T8. Audit
docs/specs/architecture/settings-system.mdforx-scope: tenant; if missing, add it to the scope taxonomy alongsidex-scope: global(verify in worktree before writing — this is a prerequisite for tenant-scoped credentials). - T9. Define TypeORM entity
TenantJobRuntimeConfiginpackages/agent/src/entities/tenant-job-runtime-config.entity.ts— columns:tenantId(PK),providerId,mode(inherit|byo|override),credentialsSecretRef,credentialVersion,enabled,createdBy,createdAt,updatedAt. (Entity moved to@ever-works/agent/entitiesper EW-654/655 cycle-avoidance — no@ManyToOneback-refs, FK enforced only at the DB layer.) - T10. Write TypeORM migration
apps/api/src/migrations/1780900000000-AddTenantJobRuntimeConfig.tsfor the new table plus partial index on(providerId) WHERE enabled(PK already coverstenantId); forward-only + idempotenthasTableguards perdocs/specs/architecture/database-migrations.md. - T11. Implement
CredentialVersionServicefor graceful drain (Q4) inpackages/agent/src/tasks/credential-version.service.ts— issues monotonic per-tenant version ids and resolves a snapshot for a given(tenantId, version)tuple so in-flight runs keep their original credentials. Full snapshot history table shipped in #1514:TenantCredentialSnapshotentity + migration +captureSnapshot()on every version bump +resolveSnapshot(tenantId, version)reads the row (FIFO history, idempotent on conflict). Bag re-issued asinline:<base64>so the existing secret-store resolver chain dereferences without a new scheme. 12 vitest history cases; 6091/6091 packages/agent green. - T12. Add audit-log table + entity for tenant runtime config changes in
packages/agent/src/entities/tenant-job-runtime-audit.entity.tsplus matching migrationapps/api/src/migrations/1781000000000-AddTenantJobRuntimeAudit.ts; records(tenantId, actorUserId, action, before, after, credentialVersion, occurredAt). - T13. Repository + unit tests under
apps/api/src/works/__tests__/tenant-job-runtime-config.repository.spec.tscovering insert, update, version bump, audit-row emission, and tenant isolation (one tenant cannot read another's row).
Phase 2 — Admin UI · [EW-742 P2]
P2.0 (REST API) ✅ Done in #1341 (cascaded #1342, #1343). P2.1 (UI) ✅ Done in #1347 (cascaded #1348, #1349). P2.2 (schema-driven form + e2e) remains.
- T14. API endpoints
GET /api/account/job-runtime/configandPUT /api/account/job-runtime/configinapps/api/src/account/tenant-job-runtime/tenant-job-runtime.controller.ts— guarded byrequireTenant()(1 User : 1 Tenant model, no:tenantIdpath param), returns redacted credentials on GET, accepts{ providerId, mode, credentials }on PUT and writes through the service + audit log. Also added:POST rotate,POST force-invalidate,DELETE config(revert to inherit). - T15. Tenant settings page in
apps/web/src/app/[locale]/(dashboard)/settings/job-runtime/page.tsx— server component that loads current config and renders the picker + credentials form. - T16. Provider picker component in
apps/web/src/components/settings/JobRuntimeSettings.tsx—Selectof operator-enabled providers (P5 wired viaavailableProvidersprop) withinheritvsBYOvsoverridemode toggle. (Filed undercomponents/settings/notcomponents/job-runtime/per the dashboard's existing settings layout.) - T17. ✅ Done in #1486. Ships
apps/web/src/components/settings/JobRuntimeCredentialsForm.tsx(alongside the settings sibling pattern, not undercomponents/job-runtime/— matches the existingJobRuntimeSettings.tsxlocation). The form reads fromjob-runtime-schemas.ts(a UI projection of each plugin'ssettingsSchema); each field renders with lock icon on secrets, reveal/hide eye toggle, env-var hint chip (BULLMQ_REDIS_URL etc.), helper text, multiline support for PEM-encoded mTLS cert/key pairs, required-fields validation, and a per-provider info banner for Trigger.dev (no tenant-form credentials). Plus a side fix toplugin-category-icons.tscovering'job-runtime'(Cog) +'secret-store-resolver'(KeyRound) so apps/web type-check is green at HEAD. - T18. Validation + error surfacing — "provider disabled by operator" surfaces via the P5 allow-list
BadRequestException+ warning banner; "credentials rejected by provider reachability probe" deferred to P4 (the probe itself doesn't exist yet); "force-invalidate in progress" surfaces via the controller's rate-limit response + button-disabled state. - T19. ✅ Done in #1486. Ships
apps/web/e2e/tenant-job-runtime.spec.tscovering: page mounts without 5xx + provider picker visible; mode toggle to byo reveals creds, inherit hides them; switching provider changes the field set (proves schema-driven rendering, not opaque); secret fields default totype="password"(no plaintext DOM leak). End-to-end submit-then-verify-audit-row is layered on once a deterministic seed fixture lands.
Phase 3 — Dispatcher routing · [EW-742 P3]
T20 + T23 + T24 ✅ Done in #1380 (EW-747 — minimal viable resolver, byo/override modes return instance default with Logger.debug deferral note until per-provider credential-binding API lands). T21 (cache) ✅ shipped in #1381. T22 (stamper helper) ✅ shipped in this PR; per-dispatcher wiring still deferred to incremental follow-ups.
-
T20. Tenant-aware resolver at
packages/agent/src/tasks/tenant-aware-runtime.resolver.ts(TenantAwareRuntimeResolver) wraps the EW-685 P0 T4 binding-factory registry:resolve(tenantId)returns the instance default fornull/ no-row / inherit / disabled; forbyo+override+enabledit logs the deferral and still returns the instance default until EW-686 P2 lands the per-provider credential-binding hook.getEffectiveBinding(tenantId)returns the metadata T22 will stamp onto the run record. -
T21. Credential cache with 15–60s TTL in
packages/agent/src/tasks/tenant-credential.cache.ts— keyed by(tenantId, providerId, credentialVersion), in-process LRU with explicit invalidate on version bump or force-invalidate. Standalone@Injectable()class with no DI dependencies (defaults:maxEntries=1024,ttlMs=30_000); insertion-order LRU (no promotion-on-read so an in-flight runs snapshot does not get kept alive past its rotation window — see ADR-017 §3 / Q4). Provided + exported viaTenantJobRuntimeModule. PR: #1381. -
T22 (helper).
RuntimeBindingStamperServiceatpackages/agent/src/tasks/runtime-binding-stamper.service.ts— standalone@Injectable()whosestamp(tenantId)returns{ providerId, credentialVersion }for byo/override+enabled tenants and{ null, null }for inherit/no-row/disabled/no-tenant (fail-open on DB hiccup withLogger.warn). Provided + exported viaTenantJobRuntimeModule. The dispatcher call sites adopt this helper incrementally — each enqueue payload type adds acredentialVersion?: numberfield on its own PR, no big-bang bus stop. The actual per-call-site wiring (theawait stamper.stamp(tenantId)inside each of the twelve dispatchers + the run-record schema column it writes into) remains open below. -
T22 (per-dispatcher wiring). Walk the dispatcher call sites in
packages/agent/src/tasks/_tasks-symbols.ts(plusKB_REEMBED_WORK_DISPATCHERwired inapps/api/src/works/works.module.tsandWEBHOOK_DELIVERY_DISPATCHERinapps/api/src/webhooks/), callawait stamper.stamp(tenantId)inside each enqueue path, and persist(providerId, credentialVersion)on the payload so the P4 worker host can later resolve the pinned snapshot viaCredentialVersionService.resolveSnapshot. Each dispatcher migration shipped as its own PR — no coordinated bus stop.10 stamped + consumed dispatchers (producer ↔ task):
# Dispatcher Resolver helper Producer PR Worker PR 1 KB_EMBED_DOCUMENT_DISPATCHERresolveForWork#1427 (PoC) #1435 2 KB_MIRROR_DOCUMENT_DISPATCHERresolveForWork#1430 (Tier 1) #1439 3 KB_NORMALIZE_MEDIA_DISPATCHER(video+audio)resolveForWork#1430 (Tier 1) #1439 4 KB_TRANSCRIBE_DISPATCHERresolveForWork#1436 #1439 5 WORK_GENERATION_DISPATCHERresolveForWork#1431 (Tier 2) #1439 6 WORK_IMPORT_DISPATCHERresolveForWork#1431 (Tier 2) #1439 7 KB_ORG_OVERLAY_FANOUT_DISPATCHERresolveForOrganization#1430 (Tier 1) #1442 8 TEMPLATE_CUSTOMIZATION_DISPATCHERresolveForCustomization#1431 (Tier 2) #1442 9 WEBHOOK_DELIVERY_DISPATCHER(apps/api)resolveForSubscription#1445 #1445 10 KB_REEMBED_WORK_DISPATCHER(pgvector→apps/api)resolveForWork#1448 #1448 Worker-host infrastructure shipped in #1432:
TenantRuntimeBindingResolverServiceatpackages/tasks/src/trigger/worker/services/tenant-runtime-binding-resolver.service.tsexposesresolve(payload, tenantId)+ 4 convenience wrappers (resolveForWork/Organization/Customization/Subscription) returning a 4-state status (no-binding/resolved/drained/error, fail-open per FR-5).CredentialVersionServiceproxied throughTriggerInternalController.remoteMapso worker tasks can callresolveSnapshotvia the existing RPC channel. The webhook-delivery task uses direct providers inTriggerWebhookDeliveryModule(avoids a pointless RPC hop since that module already has direct DB access).Intentionally out-of-scope dispatchers (do NOT need T22 stamping):
KB_BACKFILL_SKELETON_DISPATCHER— fleet-wide operator bootstrap script. Runs as the instance-default credentials by design (no tenant scope; the caller enumeratesworkIdsthat may span tenants). Adding T22 stamping here would be semantically incorrect.
Per-tenant credential application (deferred to a future PR with real BYO infra): the resolved
snapshot.credentialsis logged + threaded through the worker but the Trigger.dev SDK is still configured globally at boot. Per-tenant Trigger.dev BYO project switching (callingconfigure({ accessToken: snapshot.credentials.accessToken })per dispatch) needs an actual operator with per-tenant Trigger.dev projects to validate; building speculatively risks getting the design wrong. Today the'resolved'status is observability-only — the worker still runs against the instance default credentials. -
T23. Fallback path to instance-global default when a tenant has no overlay row — resolver returns the EW-683 instance binding unchanged; the
getActive() = nullsemantic propagates asnull(in-process dev fallback preserved). Tests in T24 prove the zero-overhead path for tenants that never opt in. -
T24. Unit tests for the resolver under
packages/agent/src/tasks/__tests__/tenant-aware-runtime.resolver.spec.ts— 12 cases covering inherit fallback, no-row fallback, kill switch, byo/override stopgap + log emission,getEffectiveBindingmetadata, and thegetActive() = nullpropagation.
Phase 3.1 — Enqueue capture · [EW-742 P3.1] ✅ DONE
- T22 (above) — per-dispatcher wiring shipped across PRs #1427 / #1430 / #1431 / #1436 / #1442 / #1445 / #1448 (producer side) and #1432 / #1435 / #1439 / #1442 / #1445 / #1448 (worker side). The T21 cache is in place to serve the snapshot lookup at run time; today the worker-host resolver consumes
(providerId, credentialVersion)from the payload and classifies asno-binding/resolved/drained/error(skip-and-ack on drained, fall-through otherwise).
Phase 3.2 — Resolver bindToTenant wiring · [EW-742 P3.2]
✅ Done in this PR. The resolver's byo/override branch now actively binds the active provider to per-tenant credentials:
- T20.1.
SecretStoreResolvercontract atpackages/agent/src/tasks/secret-store-resolver.interface.ts— operator-pluggable credential-pointer resolution.SECRET_STORE_RESOLVERDI token +resolve(pointer): Promise<Record<string, unknown> | null>interface. Contract is fail-open: implementations MUST NOT throw on missing / malformed pointers — returnnull+ log at warn instead. - T20.2.
InProcessSecretStoreResolverdefault atpackages/agent/src/tasks/in-process-secret-store-resolver.service.ts— supports onlyinline:<base64-of-json-object>for dev/test. Every other scheme (vault:,k8s:,op:) returnsnull+Logger.warntelling the operator to wire a real implementation for that scheme. - T20.3.
TenantAwareRuntimeResolver.resolve()byo/override branch now does the full bind path: (a) checkTenantCredentialCachekeyed by(tenantId, providerId, credentialVersion); (b) on miss, resolve credentials viaSecretStoreResolver; (c) callprovider.bindToTenant(snapshot)(EW-686 P2 contract); (d) cache + return the bound provider. Any null/undefined/throw anywhere in the chain falls back to the instance default with aLogger.warncarrying enough tenant context for operators to diagnose. 20+ tests cover happy path + 5 fallback paths + cache-hit behaviour. - T20.4a.
VaultSecretStoreResolveratpackages/agent/src/tasks/vault-secret-store-resolver.service.ts— HTTP-based HashiCorp Vault implementation. Supportsvault:<path>pointers, KV v1 + v2 envelope auto-detection, reads VAULT_ADDR + VAULT_TOKEN from env. Operators opt in by overriding theSECRET_STORE_RESOLVERDI binding. Default deployment unchanged. - T20.4b.
K8sSecretStoreResolveratpackages/agent/src/tasks/k8s-secret-store-resolver.service.ts— in-cluster API. Reads bearer token + CA cert + default namespace from/var/run/secrets/kubernetes.io/serviceaccount/{token,ca.crt,namespace}. Supportsk8s:<name>(default namespace) andk8s:<ns>/<name>(explicit). Base64-decodes each.data[key]to UTF-8. Opt-in via DI binding override. - T20.4c.
env:scheme added toInProcessSecretStoreResolver— reads JSON-encoded credential bag fromprocess.env[<VAR_NAME>]. Self-hoster path for operators who already inject credentials via the standard 12-factor env-var pattern. Zero-dep, ships in the default resolver alongsideinline:. (Replaces the dropped 1Password /op://plan — that vendor lock-in was a poor fit for an OSS project.) - T20.4d.
InfisicalSecretStoreResolveratpackages/agent/src/tasks/infisical-secret-store-resolver.service.ts— HTTP-based against Infisical REST API. Supportsinfisical:<workspaceId>/<env>/<secretPath>; fetches a folder of secrets and returns them as a flat credential bag. Reads INFISICAL_TOKEN + optional INFISICAL_HOST (defaults toapp.infisical.com) from env at every resolve call (rotation-friendly). Opt-in via DI binding override. - T20.4e.
DopplerSecretStoreResolveratpackages/agent/src/tasks/doppler-secret-store-resolver.service.ts— HTTP-based against Doppler REST API (/v3/configs/config/secrets). Supportsdoppler:<project>/<config>; fetches all secrets in the config and returns them as a flat bag (prefers.raw, falls back to.computed). Reads DOPPLER_TOKEN from env at every resolve call (rotation-friendly). Opt-in via DI binding override. - T20.5 (foundation).
secret-store-resolverplugin category +ISecretStoreProvidercapability contract atpackages/plugin/src/contracts/capabilities/secret-store.interface.ts. Adds the category toPLUGIN_CATEGORIES, the plugin-shapedISecretStoreProvider extends IPlugininterface, and theSECRET_STORE_CAPABILITIES.RESOLVEcapability string. Gates the migration of Vault / K8s / Infisical / Doppler from@Injectable()services inpackages/agent/src/tasks/into truepackages/plugins/secret-store-<vendor>/plugin packages (T20.6+). - T20.6.
VaultSecretStorePluginatpackages/plugins/secret-store-vault/— true plugin package with everworks.plugin manifest (category:secret-store-resolver, capabilities:[secret-store-resolve]), tsup build, vitest tests (13/13 pass). ImplementsIPlugin + ISecretStoreProvider. Old@Injectable()class atpackages/agent/src/tasks/vault-secret-store-resolver.service.tsdeleted (no consumer was registering it). - T20.7.
K8sSecretStorePluginatpackages/plugins/secret-store-k8s/— true plugin package; old@Injectable()class deleted frompackages/agent/src/tasks/. 13/13 plugin vitest pass. - T20.8.
InfisicalSecretStorePluginatpackages/plugins/secret-store-infisical/— true plugin package; old@Injectable()class deleted. 12/12 plugin vitest pass. - T20.9.
DopplerSecretStorePluginatpackages/plugins/secret-store-doppler/— true plugin package; old@Injectable()class deleted. 13/13 plugin vitest pass. - T20.10. New cloud-vendor plugin packages:
secret-store-aws-sm,secret-store-gcp-sm,secret-store-azure-kv— all shipped as plugin packages and now run the sharedISecretStoreProviderconformance suite (#1471).
Phase 4 — Worker host · [EW-742 P4]
-
T25 (reframed 2026-06-21). Per-tenant Trigger.dev isolation — per-tenant Trigger.dev project routing is NOT pursued. Trigger.dev hard-caps 10 projects per organization across all tiers, and Trigger.dev's own multi-tenant guidance explicitly calls the per-tenant-project model an anti-pattern. The original "platform auto-provisions one Trigger.dev project per tenant via the management API" framing was therefore wrong at the premise level — replacing it with the vendor-recommended runtime-scoping pattern:
- Single project per Trigger.dev account (the operator's for
inherit, the tenant's forbyo/override). - Per-call routing via
concurrencyKey: tenantId+externalId: tenantIdon everytasks.trigger(...)so each tenant gets its own queue + per-tenant concurrency budget inside the shared project. For dynamic schedules, the sameexternalId: tenantIdpattern applies per Trigger.dev's per-tenant schedules guide. - Per-tenant credential resolution through the existing P3.2 plumbing:
TenantAwareRuntimeResolver→SecretStoreResolver→provider.bindToTenant(snapshot)→TenantCredentialCache. The Trigger.dev plugin'sbindToTenantalready swapsaccessToken/secretKey/projectRef/apiUrlper the resolved snapshot (EW-742 P3.2 T21.1) sobyo/overridemode wires up without any per-tenant-project ceremony.
Implementation is split across follow-up PRs:
- Runtime stamping — emit
concurrencyKey: tenantId+externalId: tenantIdon everytasks.trigger(...)/tasks.batchTrigger(...)call site. (Tracked under thesession/1516-trigger-tenant-runtime-stampingbranch.) - BYO credentials end-to-end — wire
accessToken+secretKey+projectRef+ optionalapiUrlfrom the tenant settings form through the secret store intoTriggerJobRuntimePlugin.bindToTenant. (Tracked under thesession/1516-trigger-tenant-byo-credentialsbranch.)
The platform-side Trigger.dev webhook receiver with per-tenant HMAC verify (lookup via
TenantJobRuntimeConfig.credentials.webhookSecret) shipped in #1533 and remains the receive-side surface. See./providers.md§ Trigger.dev for the full mode matrix. - Single project per Trigger.dev account (the operator's for
-
T26. ✅ Done in #1492. Ships
tenantAwareInngestFunctionHandlerHOF wrapper inpackages/plugins/job-runtime-inngest/src/inngest-tenant-aware-handler.ts— readsevent.data._ew.tenantId(T31 namespace), resolves snapshot via optionalresolveSnapshot(or synthetic fallback), binds viaplugin.bindToTenant, hands the binding as 2nd arg to the operator handler. Fail-open per FR-5: missing tenantId → handler receives plugin itself as binding. 7 tenant-isolation tests. -
T27. ✅ Done in #1492. Ships
TenantAwareTemporalWorkerHostFactoryinpackages/plugins/job-runtime-temporal/src/temporal-tenant-aware-worker-host-factory.tsas a registry mapper — one Worker per tenant. Operator'sbuild(binding)callback receives the boundIJobRuntimeProviderview so it can configureWorker.create({ connection, namespace })against the right namespace per ADR-017 Q1.start()materialises all workers + runs concurrently;stop()shuts down all + awaits run promises. 6 tenant-isolation tests. -
T28. ✅ Done in #1492. Ships
TenantAwareBullMqWorkerHostFactoryinpackages/plugins/job-runtime-bullmq/src/bullmq-tenant-aware-worker-host-factory.ts— multiplexing worker that readsjob.opts.tenantId(the T31 stamping field, withjob.data._ew.tenantIdfallback), binds viaplugin.bindToTenant, invokes handler with the binding. Per-tenant Redis prefix isolation is operator-side atdispatchersBuildertime (one BullMqDispatcherFactory per prefix). 6 tenant-isolation tests. -
T29. ✅ Done in #1492. Ships
TenantAwarePgBossWorkerHostFactoryinpackages/plugins/job-runtime-pgboss/src/pgboss-tenant-aware-worker-host-factory.ts— readsjob.data._ew.tenantId(T31 namespace) per job (batched workers iterate per-job and route each independently), binds viaplugin.bindToTenant, invokes handler with the binding. Per-tenant Postgres schema isolation (ADR-017 Q2) is operator-side atdispatchersBuildertime (onePgBossinstance per schema). 6 tenant-isolation tests. -
T30. ✅ Done in #1492 — the multiplexing mode is the default behaviour of each
TenantAware*WorkerHostFactory: one worker process per provider routes all tenants via the per-job_ew.tenantIdmetadata (BullMQ/pg-boss/Inngest) or one-Worker-per-tenant registry (Temporal, where per-namespace connection is required). TheEVER_WORKS_JOB_RUNTIME_HOSTING={per-tenant|shared|tiered}env-var selector remains operator-side wiring — operators pickshared(multiplexing via TenantAware*) vsper-tenant(separate worker process per tenant) when launching their worker apps. -
T31 (contract).
JobEnqueueOptions.tenantId?: stringadded inpackages/plugin/src/contracts/capabilities/job-runtime.interface.tswith per-provider routing semantic in the JSDoc (Triggermetadata, Inngestdata, TemporalsearchAttributes, BullMQopts.tenantId, pg-boss payload field). 11/11 type tests pass. Per-provider stamping is still TODO — each provider's dispatcher implementation maps the field onto its native carrier on its own PR (incremental adoption, same seam pattern as T22). -
T31 (per-provider stamping). ✅ Done across 4 PRs (#1474 BullMQ, #1475 pg-boss, #1476 Temporal, #1477 Inngest). Each provider's factory adds a
mapEnqueueOptions(opts: JobEnqueueOptions)translator +.enqueue(name, payload/args, enqueueOpts, extraOpts?)convenience method that maps platform fields onto the native carrier perproviders.md:idempotencyKey→jobId/singletonKey/workflowId/event.id;tenantId→native field or_ewnamespace;concurrencyKey/tags/maxDurationSeconds/machineHintper provider. Trigger.dev already doesmetadata.tenantIdvia the bindToTenant impl shipped in EW-742 P3.2 T21.1. -
T32. ✅ Done in #1492 for the mocks-only surface. Each of the 4 non-Trigger plugins ships
__tests__/<provider>-tenant-isolation.spec.ts(6-7 cases each) verifying no cross-tenant binding leakage, route-A-distinct-from-route-B, concurrent A+B no cross-contamination, missing-tenantId fallback to platform default, and resolveSnapshot/bindToTenant memoisation. Real-infra integration tests (live Redis/Postgres/Temporal/Inngest) remain gated on CI infra availability.
Phase 5 — Plugin gating · [EW-742 P5] ✅ Done in #1350 (pending cascade)
- T33. Instance plugin allow-list config in
apps/api/src/config/constants.ts(underconfig.tenantJobRuntime.getAllowedProviders()) — operator-controlled list of bundledjob-runtimeprovider ids exposed to tenants via theEVER_WORKS_TENANT_RUNTIME_ALLOWED_PROVIDERSenv var (comma-separated). Empty/unset = all 5 bundled providers (fail-open). Q5 platform-default mode (shared/per-tenant/tiered) is a P4 worker-host concern; this gate only controls picker visibility. - T34. Picker filter in tenant admin UI —
apps/web/src/components/settings/JobRuntimeSettings.tsxonly renders providers that appear in the operator allow-list resolved from the newGET /api/account/job-runtime/available-providersendpoint. Warning banner shows when the tenant's currently-savedproviderIdis no longer in the list. - T35. Audit decoration on every overlay mutation —
emitAuditdecoratesbefore/aftersnapshots withoperatorAllowedProviders: string[]captured at write time so future audit-log reads can correlate a mutation with the active allow-list. (Per-tenantaction='operator_allowlist_change'startup row is a P5.1 follow-up.)
Phase 5.1 — Per-tenant whitelist (deferred)
- T35a. ✅ Done in #1501. Ships
TenantRuntimeProviderAllowlistentity (composite PK(tenantId, providerId)+ createdBy + createdAt) +AddTenantRuntimeProviderAllowlistmigration (FKs to tenants/users) +OperatorTenantRuntimeAllowlistController(GET/PUT/DELETE under/api/operator/tenants/:tenantId/runtime-allowlist, gated byIsPlatformAdminGuard) +tenantJobRuntime.isPerTenantGatingEnabled()config knob (default OFF — when off the table is ignored and behaviour is byte-identical to today). Service:listTenantAllowlist/replaceTenantAllowlist(atomic txn) /deleteTenantAllowlistEntry(each mutation emitsoperator_allowlist_changeaudit row) +getAvailableProvidersForTenant(tenantId)— 4-way resolver: flag OFF → global; flag ON + zero rows → global (inherit); flag ON + rows ⊆ global → intersection in global's order; flag ON + rows ⊄ global → intersection silently drops unknown (global = upper bound). - T35b. ✅ Done in #1501. Ships
TenantJobRuntimeBootAuditService(OnApplicationBootstrap) that hashes the effective allow-list + gating flag, dedups by hash, writes oneoperator_allowlist_bootrow per real change (five pods restarting on same config write ONE row, not five).RelaxTenantJobRuntimeAuditTenantNullablemigration relaxestenantIdto nullable so boot rows writetenantId=NULL. Bootstrap failures swallowed + logged so a transient DB hiccup at boot doesn't fail liveness; next pod restart retries. - T35c. ✅ Done in #1516. Operator UI for the per-tenant allow-list shipped at
/admin/tenants/<tenantId>/runtime-allowlist: server component (apps/web/src/app/[locale]/(dashboard)/admin/tenants/[tenantId]/runtime-allowlist/page.tsx) gated byauthAPI.getProfile()→notFound()when!isPlatformAdmin(defense-in-depth on top of the API guard, same pattern asadmin/usage/page.tsx); client componentTenantRuntimeAllowlistManager.tsxwith checkbox picker over the 5 runtime providers, status banner reflecting the config knob (isPerTenantGatingEnabled), delete chips per row;apps/web/src/lib/api/operator-tenant-runtime-allowlist.tsfetch client +apps/web/src/app/actions/admin/tenant-runtime-allowlist.tsserver actions thatrevalidatePathon every mutation. i18n: newdashboard.adminTenantRuntimeAllowlistnamespace (camelCase leaves, next-intl-safe).
Phase 6 — Conformance · [EW-742 P6]
- T36. ✅ Done in #1480.
runJobRuntimeTenantContractSuitepublished via@ever-works/plugin/contracts-conformancesubpath; parameterised by(tenantA, tenantB); layers cross-tenant isolation + memoisation invariants on top of the base contract. Wired into all 4 non-Trigger plugins via 3-line spec files. - T37. ✅ Done across 3 PRs. Mocks-only matrix — two CI jobs in
.github/workflows/ci.yml:job-runtime-plugin-matrix(4 cells: bullmq/pgboss/temporal/inngest) +secret-store-plugin-matrix(7 cells: vault/k8s/infisical/doppler/aws-sm/gcp-sm/azure-kv). Each cell runspnpm --filter @ever-works/<plugin> testafter building@ever-works/pluginfirst (needed for thecontracts-conformancesubpath).fail-fast: falseso one provider's failure doesn't mask the others. Real-infra portion complete across all 4 providers: BullMQ + pg-boss in #1515 (Redis 7-alpine + Postgres 16-alpine service containers + 6 smoke cases each, gated byEW_TEST_REAL_REDIS_URL/EW_TEST_REAL_PGBOSS_URL); Temporal + Inngest in #1522 (temporalio/auto-setup+ Inngest dev-server containers with the longer health-check windows, gated byEW_TEST_REAL_TEMPORAL_ADDRESS/EW_TEST_REAL_INNGEST_URL); skip-when-env-unset for local devs. Cost gate in #1529 restricts thejob-runtime-real-infrajob to direct pushes tomainonly (PRs + cascade branches skip it — saves CI minutes on every cascade hop). Cross-provider tenant isolation at the resolver contract surface closed out in #1531. - T38. ✅ Done in #1480 (as part of the tenant conformance harness). Verifies v=N+1 returns a DIFFERENT view, and the OLD view's
dispatchersreference stays callable (in-flight runs holding the old view aren't broken by rotation). Real-infra rotation-while-running test is T32 (gated on CI infra). - T39. ✅ Done in #1480 (contract-surface portion). Verifies the cache-eviction semantic: re-binding
(tenantId, v=N)afterv=N+1has been bound returns a FRESH view (cache stays bounded by tenant count, not version count). The full admin endpoint + "in-flight marked FAILED" behaviour is wired in the existing API force-invalidate route from P2.0 (POST /api/account/job-runtime/force-invalidate); the new conformance test pins the provider-side contract that backs it. - T40. ✅ Done in #1480 (contract-surface portion). Verifies cross-tenant isolation at the cache level: tenantA bind does NOT affect tenantB cache;
view.bindToTenant(otherTenant)routes through the base rather than back to self. End-to-end cross-provider isolation (tenant A on pgboss + tenant B on temporal in the same process) is real-infra T32 territory.
Phase 7 — Docs · [EW-742 P7]
T41 + T42 ✅ Done in #1352 (cascaded). T44 ✅ Done in this PR (migration guide writeable now that P3 resolver + P3.1 cache + EW-686 P2 bindToTenant are on main — the drain procedure is a real thing to document). T43 still deferred until per-provider plugin packages exist.
- T41. Tenant admin runbook in
docs/runbooks/TENANT_JOB_RUNTIME.md— how to pick a provider, enter BYO credentials, rotate, force-invalidate, revert to inherit, and read the audit log; cross-linkspec.mdandproviders.md. (Filed underdocs/runbooks/UPPERCASE per the existingEVER_WORKS_ZERO_FRICTION_FLOW.mdconvention.) - T42. Operator runbook in
docs/runbooks/OPERATOR_JOB_RUNTIME_OVERLAY.md— covers allow-list gating, force-invalidate procedure with on-call checklist, operator-driven rollback to inherit, removing a provider from the bundled set, pre-deploy checklist. (Q5 hosting modes deferred to a follow-up edit once P4 worker-host wiring is in place; doc calls this out explicitly so operators don't expect the env var to do anything yet.) - T43. Per-provider tenant-config docs — added a "Tenant overlay (EW-742)" section to each of the 4 new provider plugin READMEs (
packages/plugins/job-runtime-{bullmq,pgboss,temporal,inngest}/README.md, all newly authored in this batch) and to the Trigger.dev arm (packages/tasks/README.md). Each section documentsinherit/byo/overridemodes, the tenant credential bag shape, per-tenant routing pattern (where applicable), and cross-links to./providers.md+ ADR-017 + the conformance suite spec. - T44. Migration guide for existing tenants in
docs/runbooks/TENANT_JOB_RUNTIME_MIGRATION.md— four scenarios (inherit → BYO opt-in, BYO rotate with graceful drain, BYO → inherit rollback, force-invalidate emergency), failure-mode table, verification queries (curl + SQL). Cross-links the canonical tenant + operator runbooks + ADR-017 Q4. (Filed underdocs/runbooks/UPPERCASE per convention; original deferral note assumed the docs needed P4 worker host to be real — P3 resolver + P3.1 cache + EW-686 P2 are enough for the drain procedure to be documented honestly.)
Dependency notes
- T1–T7 (spec-kit) block everything else; this PR ships only Phase 0.
- T8 (
x-scope: tenant) blocks T9–T13 (data model needs the scope token). - T9–T13 (data model) block T14–T19 (admin UI writes through the repository) and T20–T24 (resolver reads from the repository).
- T20–T24 (dispatcher) block T25–T32 (worker host needs the tenant-aware resolver to know which credentials a run belongs to).
- T31 (
tenantIdinJobEnqueueOptions) blocks T25–T30 (all per-tenant routing depends on the metadata field). - T33–T35 (gating) can land in parallel with Phase 4 but must precede T19 going green (e2e validates the operator-gated picker).
- T36–T40 (conformance) trail each provider's worker-host task (T27/T28/T29 for self-host, T25/T26 for SaaS).
- T41–T44 (docs) trail each phase as it lands.