Task Breakdown: Tenant-Scoped Job-Runtime Overlay
Feature ID: tenant-job-runtime-overlay
Status: In progress — P0 + P1.0 + P1 + P2.0 + P2.1 + P3 (T20+T23+T24 resolver) + P3.1 (T21 cache + T22 stamper helper + T22 per-dispatcher wiring across all 10 in-platform dispatchers, with worker-host TenantRuntimeBindingResolverService consumption) + P3.2 (resolver bindToTenant wiring + SecretStoreResolver contract + InProcessSecretStoreResolver default + Trigger.dev bindToTenant minimal impl + non-inline: SecretStoreResolver plugin packages: Vault / K8s / Infisical / Doppler / AWS-SM / GCP-SM / Azure-KV + the 4 non-Trigger IJobRuntimeProvider plugin packages (Temporal / BullMQ / pg-boss / Inngest) with real bindToTenant memoisation AND operator-pluggable real dispatcher + worker-host factories (#1455 BullMQ, #1456 pg-boss, #1457 Temporal, #1458 Inngest)) + P4 (T31 contract — tenantId on JobEnqueueOptions) + P5 + P6 (T37 real-infra matrix complete across all 4 providers — BullMQ+pg-boss in #1515, Temporal+Inngest in #1522; cost-gated to direct main pushes in #1529; cross-provider tenant isolation resolver-level closeout in #1531) + P7 (T41+T42+T44) landed on develop (cascades through develop→stage→main where applicable, develop-only for the late-2026 batches). Platform-side Trigger.dev webhook receiver with per-tenant HMAC verify shipped in #1533 (T25 platform-side prep, receiver-side ready). Dependabot HIGH alerts patched in #1526; 28 of 33 MODERATE patched in #1532. P2.2 (schema-driven form + e2e), P4 (per-provider worker hosts T25–T30 + T31 per-provider stamping + T32 isolation tests), P5.1 (per-tenant whitelist), and P7 (docs T43) remain. T25 reframed (2026-06-21): the original "one Trigger.dev project per tenant" framing was rejected after the 10-project-per-org cap + Trigger.dev's own multi-tenant guidance surfaced. Full per-tenant Trigger.dev isolation is now achieved via the runtime-scoping pattern (single project + concurrencyKey: tenantId + externalId: tenantId + per-tenant credential resolution via the existing bindToTenant + TenantCredentialCache plumbing) across all 3 modes (inherit / byo / override); see T25 below and ./providers.md § Trigger.dev. Operator wiring docs for the 4 plugin packages live in ./providers.md § Operator wiring.
Last updated: 2026-06-21
Spec: ./spec.md · Plan: ./plan.md · Providers: ./providers.md · Epic: EW-742 · Story: EW-743 · ADR: ADR-017
Ordered, granular tasks with explicit paths. Phases map to
plan.md§10. Each phase is a candidate sub-PR. Mark[x]as landed. Suggested Jira child-issue grouping noted per phase (e.g.[EW-742 P1]). Builds on EW-683's instance-level job-runtime contract; nothing here re-opens ADR-017 Q1–Q5 (Temporal namespace-per-tenant, pg-boss schema-per-tenant, Inngest inherit/BYO uniform with Trigger.dev, credential rotation via graceful drain + separateforce-invalidate, hybrid operator-gated platform-defaultshared/per-tenant/tiered).
Phase 0 — Spec-Kit (EW-743) · [EW-743 P0] ✅ Done in #1332 (cascaded #1333, #1334)
- T1. Write ADR-017 at
docs/specs/decisions/017-tenant-scoped-job-runtime-overlay.mdcapturing Q1–Q5 locked decisions and the inherit/BYO overlay rationale on top of EW-683. - T2. Write
docs/specs/features/tenant-job-runtime-overlay/spec.md— behaviour-first user stories (tenant picks provider, operator gates allow-list, graceful drain on rotation, force-invalidate path) with no implementation detail. - T3. Write
docs/specs/features/tenant-job-runtime-overlay/plan.md— architecture, data model, API surface, phased rollout, constitution reconciliation; mirrors structure ofjob-runtime-providers/plan.md. - T4. Write
docs/specs/features/tenant-job-runtime-overlay/tasks.md(this file) — numbered T1..TN per phase with file paths. - T5. Write
docs/specs/features/tenant-job-runtime-overlay/providers.md— per-provider tenant-isolation matrix (Temporal namespace, pg-boss schema, BullMQ prefix, Trigger.dev project, Inngest app) and inherit-vs-BYO knob per provider. - T6. Cross-link from
docs/specs/architecture/job-runtime-providers.mdto the tenant-overlay docs by adding a "Tenant-scoped overlay" see-also section pointing atspec.md,plan.md, and ADR-017. - T7. Verify constitution gates against
.specify/memory/constitution.md— confirm Principle IV ("via the configured job-runtime provider") already covers tenant overlay; no new amendment required, document the verification inplan.md§Constitution Reconciliation.
Phase 1 — Data model + migration · [EW-742 P1] ✅ Done in #1338 (cascaded #1339, #1340). P1.0 (x-scope: tenant) shipped in #1335 (cascaded #1336, #1337).
- T8. Audit
docs/specs/architecture/settings-system.mdforx-scope: tenant; if missing, add it to the scope taxonomy alongsidex-scope: global(verify in worktree before writing — this is a prerequisite for tenant-scoped credentials). - T9. Define TypeORM entity
TenantJobRuntimeConfiginpackages/agent/src/entities/tenant-job-runtime-config.entity.ts— columns:tenantId(PK),providerId,mode(inherit|byo|override),credentialsSecretRef,credentialVersion,enabled,createdBy,createdAt,updatedAt. (Entity moved to@ever-works/agent/entitiesper EW-654/655 cycle-avoidance — no@ManyToOneback-refs, FK enforced only at the DB layer.) - T10. Write TypeORM migration
apps/api/src/migrations/1780900000000-AddTenantJobRuntimeConfig.tsfor the new table plus partial index on(providerId) WHERE enabled(PK already coverstenantId); forward-only + idempotenthasTableguards perdocs/specs/architecture/database-migrations.md. - T11. Implement
CredentialVersionServicefor graceful drain (Q4) inpackages/agent/src/tasks/credential-version.service.ts— issues monotonic per-tenant version ids and resolves a snapshot for a given(tenantId, version)tuple so in-flight runs keep their original credentials. Full snapshot history table shipped in #1514:TenantCredentialSnapshotentity + migration +captureSnapshot()on every version bump +resolveSnapshot(tenantId, version)reads the row (FIFO history, idempotent on conflict). Bag re-issued asinline:<base64>so the existing secret-store resolver chain dereferences without a new scheme. 12 vitest history cases; 6091/6091 packages/agent green. - T12. Add audit-log table + entity for tenant runtime config changes in
packages/agent/src/entities/tenant-job-runtime-audit.entity.tsplus matching migrationapps/api/src/migrations/1781000000000-AddTenantJobRuntimeAudit.ts; records(tenantId, actorUserId, action, before, after, credentialVersion, occurredAt). - T13. Repository + unit tests under
apps/api/src/works/__tests__/tenant-job-runtime-config.repository.spec.tscovering insert, update, version bump, audit-row emission, and tenant isolation (one tenant cannot read another's row).
Phase 2 — Admin UI · [EW-742 P2]
P2.0 (REST API) ✅ Done in #1341 (cascaded #1342, #1343). P2.1 (UI) ✅ Done in #1347 (cascaded #1348, #1349). P2.2 (schema-driven form + e2e) remains.
- T14. API endpoints
GET /api/account/job-runtime/configandPUT /api/account/job-runtime/configinapps/api/src/account/tenant-job-runtime/tenant-job-runtime.controller.ts— guarded byrequireTenant()(1 User : 1 Tenant model, no:tenantIdpath param), returns redacted credentials on GET, accepts{ providerId, mode, credentials }on PUT and writes through the service + audit log. Also added:POST rotate,POST force-invalidate,DELETE config(revert to inherit). - T15. Tenant settings page in
apps/web/src/app/[locale]/(dashboard)/settings/job-runtime/page.tsx— server component that loads current config and renders the picker + credentials form. - T16. Provider picker component in
apps/web/src/components/settings/JobRuntimeSettings.tsx—Selectof operator-enabled providers (P5 wired viaavailableProvidersprop) withinheritvsBYOvsoverridemode toggle. (Filed undercomponents/settings/notcomponents/job-runtime/per the dashboard's existing settings layout.) - T17. ✅ Done in #1486. Ships
apps/web/src/components/settings/JobRuntimeCredentialsForm.tsx(alongside the settings sibling pattern, not undercomponents/job-runtime/— matches the existingJobRuntimeSettings.tsxlocation). The form reads fromjob-runtime-schemas.ts(a UI projection of each plugin'ssettingsSchema); each field renders with lock icon on secrets, reveal/hide eye toggle, env-var hint chip (BULLMQ_REDIS_URL etc.), helper text, multiline support for PEM-encoded mTLS cert/key pairs, required-fields validation, and a per-provider info banner for Trigger.dev (no tenant-form credentials). Plus a side fix toplugin-category-icons.tscovering'job-runtime'(Cog) +'secret-store-resolver'(KeyRound) so apps/web type-check is green at HEAD. - T18. Validation + error surfacing — "provider disabled by operator" surfaces via the P5 allow-list
BadRequestException+ warning banner; "credentials rejected by provider reachability probe" deferred to P4 (the probe itself doesn't exist yet); "force-invalidate in progress" surfaces via the controller's rate-limit response + button-disabled state. - T19. ✅ Done in #1486. Ships
apps/web/e2e/tenant-job-runtime.spec.tscovering: page mounts without 5xx + provider picker visible; mode toggle to byo reveals creds, inherit hides them; switching provider changes the field set (proves schema-driven rendering, not opaque); secret fields default totype="password"(no plaintext DOM leak). End-to-end submit-then-verify-audit-row is layered on once a deterministic seed fixture lands.
Phase 3 — Dispatcher routing · [EW-742 P3]
T20 + T23 + T24 ✅ Done in #1380 (EW-747 — minimal viable resolver, byo/override modes return instance default with Logger.debug deferral note until per-provider credential-binding API lands). T21 (cache) ✅ shipped in #1381. T22 (stamper helper) ✅ shipped in this PR; per-dispatcher wiring still deferred to incremental follow-ups.
-
T20. Tenant-aware resolver at
packages/agent/src/tasks/tenant-aware-runtime.resolver.ts(TenantAwareRuntimeResolver) wraps the EW-685 P0 T4 binding-factory registry:resolve(tenantId)returns the instance default fornull/ no-row / inherit / disabled; forbyo+override+enabledit logs the deferral and still returns the instance default until EW-686 P2 lands the per-provider credential-binding hook.getEffectiveBinding(tenantId)returns the metadata T22 will stamp onto the run record. -
T21. Credential cache with 15–60s TTL in
packages/agent/src/tasks/tenant-credential.cache.ts— keyed by(tenantId, providerId, credentialVersion), in-process LRU with explicit invalidate on version bump or force-invalidate. Standalone@Injectable()class with no DI dependencies (defaults:maxEntries=1024,ttlMs=30_000); insertion-order LRU (no promotion-on-read so an in-flight runs snapshot does not get kept alive past its rotation window — see ADR-017 §3 / Q4). Provided + exported viaTenantJobRuntimeModule. PR: #1381. -
T22 (helper).
RuntimeBindingStamperServiceatpackages/agent/src/tasks/runtime-binding-stamper.service.ts— standalone@Injectable()whosestamp(tenantId)returns{ providerId, credentialVersion }for byo/override+enabled tenants and{ null, null }for inherit/no-row/disabled/no-tenant (fail-open on DB hiccup withLogger.warn). Provided + exported viaTenantJobRuntimeModule. The dispatcher call sites adopt this helper incrementally — each enqueue payload type adds acredentialVersion?: numberfield on its own PR, no big-bang bus stop. The actual per-call-site wiring (theawait stamper.stamp(tenantId)inside each of the twelve dispatchers + the run-record schema column it writes into) remains open below. -
T22 (per-dispatcher wiring). Walk the dispatcher call sites in
packages/agent/src/tasks/_tasks-symbols.ts(plusKB_REEMBED_WORK_DISPATCHERwired inapps/api/src/works/works.module.tsandWEBHOOK_DELIVERY_DISPATCHERinapps/api/src/webhooks/), callawait stamper.stamp(tenantId)inside each enqueue path, and persist(providerId, credentialVersion)on the payload so the P4 worker host can later resolve the pinned snapshot viaCredentialVersionService.resolveSnapshot. Each dispatcher migration shipped as its own PR — no coordinated bus stop.10 stamped + consumed dispatchers (producer ↔ task):
# Dispatcher Resolver helper Producer PR Worker PR 1 KB_EMBED_DOCUMENT_DISPATCHERresolveForWork#1427 (PoC) #1435 2 KB_MIRROR_DOCUMENT_DISPATCHERresolveForWork#1430 (Tier 1) #1439 3 KB_NORMALIZE_MEDIA_DISPATCHER(video+audio)resolveForWork#1430 (Tier 1) #1439 4 KB_TRANSCRIBE_DISPATCHERresolveForWork#1436 #1439 5 WORK_GENERATION_DISPATCHERresolveForWork#1431 (Tier 2) #1439 6 WORK_IMPORT_DISPATCHERresolveForWork#1431 (Tier 2) #1439 7 KB_ORG_OVERLAY_FANOUT_DISPATCHERresolveForOrganization#1430 (Tier 1) #1442 8 TEMPLATE_CUSTOMIZATION_DISPATCHERresolveForCustomization#1431 (Tier 2) #1442 9 WEBHOOK_DELIVERY_DISPATCHER(apps/api)resolveForSubscription#1445 #1445 10 KB_REEMBED_WORK_DISPATCHER(pgvector→apps/api)resolveForWork#1448 #1448 Worker-host infrastructure shipped in #1432:
TenantRuntimeBindingResolverServiceatpackages/tasks/src/trigger/worker/services/tenant-runtime-binding-resolver.service.tsexposesresolve(payload, tenantId)+ 4 convenience wrappers (resolveForWork/Organization/Customization/Subscription) returning a 4-state status (no-binding/resolved/drained/error, fail-open per FR-5).CredentialVersionServiceproxied throughTriggerInternalController.remoteMapso worker tasks can callresolveSnapshotvia the existing RPC channel. The webhook-delivery task uses direct providers inTriggerWebhookDeliveryModule(avoids a pointless RPC hop since that module already has direct DB access).Intentionally out-of-scope dispatchers (do NOT need T22 stamping):
KB_BACKFILL_SKELETON_DISPATCHER— fleet-wide operator bootstrap script. Runs as the instance-default credentials by design (no tenant scope; the caller enumeratesworkIdsthat may span tenants). Adding T22 stamping here would be semantically incorrect.
Per-tenant credential application (deferred to a future PR with real BYO infra): the resolved
snapshot.credentialsis logged + threaded through the worker but the Trigger.dev SDK is still configured globally at boot. Per-tenant Trigger.dev BYO project switching (callingconfigure({ accessToken: snapshot.credentials.accessToken })per dispatch) needs an actual operator with per-tenant Trigger.dev projects to validate; building speculatively risks getting the design wrong. Today the'resolved'status is observability-only — the worker still runs against the instance default credentials. -
Router consumer (2026-09-25,
05e4b0236; bounds 2026-09-26,68ffb9a97).PluginExecutionRouterService(dynamic-plugin-distribution T26) is the first production caller ofTenantAwareRuntimeResolver.resolve: a long-running plugin operation with atenantIddispatches and reads through the tenant's bound view, and its payload carries the stamper's(providerId, credentialVersion). Fixes that came with it:TenantJobRuntimeModuleno longer binds its ownJOB_RUNTIME_PROVIDER_REGISTRY. The local, never-registered in-memory registry shadowed the@GlobalTriggerModuleone, soresolve()answerednullfor every tenant.apps/api/src/account/tenant-job-runtime/tenant-job-runtime.module.spec.tspins this.- A BYO bound view's stamping Proxy no longer uses the frozen BYO dispatcher map as its target. Before, every BYO dispatch through the view threw.
Known limits:
- The resolver binds the ACTIVE provider whatever the row's
providerIdsays, so an overlay naming node or bullmq still sends plugin operations to Trigger.dev, the only runtime that implements them. - BYO runs are untagged. A BYO tenant's plugin-operation run goes to the tenant's own Trigger.dev project but carries no
tenant:<id>tag and no tenant concurrency key: the view sets the stamp, but the BYO dispatcher map never reads it. Only inherit-shaped (singleton) views add them, throughTriggerService.stampTenantOptions. - Start and poll can land on different projects. The start and each poll or cancel resolve the tenant separately. One can land on the platform while another reaches the tenant's BYO view when the resolver threw and the router failed open; when the resolver fell back on its own without throwing (the secret store threw or answered nothing, or
bindToTenantthrew or refused the snapshot; none of these is cached, so a brief secret-store miss at the start followed by a good bind at the next poll is the likeliest path); or when the overlay row changed in between (including a credential rotation). The run then readsunknownon every poll and a wait ends asJOB_RUNTIME_RUN_UNREADABLE; it is never reported failed and never dispatched twice. Returning which view served the start next to therunIdwould close it. Graceful-drain reads throughCredentialVersionService.resolveSnapshotare not wired on this path. - The stamp is the row's, not the provider's. The FR-5 stamp (
providerId,credentialVersion) records the overlay row's values, not the provider that actually ran. After a fail-open to the platform, or with an overlay naming node or bullmq while the active runtime is Trigger.dev, it is wrong. The worker ignores it today; a future credential-rotation drain must check it before relying on it. - Start and abort limits.
startLongRunningbounds the tenant lookup plus the stamp to about 20 s (the stamp always gets at least 1 s). A lookup timeout isJOB_RUNTIME_DISPATCH_FAILEDwith nothing dispatched; a stamp timeout dispatches unstamped.dispatchLongRunning(background callers) does not bound the lookup; an abort before or during it dispatches nothing (JOB_RUNTIME_WAIT_ABORTED, norunId), and an abort during or after the dispatch keeps therunId, whichManagedAgentSandboxRunnerService.run()cancels by. - A resolver error sends a BYO tenant's run to the platform project; that is the resolver's documented fail-open behaviour.
- Follow-up, not fixed (a behaviour change): in
packages/tasks/src/trigger/trigger-job-runtime.provider.ts,bindToTenantcaches the view even whensafeBuildTenantClientfailed and fell back to the platform, and the resolver then caches it as a successful bind. So a BYO tenant whose client failed to build once stays on the platform project until its credential version changes. Start and poll agree, so this is not the mismatch above, but the tenant runs in the wrong project for a long time. A fix would skiptenantViews.setwhen a BYO snapshot fell back to the platform.
-
T23. Fallback path to instance-global default when a tenant has no overlay row — resolver returns the EW-683 instance binding unchanged; the
getActive() = nullsemantic propagates asnull(in-process dev fallback preserved). Tests in T24 prove the zero-overhead path for tenants that never opt in. -
T24. Unit tests for the resolver under
packages/agent/src/tasks/__tests__/tenant-aware-runtime.resolver.spec.ts— 12 cases covering inherit fallback, no-row fallback, kill switch, byo/override stopgap + log emission,getEffectiveBindingmetadata, and thegetActive() = nullpropagation.
Phase 3.1 — Enqueue capture · [EW-742 P3.1] ✅ DONE
- T22 (above) — per-dispatcher wiring shipped across PRs #1427 / #1430 / #1431 / #1436 / #1442 / #1445 / #1448 (producer side) and #1432 / #1435 / #1439 / #1442 / #1445 / #1448 (worker side). The T21 cache is in place to serve the snapshot lookup at run time; today the worker-host resolver consumes
(providerId, credentialVersion)from the payload and classifies asno-binding/resolved/drained/error(skip-and-ack on drained, fall-through otherwise).
Phase 3.2 — Resolver bindToTenant wiring · [EW-742 P3.2]
✅ Done in this PR. The resolver's byo/override branch now actively binds the active provider to per-tenant credentials:
- T20.1.
SecretStoreResolvercontract atpackages/agent/src/tasks/secret-store-resolver.interface.ts— operator-pluggable credential-pointer resolution.SECRET_STORE_RESOLVERDI token +resolve(pointer): Promise<Record<string, unknown> | null>interface. Contract is fail-open: implementations MUST NOT throw on missing / malformed pointers — returnnull+ log at warn instead. - T20.2.
InProcessSecretStoreResolverdefault atpackages/agent/src/tasks/in-process-secret-store-resolver.service.ts— supports onlyinline:<base64-of-json-object>for dev/test. Every other scheme (vault:,k8s:,op:) returnsnull+Logger.warntelling the operator to wire a real implementation for that scheme. - T20.3.
TenantAwareRuntimeResolver.resolve()byo/override branch now does the full bind path: (a) checkTenantCredentialCachekeyed by(tenantId, providerId, credentialVersion); (b) on miss, resolve credentials viaSecretStoreResolver; (c) callprovider.bindToTenant(snapshot)(EW-686 P2 contract); (d) cache + return the bound provider. Any null/undefined/throw anywhere in the chain falls back to the instance default with aLogger.warncarrying enough tenant context for operators to diagnose. 20+ tests cover happy path + 5 fallback paths + cache-hit behaviour. - T20.4a.
VaultSecretStoreResolveratpackages/agent/src/tasks/vault-secret-store-resolver.service.ts— HTTP-based HashiCorp Vault implementation. Supportsvault:<path>pointers, KV v1 + v2 envelope auto-detection, reads VAULT_ADDR + VAULT_TOKEN from env. Operators opt in by overriding theSECRET_STORE_RESOLVERDI binding. Default deployment unchanged. - T20.4b.
K8sSecretStoreResolveratpackages/agent/src/tasks/k8s-secret-store-resolver.service.ts— in-cluster API. Reads bearer token + CA cert + default namespace from/var/run/secrets/kubernetes.io/serviceaccount/{token,ca.crt,namespace}. Supportsk8s:<name>(default namespace) andk8s:<ns>/<name>(explicit). Base64-decodes each.data[key]to UTF-8. Opt-in via DI binding override. - T20.4c.
env:scheme added toInProcessSecretStoreResolver— reads JSON-encoded credential bag fromprocess.env[<VAR_NAME>]. Self-hoster path for operators who already inject credentials via the standard 12-factor env-var pattern. Zero-dep, ships in the default resolver alongsideinline:. (Replaces the dropped 1Password /op://plan — that vendor lock-in was a poor fit for an OSS project.) - T20.4d.
InfisicalSecretStoreResolveratpackages/agent/src/tasks/infisical-secret-store-resolver.service.ts— HTTP-based against Infisical REST API. Supportsinfisical:<workspaceId>/<env>/<secretPath>; fetches a folder of secrets and returns them as a flat credential bag. Reads INFISICAL_TOKEN + optional INFISICAL_HOST (defaults toapp.infisical.com) from env at every resolve call (rotation-friendly). Opt-in via DI binding override. - T20.4e.
DopplerSecretStoreResolveratpackages/agent/src/tasks/doppler-secret-store-resolver.service.ts— HTTP-based against Doppler REST API (/v3/configs/config/secrets). Supportsdoppler:<project>/<config>; fetches all secrets in the config and returns them as a flat bag (prefers.raw, falls back to.computed). Reads DOPPLER_TOKEN from env at every resolve call (rotation-friendly). Opt-in via DI binding override. - T20.5 (foundation).
secret-store-resolverplugin category +ISecretStoreProvidercapability contract atpackages/plugin/src/contracts/capabilities/secret-store.interface.ts. Adds the category toPLUGIN_CATEGORIES, the plugin-shapedISecretStoreProvider extends IPlugininterface, and theSECRET_STORE_CAPABILITIES.RESOLVEcapability string. Gates the migration of Vault / K8s / Infisical / Doppler from@Injectable()services inpackages/agent/src/tasks/into truepackages/plugins/secret-store-<vendor>/plugin packages (T20.6+). - T20.6.
VaultSecretStorePluginatpackages/plugins/secret-store-vault/— true plugin package with everworks.plugin manifest (category:secret-store-resolver, capabilities:[secret-store-resolve]), tsup build, vitest tests (13/13 pass). ImplementsIPlugin + ISecretStoreProvider. Old@Injectable()class atpackages/agent/src/tasks/vault-secret-store-resolver.service.tsdeleted (no consumer was registering it). - T20.7.
K8sSecretStorePluginatpackages/plugins/secret-store-k8s/— true plugin package; old@Injectable()class deleted frompackages/agent/src/tasks/. 13/13 plugin vitest pass. - T20.8.
InfisicalSecretStorePluginatpackages/plugins/secret-store-infisical/— true plugin package; old@Injectable()class deleted. 12/12 plugin vitest pass. - T20.9.
DopplerSecretStorePluginatpackages/plugins/secret-store-doppler/— true plugin package; old@Injectable()class deleted. 13/13 plugin vitest pass. - T20.10. New cloud-vendor plugin packages:
secret-store-aws-sm,secret-store-gcp-sm,secret-store-azure-kv— all shipped as plugin packages and now run the sharedISecretStoreProviderconformance suite (#1471).
Phase 4 — Worker host · [EW-742 P4]
-
T25 (reframed 2026-06-21). Per-tenant Trigger.dev isolation — per-tenant Trigger.dev project routing is NOT pursued. Trigger.dev hard-caps 10 projects per organization across all tiers, and Trigger.dev's own multi-tenant guidance explicitly calls the per-tenant-project model an anti-pattern. The original "platform auto-provisions one Trigger.dev project per tenant via the management API" framing was therefore wrong at the premise level — replacing it with the vendor-recommended runtime-scoping pattern:
- Single project per Trigger.dev account (the operator's for
inherit, the tenant's forbyo/override). - Per-call routing via
concurrencyKey: tenantId+externalId: tenantIdon everytasks.trigger(...)so each tenant gets its own queue + per-tenant concurrency budget inside the shared project. For dynamic schedules, the sameexternalId: tenantIdpattern applies per Trigger.dev's per-tenant schedules guide. - Per-tenant credential resolution through the existing P3.2 plumbing:
TenantAwareRuntimeResolver→SecretStoreResolver→provider.bindToTenant(snapshot)→TenantCredentialCache. The Trigger.dev plugin'sbindToTenantalready swapsaccessToken/secretKey/projectRef/apiUrlper the resolved snapshot (EW-742 P3.2 T21.1) sobyo/overridemode wires up without any per-tenant-project ceremony.
Implementation is split across follow-up PRs:
- Runtime stamping — emit
concurrencyKey: tenantId+externalId: tenantIdon everytasks.trigger(...)/tasks.batchTrigger(...)call site. (Tracked under thesession/1516-trigger-tenant-runtime-stampingbranch.) - BYO credentials end-to-end — wire
accessToken+secretKey+projectRef+ optionalapiUrlfrom the tenant settings form through the secret store intoTriggerJobRuntimePlugin.bindToTenant. (Tracked under thesession/1516-trigger-tenant-byo-credentialsbranch.)
The platform-side Trigger.dev webhook receiver with per-tenant HMAC verify (lookup via
TenantJobRuntimeConfig.credentials.webhookSecret) shipped in #1533 and remains the receive-side surface. See./providers.md§ Trigger.dev for the full mode matrix. - Single project per Trigger.dev account (the operator's for
-
T26. ✅ Done in #1492. Ships
tenantAwareInngestFunctionHandlerHOF wrapper inpackages/plugins/job-runtime-inngest/src/inngest-tenant-aware-handler.ts— readsevent.data._ew.tenantId(T31 namespace), resolves snapshot via optionalresolveSnapshot(or synthetic fallback), binds viaplugin.bindToTenant, hands the binding as 2nd arg to the operator handler. Fail-open per FR-5: missing tenantId → handler receives plugin itself as binding. 7 tenant-isolation tests. -
T27. ✅ Done in #1492. Ships
TenantAwareTemporalWorkerHostFactoryinpackages/plugins/job-runtime-temporal/src/temporal-tenant-aware-worker-host-factory.tsas a registry mapper — one Worker per tenant. Operator'sbuild(binding)callback receives the boundIJobRuntimeProviderview so it can configureWorker.create({ connection, namespace })against the right namespace per ADR-017 Q1.start()materialises all workers + runs concurrently;stop()shuts down all + awaits run promises. 6 tenant-isolation tests. -
T28. ✅ Done in #1492. Ships
TenantAwareBullMqWorkerHostFactoryinpackages/plugins/job-runtime-bullmq/src/bullmq-tenant-aware-worker-host-factory.ts— multiplexing worker that readsjob.opts.tenantId(the T31 stamping field, withjob.data._ew.tenantIdfallback), binds viaplugin.bindToTenant, invokes handler with the binding. Per-tenant Redis prefix isolation is operator-side atdispatchersBuildertime (one BullMqDispatcherFactory per prefix). 6 tenant-isolation tests. -
T29. ✅ Done in #1492. Ships
TenantAwarePgBossWorkerHostFactoryinpackages/plugins/job-runtime-pgboss/src/pgboss-tenant-aware-worker-host-factory.ts— readsjob.data._ew.tenantId(T31 namespace) per job (batched workers iterate per-job and route each independently), binds viaplugin.bindToTenant, invokes handler with the binding. Per-tenant Postgres schema isolation (ADR-017 Q2) is operator-side atdispatchersBuildertime (onePgBossinstance per schema). 6 tenant-isolation tests. -
T30. ✅ Done in #1492 — the multiplexing mode is the default behaviour of each
TenantAware*WorkerHostFactory: one worker process per provider routes all tenants via the per-job_ew.tenantIdmetadata (BullMQ/pg-boss/Inngest) or one-Worker-per-tenant registry (Temporal, where per-namespace connection is required). TheEVER_WORKS_JOB_RUNTIME_HOSTING={per-tenant|shared|tiered}env-var selector remains operator-side wiring — operators pickshared(multiplexing via TenantAware*) vsper-tenant(separate worker process per tenant) when launching their worker apps. -
T31 (contract).
JobEnqueueOptions.tenantId?: stringadded inpackages/plugin/src/contracts/capabilities/job-runtime.interface.tswith per-provider routing semantic in the JSDoc (Triggermetadata, Inngestdata, TemporalsearchAttributes, BullMQopts.tenantId, pg-boss payload field). 11/11 type tests pass. Per-provider stamping is still TODO — each provider's dispatcher implementation maps the field onto its native carrier on its own PR (incremental adoption, same seam pattern as T22). -
T31 (per-provider stamping). ✅ Done across 4 PRs (#1474 BullMQ, #1475 pg-boss, #1476 Temporal, #1477 Inngest). Each provider's factory adds a
mapEnqueueOptions(opts: JobEnqueueOptions)translator +.enqueue(name, payload/args, enqueueOpts, extraOpts?)convenience method that maps platform fields onto the native carrier perproviders.md:idempotencyKey→jobId/singletonKey/workflowId/event.id;tenantId→native field or_ewnamespace;concurrencyKey/tags/maxDurationSeconds/machineHintper provider. Trigger.dev already doesmetadata.tenantIdvia the bindToTenant impl shipped in EW-742 P3.2 T21.1. -
T32. ✅ Done in #1492 for the mocks-only surface. Each of the 4 non-Trigger plugins ships
__tests__/<provider>-tenant-isolation.spec.ts(6-7 cases each) verifying no cross-tenant binding leakage, route-A-distinct-from-route-B, concurrent A+B no cross-contamination, missing-tenantId fallback to platform default, and resolveSnapshot/bindToTenant memoisation. Real-infra integration tests (live Redis/Postgres/Temporal/Inngest) remain gated on CI infra availability.
Phase 5 — Plugin gating · [EW-742 P5] ✅ Done in #1350 (pending cascade)
- T33. Instance plugin allow-list config in
apps/api/src/config/constants.ts(underconfig.tenantJobRuntime.getAllowedProviders()) — operator-controlled list of bundledjob-runtimeprovider ids exposed to tenants via theEVER_WORKS_TENANT_RUNTIME_ALLOWED_PROVIDERSenv var (comma-separated). Empty/unset = all 5 bundled providers (fail-open). Q5 platform-default mode (shared/per-tenant/tiered) is a P4 worker-host concern; this gate only controls picker visibility. - T34. Picker filter in tenant admin UI —
apps/web/src/components/settings/JobRuntimeSettings.tsxonly renders providers that appear in the operator allow-list resolved from the newGET /api/account/job-runtime/available-providersendpoint. Warning banner shows when the tenant's currently-savedproviderIdis no longer in the list. - T35. Audit decoration on every overlay mutation —
emitAuditdecoratesbefore/aftersnapshots withoperatorAllowedProviders: string[]captured at write time so future audit-log reads can correlate a mutation with the active allow-list. (Per-tenantaction='operator_allowlist_change'startup row is a P5.1 follow-up.)
Phase 5.1 — Per-tenant whitelist (deferred)
- T35a. ✅ Done in #1501. Ships
TenantRuntimeProviderAllowlistentity (composite PK(tenantId, providerId)+ createdBy + createdAt) +AddTenantRuntimeProviderAllowlistmigration (FKs to tenants/users) +OperatorTenantRuntimeAllowlistController(GET/PUT/DELETE under/api/operator/tenants/:tenantId/runtime-allowlist, gated byIsPlatformAdminGuard) +tenantJobRuntime.isPerTenantGatingEnabled()config knob (default OFF — when off the table is ignored and behaviour is byte-identical to today). Service:listTenantAllowlist/replaceTenantAllowlist(atomic txn) /deleteTenantAllowlistEntry(each mutation emitsoperator_allowlist_changeaudit row) +getAvailableProvidersForTenant(tenantId)— 4-way resolver: flag OFF → global; flag ON + zero rows → global (inherit); flag ON + rows ⊆ global → intersection in global's order; flag ON + rows ⊄ global → intersection silently drops unknown (global = upper bound). - T35b. ✅ Done in #1501. Ships
TenantJobRuntimeBootAuditService(OnApplicationBootstrap) that hashes the effective allow-list + gating flag, dedups by hash, writes oneoperator_allowlist_bootrow per real change (five pods restarting on same config write ONE row, not five).RelaxTenantJobRuntimeAuditTenantNullablemigration relaxestenantIdto nullable so boot rows writetenantId=NULL. Bootstrap failures swallowed + logged so a transient DB hiccup at boot doesn't fail liveness; next pod restart retries. - T35c. ✅ Done in #1516. Operator UI for the per-tenant allow-list shipped at
/admin/tenants/<tenantId>/runtime-allowlist: server component (apps/web/src/app/[locale]/(dashboard)/admin/tenants/[tenantId]/runtime-allowlist/page.tsx) gated byauthAPI.getProfile()→notFound()when!isPlatformAdmin(defense-in-depth on top of the API guard, same pattern asadmin/usage/page.tsx); client componentTenantRuntimeAllowlistManager.tsxwith checkbox picker over the 5 runtime providers, status banner reflecting the config knob (isPerTenantGatingEnabled), delete chips per row;apps/web/src/lib/api/operator-tenant-runtime-allowlist.tsfetch client +apps/web/src/app/actions/admin/tenant-runtime-allowlist.tsserver actions thatrevalidatePathon every mutation. i18n: newdashboard.adminTenantRuntimeAllowlistnamespace (camelCase leaves, next-intl-safe).
Phase 6 — Conformance · [EW-742 P6]
- T36. ✅ Done in #1480.
runJobRuntimeTenantContractSuitepublished via@ever-works/plugin/contracts-conformancesubpath; parameterised by(tenantA, tenantB); layers cross-tenant isolation + memoisation invariants on top of the base contract. Wired into all 4 non-Trigger plugins via 3-line spec files. - T37. ✅ Done across 3 PRs. Mocks-only matrix — two CI jobs in
.github/workflows/ci.yml:job-runtime-plugin-matrix(4 cells: bullmq/pgboss/temporal/inngest) +secret-store-plugin-matrix(7 cells: vault/k8s/infisical/doppler/aws-sm/gcp-sm/azure-kv). Each cell runspnpm --filter @ever-works/<plugin> testafter building@ever-works/pluginfirst (needed for thecontracts-conformancesubpath).fail-fast: falseso one provider's failure doesn't mask the others. Real-infra portion complete across all 4 providers: BullMQ + pg-boss in #1515 (Redis 7-alpine + Postgres 16-alpine service containers + 6 smoke cases each, gated byEW_TEST_REAL_REDIS_URL/EW_TEST_REAL_PGBOSS_URL); Temporal + Inngest in #1522 (temporalio/auto-setup+ Inngest dev-server containers with the longer health-check windows, gated byEW_TEST_REAL_TEMPORAL_ADDRESS/EW_TEST_REAL_INNGEST_URL); skip-when-env-unset for local devs. Cost gate in #1529 restricts thejob-runtime-real-infrajob to direct pushes tomainonly (PRs + cascade branches skip it — saves CI minutes on every cascade hop). Cross-provider tenant isolation at the resolver contract surface closed out in #1531. - T38. ✅ Done in #1480 (as part of the tenant conformance harness). Verifies v=N+1 returns a DIFFERENT view, and the OLD view's
dispatchersreference stays callable (in-flight runs holding the old view aren't broken by rotation). Real-infra rotation-while-running test is T32 (gated on CI infra). - T39. ✅ Done in #1480 (contract-surface portion). Verifies the cache-eviction semantic: re-binding
(tenantId, v=N)afterv=N+1has been bound returns a FRESH view (cache stays bounded by tenant count, not version count). The full admin endpoint + "in-flight marked FAILED" behaviour is wired in the existing API force-invalidate route from P2.0 (POST /api/account/job-runtime/force-invalidate); the new conformance test pins the provider-side contract that backs it. - T40. ✅ Done in #1480 (contract-surface portion). Verifies cross-tenant isolation at the cache level: tenantA bind does NOT affect tenantB cache;
view.bindToTenant(otherTenant)routes through the base rather than back to self. End-to-end cross-provider isolation (tenant A on pgboss + tenant B on temporal in the same process) is real-infra T32 territory.
Phase 7 — Docs · [EW-742 P7]
T41 + T42 ✅ Done in #1352 (cascaded). T44 ✅ Done in this PR (migration guide writeable now that P3 resolver + P3.1 cache + EW-686 P2 bindToTenant are on main — the drain procedure is a real thing to document). T43 still deferred until per-provider plugin packages exist.
- T41. Tenant admin runbook in
docs/runbooks/TENANT_JOB_RUNTIME.md— how to pick a provider, enter BYO credentials, rotate, force-invalidate, revert to inherit, and read the audit log; cross-linkspec.mdandproviders.md. (Filed underdocs/runbooks/UPPERCASE per the existingEVER_WORKS_ZERO_FRICTION_FLOW.mdconvention.) - T42. Operator runbook in
docs/runbooks/OPERATOR_JOB_RUNTIME_OVERLAY.md— covers allow-list gating, force-invalidate procedure with on-call checklist, operator-driven rollback to inherit, removing a provider from the bundled set, pre-deploy checklist. (Q5 hosting modes deferred to a follow-up edit once P4 worker-host wiring is in place; doc calls this out explicitly so operators don't expect the env var to do anything yet.) - T43. Per-provider tenant-config docs — added a "Tenant overlay (EW-742)" section to each of the 4 new provider plugin READMEs (
packages/plugins/job-runtime-{bullmq,pgboss,temporal,inngest}/README.md, all newly authored in this batch) and to the Trigger.dev arm (packages/tasks/README.md). Each section documentsinherit/byo/overridemodes, the tenant credential bag shape, per-tenant routing pattern (where applicable), and cross-links to./providers.md+ ADR-017 + the conformance suite spec. - T44. Migration guide for existing tenants in
docs/runbooks/TENANT_JOB_RUNTIME_MIGRATION.md— four scenarios (inherit → BYO opt-in, BYO rotate with graceful drain, BYO → inherit rollback, force-invalidate emergency), failure-mode table, verification queries (curl + SQL). Cross-links the canonical tenant + operator runbooks + ADR-017 Q4. (Filed underdocs/runbooks/UPPERCASE per convention; original deferral note assumed the docs needed P4 worker host to be real — P3 resolver + P3.1 cache + EW-686 P2 are enough for the drain procedure to be documented honestly.)
Dependency notes
- T1–T7 (spec-kit) block everything else; this PR ships only Phase 0.
- T8 (
x-scope: tenant) blocks T9–T13 (data model needs the scope token). - T9–T13 (data model) block T14–T19 (admin UI writes through the repository) and T20–T24 (resolver reads from the repository).
- T20–T24 (dispatcher) block T25–T32 (worker host needs the tenant-aware resolver to know which credentials a run belongs to).
- T31 (
tenantIdinJobEnqueueOptions) blocks T25–T30 (all per-tenant routing depends on the metadata field). - T33–T35 (gating) can land in parallel with Phase 4 but must precede T19 going green (e2e validates the operator-gated picker).
- T36–T40 (conformance) trail each provider's worker-host task (T27/T28/T29 for self-host, T25/T26 for SaaS).
- T41–T44 (docs) trail each phase as it lands.