Implementation Plan: Tenant Job-Runtime Overlay
Translates
spec.mdinto architecture and tech choices for tenant-scoped overlays atop the instance-global job-runtime selection of EW-683. Behaviour lives in the spec; this owns the "how." Per-provider research stays in the parent feature'sproviders.md. Rationale: ADR-017. Deep architecture (the seam this extends):architecture/job-runtime-providers.md.
Feature ID: tenant-job-runtime-overlay
Spec: ./spec.md
Tasks: ./tasks.md
Status: Draft
Last updated: 2026-06-18
Epic: EW-742 · Spec-kit story: EW-743 · Parent (instance-global runtime): EW-683 · Dispatcher seam: EW-685 · Worker host: EW-686
1. Architecture Summary
The instance-global seam built in EW-683 / EW-685 stays load-bearing. This feature inserts a TenantAwareRuntimeResolver in front of the existing binding factory: on every dispatch the resolver consults tenant_job_runtime_config for the tenant in scope; if a row exists and is enabled, its provider+credentials win; otherwise the call falls through to the instance-global provider with mode = inherit. Credential identity (credential_version) is captured into WorkGenerationHistory at enqueue so an in-flight run keeps using the credentials it started with, even if the tenant rotates them mid-run (Q4 graceful drain).
2. Tech Choices
| Concern | Choice | Rationale |
|---|---|---|
| Persistence | TypeORM entity TenantJobRuntimeConfig + forward-only migration in apps/api/src/migrations/ | Matches platform database.md rules; self-applied at API boot via migrationsRun: true |
| Secret storage | secret_ref pointer into the existing AES-256-GCM secrets jsonb envelope used by plugin settings | Reuses PLUGIN_SECRETS_ENCRYPTION_KEY rotation envelope from settings-system.md §5 |
| Settings scope | NEW x-scope: tenant value in plugin schema extensions | First sub-story of P1; x-scope today is global/user/work only (settings-system.md §3) |
| Resolver placement | NestJS request-scoped service TenantAwareRuntimeResolver in packages/agent/src/tasks/ | Sits in front of EW-685's binding factory; constructor-injected per dispatcher symbol |
| Tenant credential cache | In-memory LRU keyed by (tenant_id, credential_version), TTL ≤ 60s | Avoids per-dispatch decrypt; version pinning makes invalidation deterministic |
| Temporal multi-tenancy | One namespace per tenant (Q1 locked); namespace name derived from tenant_id | Native isolation; control-plane quota risk tracked in §end |
| pg-boss multi-tenancy | One schema per tenant (Q2 locked); schema name pgboss_<tenant_id> | Reuses tenant's existing Postgres; isolated migrations per schema |
| Inngest tenancy mode | Uniform inherit / BYO with Trigger.dev (Q3 locked) | Both push-model SaaS; treated identically |
| Push-model webhooks | Per-tenant routes /api/jobs/webhook/:tenantId/... (Trigger.dev, Inngest) | Lets SaaS provider's webhook hit the right tenant context; signature verified per tenant signing key |
| Pull-model workers | Multiplexing worker reads tenant_id from job metadata + injects per-tenant credentials at handler entry (BullMQ, pg-boss). Temporal uses per-namespace pollers (one Worker per active namespace; cold-start lazily) | Multiplex avoids N worker processes for queue-based runtimes; Temporal SDK binds a Worker to one namespace by design |
| Inherit default mode | Operator-gated shared / per-tenant / tiered (Q5 locked, env: EVER_WORKS_INHERIT_DEFAULT_MODE) | Lets ops trade signup latency vs blast-radius without code changes |
| Plugin gating | Reuse existing instance operator allow-list from EW-683 | Per-tenant whitelist/blacklist deferred to v2 behind a flag |
3. Data Model
New entities
TenantJobRuntimeConfig (apps/api/src/works/entities/tenant-job-runtime-config.entity.ts):
| Column | Type | Notes |
|---|---|---|
tenant_id | uuid PRIMARY KEY | One row per tenant; FK to tenants.id ON DELETE CASCADE |
provider_id | varchar(64) NOT NULL | Matches IJobRuntimeProvider.runtimeId (trigger/temporal/bullmq/pgboss/inngest) |
credentials_secret_ref | varchar(128) NULL | Pointer into the encrypted secrets store; NULL when mode = inherit |
credential_version | integer NOT NULL DEFAULT 1 | Bumped on every credential rotation; captured into WorkGenerationHistory |
mode | enum('inherit','byo','override') NOT NULL | inherit = use instance defaults; byo = tenant-owned creds; override = ops-imposed override |
enabled | boolean NOT NULL DEFAULT true | Soft-disable without losing config; resolver treats false as inherit |
created_by | uuid NULL | FK to users.id; NULL for system-created rows |
created_at | timestamptz NOT NULL DEFAULT now() | |
updated_at | timestamptz NOT NULL DEFAULT now() | Bumped by @UpdateDateColumn |
Indexes: PK on tenant_id; partial index WHERE enabled = true on (provider_id) for ops dashboards.
Extensions to existing entities
WorkGenerationHistory (and the other *_history tables that mirror it) gains:
| Column | Type | Notes |
|---|---|---|
runtime_provider_id | varchar(64) NULL | Provider that owned the enqueue; NULL = legacy/in-process |
runtime_credential_version | integer NULL | Snapshot of tenant_job_runtime_config.credential_version at enqueue |
tenant_id | uuid NULL | Denormalised for fast tenant-filtered history queries |
Migrations
- Generated via
pnpm typeorm migration:generate -d typeorm.config.ts src/migrations/TenantJobRuntimeOverlayper platform CLAUDE.md. - Forward-only, two-phase for the
*_historycolumn adds (perdatabase.md§6.2 + workstation NN #16): add columns nullable → backfillruntime_provider_idfrom existing rows (set totriggerfor any row with non-nulltriggerRunId) → switch reads → leave nullable forever. - Tenant-side provider schemas (Temporal namespaces, pg-boss schemas) are provisioned on first use by P4 worker code, not by a TypeORM migration — they live in the runtime's own control plane.
DTOs / contracts
New types in packages/plugin/src/contracts/capabilities/job-runtime-tenant.interface.ts (sibling to the EW-685 P0 job-runtime.interface.ts; lands with P3 alongside the tenant-aware resolver):
export type TenantRuntimeMode = 'inherit' | 'byo' | 'override';
export interface TenantRuntimeBinding {
tenantId: string;
providerId: JobRuntimeId;
mode: TenantRuntimeMode;
credentialVersion: number;
resolvedAt: Date;
}
export interface ITenantRuntimeResolver {
resolve(tenantId: string): Promise<TenantRuntimeBinding>;
invalidate(tenantId: string): Promise<void>;
}
@ever-works/contracts is unchanged for external API consumers.