Architecture: Monitoring & Observability

Status: Active Last updated: 2026-05-02 Audience: AI agents and engineers debugging production issues, adding new metrics, configuring alert rules, or extending the observability surface.

---

1. Purpose

The platform's observability surface is two providers wrapped in one NestJS module:

• Sentry — error tracking, traces, breadcrumbs, performance profiling.
• PostHog — product analytics, feature flags, session replay metadata.

The @ever-works/monitoring package is the single integration point both apps and packages depend on. Plugins never import Sentry or PostHog SDKs directly; they emit events that flow through the monitoring module's interceptors and helpers, so swapping providers or turning the whole thing off is a config change rather than a code change.

This spec covers the module composition, interceptor model, event taxonomy, secret hygiene at the observability boundary, and the disabled-by-default behaviour for self-hosted deploys.

2. Module Layout

packages/monitoring/src/
├── index.ts
├── monitoring.module.ts          # Top-level NestJS module
├── interceptors/                 # Request-scoped wrappers
│   ├── sentry.interceptor.ts     # Wraps every controller in a Sentry span
│   ├── posthog.interceptor.ts    # Captures structured events post-handler
│   └── index.ts
├── sentry/
│   ├── sentry.module.ts          # Sentry NestJS provider
│   ├── sentry.config.ts          # DSN, environment, sample rates, integrations
│   └── index.ts
├── posthog/
│   ├── posthog.module.ts
│   ├── posthog.config.ts         # Project key, host, batching
│   └── index.ts
├── services/                     # Helpers consumed by domain code
└── types/                        # Shared event-shape types

monitoring.module.ts composes both providers into a single import:

@Module({
	imports: [SentryModule.forRoot(), PostHogModule.forRoot()],
	providers: [SentryInterceptor, PostHogInterceptor],
	exports: [SentryInterceptor, PostHogInterceptor]
})
export class MonitoringModule {}

Apps register MonitoringModule once at the root; the global interceptor list (APP_INTERCEPTOR) wires both interceptors so every controller is covered without per-route opt-in.

3. Configuration & Disabled-by-Default

Monitoring providers are off unless their respective env vars are set:

Provider	Required env vars	Disabled behaviour
Sentry	SENTRY_DSN	SentryInterceptor becomes a no-op pass-through
PostHog	POSTHOG_PROJECT_KEY, POSTHOG_HOST	PostHogInterceptor and ActivityLogAnalyticsDispatcher no-op

sentry.config.ts and posthog.config.ts validate env at startup and emit a clear log line on each startup (Sentry enabled: dsn=... environment=production or Sentry disabled: SENTRY_DSN not set) so ops can tell at a glance whether the deploy is reporting.

This matches the subscriptions "self-hosted single-tenant" pattern — a fresh deploy works without external provider accounts.

4. Sentry Integration

4.1 What Sentry captures

Surface	What
Unhandled exceptions	Full stack traces with TypeScript source-maps
HTTP request errors	Captured by SentryInterceptor with operation name + user context
Trigger.dev tasks	Wrapped at the worker level; failures attach the task id and run id
Pipeline step errors	Step-name + step-state context attached as Sentry tags
Performance traces	Sampled requests + Trigger.dev runs; sample rate per environment
Breadcrumbs	Manually-added context (cache hits/misses, facade calls, plugin lookups)

4.2 Tags & context

The interceptor sets these on every event automatically:

Tag	Source
app	ever-works-api / ever-works-mcp / trigger-worker
userId	req.user.userId if authenticated
workId	Route param when present
pluginId	Set by facade interceptors when routing through a plugin
pipelineStep	Set by the pipeline executor on step events
triggerRunId	Set by the Trigger.dev task wrapper
release	Git SHA of the deployed build
environment	production / staging / development

4.3 Sample rates

Environment	Trace sample rate	Profiler sample rate
production	0.1 (10%)	0.05 (5%)
staging	0.5 (50%)	0.25 (25%)
development	1.0 (100%)	1.0 (100%)

Configurable via SENTRY_TRACES_SAMPLE_RATE and SENTRY_PROFILES_SAMPLE_RATE env vars.

4.4 Integration list

Sentry is configured with these integrations enabled by default:

• Http — outbound HTTP request spans.
• Express / Nest — automatic transaction naming.
• Postgres (when applicable) — DB query spans.
• OnUncaughtException, OnUnhandledRejection — guarantee no error escapes silently.
• Disabled: LocalVariables (memory cost in production), Anr (still flaky in NestJS workers).

5. PostHog Integration

5.1 What PostHog captures

PostHog is product analytics, not error tracking. It receives:

• Activity-log events — see activity-log. Every ActivityLog row dispatched to PostHog as a typed event.
• Page views (from the dashboard, with locale + page name).
• Feature usage (custom events emitted by domain code).
• Plan changes — upgrade, downgrade, cancellation.
• Generation cost / outcome — aggregated per work per period.

5.2 Event naming convention

snake_case, derived from ActivityActionType:

ActivityActionType	PostHog event name
WORK_GENERATED	work_generated
ITEM_ADDED	item_added
OAUTH_LINKED	oauth_linked

Event properties mirror the row's details block but with PII stripped (no email, no IP, no user-agent — only userId, opaque UUIDs, and action context).

5.3 Identification

Users are identified by userId (UUID). The platform never sends email or name to PostHog. The dashboard alias-creates the user on first login — posthog.identify(userId, {plan, locale}) — and the PostHogInterceptor includes distinct_id: userId on every event.

5.4 Feature flags

PostHog feature flags are read on the API side via posthog.getFeatureFlag(flagKey, distinctId). Used today for:

• A/B testing pipeline tweaks.
• Gradual rollouts of new pipelines (e.g. claude-managed-agent rollout).
• "Beta tester" feature gates.

6. Structured Logging

NestJS's Logger is the canonical logger. The platform doesn't add a parallel logging library. Conventions:

• Context — every service constructor sets a Logger with the class name (new Logger(MyService.name)).
• Levels — verbose < debug < log < warn < error.
• Structured payloads — logger.log('work generated', { workId, durationMs }).
• No sensitive values — never log API keys, OAuth tokens, JWT payloads, or user emails.
• JSON output in production — LOGGER_FORMAT=json flips the log formatter to NDJSON for downstream aggregation (Datadog, Loki, CloudWatch, etc.).

7. The Two Interceptors

Both run on every controller via APP_INTERCEPTOR:

7.1 SentryInterceptor

• Starts a Sentry span named after the route.
• Attaches the user / work / plugin tags described in §4.2.
• On error, captures the exception with the active span.
• On success, finishes the span with the response status.

7.2 PostHogInterceptor

• Skips @Public() routes by default (no PostHog events for unauthenticated traffic — they'd flood with bot scans).
• After the handler finishes, captures a request_completed event with path, method, status, durationMs, userId.
• Domain-specific events come from ActivityLogService — the interceptor is the catch-all for "something happened on a request" metrics.

8. Plugin-Side Observability

Plugins receive a logger via their PluginContext. The logger is namespaced as [plugin:<id>] so log lines from a plugin can be filtered to one source. Plugins don't call Sentry / PostHog SDKs — the platform's interceptors capture errors and successes for them.

For plugin-specific events that matter to product analytics, plugins emit them via the activity-log service:

ctx.events.emit('plugin:setting-tested', {
	pluginId: 'openai',
	success: true,
	tier: 'complex'
});

The activity-log service picks up plugin:* events and records them as PLUGIN_TESTED activity-log rows — which then flow to PostHog via the dispatcher.

9. Trigger.dev Worker Observability

The Trigger.dev worker is its own NestJS app context (see trigger-integration). Each task:

• Boots a NestJS application context with MonitoringModule.
• Wraps task.run in a Sentry span named after the task id.
• Adds triggerRunId and cronId (when scheduled) as tags.
• On error, captures with task context.
• Closes the application context in finally so resources free up.

A failed Trigger.dev run shows up in Sentry as a transaction tagged with the task id, the work id (when applicable), and the failing step — clickable straight to the offending pipeline step.

10. Health Checks

The API exposes /api/health (a @Public() route):

{
	"status": "ok",
	"uptime": 12345,
	"checks": {
		"database": "ok",
		"cache": "ok",
		"sentry": "enabled",
		"posthog": "enabled",
		"subscriptions": "enabled",
		"trigger": "enabled"
	}
}

Each check is a fast read (no external API call) — it inspects the in-process module wiring. Returns 503 when any check is failed; returns 200 with a degraded subset list when any check is disabled. K8s liveness probes consume this endpoint.

A separate /api/health/deep (admin-only) actually pings the DB, cache, Sentry, PostHog, and Trigger.dev — used for incident response.

10.1 External uptime monitor — alert presentation contract

.github/workflows/external-uptime-monitor.yml probes the public surfaces every 15 minutes from a GitHub-hosted runner, i.e. from outside the homelab uplink that every in-cluster probe shares. Its header comment explains why that vantage point is load-bearing; this section covers only how a failure is presented, which is a separate concern and has its own failure mode.

The monitor keeps exactly one open uptime-alert issue. Issue spam is the main reason monitors get muted, so a second issue is never opened. The cost of that choice — logged as EW-760 — is that a new, worse outage can fold into an issue a reader has already triaged. On 2026-08-23 the entire web app went down while an alert issue stood open for demo.ever.works; the escalation was invisible for ~18 hours. Detection worked the whole time. Presentation did not.

Four properties keep the single-issue design honest. All of them work by changing what the issue says, never by opening another one.

1. The title names the current failing set. It is recomputed every run from the failing URLs and re-applied when it differs from the live title. A comment appended to a long-open issue is low-salience; the title is what people re-read.
2. The title shouts for customer-facing surfaces. A separate CRITICAL_URLS list (the marketing site, the app login and health routes, the API and its version endpoint) switches the title between 🔴 Uptime: … and 🚨 Uptime: CRITICAL — …. It is a separate array, not a fourth TARGETS field, because the target parser treats everything after the second | as the body marker — a trailing severity field would be swallowed and every target would read as non-critical.
3. Duration escalates. The workflow has no store, so the outage clock rides in an HTML comment on the last report it wrote: <!-- uptime-state: fp=… since=… tier=… urls=… -->. On a run where the fingerprint still matches, since is carried forward; when the failing set changes the clock restarts, because a different set is a new event. At 60 minutes the alert reaches tier 1 and at 240 minutes tier 2. Each escalation appends (down 1h15m+) to the title (bucketed to the 15-minute probe interval so the title moves on severity, not on every run), adds an uptime-sev<tier> label, and is allowed past the one-comment-per-hour throttle exactly once.
4. A partial recovery is named. The urls= field records the failing set, so the next run can diff it and lead its comment with ### ✅ Recovered since the last report. Without this an issue lists only what is broken now, and a reader cannot tell a partial recovery from a target that was never mentioned. Full auto-close stays all-or-nothing: the issue closes when every target is green.

The state marker is written alongside the pre-existing <!-- uptime-fingerprint: … --> marker, never instead of it — the comment throttle greps for that marker verbatim.

The parse-decide-reemit logic lives between the sentinels # >>> uptime-escalation-state and # <<< uptime-escalation-state and is pure text plus arithmetic: no gh, no date. That is what makes it testable. apps/node/src/core/external-uptime-monitor.contract.spec.ts lifts the block straight out of the YAML and runs it under the same Bash flags GitHub gives a run: step (bash -e plus set -uo pipefail), covering a fresh outage, each escalation threshold, a changed fingerprint resetting the clock, a partial recovery, a corrupt marker, and the empty input. The empty case is not hypothetical — an all-green run once failed the whole monitor because grep on an empty list returned non-zero under pipefail.

11. Sentry-PostHog Cross-Linking

When both providers are enabled, the platform threads each Sentry event with a posthog_session_id so support can pivot from a Sentry error → PostHog session replay. The reverse pivot uses sentry_event_id recorded on PostHog events at error time.

12. Secret Hygiene at the Observability Boundary

Two scrubbing layers ensure secrets never leave the process:

1. Sentry beforeSend hook — strips fields matching the settings-system x-secret patterns from breadcrumbs, request payloads, and error context.
2. PostHog property denylist — email, apiKey, token, password, anything matching *_secret / *_password / *Token is dropped before sending.

Both hooks are applied at the SDK level — every event the platform sends goes through them, regardless of caller.

13. Constitution Reconciliation

Principle	How monitoring respects it
I — Plugin-first	Plugins use ctx.logger and ctx.events; never Sentry / PostHog SDKs directly.
II — Capability-driven	Provider abstraction means swapping Sentry / PostHog is contained.
III — Source-of-truth repos	Monitoring is platform-side; never writes to user repos.
IV — Trigger.dev	Trigger.dev worker has the same MonitoringModule wiring.
V — Forward-only migrations	No DB schema.
VI — Tests	Interceptors covered by unit tests; integration paths e2e-tested.
VII — Secret hygiene	Two scrubbing layers (Sentry beforeSend + PostHog denylist).
VIII — Plugin counts	Plugin counts are reported as PostHog events for adoption tracking.
IX — Behaviour-first	This spec describes observable behaviour.
X — Backwards-compat	New tags + events are additive.

14. References

• Source:
  • packages/monitoring/src/
  • packages/monitoring/src/sentry/sentry.config.ts
  • packages/monitoring/src/posthog/posthog.config.ts
  • .github/workflows/external-uptime-monitor.yml (see §10.1)
  • apps/node/src/core/external-uptime-monitor.contract.spec.ts
• Related specs:
  • activity-log
  • auth
  • trigger-integration
  • settings-system
• User docs: docs/devops/monitoring.md