Production Readiness

@objectstack/runtime ships first-class primitives for the cross-cutting concerns every production deployment needs: HTTP hardening, rate limiting, metrics, error reporting, and request-id correlation. Everything is pluggable — defaults are safe, opt-in is gradual, and the framework takes no hard dependency on Prometheus / OTel / Sentry / Redis.

This guide is the high-level overview. For the full reference (header defaults, adapter recipes, go-live checklist) follow the deep-dive links.

Wiring everything together

A typical production server enables all four:

import { createDispatcherPlugin } from '@objectstack/runtime';
import { promMetrics } from './adapters/prom-metrics.js';     // your code
import { sentryReporter } from './adapters/sentry.js';        // your code

const dispatcher = createDispatcherPlugin({
  // Hardening
  securityHeaders: {
    hsts: { maxAge: 31536000, includeSubDomains: true, preload: true },
    // CSP defaults to deny-all (API server). Override here if you serve
    // a SPA from the same origin.
  },

  // Observability
  observability: {
    metrics: promMetrics,
    errorReporter: sentryReporter,
  },
});

That's it — every route the dispatcher mounts now:

1. Sends conservative security headers.
2. Echoes X-Request-Id (honored from caller, or freshly minted).
3. Reports 5xx exceptions to Sentry (or your reporter) with the request id attached.

The http_requests_total{method,route,status} counter and the http_request_duration_ms{method,route} histogram are emitted at the transport (the IHttpServer.afterResponse observation seam), so they cover every inbound request on the server — auth, the REST data API and other raw-app mounts included, not only the dispatcher's own routes. The dispatcher offers its metrics registry to that seam automatically, and exactly one emitter is armed per server per family, so the wiring above is complete on its own and wiring the transport plugin too never double-counts. The route label is the registered pattern (/api/v1/data/:id), never the concrete path.

http_request_duration_ms measures the request as the transport sees it (middleware chain and body parse included), not the handler alone — so p95 is the latency a caller experiences minus the network.

http_request_errors_total was retired in 17.2.0 (#9834). Nothing declares or emits it any more, so a dashboard keyed on that name reads a flat zero — that is the removal, not a healthy server. Alert on http_requests_total{status=~"5.."} instead: the transport emits it for every inbound surface, whereas the retired counter only ever saw the dispatcher's own routes and only when a handler threw, missing every error answered politely with a status. Unhandled exceptions specifically still reach the errorReporter, which is the signal that counter was mistaken for.

A transport that does not implement the seam reports no HTTP metrics: zero on http_requests_total there means "not instrumented", never "no traffic". The shipped Hono adapter implements it; verify a custom adapter with typeof server.afterResponse === 'function' before trusting its numbers.

A flat cache_* means "no configured consumer", and that reading is the opposite of the one above: the @objectstack/service-cache adapters are instrumented and do hold the host's registry, so a zero on cache_lookups_total is a true count — of a service nothing asked anything of. Nothing consults the cache service unconditionally; every production consumer is a rate-limit or budget counter store gated on an explicit declaration (better-auth's rate_limit_max / rate_limit_window_seconds, the dispatcher's inbound and per-endpoint rateLimit budgets, the per-number OTP send budget). Declare none and the family sits at 0 under normal traffic, so treat a cache hit-rate panel as reporting on your configuration before you treat it as reporting on your cache.

Rate limiting is the one piece you wire at the adapter layer (Fastify preHandler, Hono middleware, etc.) because that's where you have reliable access to the caller's IP and authenticated identity. See the HARDENING.md recipes.

Go-live checklist

• HSTS enabled (after TLS is confirmed and you accept the lock-in).
• CSP tuned for your serving topology (deny-all API, looser for SPA-on-same-origin).
• Rate limit configured at the adapter for /auth/*, write verbs, and read verbs (auth bucket should be ~10/min/IP).
• Metrics adapter wired; /metrics endpoint (Prom) or OTel exporter producing data; alerts wired for error rate + p95 latency per route.
• Error reporter wired; synthetic 5xx test reaches Sentry / Datadog / Rollbar.
• Log records include requestId and cross-check with the response X-Request-Id header.
• better-auth session TTL / refresh / revoke verified (curl checklist in HARDENING.md).
• Cross-tenant negative tests in CI.
• Multi-org deployments: tenant isolation is active, not degraded — confirm features.degradedTenancy is false in /auth/config. A deployment requesting multi-org without @objectstack/organizations now refuses to boot unless OS_ALLOW_DEGRADED_TENANCY=1; never set that flag in production. See Tenancy Modes & Membership.
• OS_ALLOW_DRIVER_CONNECT_FAILURE is unset. A data driver that cannot connect at startup refuses the boot, so a bad OS_DATABASE_URL, a rotated password, or a closed network path surfaces as a failed deploy instead of a "started" server that 500s every request. The same applies to a declared datasource that objects bind to explicitly (datasource: 'analytics'): those objects have no fallback, so a misconfigured analytics URL fails the deploy instead of leaving a server where most pages work and that one subset errors. Setting the flag boots without the database and never reconnects; never set it in production.
• Backup / restore drill documented and tested.
• Data-retention windows reviewed (ADR-0057): the platform's default lifecycle declarations bound telemetry (activity 14d, job runs 30d, notifications 90d, audit 90d-hot); extend per environment/tenant via the lifecycle.retention_overrides setting before go-live if your compliance regime needs longer, and register an archive datasource if audit data must move to cold storage instead of being retained hot.

CDN / reverse-proxy caching of /meta object schemas

Object schemas are the hottest metadata read — every list, form and detail render fetches one — so they are the first thing an operator reaches for when putting a CDN or reverse proxy in front of the runtime. Since ADR-0106 those responses are per-caller, and the rules below are what keep that safe.

• The response is already marked private. GET /meta/object/:name answers Cache-Control: private, no-cache — store it in the client, revalidate on every use, never in a shared cache. A proxy configured to ignore private (proxy_ignore_headers Cache-Control, an "override TTL" rule, an edge worker that caches by URL alone) would serve one user's projection to another. Do not add such a rule for /meta or /metadata.
• The ETag carries the caller's field visibility. The validator is the shared document hash plus, for a restricted caller, a ~<fingerprint> suffix hashing the set of fields that caller cannot read. Consequences worth knowing before tuning anything:
  • A caller who can read every field produces no suffix, so their ETag and body are byte-identical to pre-ADR-0106 releases — existing cache hit rates on unrestricted deployments are unchanged.
  • Callers in the same permission cohort share one validator, and therefore share 304s. Cache storage stays O(objects), not O(users × objects).
  • A permission change moves the fingerprint, so a stale 304 self-invalidates on the next revalidation. There is nothing to purge after editing a permission set.
  • Never strip or rewrite ETag on these routes (some proxies drop it when they compress). Without it every read falls back to a full body, and a proxy that replaces it with its own hash of the served body erases the cohort dimension.
• A private, no-store response means field visibility could not be resolved. In that degraded window the runtime serves the unmasked schema deliberately — failing closed would brick every render of the object and can deadlock console bootstrap, since permission sets are themselves metadata — and marks it un-storable so it cannot be replayed to anyone else. It also emits a structured warning and increments objectstack_meta_field_visibility_undetermined_total. Alert on that counter: a deployment sitting in this state is disclosing more than it intends, and it is meant to be a hydration blip, not a resting state.
• A 503 FIELD_VISIBILITY_UNRESOLVED is the security service being unhealthy, not the metadata store. The runtime refuses the read rather than disclose an unmasked schema. It is safe to retry and must not be cached.
• To deliberately restore the old, unmasked behaviour on a trusted deployment, the supported switch is OS_ALLOW_UNMASKED_OBJECT_METADATA=1 (or per-server metadata.maskObjectFields: false) — not a caching rule that hides the per-caller dimension.

What's NOT in the runtime (yet)

• OTel context propagation. We export parseTraceparent / formatTraceparent so the host can wire traceparent into its OTel SDK, but we don't bundle the SDK itself. The OTel API surface is large and host-specific (Node vs. edge vs. browser).
• A Prometheus /metrics endpoint. Adapters mount their own scrape endpoint after wiring the MetricsRegistry. We give you the recipe, not the endpoint, because TLS / auth / path conventions vary.
• A built-in audit log immutability layer. @objectstack/plugin-audit writes sys_audit_log records via ObjectQL afterInsert/afterUpdate/afterDelete hooks; durable storage and tamper-evidence are the host's choice (append-only S3, immudb, etc.).