Cluster semantics — multi-node runtimes

Status: Accepted · Audience: Plugin authors, runtime engineers

TL;DR — ObjectStack runs identically on a laptop and on a 10-node cluster. To make that true, a small number of runtime primitives need explicit semantics in the protocol layer: how events propagate, whether a service is one-per-node or one-per-cluster, and how metadata changes invalidate caches everywhere at once. This document fixes those semantics.

1. Why this document exists

The protocol layer (@objectstack/spec) was designed top-down: it describes what the system looks like (objects, fields, views, flows, agents) and deliberately stays silent about where code runs. That silence was correct during bootstrap — it let us ship the metadata-first vision without prematurely locking in a runtime topology.

It is no longer correct. As ObjectStack approaches its first production deployments, three classes of bugs become inevitable on any multi-node setup:

1. Stale caches. Node A updates an object's label; Node B continues to serve the old value from its in-process metadata cache because the change event never crossed the process boundary.
2. Duplicated work. A schedule-triggered Flow fires once on every node; a single nightly job runs N times.
3. Lost or doubled deliveries. Two webhook dispatchers pick up the same pending row; an external system receives the same event twice with no safe way to deduplicate.

These are not implementation bugs. They are protocol omissions — the spec never told plugin authors what "emit an event" or "register a job service" should mean when more than one process is involved.

This document closes those omissions. It is the source of truth for cluster behaviour and the basis for the kernel/cluster.zod.ts schema that codifies it.

2. Design principles

Every decision below follows three non-negotiable rules.

P1 — Single-machine remains zero-config. A developer who runs pnpm dev must never need Redis, a message broker, or any external state. The default driver for every cluster primitive is in-process; the API surface is identical to the distributed driver.

P2 — Distributed semantics live in the protocol, not in the implementation. A plugin author should be able to write one version of their code that is correct on both topologies. They achieve that by declaring intent ("this job runs once per cluster", "this event should reach every node") in metadata or through context APIs — never by branching on if (cluster) { ... }.

P3 — Default to the safe semantic, not the cheap one. When in doubt, events stay node-local, services stay node-singletons, and leader-elected work is opt-in. This guarantees that a plugin written without thinking about clustering still behaves correctly when deployed to one — it just may not benefit from horizontal scale until annotated.

3. The four cluster primitives

The protocol exposes exactly four primitives. Everything else (caches, queues, schedulers, realtime fan-out, webhook delivery, audit shipping) is built on top of them.

These four are the minimum sufficient set. Higher-level facilities (runOnce, withLock, broadcast, leaderElect, cache invalidation) are derived; their semantics are defined in §5.

Why these four and not more

• No native job queue primitive. A job queue is KV + Lock + PubSub composed; expressing it directly in the cluster layer would couple the protocol to a specific queue shape. service-queue is expected to build on the primitives instead — today it ships memory and DB adapters only and does not consume the cluster service.
• No native cache primitive. Caches are KV + PubSub (read the value, subscribe to the invalidation channel). service-cache is expected to build on the primitives instead — today it ships a memory adapter plus a Redis adapter skeleton that throws RedisCacheAdapter not yet implemented, and it does not consume the cluster service either.
• No native stream/Kafka primitive. Streaming is intentionally out of scope for v1. It can be added as a fifth primitive later without changing the four above.

4. Event scope and delivery semantics

The single biggest source of cluster bugs is "I published an event and assumed everyone would hear it". Every emit must answer three questions:

Status: protocol only. The three fields below exist as EventMetadata.cluster (EventClusterOptionsSchema in kernel/cluster.zod.ts, wired into kernel/events/core.zod.ts), but no runtime consumes them — there is no eventBus object in the codebase at all. The kernel's plugin event API is ctx.hook(name, handler) / ctx.trigger(name, ...args), neither of which takes cluster metadata. The shipped cross-node path today is a direct cluster.pubsub.publish(...) call (§6.2, §7.3). §4.1-§4.4 define what these enum values are supposed to mean once a bus reads them.

4.1 Scope — who receives this event?

scope: 'local' | 'cluster' | 'tenant'

• local (default) — Delivered only to handlers in the emitting process. Use for: in-request hooks, derived-state recomputation that the emitting node will use itself, anything cheap to recompute.
• cluster — Delivered to every node in the cluster, including the emitter. Use for: cache invalidations, configuration reloads, metadata changes, anything where every node must converge to the same view.
• tenant — Same as cluster, but delivery is partitioned by tenantId so only nodes currently serving that tenant receive it. Use for: tenant- scoped cache invalidations on large multi-tenant deployments where broadcasting to all nodes would waste bandwidth.

Default rationale. local is safe by accident: a plugin author who never thinks about clustering produces code that runs identically on one or N nodes — it just doesn't gain any cluster-aware behaviour until they opt in.

4.2 Delivery semantics — what guarantees does the bus offer?

deliverySemantics: 'best-effort' | 'at-least-once' | 'exactly-once'

• best-effort (default for local) — In-process delivery. Lost if a handler throws or the process crashes before the handler runs. Use for: UI hints, telemetry, anything where loss is recoverable.
• at-least-once (default for cluster / tenant) — Persisted to the transport (Redis Streams, Postgres outbox table) before publish returns. Survives node crash. Handlers must be idempotent — duplicates are possible during retry. Use for: webhook outbox, audit shipping, async job enqueue. No shipped driver provides this yet. The redis driver publishes over plain Redis pub/sub, which is at-most-once — fire-and-forget, no persistence, no replay for a node that was down at publish time. Only route cache-invalidation hints through it; anything that must survive a crash needs a durable outbox of its own.
• exactly-once — Reserved keyword; not implemented in v1. The protocol accepts the enum value so future runtimes can add it without a breaking change. Startup rejection of this value is intended behaviour but is not yet implemented — the bus does not currently inspect or reject it.

4.3 Partition key — what ordering does the bus preserve?

partitionKey?: string

When set, the bus guarantees that two events with the same partitionKey are delivered to handlers in emit order, even with multiple consumers. Events with different partition keys may interleave freely. Typical partition keys: record id, tenant id, conversation id.

When unset, no ordering is guaranteed — handlers may run in parallel and out of emit order.

Status: accepted but ignored. PublishOptions.partitionKey is part of the IPubSub contract and callers already pass it (the metadata manager keys on `${type}:${name}`), but both shipped drivers take the options bag as an unused _opts parameter. The memory driver's synchronous single-process fan-out preserves emit order anyway; the redis driver gives you whatever ordering Redis pub/sub does. Do not rely on cross-key ordering guarantees yet.

4.4 The combined contract

The three fields compose into one EventMetadata.cluster block:

// target shape — parses today, nothing reads it yet
const metadata = {
  source: 'my-plugin',
  timestamp: new Date().toISOString(),
  cluster: {
    scope: 'cluster',
    deliverySemantics: 'at-least-once',
    partitionKey: payload.id,
  },
}

Once a bus honours it, that combination is the intended way to invalidate a per-record cache entry across a cluster: the cluster scope reaches every node, at-least-once guarantees no node misses the invalidation across a crash, and partitionKey ensures that two rapid updates to the same record are applied in order.

Until then, the working equivalent is a direct publish on the cluster service — it accepts partitionKey (see the §4.3 caveat) and gives best-effort delivery only:

import type { IClusterService } from '@objectstack/spec/contracts'

const cluster = ctx.getService<IClusterService>('cluster')
await cluster.pubsub.publish('account.updated', payload, {
  partitionKey: payload.id,
})

5. Service scope and leader election

Every service registered with the kernel has a cluster scope. The annotation is optional — omitting it is equivalent to { clusterScope: 'node' }. This is independent of DI lifecycle scope (singleton / transient / scoped); it answers a different question.

A cluster-scoped service must declare how the single-active-instance invariant is maintained:

leaderStrategy: 'leader-elected' | 'partitioned' | 'idempotent-broadcast'

• leader-elected — Exactly one node holds the leadership lock at any time; non-leader nodes idle. The runtime handles election via the Lock primitive (heartbeat + TTL). Use for: cron schedulers, anything that must run "once per tick".
• partitioned — Every node runs the service, but each instance owns a disjoint partition of the work (typically hashed on partitionKey). Use for: webhook dispatchers (partition by webhook_id), high-throughput workers.
• idempotent-broadcast — Every node runs the service on every input, and the work itself is idempotent (writes use UPSERT, side effects are keyed). Use for: cache invalidation handlers, projection rebuilders.

These annotations are nested under a cluster key on the service registration (ServiceMetadata.cluster / ServiceFactoryRegistration.cluster = ServiceClusterAnnotations), not declared as flat top-level fields:

{
  name: 'cron-scheduler',
  cluster: {
    clusterScope: 'cluster',
    leaderStrategy: 'leader-elected',
    // clusterId: 'scheduler',  // optional — share one leadership lock
    //   across physically different services (e.g. safe rolling upgrades);
    //   defaults to the service name.
  },
}

Default rationale. node is the only safe default. A plugin author who forgets to declare scope gets the single-machine behaviour, which is always correct; opting up to cluster is an explicit decision.

5.1 What the runtime will give you for free

Status: not yet implemented (Phase 4). The schema accepts clusterScope / leaderStrategy annotations today, but no runtime code yet consumes those annotations to wire leadership automatically — the four behaviours below are the intended Phase 4 (see §10) contract. Leader election itself already works via the Lock primitive (service-job's cron scheduler calls cluster.lock.acquire(...) directly so only one node fires each scheduled job); what Phase 4 adds is the kernel doing it for you from a service's declared clusterScope.

When a service declares clusterScope: 'cluster', the runtime will automatically:

1. Wrap the plugin's start(ctx) so the work only begins after lock acquisition (for leader-elected).
2. Refresh the lock TTL via heartbeat while the node is healthy.
3. Release the lock on destroy() / the kernel:shutdown hook.
4. Emit a leadership-change event so dependent services can react.

(The kernel's plugin contract is init(ctx) / start(ctx) / destroy(). The onEnable / onDisable family was retired from the protocol — it never had an invocation site — so do not reach for it here.)

A plugin author writing a cron scheduler will not write any lock code — they declare scope and strategy, and the runtime does the rest.

6. Metadata versioning and cache coherence

Metadata is the most cache-hot data in the system. Every read path — ObjectQL planning, REST routing, UI rendering, permission checks — touches it. Making it correct across a cluster needs a well-defined invalidation signal per state owner — the three lanes of §6.2 — and, in this protocol, deliberately no per-item version on the wire (§6.1).

6.1 Monotonic version

A version column is already present on every persisted metadata record in system/metadata-persistence.zod.ts (version: z.number(), used today for optimistic concurrency). Earlier revisions of this page proposed formalising it as a monotonic per-item version, widening it to bigint, and having every cache compare it on receipt.

Status: not declared. The column is version: number in the schema, and nothing in the protocol carries a version across nodes. The version-compare invalidation this section once described as the target design — caches storing {value, version} and ignoring a notification whose version is not strictly greater than the cached one — went with the retired MetadataChangedEventPayloadSchema (§6.2, lane 1; ADR-0049 enforce-or-remove, ADR-0087 entry metadata-changed-event-payload-retired). The shipped lanes are address-only signals whose receiver re-reads its own store, and a re-ordered or duplicated delivery converges to the same state as a single one — which is what removes the "old invalidation evicts a newer value" class of bug without a version to compare. A version-stamped protocol returns only via the ENFORCE route of ADR-0049: the emitter and the reader first, the declaration with them.

6.2 The cluster-invalidation family

Cross-node metadata invalidation is not one channel — it is a family of three independent PubSub lanes, late-bound by MetadataClusterBridgePlugin (@objectstack/service-cluster) at kernel:ready, one per state owner that goes stale when metadata changes on a different node:

Lanes 2 and 3 share one contract, stated once here rather than in each channel's own doc comment:

• Address-only signal. No message body ever rides these channels — the payload names only what changed (a metadata address, or a datasource name), never the changed value.
• Receipt runs convergence, not trust. A peer that receives a signal does not apply the payload as content; it re-reads its OWN copy of the source of truth (sys_metadata for lane 2, the shared datasource record for lane 3) and converges its local registry from that read. A duplicate or re-ordered delivery converges to the same state as a single one.
• originNode loopback suppression. Every payload carries the publishing node's id; a peer that receives its own originNode drops the message — the write-through or pool rebuild already ran, synchronously, at the door that persisted the change.
• At-most-once, bounded by boot rehydration. No shipped IPubSub driver exceeds at-most-once delivery (§4.2), so a lost message is not retried. Each lane narrows the staleness window this always had — "until the next full reload on restart" — to "one network hop"; it does not promise more.

All three lanes skip their fan-out on the same condition — the in-process memory cluster driver, described for lane 1 immediately below, applies identically to lanes 2 and 3 (#14021).

Lane 1 — metadata.changed (the metadata service cache)

Current behaviour. Cross-node metadata invalidation already works, but not through any event bus (there isn't one — see §4). The metadata manager fans out over the cluster PubSub channel metadata.changed (note: dot, not colon) with the payload shape ClusterMetadataChangedPayload:

// packages/metadata/src/metadata-manager.ts — notifyWatchers()
this.clusterPubSub.publish('metadata.changed', {
  originNode: this.clusterNodeId, // used for loopback suppression
  type,                           // 'object' | 'view' | 'dashboard' | …
  event,                          // the local watch event, replayed verbatim on peers
}, { partitionKey: `${type}:${event.name ?? ''}` })

The transport is handed to the manager by MetadataClusterBridgePlugin (@objectstack/service-cluster), which calls metadata.attachClusterPubSub(cluster.pubsub, cluster.nodeId) on kernel:ready. Runtime registers that bridge automatically alongside the cluster service. It skips that call when the resolved driver is in-process (memory) — such a bus fans out to no peer, so the bridge states that at debug rather than reporting itself as "bridged" (#14021).

On receipt a peer suppresses its own messages by originNode, then — first, synchronously — invalidates its local caches for that type: it drops the registry entry named by the nested watch event and the list(type) cache (invalidateForForeignWrite). Only then does it replay the watch event into its local watch hub, deferred by one tick so a slow consumer callback cannot back-pressure the pubsub dispatch loop. That order is what lets a woken consumer answer by re-reading through list() and see the write rather than the pre-write set (#5109). The registry entry is deleted, never pre-filled from the payload — the peer re-reads the shared store, which is the source of truth.

At the payload's top level there is no version / name / tenantId / operation field and no version comparison — by design, not by omission; the item's name is carried only inside the replayed event.

No richer payload is declared. The spec once carried a version-stamped target shape for this lane — MetadataChangedEventPayloadSchema in kernel/cluster.zod.ts (type / name / tenantId / version: bigint / operation / correlationId), whose docblock said every persistence layer MUST emit it and every reader MUST compare version on receipt. Nothing ever emitted or subscribed to it, and its bigint version could not have crossed a JSON transport without a codec no driver ships, so it was retired under ADR-0049 enforce-or-remove (ADR-0087 entry metadata-changed-event-payload-retired, RETIRED_DEFS_BY_MAJOR[18], together with the MetadataChangeOperationSchema enum that typed its operation field). The shape above is the lane-1 contract, and its receipt semantics are the family's: drop the cache entry, re-read the shared store, no version comparison.

Lane 2 — metadata.mutated (the ObjectQL object registry)

The metadata protocol (@objectstack/metadata-protocol) fans out on a second, independent channel — METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated' (#13331) — with payload type ClusterMetadataMutationPayload:

// packages/metadata-protocol/src/protocol.ts
export interface ClusterMetadataMutationPayload {
  originNode?: string;          // used for loopback suppression
  event: MetadataMutationEvent; // address-only: type/name/state/org scope, no body
}

This lane exists because lane 1 replays into the metadata service's own caches, not into the ObjectQL object registry the runtime authoring path (saveMetaItem → applyRegistryWriteThrough) mutates behind the data plane. Without it, an object authored at runtime through PUT /api/v1/meta/* answers OBJECT_NOT_FOUND on every replica that did not perform the write, until restart — measured on a 3-replica EE deployment: 200 concurrent creates through the load balancer gave 67×201 / 133×404.

On receipt, a peer suppresses its own originNode, then re-reads the row from its OWN sys_metadata and re-runs the registry write-through from that read (applyRemoteMetadataMutation) rather than applying the payload as content. Attached by MetadataClusterBridgePlugin's protocol lane at kernel:ready, duck-typed against protocol.attachMetadataMutationPubSub() so this package takes no dependency on @objectstack/metadata-protocol.

Lane 3 — datasource.mutated (the ObjectQL driver registry)

The datasource admin service (@objectstack/service-datasource) fans out on a third channel — DATASOURCE_MUTATION_CLUSTER_CHANNEL = 'datasource.mutated' (#13805) — with payload type ClusterDatasourceMutationPayload:

// packages/services/service-datasource/src/datasource-admin-service.ts
export interface ClusterDatasourceMutationPayload {
  originNode?: string; // used for loopback suppression
  name: string;        // the datasource whose shared record was just written
}

Lane 2's family, adopted by the driver registry: registerPool / unregisterPool mutate the ObjectQL DRIVER registry on the serving replica only, so before this channel a DELETE /api/v1/datasources/:name recovered /api/v1/ready on the one replica that served it, while every other replica kept the stuck driver until restart. Symmetric by construction — create, update and delete all publish this one shape, and a receiver cannot tell which it was without reading, which is the point.

On receipt, a peer suppresses its own originNode, then re-reads the shared datasource record for name and converges its own pool from that read (builds what the read says should be live, rebuilds what changed, evicts what is gone); convergence for the same name is serialized so two signals in quick succession — a create chased by an update — apply in order. Attached by MetadataClusterBridgePlugin's datasource lane at kernel:ready, duck-typed against datasource-admin.attachDatasourceMutationPubSub() so this package takes no dependency on @objectstack/service-datasource, and @objectstack/objectql — the driver registry's owner — is handed no bus at all; the admin service publishes on the write doors it already owns.

6.3 Reader contract

The reader contract is the one §6.2 states for the family, and it is what ships — there is no version-stamped variant behind it (the payload that carried one was retired, §6.1). A metadata reader that keeps its own long-lived cache:

1. Subscribes to the lane whose state it mirrors — metadata.changed for a mirror of the metadata service's own caches; lanes 2 and 3 already keep ObjectQL's object and driver registries converged (§7.3).
2. On receipt drops its own originNode messages, then evicts — never pre-fills from the payload.
3. Re-reads the shared store on the next access; a duplicate or re-ordered delivery converges to the same state as a single one.

7. Plugin author guidance

This section is a quick reference for plugin authors. The full details live in §4-§6; this is the cheat sheet.

7.1 "I want to fire an event"

Target design only — no runtime reads EventMetadata.cluster today (§4). Pick scope first, then delivery, then partition key:

If none of the above apply, stay with the defaults (local + best-effort) — they are always correct on a single node and never wrong on a cluster, they just don't propagate.

7.2 "I want to register a service"

Ask: if I deploy 5 nodes, should there be 5 of these or 1?

• 5 → clusterScope: 'node' (default). You're done.
• 1 → clusterScope: 'cluster' + pick a leaderStrategy:
  • "Wakes on a timer / cron" → leader-elected
  • "Pulls from a shared queue" → partitioned
  • "Reacts to events with idempotent writes" → idempotent-broadcast

Declaring cluster does not yet do anything (§5.1) — until Phase 4 lands you still have to hold the invariant yourself, either by wrapping the work in cluster.lock.withLock(key, fn) or by acquiring and releasing by hand the way service-job's cron scheduler does (cluster.lock.acquire(key, { waitMs: 0 }), bail out when it returns null, release in a finally).

7.3 "I want to read metadata in a long-lived cache"

There is no cluster.cache(...) factory, and there is no ctx.cluster property either — PluginContext exposes the service-registry family (registerService / registerServiceFactory / getService / getServiceScoped / getServices / replaceService), hook / trigger, logger and getKernel. No cluster handle is among them, so resolve the cluster service by name:

import type { IClusterService } from '@objectstack/spec/contracts'

const cluster = ctx.getService<IClusterService>('cluster')

It exposes only the four primitives (cluster.{pubsub, lock, kv, counter}, see §3) plus nodeId / driver / close(). Caches are meant to be derived from KV + PubSub; they are not a built-in cluster method.

To keep an in-process metadata cache coherent today, subscribe to the metadata change channel via PubSub and invalidate on each notification:

cluster.pubsub.subscribe('metadata.changed', (msg) => {
  const payload = msg.payload // { originNode?, type, event }
  if (payload.type === 'object') {
    myObjectCache.invalidate(payload.event)
  }
})

Already covered? The subscription above is for a cache that mirrors the metadata service's own state (lane 1, §6.2). If what you are caching is ObjectQL's object or driver registry instead, check §6.2 first — lanes 2 (metadata.mutated) and 3 (datasource.mutated) already keep those registries converged across replicas, and a second subscription would be redundant.

There is no higher-level cache factory and no version comparison to automate — the version-stamped payload was retired (§6.1) and the lanes are address-only, so the subscribe-and-evict shape above is the whole contract. No if (cluster) branches are needed either way — the memory driver makes the subscribe call a no-op-equivalent local fan-out on a single node.

7.4 What you must never do

• Never call setInterval for cluster-wide periodic work — gate each tick on cluster.lock (and declare clusterScope: 'cluster' + leader-elected so Phase 4 can take the wiring over from you).
• Never SELECT … WHERE status = 'pending' LIMIT 1 without a row lock — use SELECT … FOR UPDATE SKIP LOCKED (Postgres) or UPDATE … RETURNING with a claimed_by column.
• Never write into a process-local Map and assume other nodes see it — use cluster.kv for shared state.
• Never import Redis from 'ioredis' directly — go through cluster.{pubsub, lock, kv, counter}.

8. Configuration surface

Cluster behaviour is configured once, at the runtime level. The shape is ClusterCapabilityConfig (kernel/cluster.zod.ts), and it is passed on the Runtime constructor:

import { Runtime } from '@objectstack/runtime'

const runtime = new Runtime({
  cluster: {
    driver: 'memory',   // default — single-process
    // driver: 'redis', url: process.env.OS_REDIS_URL,
    // driver: 'custom',  // resolved from registerClusterDriver()

    nodeId: process.env.NODE_ID,           // optional, auto-generated if absent
    heartbeatMs: 5000,                     // leader-election heartbeat
    lockTtlMs: 15000,                      // 3× heartbeat is the safe ratio

    tenantIsolation: 'channel-prefix',     // 'channel-prefix' | 'none'
    // driverOptions: { /* opaque, driver-specific options */ },
  },
})

The same object can be passed lower down when you are wiring the kernel by hand, or used to build a bare service:

import { ClusterServicePlugin, defineCluster } from '@objectstack/service-cluster'

kernel.use(new ClusterServicePlugin({ config: { driver: 'memory', nodeId: 'web-1' } }))

// …or standalone, outside a kernel
const cluster = defineCluster({ driver: 'memory' })

Under os serve the driver is selected from the environment instead: OS_CLUSTER_DRIVER (anything other than memory triggers a dynamic import of @objectstack/service-cluster-<driver>) and OS_REDIS_URL for the connection string. A distribution may register a multi-node gate; when it denies, os serve logs a warning and downgrades to the single-node in-memory cluster rather than failing.

A gate may instead admit fewer nodes than the deployment declares — a licensed node-cap overflow. That is a different verdict from a denial and is deliberately not a downgrade: the cluster is entitled to run, it simply asked for more nodes than it paid for. os serve passes the declared OS_CLUSTER_REPLICAS to the gate and, when the declaration exceeds what the gate admits, prints an advisory naming both counts:

[cluster] licensed node cap exceeded: the licence admits 3 node(s), but
OS_CLUSTER_REPLICAS declares 5 — 2 beyond the cap.
[cluster] This cap is ADVISORY and is not enforced yet: nothing is refused, and
all 5 replicas will still join the cluster.
[cluster] Reduce OS_CLUSTER_REPLICAS to 3, or raise the licensed node limit.

Note this reads the operator-declared count, not live membership: no membership view exists at boot. It is the right input for telling an operator about the configuration they wrote, and is deliberately not sufficient for enforcement.

Every cluster primitive selects its implementation from cluster.driver. When the field is absent, Runtime auto-registers the in-memory, single-node driver (pass cluster: false to skip that auto-registration). No production warning is emitted for that case. There is a split-brain guard: if the operator declares a multi-node topology via OS_EXPECT_MULTI_NODE=true or OS_CLUSTER_REPLICAS > 1 while the resolved driver is memory, startup throws — override with OS_ALLOW_MEMORY_CLUSTER_MULTINODE=true.

8.1 Driver matrix

ClusterDriverSchema accepts exactly the drivers that ship — memory and redis — plus custom, which resolves whatever a plugin registered via registerClusterDriver(name, factory). It used to also accept postgres and nats with no package behind either, so a schema-valid config reached defineCluster()'s unconditional Cluster driver "<name>" is not registered throw; both values were removed from the enum (maintainer ruling on cloud#1626, 2026-08-24 — a schema-valid value must not be an unconditional runtime throw). Their design sketches — postgres as LISTEN/NOTIFY + advisory locks + a KV table + a sequence, nats as subjects + JetStream KV — stay recorded in Phase 5 below for whoever builds one.

The redis driver is the recommended starting point for production ObjectStack deployments: it accepts a pre-built ioredis client via driverOptions.client, so it can share one connection pool with any other Redis consumer (the service-cache Redis adapter is still a skeleton, so there is nothing to share with yet); it gives native semantics for every primitive (SET NX/Lua/INCRBY/PUBLISH); and it decouples cluster transport from the choice of database driver (SQL / Turso / Postgres / SQLite all work the same).

A postgres driver would remain a viable "single binary, single dependency" choice for small deployments that already run Postgres and want to avoid a Redis dependency, but nobody has built one — it is deferred to a community/optional driver against the same registerClusterDriver() SPI.

9. Industry alignment

These choices follow patterns proven across the enterprise platform ecosystem; none are novel.

• Salesforce Platform Events — cluster-wide pub/sub with at-least-once delivery; tenant-partitioned channels. Maps directly to our scope: 'cluster' + 'tenant'.
• ServiceNow Cluster Cache + Event Management — version-stamped cache with invalidation broadcast; leader-elected schedulers. Maps to §6 and leader-elected strategy.
• Odoo bus.bus — Postgres LISTEN/NOTIFY for cross-worker fan-out. Validates LISTEN/NOTIFY as a sound transport for the deferred postgres driver (§8.1, Phase 5) — ObjectStack shipped redis first instead.
• Notion / Linear — Redis pub/sub for realtime fan-out with per-document partitioning. Validates partitionKey ordering as the right primitive.
• Kubernetes leader election — Lease objects with TTL + heartbeat. The leader-elected strategy in §5 follows the same pattern.

10. Migration plan

This ADR is additive. Existing plugins continue to compile and run unchanged. The protocol additions (a new cluster.zod.ts file plus optional fields on existing schemas) ship in spec v5.x without breaking v4 consumers.

Phase 1 — Protocol (this work, ~1 day)

1. Add kernel/cluster.zod.ts — schemas for EventScope, DeliverySemantics, ClusterScope, LeaderStrategy, ClusterConfig.
2. Extend events/core.zod.ts EventMetadataSchema with an optional nested cluster object (scope, deliverySemantics, partitionKey).
3. Extend kernel/service-registry.zod.ts ServiceMetadataSchema and ServiceFactoryRegistrationSchema with an optional nested cluster object (clusterScope, leaderStrategy).
4. Add optional cluster field to stack.zod.ts. Dropped — cluster config landed on RuntimeConfig instead (see §8); stack.zod.ts has no cluster field.

Steps 1-3 shipped. No runtime changes in this phase: the protocol describes cluster semantics; nothing enforces them.

Phase 2 — In-memory runtime (~2 days) ✅

Implement service-cluster with the memory driver only. Every cluster primitive works; behaviour on one node is identical to today. This phase proves the API shape against real callers before committing to a wire format. In practice the only callers wired so far are MetadataClusterBridgePlugin, service-job's cron scheduler, and service-messaging's dispatchers — service-cache and service-queue still do not consume the cluster service.

Phase 3 — Redis driver (~2 days) ✅

Implement redis driver via @objectstack/service-cluster-redis (PUBLISH/SUBSCRIBE + SET NX PX/Lua locks + WATCH/MULTI KV + INCRBY counter). Ships as a sibling package so adopters opt in by importing it — the import self-registers the driver via registerClusterDriver('redis', …). It can share an ioredis client with other consumers by passing driverOptions.client, though nothing shares one with it today.

Note: this was switched from "Postgres first" — see commit history. Rationale: Postgres isn't a fixed dependency in the ObjectStack stack, so binding cluster to PG would couple the two unnecessarily. Redis is already the planned cache backend.

Phase 4 — Existing services migrate (~3 days, opportunistic)

Not started. service-job already leader-elects by calling cluster.lock.acquire('job:<name>') directly, but that is hand-rolled — none of the declarative wiring below exists yet.

• service-job declares clusterScope: 'cluster' / 'leader-elected'.
• service-cache gains a cluster-backed cache derived from KV + PubSub.
• service-realtime adds cross-node fan-out via PubSub.
• Webhook dispatcher declares the partitioned strategy. (The behaviour already ships hand-rolled: webhook deliveries drain through service-messaging's HttpDispatcher, which walks partitionCount partitions behind per-partition cluster.lock keys — only the declarative annotation is missing.)

Phase 5 — Postgres / NATS drivers (deferred, optional)

Postgres driver downgraded to community/optional — useful for the "one binary, one container" deployment archetype that wants to skip Redis. NATS deferred until a customer reports throughput needs that exceed Redis PUBSUB. A community driver ships under driver: 'custom' via registerClusterDriver() with no protocol change; a first-party driver restores its enum value in the same release that ships the implementation (the cloud#1626 ruling records that reversal condition — the value returns only with an implementation behind it).

11. Non-goals (v1)

• Exactly-once delivery — accepted as a keyword in the protocol; startup rejection of the value is intended but not yet implemented. Implementing it correctly requires distributed transactions or idempotency tokens that are out of scope for v1.
• Cross-region replication — single-region clusters only. Multi-region is a future ADR.
• Streaming protocol — Kafka-style consumer groups not modelled here. Will arrive as a fifth primitive when needed.
• Schemaful event registry enforcement at the bus layer — event names are validated by metadata, not by the bus. The bus stays string-typed for plugin extensibility.