Cluster Semantics

Status: Accepted · Audience: Plugin authors, runtime engineers

TL;DR — ObjectStack runs identically on a laptop and on a 10-node cluster. To make that true, a small number of runtime primitives need explicit semantics in the protocol layer: how events propagate, whether a service is one-per-node or one-per-cluster, and how metadata changes invalidate caches everywhere at once. This document fixes those semantics.

1. Why this document exists

The protocol layer (@objectstack/spec) was designed top-down: it describes what the system looks like (objects, fields, views, flows, agents) and deliberately stays silent about where code runs. That silence was correct during bootstrap — it let us ship the metadata-first vision without prematurely locking in a runtime topology.

It is no longer correct. As ObjectStack approaches its first production deployments, three classes of bugs become inevitable on any multi-node setup:

1. Stale caches. Node A updates an object's label; Node B continues to serve the old value from its in-process metadata cache because the change event never crossed the process boundary.
2. Duplicated work. A schedule-triggered Flow fires once on every node; a single nightly job runs N times.
3. Lost or doubled deliveries. Two webhook dispatchers pick up the same pending row; an external system receives the same event twice with no safe way to deduplicate.

These are not implementation bugs. They are protocol omissions — the spec never told plugin authors what "emit an event" or "register a job service" should mean when more than one process is involved.

This document closes those omissions. It is the source of truth for cluster behaviour and the basis for the kernel/cluster.zod.ts schema that codifies it.

2. Design principles

Every decision below follows three non-negotiable rules.

P1 — Single-machine remains zero-config. A developer who runs pnpm dev must never need Redis, a message broker, or any external state. The default driver for every cluster primitive is in-process; the API surface is identical to the distributed driver.

P2 — Distributed semantics live in the protocol, not in the implementation. A plugin author should be able to write one version of their code that is correct on both topologies. They achieve that by declaring intent ("this job runs once per cluster", "this event should reach every node") in metadata or through context APIs — never by branching on if (cluster) { ... }.

P3 — Default to the safe semantic, not the cheap one. When in doubt, events stay node-local, services stay node-singletons, and leader-elected work is opt-in. This guarantees that a plugin written without thinking about clustering still behaves correctly when deployed to one — it just may not benefit from horizontal scale until annotated.

3. The four cluster primitives

The protocol exposes exactly four primitives. Everything else (caches, queues, schedulers, realtime fan-out, webhook delivery, audit shipping) is built on top of them.

These four are the minimum sufficient set. Higher-level facilities (runOnce, withLock, broadcast, leaderElect, cache invalidation) are derived; their semantics are defined in §5.

Why these four and not more

• No native job queue primitive. A job queue is KV + Lock + PubSub composed; expressing it directly in the cluster layer would couple the protocol to a specific queue shape. service-queue is expected to build on the primitives instead — today it ships memory and DB adapters only and does not consume the cluster service.
• No native cache primitive. Caches are KV + PubSub (read the value, subscribe to the invalidation channel). service-cache is expected to build on the primitives instead — today it ships a memory adapter plus a Redis adapter skeleton that throws RedisCacheAdapter not yet implemented, and it does not consume the cluster service either.
• No native stream/Kafka primitive. Streaming is intentionally out of scope for v1. It can be added as a fifth primitive later without changing the four above.

4. Event scope and delivery semantics

The single biggest source of cluster bugs is "I published an event and assumed everyone would hear it". Every emit must answer three questions:

Status: protocol only. The three fields below exist as EventMetadata.cluster (EventClusterOptionsSchema in kernel/cluster.zod.ts, wired into kernel/events/core.zod.ts), but no runtime consumes them — there is no eventBus object in the codebase at all. The kernel's plugin event API is ctx.hook(name, handler) / ctx.trigger(name, ...args), neither of which takes cluster metadata. The shipped cross-node path today is a direct cluster.pubsub.publish(...) call (§6.2, §7.3). §4.1-§4.4 define what these enum values are supposed to mean once a bus reads them.

4.1 Scope — who receives this event?

scope: 'local' | 'cluster' | 'tenant'

• local (default) — Delivered only to handlers in the emitting process. Use for: in-request hooks, derived-state recomputation that the emitting node will use itself, anything cheap to recompute.
• cluster — Delivered to every node in the cluster, including the emitter. Use for: cache invalidations, configuration reloads, metadata changes, anything where every node must converge to the same view.
• tenant — Same as cluster, but delivery is partitioned by tenantId so only nodes currently serving that tenant receive it. Use for: tenant- scoped cache invalidations on large multi-tenant deployments where broadcasting to all nodes would waste bandwidth.

Default rationale. local is safe by accident: a plugin author who never thinks about clustering produces code that runs identically on one or N nodes — it just doesn't gain any cluster-aware behaviour until they opt in.

4.2 Delivery semantics — what guarantees does the bus offer?

deliverySemantics: 'best-effort' | 'at-least-once' | 'exactly-once'

• best-effort (default for local) — In-process delivery. Lost if a handler throws or the process crashes before the handler runs. Use for: UI hints, telemetry, anything where loss is recoverable.
• at-least-once (default for cluster / tenant) — Persisted to the transport (Redis Streams, Postgres outbox table) before publish returns. Survives node crash. Handlers must be idempotent — duplicates are possible during retry. Use for: webhook outbox, audit shipping, async job enqueue. No shipped driver provides this yet. The redis driver publishes over plain Redis pub/sub, which is at-most-once — fire-and-forget, no persistence, no replay for a node that was down at publish time. Only route cache-invalidation hints through it; anything that must survive a crash needs a durable outbox of its own.
• exactly-once — Reserved keyword; not implemented in v1. The protocol accepts the enum value so future runtimes can add it without a breaking change. Startup rejection of this value is intended behaviour but is not yet implemented — the bus does not currently inspect or reject it.

4.3 Partition key — what ordering does the bus preserve?

partitionKey?: string

When set, the bus guarantees that two events with the same partitionKey are delivered to handlers in emit order, even with multiple consumers. Events with different partition keys may interleave freely. Typical partition keys: record id, tenant id, conversation id.

When unset, no ordering is guaranteed — handlers may run in parallel and out of emit order.

Status: accepted but ignored. PublishOptions.partitionKey is part of the IPubSub contract and callers already pass it (the metadata manager keys on `${type}:${name}`), but both shipped drivers take the options bag as an unused _opts parameter. The memory driver's synchronous single-process fan-out preserves emit order anyway; the redis driver gives you whatever ordering Redis pub/sub does. Do not rely on cross-key ordering guarantees yet.

4.4 The combined contract

The three fields compose into one EventMetadata.cluster block:

// target shape — parses today, nothing reads it yet
const metadata = {
  source: 'my-plugin',
  timestamp: new Date().toISOString(),
  cluster: {
    scope: 'cluster',
    deliverySemantics: 'at-least-once',
    partitionKey: payload.id,
  },
}

Once a bus honours it, that combination is the intended way to invalidate a per-record cache entry across a cluster: the cluster scope reaches every node, at-least-once guarantees no node misses the invalidation across a crash, and partitionKey ensures that two rapid updates to the same record are applied in order.

Until then, the working equivalent is a direct publish on the cluster service — it accepts partitionKey (see the §4.3 caveat) and gives best-effort delivery only:

import type { IClusterService } from '@objectstack/spec/contracts'

const cluster = ctx.getService<IClusterService>('cluster')
await cluster.pubsub.publish('account.updated', payload, {
  partitionKey: payload.id,
})

5. Service scope and leader election

Every service registered with the kernel has a cluster scope. The annotation is optional — omitting it is equivalent to { clusterScope: 'node' }. This is independent of DI lifecycle scope (singleton / transient / scoped); it answers a different question.

A cluster-scoped service must declare how the single-active-instance invariant is maintained:

leaderStrategy: 'leader-elected' | 'partitioned' | 'idempotent-broadcast'

• leader-elected — Exactly one node holds the leadership lock at any time; non-leader nodes idle. The runtime handles election via the Lock primitive (heartbeat + TTL). Use for: cron schedulers, anything that must run "once per tick".
• partitioned — Every node runs the service, but each instance owns a disjoint partition of the work (typically hashed on partitionKey). Use for: webhook dispatchers (partition by webhook_id), high-throughput workers.
• idempotent-broadcast — Every node runs the service on every input, and the work itself is idempotent (writes use UPSERT, side effects are keyed). Use for: cache invalidation handlers, projection rebuilders.

These annotations are nested under a cluster key on the service registration (ServiceMetadata.cluster / ServiceFactoryRegistration.cluster = ServiceClusterAnnotations), not declared as flat top-level fields:

{
  name: 'cron-scheduler',
  cluster: {
    clusterScope: 'cluster',
    leaderStrategy: 'leader-elected',
    // clusterId: 'scheduler',  // optional — share one leadership lock
    //   across physically different services (e.g. safe rolling upgrades);
    //   defaults to the service name.
  },
}

Default rationale. node is the only safe default. A plugin author who forgets to declare scope gets the single-machine behaviour, which is always correct; opting up to cluster is an explicit decision.

5.1 What the runtime will give you for free

Status: not yet implemented (Phase 4). The schema accepts clusterScope / leaderStrategy annotations today, but no runtime code yet consumes those annotations to wire leadership automatically — the four behaviours below are the intended Phase 4 (see §10) contract. Leader election itself already works via the Lock primitive (service-job's cron scheduler calls cluster.lock.acquire(...) directly so only one node fires each scheduled job); what Phase 4 adds is the kernel doing it for you from a service's declared clusterScope.

When a service declares clusterScope: 'cluster', the runtime will automatically:

1. Wrap the plugin's start(ctx) so the work only begins after lock acquisition (for leader-elected).
2. Refresh the lock TTL via heartbeat while the node is healthy.
3. Release the lock on destroy() / the kernel:shutdown hook.
4. Emit a leadership-change event so dependent services can react.

(The kernel's plugin contract is init(ctx) / start(ctx) / destroy(). The onEnable / onDisable family was retired from the protocol — it never had an invocation site — so do not reach for it here.)

A plugin author writing a cron scheduler will not write any lock code — they declare scope and strategy, and the runtime does the rest.

6. Metadata versioning and cache coherence

Metadata is the most cache-hot data in the system. Every read path — ObjectQL planning, REST routing, UI rendering, permission checks — touches it. Making it correct across a cluster requires two things the current protocol lacks: a monotonic version per item and a well-defined invalidation event.

6.1 Monotonic version

A version column is already present on every persisted metadata record in system/metadata-persistence.zod.ts (version: z.number(), used today for optimistic concurrency). This ADR proposes formalising it as a monotonic per-item version and widening it to bigint so caches can compare freshness across long-running clusters without overflow.

Status: planned. The column is still version: number in the schema; the widening and the cache-comparison contract below describe the target design, not current runtime behaviour.

Once wired, caches store {value, version}. On any incoming change notification the cache compares the incoming version with the stored one:

• Incoming version > stored → invalidate (apply new value or evict)
• Incoming version ≤ stored → ignore (out-of-order notification, already superseded)

This eliminates a whole class of bugs where a slow-arriving "old" invalidation evicts a "newer" value the node has already learned about.

6.2 The metadata change event

Current behaviour. Cross-node metadata invalidation already works, but not through any event bus (there isn't one — see §4). The metadata manager fans out over the cluster PubSub channel metadata.changed (note: dot, not colon) with the payload shape ClusterMetadataChangedPayload:

// packages/metadata/src/metadata-manager.ts — notifyWatchers()
this.clusterPubSub.publish('metadata.changed', {
  originNode: this.clusterNodeId, // used for loopback suppression
  type,                           // 'object' | 'view' | 'dashboard' | …
  event,                          // the local watch event, replayed verbatim on peers
}, { partitionKey: `${type}:${event.name ?? ''}` })

The transport is handed to the manager by MetadataClusterBridgePlugin (@objectstack/service-cluster), which calls metadata.attachClusterPubSub(cluster.pubsub, cluster.nodeId) on kernel:ready. Runtime registers that bridge automatically alongside the cluster service.

On receipt a peer suppresses its own messages by originNode, then — first, synchronously — invalidates its local caches for that type: it drops the registry entry named by the nested watch event and the list(type) cache (invalidateForForeignWrite). Only then does it replay the watch event into its local watch hub, deferred by one tick so a slow consumer callback cannot back-pressure the pubsub dispatch loop. That order is what lets a woken consumer answer by re-reading through list() and see the write rather than the pre-write set (#5109). The registry entry is deleted, never pre-filled from the payload — the peer re-reads the shared store, which is the source of truth.

At the payload's top level there is still no version / name / tenantId / operation field and no version comparison; the item's name is carried only inside the replayed event.

Target spec (planned). The richer, version-stamped payload below is defined as MetadataChangedEventPayloadSchema in kernel/cluster.zod.ts but is not yet wired into the runtime:

// MetadataChangedEventPayloadSchema — target shape, not yet emitted
{
  type: 'object' | 'view' | 'flow' | …,
  name: '<machine-name>',
  tenantId?: '<tenant>',
  version: <new-version>,
  operation: 'create' | 'update' | 'delete' | 'publish',
}

When wired with scope: 'cluster' + deliverySemantics: 'at-least-once' and a partitionKey of `${type}:${name}`, the partition key would guarantee that two rapid updates to the same item are applied to every node's cache in order, and the at-least-once guarantee would let a briefly partitioned node catch up on reconnect.

6.3 Reader contract

Status: planned. Today readers are invalidated by the metadata.changed PubSub fan-out described in §6.2, which replays watch events verbatim without version comparison. The contract below is the target design that goes with the version-stamped payload.

All metadata readers (registry, loader, query engine) should:

1. Subscribe to the metadata change channel on startup.
2. Compare incoming version with cached version before evicting.
3. Treat missing version as 0 (legacy compatibility).

7. Plugin author guidance

This section is a quick reference for plugin authors. The full details live in §4-§6; this is the cheat sheet.

7.1 "I want to fire an event"

Target design only — no runtime reads EventMetadata.cluster today (§4). Pick scope first, then delivery, then partition key:

If none of the above apply, stay with the defaults (local + best-effort) — they are always correct on a single node and never wrong on a cluster, they just don't propagate.

7.2 "I want to register a service"

Ask: if I deploy 5 nodes, should there be 5 of these or 1?

• 5 → clusterScope: 'node' (default). You're done.
• 1 → clusterScope: 'cluster' + pick a leaderStrategy:
  • "Wakes on a timer / cron" → leader-elected
  • "Pulls from a shared queue" → partitioned
  • "Reacts to events with idempotent writes" → idempotent-broadcast

Declaring cluster does not yet do anything (§5.1) — until Phase 4 lands you still have to hold the invariant yourself, either by wrapping the work in cluster.lock.withLock(key, fn) or by acquiring and releasing by hand the way service-job's cron scheduler does (cluster.lock.acquire(key, { waitMs: 0 }), bail out when it returns null, release in a finally).

7.3 "I want to read metadata in a long-lived cache"

There is no cluster.cache(...) factory, and there is no ctx.cluster property either — PluginContext exposes the service-registry family (registerService / registerServiceFactory / getService / getServiceScoped / getServices / replaceService), hook / trigger, logger and getKernel. No cluster handle is among them, so resolve the cluster service by name:

import type { IClusterService } from '@objectstack/spec/contracts'

const cluster = ctx.getService<IClusterService>('cluster')

It exposes only the four primitives (cluster.{pubsub, lock, kv, counter}, see §3) plus nodeId / driver / close(). Caches are meant to be derived from KV + PubSub; they are not a built-in cluster method.

To keep an in-process metadata cache coherent today, subscribe to the metadata change channel via PubSub and invalidate on each notification:

cluster.pubsub.subscribe('metadata.changed', (msg) => {
  const payload = msg.payload // { originNode?, type, event }
  if (payload.type === 'object') {
    myObjectCache.invalidate(payload.event)
  }
})

A higher-level cache factory (with automatic version comparison) is part of the Phase 4 / §6 target design but is not yet implemented. No if (cluster) branches are needed either way — the memory driver makes the subscribe call a no-op-equivalent local fan-out on a single node.

7.4 What you must never do

• Never call setInterval for cluster-wide periodic work — gate each tick on cluster.lock (and declare clusterScope: 'cluster' + leader-elected so Phase 4 can take the wiring over from you).
• Never SELECT … WHERE status = 'pending' LIMIT 1 without a row lock — use SELECT … FOR UPDATE SKIP LOCKED (Postgres) or UPDATE … RETURNING with a claimed_by column.
• Never write into a process-local Map and assume other nodes see it — use cluster.kv for shared state.
• Never import Redis from 'ioredis' directly — go through cluster.{pubsub, lock, kv, counter}.

8. Configuration surface

Cluster behaviour is configured once, at the runtime level. The shape is ClusterCapabilityConfig (kernel/cluster.zod.ts), and it is passed on the Runtime constructor:

import { Runtime } from '@objectstack/runtime'

const runtime = new Runtime({
  cluster: {
    driver: 'memory',   // default — single-process
    // driver: 'redis', url: process.env.OS_REDIS_URL,
    // driver: 'custom',  // resolved from registerClusterDriver()

    nodeId: process.env.NODE_ID,           // optional, auto-generated if absent
    heartbeatMs: 5000,                     // leader-election heartbeat
    lockTtlMs: 15000,                      // 3× heartbeat is the safe ratio

    tenantIsolation: 'channel-prefix',     // 'channel-prefix' | 'none'
    // driverOptions: { /* opaque, driver-specific options */ },
  },
})

The same object can be passed lower down when you are wiring the kernel by hand, or used to build a bare service:

import { ClusterServicePlugin, defineCluster } from '@objectstack/service-cluster'

kernel.use(new ClusterServicePlugin({ config: { driver: 'memory', nodeId: 'web-1' } }))

// …or standalone, outside a kernel
const cluster = defineCluster({ driver: 'memory' })

Under os serve the driver is selected from the environment instead: OS_CLUSTER_DRIVER (anything other than memory triggers a dynamic import of @objectstack/service-cluster-<driver>) and OS_REDIS_URL for the connection string. A distribution may register a multi-node gate; when it denies, os serve logs a warning and downgrades to the single-node in-memory cluster rather than failing.

Every cluster primitive selects its implementation from cluster.driver. When the field is absent, Runtime auto-registers the in-memory, single-node driver (pass cluster: false to skip registration entirely). No production warning is emitted for that case. There is a split-brain guard: if the operator declares a multi-node topology via OS_EXPECT_MULTI_NODE=true or OS_CLUSTER_REPLICAS > 1 while the resolved driver is memory, startup throws — override with OS_ALLOW_MEMORY_CLUSTER_MULTINODE=true.

8.1 Driver matrix

Only memory and redis are implemented. postgres and nats are accepted by ClusterDriverSchema but no package provides them — defineCluster() throws Cluster driver "<name>" is not registered for either. The custom driver value resolves whatever a plugin registered via registerClusterDriver(name, factory).

The redis driver is the recommended starting point for production ObjectStack deployments: it accepts a pre-built ioredis client via driverOptions.client, so it can share one connection pool with any other Redis consumer (the service-cache Redis adapter is still a skeleton, so there is nothing to share with yet); it gives native semantics for every primitive (SET NX/Lua/INCRBY/PUBLISH); and it decouples cluster transport from the choice of database driver (SQL / Turso / Postgres / SQLite all work the same).

A postgres driver would remain a viable "single binary, single dependency" choice for small deployments that already run Postgres and want to avoid a Redis dependency, but nobody has built one — it is deferred to a community/optional driver against the same registerClusterDriver() SPI.

9. Industry alignment

These choices follow patterns proven across the enterprise platform ecosystem; none are novel.

• Salesforce Platform Events — cluster-wide pub/sub with at-least-once delivery; tenant-partitioned channels. Maps directly to our scope: 'cluster' + 'tenant'.
• ServiceNow Cluster Cache + Event Management — version-stamped cache with invalidation broadcast; leader-elected schedulers. Maps to §6 and leader-elected strategy.
• Odoo bus.bus — Postgres LISTEN/NOTIFY for cross-worker fan-out. Validates LISTEN/NOTIFY as a sound transport for the deferred postgres driver (§8.1, Phase 5) — ObjectStack shipped redis first instead.
• Notion / Linear — Redis pub/sub for realtime fan-out with per-document partitioning. Validates partitionKey ordering as the right primitive.
• Kubernetes leader election — Lease objects with TTL + heartbeat. The leader-elected strategy in §5 follows the same pattern.

10. Migration plan

This ADR is additive. Existing plugins continue to compile and run unchanged. The protocol additions (a new cluster.zod.ts file plus optional fields on existing schemas) ship in spec v5.x without breaking v4 consumers.

Phase 1 — Protocol (this work, ~1 day)

1. Add kernel/cluster.zod.ts — schemas for EventScope, DeliverySemantics, ClusterScope, LeaderStrategy, ClusterConfig.
2. Extend events/core.zod.ts EventMetadataSchema with an optional nested cluster object (scope, deliverySemantics, partitionKey).
3. Extend kernel/service-registry.zod.ts ServiceMetadataSchema and ServiceFactoryRegistrationSchema with an optional nested cluster object (clusterScope, leaderStrategy).
4. Add optional cluster field to stack.zod.ts. Dropped — cluster config landed on RuntimeConfig instead (see §8); stack.zod.ts has no cluster field.

Steps 1-3 shipped. No runtime changes in this phase: the protocol describes cluster semantics; nothing enforces them.

Phase 2 — In-memory runtime (~2 days) ✅

Implement service-cluster with the memory driver only. Every cluster primitive works; behaviour on one node is identical to today. This phase proves the API shape against real callers before committing to a wire format. In practice the only callers wired so far are MetadataClusterBridgePlugin, service-job's cron scheduler, and service-messaging's dispatchers — service-cache and service-queue still do not consume the cluster service.

Phase 3 — Redis driver (~2 days) ✅

Implement redis driver via @objectstack/service-cluster-redis (PUBLISH/SUBSCRIBE + SET NX PX/Lua locks + WATCH/MULTI KV + INCRBY counter). Ships as a sibling package so adopters opt in by importing it — the import self-registers the driver via registerClusterDriver('redis', …). It can share an ioredis client with other consumers by passing driverOptions.client, though nothing shares one with it today.

Note: this was switched from "Postgres first" — see commit history. Rationale: Postgres isn't a fixed dependency in the ObjectStack stack, so binding cluster to PG would couple the two unnecessarily. Redis is already the planned cache backend.

Phase 4 — Existing services migrate (~3 days, opportunistic)

Not started. service-job already leader-elects by calling cluster.lock.acquire('job:<name>') directly, but that is hand-rolled — none of the declarative wiring below exists yet.

• service-job declares clusterScope: 'cluster' / 'leader-elected'.
• service-cache gains a cluster-backed cache derived from KV + PubSub.
• service-realtime adds cross-node fan-out via PubSub.
• Webhook dispatcher declares the partitioned strategy. (The behaviour already ships hand-rolled: webhook deliveries drain through service-messaging's HttpDispatcher, which walks partitionCount partitions behind per-partition cluster.lock keys — only the declarative annotation is missing.)

Phase 5 — Postgres / NATS drivers (deferred, optional)

Postgres driver downgraded to community/optional — useful for the "one binary, one container" deployment archetype that wants to skip Redis. NATS deferred until a customer reports throughput needs that exceed Redis PUBSUB. No protocol changes required at that point — only a new implementation that calls registerClusterDriver().

11. Non-goals (v1)

• Exactly-once delivery — accepted as a keyword in the protocol; startup rejection of the value is intended but not yet implemented. Implementing it correctly requires distributed transactions or idempotency tokens that are out of scope for v1.
• Cross-region replication — single-region clusters only. Multi-region is a future ADR.
• Streaming protocol — Kafka-style consumer groups not modelled here. Will arrive as a fifth primitive when needed.
• Schemaful event registry enforcement at the bus layer — event names are validated by metadata, not by the bus. The bus stays string-typed for plugin extensibility.