Cluster Semantics

Status: Accepted · Audience: Plugin authors, runtime engineers

TL;DR — ObjectStack runs identically on a laptop and on a 10-node cluster. To make that true, a small number of runtime primitives need explicit semantics in the protocol layer: how events propagate, whether a service is one-per-node or one-per-cluster, and how metadata changes invalidate caches everywhere at once. This document fixes those semantics.

1. Why this document exists

The protocol layer (@objectstack/spec) was designed top-down: it describes what the system looks like (objects, fields, views, flows, agents) and deliberately stays silent about where code runs. That silence was correct during bootstrap — it let us ship the metadata-first vision without prematurely locking in a runtime topology.

It is no longer correct. As ObjectStack approaches its first production deployments, three classes of bugs become inevitable on any multi-node setup:

1. Stale caches. Node A updates an object's label; Node B continues to serve the old value from its in-process metadata cache because the change event never crossed the process boundary.
2. Duplicated work. A schedule-triggered Flow fires once on every node; a single nightly job runs N times.
3. Lost or doubled deliveries. Two webhook dispatchers pick up the same pending row; an external system receives the same event twice with no safe way to deduplicate.

These are not implementation bugs. They are protocol omissions — the spec never told plugin authors what "emit an event" or "register a job service" should mean when more than one process is involved.

This document closes those omissions. It is the source of truth for cluster behaviour and the basis for the kernel/cluster.zod.ts schema that codifies it.

2. Design principles

Every decision below follows three non-negotiable rules.

P1 — Single-machine remains zero-config. A developer who runs pnpm dev must never need Redis, a message broker, or any external state. The default driver for every cluster primitive is in-process; the API surface is identical to the distributed driver.

P2 — Distributed semantics live in the protocol, not in the implementation. A plugin author should be able to write one version of their code that is correct on both topologies. They achieve that by declaring intent ("this job runs once per cluster", "this event should reach every node") in metadata or through context APIs — never by branching on if (cluster) { ... }.

P3 — Default to the safe semantic, not the cheap one. When in doubt, events stay node-local, services stay node-singletons, and leader-elected work is opt-in. This guarantees that a plugin written without thinking about clustering still behaves correctly when deployed to one — it just may not benefit from horizontal scale until annotated.

3. The four cluster primitives

The protocol exposes exactly four primitives. Everything else (caches, queues, schedulers, realtime fan-out, webhook delivery, audit shipping) is built on top of them.

These four are the minimum sufficient set. Higher-level facilities (runOnce, withLock, broadcast, leaderElect, cache invalidation) are derived; their semantics are defined in §5.

Why these four and not more

• No native job queue primitive. A job queue is KV + Lock + PubSub composed; expressing it directly in the cluster layer would couple the protocol to a specific queue shape. service-queue builds on the primitives instead.
• No native cache primitive. Caches are KV + PubSub (read the value, subscribe to the invalidation channel). service-cache builds on the primitives instead.
• No native stream/Kafka primitive. Streaming is intentionally out of scope for v1. It can be added as a fifth primitive later without changing the four above.

4. Event scope and delivery semantics

The single biggest source of cluster bugs is "I called eventBus.emit(...) and assumed everyone would hear it". Every emit must answer three questions:

4.1 Scope — who receives this event?

scope: 'local' | 'cluster' | 'tenant'

• local (default) — Delivered only to handlers in the emitting process. Use for: in-request hooks, derived-state recomputation that the emitting node will use itself, anything cheap to recompute.
• cluster — Delivered to every node in the cluster, including the emitter. Use for: cache invalidations, configuration reloads, metadata changes, anything where every node must converge to the same view.
• tenant — Same as cluster, but delivery is partitioned by tenantId so only nodes currently serving that tenant receive it. Use for: tenant- scoped cache invalidations on large multi-tenant deployments where broadcasting to all nodes would waste bandwidth.

Default rationale. local is safe by accident: a plugin author who never thinks about clustering produces code that runs identically on one or N nodes — it just doesn't gain any cluster-aware behaviour until they opt in.

4.2 Delivery semantics — what guarantees does the bus offer?

deliverySemantics: 'best-effort' | 'at-least-once' | 'exactly-once'

• best-effort (default for local) — In-process delivery. Lost if a handler throws or the process crashes before the handler runs. Use for: UI hints, telemetry, anything where loss is recoverable.
• at-least-once (default for cluster / tenant) — Persisted to the transport (Redis Streams, Postgres outbox table) before publish returns. Survives node crash. Handlers must be idempotent — duplicates are possible during retry. Use for: webhook outbox, audit shipping, async job enqueue.
• exactly-once — Reserved keyword; not implemented in v1. The protocol accepts the enum value so future runtimes can add it without a breaking change. Startup rejection of this value is intended behaviour but is not yet implemented — the bus does not currently inspect or reject it.

4.3 Partition key — what ordering does the bus preserve?

partitionKey?: string

When set, the bus guarantees that two events with the same partitionKey are delivered to handlers in emit order, even with multiple consumers. Events with different partition keys may interleave freely. Typical partition keys: record id, tenant id, conversation id.

When unset, no ordering is guaranteed — handlers may run in parallel and out of emit order.

4.4 The combined contract

eventBus.emit('account.updated', payload, {
  cluster: {
    scope: 'cluster',
    deliverySemantics: 'at-least-once',
    partitionKey: payload.id,
  },
})

This is the only correct way to invalidate a per-record cache entry across a cluster: the cluster scope reaches every node, at-least-once guarantees no node misses the invalidation across a crash, and partitionKey ensures that two rapid updates to the same record are applied in order.

5. Service scope and leader election

Every service registered with the kernel must declare its cluster scope. This is independent of DI lifecycle scope (singleton / transient / scoped); it answers a different question.

A cluster-scoped service must declare how the single-active-instance invariant is maintained:

leaderStrategy: 'leader-elected' | 'partitioned' | 'idempotent-broadcast'

• leader-elected — Exactly one node holds the leadership lock at any time; non-leader nodes idle. The runtime handles election via the Lock primitive (heartbeat + TTL). Use for: cron schedulers, anything that must run "once per tick".
• partitioned — Every node runs the service, but each instance owns a disjoint partition of the work (typically hashed on partitionKey). Use for: webhook dispatchers (partition by webhook_id), high-throughput workers.
• idempotent-broadcast — Every node runs the service on every input, and the work itself is idempotent (writes use UPSERT, side effects are keyed). Use for: cache invalidation handlers, projection rebuilders.

These annotations are nested under a cluster key on the service registration (ServiceMetadata.cluster / ServiceFactoryRegistration.cluster = ServiceClusterAnnotations), not declared as flat top-level fields:

{
  name: 'cron-scheduler',
  cluster: {
    clusterScope: 'cluster',
    leaderStrategy: 'leader-elected',
    // clusterId: 'scheduler',  // optional — share one leadership lock
    //   across physically different services (e.g. safe rolling upgrades);
    //   defaults to the service name.
  },
}

Default rationale. node is the only safe default. A plugin author who forgets to declare scope gets the single-machine behaviour, which is always correct; opting up to cluster is an explicit decision.

5.1 What the runtime will give you for free

Status: not yet implemented (Phase 4). The schema accepts clusterScope / leaderStrategy annotations today, but no runtime code yet consumes them or performs leader election. The behaviour below is the intended contract once Phase 4 (see §10) lands.

When a service declares clusterScope: 'cluster', the runtime will automatically:

1. Wrap onEnable so the work only starts after lock acquisition (for leader-elected).
2. Refresh the lock TTL via heartbeat while the node is healthy.
3. Release the lock on onDisable or graceful shutdown.
4. Emit a leadership-change event so dependent services can react.

A plugin author writing a cron scheduler will not write any lock code — they declare scope and strategy, and the runtime does the rest.

6. Metadata versioning and cache coherence

Metadata is the most cache-hot data in the system. Every read path — ObjectQL planning, REST routing, UI rendering, permission checks — touches it. Making it correct across a cluster requires two things the current protocol lacks: a monotonic version per item and a well-defined invalidation event.

6.1 Monotonic version

A version column is already present on every persisted metadata record in system/metadata-persistence.zod.ts (version: z.number(), used today for optimistic concurrency). This ADR proposes formalising it as a monotonic per-item version and widening it to bigint so caches can compare freshness across long-running clusters without overflow.

Status: planned. The column is still version: number in the schema; the widening and the cache-comparison contract below describe the target design, not current runtime behaviour.

Once wired, caches store {value, version}. On any incoming change notification the cache compares the incoming version with the stored one:

• Incoming version > stored → invalidate (apply new value or evict)
• Incoming version ≤ stored → ignore (out-of-order notification, already superseded)

This eliminates a whole class of bugs where a slow-arriving "old" invalidation evicts a "newer" value the node has already learned about.

6.2 The metadata change event

Current behaviour. Cross-node metadata invalidation already works, but it does not go through eventBus.emit. The metadata manager fans out over the cluster PubSub channel metadata.changed (note: dot, not colon) with the payload shape ClusterMetadataChangedPayload:

// packages/metadata/src/metadata-manager.ts
ctx.cluster.pubsub.publish('metadata.changed', {
  originNode,        // used for loopback suppression
  type,              // 'object' | 'view' | 'dashboard' | …
  event,             // the local watch event, replayed verbatim on peers
})

Peers suppress their own messages by originNode and replay the watch event locally — there is currently no version / name / tenantId / operation field and no version comparison.

Target spec (planned). The richer, version-stamped payload below is defined as MetadataChangedEventPayloadSchema in kernel/cluster.zod.ts but is not yet wired into the runtime:

// MetadataChangedEventPayloadSchema — target shape, not yet emitted
{
  type: 'object' | 'view' | 'flow' | …,
  name: '<machine-name>',
  tenantId?: '<tenant>',
  version: <new-version>,
  operation: 'create' | 'update' | 'delete' | 'publish',
}

When wired with scope: 'cluster' + deliverySemantics: 'at-least-once' and a partitionKey of `${type}:${name}`, the partition key would guarantee that two rapid updates to the same item are applied to every node's cache in order, and the at-least-once guarantee would let a briefly partitioned node catch up on reconnect.

6.3 Reader contract

Status: planned. Today readers are invalidated by the metadata.changed PubSub fan-out described in §6.2, which replays watch events verbatim without version comparison. The contract below is the target design that goes with the version-stamped payload.

All metadata readers (registry, loader, query engine) should:

1. Subscribe to the metadata change channel on startup.
2. Compare incoming version with cached version before evicting.
3. Treat missing version as 0 (legacy compatibility).

7. Plugin author guidance

This section is a quick reference for plugin authors. The full details live in §4-§6; this is the cheat sheet.

7.1 "I want to fire an event"

Pick scope first, then delivery, then partition key:

If none of the above apply, stay with the defaults (local + best-effort) — they are always correct on a single node and never wrong on a cluster, they just don't propagate.

7.2 "I want to register a service"

Ask: if I deploy 5 nodes, should there be 5 of these or 1?

• 5 → clusterScope: 'node' (default). You're done.
• 1 → clusterScope: 'cluster' + pick a leaderStrategy:
  • "Wakes on a timer / cron" → leader-elected
  • "Pulls from a shared queue" → partitioned
  • "Reacts to events with idempotent writes" → idempotent-broadcast

7.3 "I want to read metadata in a long-lived cache"

There is no ctx.cluster.cache(...) factory. The cluster service exposes only the four primitives (ctx.cluster.{pubsub, lock, kv, counter}, see §3) — caches are derived from KV + PubSub by service-cache, they are not a built-in cluster method.

To keep an in-process metadata cache coherent today, subscribe to the metadata change channel via PubSub and invalidate on each notification:

ctx.cluster.pubsub.subscribe('metadata.changed', (payload) => {
  // payload: { originNode?, type, event }
  if (payload.type === 'object') {
    myObjectCache.invalidate(payload.event)
  }
})

A higher-level cache factory (with automatic version comparison) is part of the Phase 4 / §6 target design but is not yet implemented. No if (cluster) branches are needed either way — the memory driver makes the subscribe call a no-op-equivalent local fan-out on a single node.

7.4 What you must never do

• Never call setInterval for cluster-wide periodic work — declare a cluster-scoped service with leader-elected.
• Never SELECT … WHERE status = 'pending' LIMIT 1 without a row lock — use SELECT … FOR UPDATE SKIP LOCKED (Postgres) or UPDATE … RETURNING with a claimed_by column.
• Never write into a process-local Map and assume other nodes see it — use ctx.cluster.kv for shared state.
• Never import Redis from 'redis' directly — go through ctx.cluster.{pubsub, lock, kv, counter}.

8. Configuration surface

Cluster behaviour is configured once, at the stack level. The protocol ships a single optional field on defineStack():

defineStack({
  // …existing fields…
  cluster: {
    driver: 'memory',   // default — single-process
    // driver: 'redis', url: process.env.REDIS_URL,
    // driver: 'postgres', useExistingPool: true,
    // driver: 'nats', url: process.env.NATS_URL,

    nodeId: process.env.NODE_ID,           // optional, auto-generated if absent
    heartbeatMs: 5000,                     // leader-election heartbeat
    lockTtlMs: 15000,                      // 3× heartbeat is the safe ratio

    tenantIsolation: 'channel-prefix',     // 'channel-prefix' | 'none'
    // driverOptions: { /* opaque, driver-specific options */ },
  },
})

Every cluster primitive selects its implementation from cluster.driver. When the field is absent, the kernel silently auto-registers the in-memory, single-node driver. Emitting a production warning in this case (so operators notice that horizontal scale will be incorrect until cluster.driver is configured) is intended but not yet implemented — no such warning is logged today.

8.1 Driver matrix

The redis driver is the recommended starting point for production ObjectStack deployments: it shares its connection pool with the planned service-cache Redis adapter (no extra infrastructure once cache is on Redis), gives native semantics for every primitive (SETNX/Lua/INCR/PUBSUB), and decouples cluster transport from the choice of database driver (SQL / Turso / Postgres / SQLite all work the same).

The postgres driver remains a viable "single binary, single dependency" choice for small deployments that already run Postgres and want to avoid a Redis dependency. It is deferred to a community/optional driver.

9. Industry alignment

These choices follow patterns proven across the enterprise platform ecosystem; none are novel.

• Salesforce Platform Events — cluster-wide pub/sub with at-least-once delivery; tenant-partitioned channels. Maps directly to our scope: 'cluster' + 'tenant'.
• ServiceNow Cluster Cache + Event Management — version-stamped cache with invalidation broadcast; leader-elected schedulers. Maps to §6 and leader-elected strategy.
• Odoo bus.bus — Postgres LISTEN/NOTIFY for cross-worker fan-out. Validates our postgres driver as the right default starting point.
• Notion / Linear — Redis pub/sub for realtime fan-out with per-document partitioning. Validates partitionKey ordering as the right primitive.
• Kubernetes leader election — Lease objects with TTL + heartbeat. The leader-elected strategy in §5 follows the same pattern.

10. Migration plan

This ADR is additive. Existing plugins continue to compile and run unchanged. The protocol additions (a new cluster.zod.ts file plus optional fields on existing schemas) ship in spec v5.x without breaking v4 consumers.

Phase 1 — Protocol (this work, ~1 day)

1. Add kernel/cluster.zod.ts — schemas for EventScope, DeliverySemantics, ClusterScope, LeaderStrategy, ClusterConfig.
2. Extend events/core.zod.ts EventMetadataSchema with optional scope, deliverySemantics, partitionKey.
3. Extend kernel/service-registry.zod.ts ServiceMetadataSchema and ServiceFactoryRegistrationSchema with optional clusterScope, leaderStrategy.
4. Add optional cluster field to stack.zod.ts.

No runtime changes. After this phase the protocol describes cluster semantics; nothing enforces them.

Phase 2 — In-memory runtime (~2 days)

Implement service-cluster with the memory driver only. Every cluster primitive works; behaviour on one node is identical to today. This phase proves the API shape against real callers (EventBus, service-cache, service-job) before committing to a wire format.

Phase 3 — Redis driver (~2 days) ✅

Implement redis driver via @objectstack/service-cluster-redis (PUBSUB + SETNX/Lua locks + WATCH/MULTI KV + INCR counter). Ships as a sibling package so adopters opt in by importing it. Shares its ioredis client pool with service-cache to avoid double connections.

Note: this was switched from "Postgres first" — see commit history. Rationale: Postgres isn't a fixed dependency in the ObjectStack stack, so binding cluster to PG would couple the two unnecessarily. Redis is already the planned cache backend.

Phase 4 — Existing services migrate (~3 days, opportunistic)

• service-job declares clusterScope: 'cluster' / 'leader-elected'.
• service-cache switches to ctx.cluster.cache(...) factory.
• service-realtime adds cross-node fan-out via PubSub.
• Webhook dispatcher (planned) uses partitioned strategy from day one.

Phase 5 — Postgres / NATS drivers (deferred, optional)

Postgres driver downgraded to community/optional — useful for the "one binary, one container" deployment archetype that wants to skip Redis. NATS deferred until a customer reports throughput needs that exceed Redis PUBSUB. No protocol changes required at that point — only a new implementation that calls registerClusterDriver().

11. Non-goals (v1)

• Exactly-once delivery — accepted as a keyword in the protocol; startup rejection of the value is intended but not yet implemented. Implementing it correctly requires distributed transactions or idempotency tokens that are out of scope for v1.
• Cross-region replication — single-region clusters only. Multi-region is a future ADR.
• Streaming protocol — Kafka-style consumer groups not modelled here. Will arrive as a fifth primitive when needed.
• Schemaful event registry enforcement at the bus layer — event names are validated by metadata, not by the bus. The bus stays string-typed for plugin extensibility.