Webhook Delivery

Status: Shipped · Audience: Plugin authors, runtime engineers, integration partners

TL;DR — Webhooks are the lingua franca of SaaS integration. ObjectStack defines a Webhook config schema; the delivery runtime that turns config into actual HTTP requests is provided by @objectstack/service-messaging's shared outbound-HTTP outbox (sys_http_delivery). The plugin-webhooks plugin owns only the sys_webhook configuration object and an auto-enqueuer that fans record events onto that outbox; signing, retries, dispatch, and idempotency are inherited from the messaging substrate (ADR-0018 M3). This page documents the shipped behaviour.

1. Why this document exists

The protocol already has WebhookSchema in automation/webhook.zod.ts (URL, method, headers, retry policy, secret, triggers). That schema describes a configuration — what a user wants to happen — but says nothing about the runtime that makes it happen. Today there is no single answer to:

• When account is updated on Node A, does Node B's webhook config fire? Or only Node A's?
• If the receiver returns 502, who retries? After how long? How many times?
• If we crash mid-delivery, does the customer receive 0, 1, or 2 copies?
• How does the receiver know the request really came from us and not from an attacker spoofing our hostname?
• If 1,000 webhooks fire from a bulk import, do they all hit the external system in 200ms (and get rate-limited away), or do we shape the traffic?

These are not optional features — every production webhook implementation in the industry (Stripe, GitHub, Slack, Shopify) had to answer them. We codify the answers here so plugin authors and integration partners can rely on a stable contract.

2. Design principles

P1 — At-least-once, never at-most-once. Receivers MUST be prepared for duplicates. Once a delivery is enqueued as a sys_http_delivery row we never silently drop it, but we also never guarantee uniqueness — that requires receiver-side idempotency keys, which we provide but cannot enforce. (The enqueue step itself is not yet this durable across nodes — see §4.1.)

P2 — Durable before fast. Every delivery is persisted before it is attempted. A node crash mid-flight loses at most the in-flight HTTP attempt; the next node picks the row up from the queue.

P3 — Signed when configured. Whenever a webhook's definition_json carries a secret, every outbound request for it carries a signature the receiver can verify with that shared secret (see §6). The secret field is optional, not auto-generated — a webhook created without one is delivered unsigned. Customers can detect spoofing without writing custom auth, once they set a secret.

P4 — Safe egress. Webhooks are a textbook SSRF vector, and the design goal is to refuse localhost, private IP ranges, and cloud-provider metadata endpoints by default. This is not shipped yet — see §7: the dispatcher today POSTs to whatever URL is configured, with no blocklisting. Treat webhook URL configuration as privileged until this lands.

P5 — Observable. Every delivery attempt is visible to the owner of the webhook: status, response code, retry schedule, manual redelivery. (Per-attempt latency is computed in memory during the HTTP call but is not persisted to sys_http_delivery or surfaced today.) The Studio surfaces this without custom UI code by reusing the standard object/view machinery.

3. Data model

Two persisted objects underpin the runtime, owned by different packages. sys_webhook (the configuration record) lives in plugin-webhooks; sys_http_delivery (the durable outbox) lives in @objectstack/service-messaging and is shared with the Flow http node. Both are defined as ObjectSchema.create() schemas so they participate in CRUD, permissions, audit, and Studio UI without bespoke code.

3.1 sys_webhook

The subscription record. One row per "I want webhook X to fire for object Y". The full transport configuration (headers, secret, timeout, method) is carried in definition_json, a serialised Webhook JSON (canonical schema: WebhookSchema, exported from @objectstack/spec/automation).

Matching at runtime is purely object_name + the comma-separated triggers list; the headers, signing secret, and per-attempt timeout are parsed out of definition_json when an event is enqueued. There is no per-row org/tenant column, no events[] glob field, no stored retry_policy, and no secret_hint — see §6 for the actual signing model and §11 for the (single) retry budget.

3.2 sys_http_delivery

The outbox + audit. One row per delivery. Persisted before any HTTP call; updated after each attempt. This table is the durability boundary — if it has a pending row, the dispatcher will eventually deliver it.

It is not webhook-specific: sys_http_delivery is owned by @objectstack/service-messaging (ADR-0018 M3) and shared by the Flow http node executor and webhook fan-out, so both inherit retry / idempotency / dead-letter from one substrate. Webhook deliveries are the rows with source = 'webhook'. It generalises the old design's per-webhook table: webhook_id → ref_id, event_id → dedup_key, event_type → label, secret → signing_secret. Rows are managed by the platform and not directly writable.

Why store full payload? Receivers may be down for hours; we must retry the exact bytes we promised to send. Recomputing payload from source data risks drift if the underlying record changed.

4. The delivery pipeline

Five stages, each implemented as a thin layer over an existing primitive.

┌─────────────────────────────────────────────────────────────────┐
│ 1. Event   realtimeService.publish({ type, object, payload,     │
│            timestamp }) — plain in-process pub/sub; no          │
│            cluster / retry options are attached (see §4.1)      │
└──────────────────────────┬──────────────────────────────────────┘
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 2. Match   subscriber filters sys_webhook by events[]/object,   │
│            multiplies one event into N delivery rows            │
└──────────────────────────┬──────────────────────────────────────┘
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 3. Persist INSERT sys_http_delivery (status=pending)            │
│            ── this is the durability boundary ──                │
└──────────────────────────┬──────────────────────────────────────┘
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 4. Dispatch  worker holds a per-partition cluster lock, then    │
│              atomic UPDATE pending→in_flight claims a batch;     │
│              issues POST; writes back result                    │
└──────────────────────────┬──────────────────────────────────────┘
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 5. Retry   on failure: compute next_retry_at = now + backoff,   │
│            status=pending. on terminal failure: status=dead.    │
└─────────────────────────────────────────────────────────────────┘

4.1 Stage 1 — Event emission

Producers (the ObjectQL engine's insert/update/delete handlers) call IRealtimeService.publish(event) after the write commits, where event is a plain { type, object, payload, timestamp } record — e.g. { type: 'data.record.updated', object: 'account', payload: { recordId, changes, after }, timestamp: <ISO 8601 string> }.

Not yet cluster-aware. The only shipped IRealtimeService implementation is InMemoryRealtimeAdapter, an in-process, single-node pub/sub with no persistence — its own documentation calls this out as the "v1 deployment contract: single-instance only" (packages/services/service-realtime/src/realtime-service-plugin.ts). Concretely, today:

• There is no eventBus.emit() API and no scope / deliverySemantics / partitionKey options on the call — publish() takes only the event.
• Events published on one node are not delivered to subscribers (including the webhook auto-enqueuer) on another node; cross-node fan-out is future work.
• Delivery is synchronous and unpersisted before Stage 3 — if the process crashes between the write and publish() returning, the event does not survive.
• There is no explicit partitionKey ordering primitive; two updates to the same record only stay in order because a single node runs JavaScript single-threaded.

Everything from Stage 3 onward (the sys_http_delivery outbox and HttpDispatcher) is durable and cluster-aware as described below — the gap is upstream, between the CRUD write and the enqueue in Stage 2.

4.2 Stage 2 — Matching

The subscriber (AutoEnqueuer) is a plain service registered per-node via ctx.registerService('webhook.autoEnqueuer', ...) — it runs independently on every node with no clusterScope / leaderStrategy annotation, subscribing to its own node-local realtime events only (see §4.1). Duplicate enqueues (e.g. from a periodic cache refresh re-matching the same event) still collapse safely because the Stage 3 INSERT is keyed by the (source, dedup_key) UNIQUE constraint.

For each incoming event the subscriber:

1. Loads sys_webhook rows where active = true AND object_name matches the event's object AND the event's action is in the row's comma-separated triggers.
2. For each match, builds the payload (§5).
3. Enqueues a sys_http_delivery row (source = 'webhook').

4.3 Stage 3 — Persist

A single enqueue per match. The (source, dedup_key) uniqueness constraint — where dedup_key is <webhookId>:<object>:<recordId>:<action>:<ts> — absorbs duplicate event deliveries from at-least-once semantics. After this stage the event producer is no longer involved; the rest is the dispatcher's problem.

4.4 Stage 4 — Dispatcher worker

HttpDispatcher (in @objectstack/service-messaging) ticks on a timer and, for each partition, attempts to acquire a per-partition cluster lock (http.dispatcher.partition.<index>). Partition affinity is on the delivery's ref_id (the webhook id), so the same webhook's deliveries always land on the same partition — useful for in-order delivery and connection reuse. On a single-node runtime the lock is an always-grant stub.

Within a held partition, the lock-holder claims a batch with an atomic conditional UPDATE rather than SELECT … FOR UPDATE SKIP LOCKED:

-- 1. select candidate ids for this partition
SELECT id FROM sys_http_delivery
WHERE status = 'pending'
  AND partition_key = $partition_index
  AND (next_retry_at IS NULL OR next_retry_at <= $now)
ORDER BY next_retry_at
LIMIT $batch_size;

-- 2. atomic claim: only rows still pending flip to in_flight
UPDATE sys_http_delivery
SET status = 'in_flight', claimed_by = $node_id, claimed_at = $now
WHERE id IN (...) AND status = 'pending';

The WHERE … AND status = 'pending' predicate on the UPDATE is the exactly-once claim: two workers racing on the same row cannot both flip it. The held partition lock keeps the race window small in the first place.

After each HTTP attempt the row is updated with the result. If the request succeeded (2xx), status = 'success'. Otherwise status returns to pending (with bumped attempts and next_retry_at) or, once the retry budget is exhausted, becomes dead.

4.5 Stage 5 — Retry backoff

The default retry policy is:

This matches Stripe's published schedule almost exactly and gives ~1.5 days of recovery window — enough for receivers to come back from weekend-long outages without manual intervention.

Each delay is multiplied by a jitter factor uniformly distributed in [0.8, 1.2] to avoid retry stampedes when many receivers recover at the same time.

5. Payload format

The POST body is the realtime event payload with a small fixed prefix merged on top:

{
  "object": "account",
  "recordId": "acc_123",
  "action": "updated",
  "timestamp": "2024-10-27T00:00:00.000Z"
  // ...remaining fields from the originating event payload
}

Notes:

• object — short object name the event came from.
• recordId — id of the affected record.
• action — created / updated / deleted / undeleted.
• timestamp — event timestamp, an ISO 8601 string (passed through unchanged from the originating realtime event's timestamp field — not an epoch-ms number).
• Any additional fields the event carried (e.g. record snapshot data) are spread in after these four.

Idempotency and event correlation are carried in headers, not the body. Every attempt sends:

Requests also set Content-Type: application/json and User-Agent: ObjectStack-Http/1.0.

6. Signing & verification

GitHub-style HMAC over the raw request body. When the webhook's definition_json carries a secret, every outbound request is signed.

6.1 Outbound header

X-Objectstack-Signature: sha256=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd

Where the hex value is HMAC-SHA256( secret, raw_request_body ). There is no timestamp, no {t}.{body} concatenation, no version (v1) prefix, and no replay window — the signature covers the body bytes only.

6.2 Receiver-side verification

Receivers MUST:

1. Read the sha256=<hex> value from X-Objectstack-Signature.
2. Recompute HMAC-SHA256(secret, raw_body) over the raw request body bytes (no JSON re-encode — whitespace matters).
3. Compare in constant time (e.g. hmac.compare_digest).
4. Reject if not equal.

Use the X-Objectstack-Delivery header (the delivery row id) as the idempotency key when deduplicating retries.

7. Egress safety

Not yet implemented. URL blocklisting (localhost / RFC 1918 / link-local / cloud-metadata rejection, allowPrivate, https-only enforcement) and DNS-rebinding re-validation are not part of the shipped runtime. The sender POSTs to the configured URL with no SSRF filtering. Treat webhook URL configuration as a privileged operation and gate it with RBAC accordingly; network-level egress filtering is future work.

7.1 Response body capping

The recorded response_body is truncated to 16 KB before persistence. This is the one egress safeguard that is implemented today: it bounds the worst-case exfiltration through a single delivery row even if a hostile URL is configured.

8. REST surface

CRUD for sys_webhook (and read access to sys_http_delivery) comes for free from the generic object/REST surface — there is no bespoke webhook CRUD router. The plugin mounts exactly one custom endpoint:

It delegates to messaging.redeliverHttp(deliveryId) and is guarded by the better-auth session (any authenticated user). Responses: 200 on success, 401 unauthenticated, 400 missing/invalid body, 404 not_found, 409 not_eligible. There are no /webhooks/:id/test, /rotate, /deliveries, or per-delivery redeliver routes.

9. Concurrency & rate-limiting

Outbound load is shaped by the claim loop, not an explicit concurrency knob: within a held partition the dispatcher claims up to batchSize rows (default 32) per tick and POSTs them sequentially, so at most one request per partition is in flight at a time (default 8 partitions per node). Both batchSize, partitionCount, and the tick intervalMs are configurable via HttpDispatcherOptions.

Not yet implemented. Per-receiver rate limiting (tracking request rate per hostname(url) and delaying further requests to a busy host) is future work — see Phase 4 of the plan below. Today nothing shields a receiver from bulk traffic beyond the sequential-per-partition send loop.

10. Observability

1. sys_http_delivery table — the source of truth. It ships standard ObjectStack list views (Recent, Failures, Pending) showing source, url, status, attempts, response code, and timestamps; Studio surfaces these under the HTTP Deliveries nav. The X-Objectstack-Signature header (note the lowercase s in stack), X-Objectstack-Delivery, X-Objectstack-Event, and X-Objectstack-Attempt headers correlate a received request back to its row.
2. onAttempt hook — HttpDispatcher fires an onAttempt(delivery, success) callback after every attempt, the one programmatic observability hook. Wire it to whatever metrics/tracing backend your deployment uses.

Future work. Built-in OpenTelemetry spans and Prometheus metrics are not yet emitted; the onAttempt hook is the integration point until they are.

11. Multi-tenancy

Not yet implemented. sys_webhook has no owner_org_id column, there is no retry_policy field, and dispatcher partitioning is on ref_id (the webhook id) only — there is no (org_id, webhook_id) partitioning or per-tenant egress isolation. The retry budget is a single fixed 7-step schedule (§4.5), not a per-plan quota. Per-tenant webhook ownership and isolation are future work; until then, isolate access at the RBAC layer on sys_webhook.

12. Failure modes & guarantees

A precise table of what the runtime promises and what it does not.

13. Plugin location & layering

packages/plugins/plugin-webhooks/
└── src/
    ├── webhook-outbox-plugin.ts  # plugin: registers sys_webhook + nav,
    │                             #   starts the auto-enqueuer, mounts redeliver
    ├── sys-webhook.object.ts     # ObjectSchema.create() — sys_webhook config
    ├── auto-enqueuer.ts          # realtime data.record.* → enqueue onto outbox
    ├── auto-enqueuer.test.ts     # tests are colocated in src/, not a separate test/ dir
    └── schema.ts                 # shared types

The delivery runtime — sys_http_delivery, HttpDispatcher, the HTTP sender, signing, retry classification — lives in @objectstack/service-messaging, not in this plugin. plugin-webhooks owns only the sys_webhook configuration object and the auto-enqueuer that fans record events onto the shared outbox.

The plugin depends on:

• @objectstack/core
• @objectstack/service-messaging (declared as plugin dependency com.objectstack.service.messaging) — provides the outbox object and dispatcher.
• @objectstack/spec for WebhookSchema.

There is no dependency on service-cluster or service-queue; cross-node coordination is the dispatcher's per-partition cluster lock inside service-messaging.

14. Industry alignment

Every design choice matches one or more established systems:

• Stripe webhooks — retry schedule shape (1s → 24h, ~1.5-day window) with jitter.
• GitHub webhooks — X-…-Signature: sha256=<hmac-over-raw-body> signing scheme, redelivery, per-delivery detail with request/response bytes.
• Shopify webhooks — HMAC-SHA256 header, X-Shopify-Topic event hint (we carry the event hint in the X-Objectstack-Event header instead).
• AWS EventBridge / SQS — at-least-once durability, dead-letter queue model (our status = dead).
• Kubernetes admission webhooks — egress validation and TLS-only enforcement (design inspiration only — per §7 this is not yet enforced by our dispatcher).

15. Migration plan

Four phases. Each phase delivers visible user value.

How it actually shipped. The implementation did not introduce new system/* Zod schemas. There is no system/webhook-delivery.zod.ts and no SysWebhookSchema / SysWebhookDeliverySchema / WebhookSignatureHeaderSchema / WebhookEventPayloadSchema. Instead it reused the existing WebhookSchema (and WebhookReceiverSchema) in automation/webhook.zod.ts, defined sys_webhook as an ObjectSchema.create() in the plugin, and reused service-messaging's sys_http_delivery outbox for deliveries (ADR-0018 M3). The phase list below is the original plan; read it as design intent, not the current artifact layout.

Phase 1 — Spec (~1 day)

1. Persistence + config schemas. (As shipped: sys_webhook is an ObjectSchema, deliveries reuse sys_http_delivery, and the config schema is the existing WebhookSchema.)
2. Documentation cross-links between "static config" and "runtime subscription".

Phase 2 — Plugin skeleton + dispatcher (~2 days)

• plugin-webhooks package with sys_webhook wired and the auto-enqueuer fanning record events onto the shared outbox.
• Signing + retry backoff working through service-messaging's HTTP sender.

Phase 3 — Production dispatcher (~2 days)

• Atomic conditional-UPDATE claim loop under a per-partition cluster lock (as shipped; the original plan called for FOR UPDATE SKIP LOCKED).
• Stuck in_flight rows revert after the claim TTL.
• Partitioned dispatch on ref_id.
• Studio UI: HTTP Deliveries list views + redeliver endpoint (reuses existing object/view machinery — zero custom React).
• Future: built-in Prometheus metrics + OTel spans (today only the onAttempt hook).

Phase 4 — Hardening (opportunistic)

• Per-receiver rate limiting.
• Webhook receiver reference verifiers (TS / Python / Go) in docs.
• Bulk-import shaping (dispatcher slows when queue depth > threshold).

16. Non-goals (v1)

• Inbound webhooks. Receiving HTTP from third parties is a separate protocol (WebhookReceiverSchema already exists). The two systems share no runtime.
• Webhook chaining / fan-in. No "if receiver returns a payload, emit a new event". Out of scope.
• GraphQL subscriptions / WebSocket fan-out. Different transport, different durability model, separate ADR.
• Stream / batch deliveries. Each event is one HTTP request. Batching is a future enhancement.
• Custom payload templates. v1 ships one canonical payload format (§5). Per-webhook templates are a future feature.