System Lifecycle

Boot sequence, plugin installation, zero-downtime upgrades, and rollback strategies

Protocol spec. The boot sequence, schema-evolution model, and graceful shutdown described below reflect the current @objectstack/core and CLI behaviour. The plugin upgrade flow (objectstack upgrade @vendor/pkg --strategy blue-green, multi-instance rolling upgrades, automatic backup & rollback) describes the target lifecycle for hosted ObjectStack — today it is partially implemented in the control plane and exposed only on hosted plans. Treat those command snippets as design intent.

ObjectStack manages the complete lifecycle of the platform runtime—from initial boot to plugin installation, upgrades, and rollbacks. Every operation is declarative, idempotent, and auditable.

Boot Sequence

The ObjectStack boot process follows a strict order to ensure dependencies are satisfied before services start.

Boot Phases

┌─────────────────────────────────────────────────────────────────┐
│ Phase 1: INITIALIZE                                             │
│  └─ Load environment variables                                  │
│  └─ Validate runtime requirements (Node.js version, memory)     │
│  └─ Initialize logging infrastructure                           │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 2: CONFIGURE                                              │
│  └─ Load configuration files (objectstack.config.yml)           │
│  └─ Merge config sources (env → file → defaults)                │
│  └─ Validate configuration schema                               │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 3: CONNECT                                                │
│  └─ Establish database connections (PostgreSQL, Redis)          │
│  └─ Run health checks                                           │
│  └─ Initialize connection pools                                 │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 4: LOAD PLUGINS                                           │
│  └─ Discover installed plugins                                  │
│  └─ Resolve dependency graph                                    │
│  └─ Load plugins in topological order                           │
│  └─ Execute onBoot() hooks                                      │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 5: REGISTER METADATA                                      │
│  └─ Register ObjectQL schemas (objects, fields)                 │
│  └─ Register ObjectUI layouts (views, dashboards)               │
│  └─ Register permissions and validation rules                   │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 6: START SERVICES                                         │
│  └─ Start event bus                                             │
│  └─ Start job scheduler                                         │
│  └─ Start audit logger                                          │
│  └─ Start HTTP server                                           │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 7: READY                                                  │
│  └─ Mark instance as healthy                                    │
│  └─ Begin accepting requests                                    │
│  └─ Log boot time metrics                                       │
└─────────────────────────────────────────────────────────────────┘

Boot Configuration

Boot behaviour is controlled by the host runtime (kernel options + env vars), not by metadata. The relevant knobs:

Concern	Where it lives
Plugin startup timeout	`new ObjectKernel({ defaultStartupTimeout: 60_000 })`
Fail-fast on plugin error	`new ObjectKernel({ rollbackOnFailure: true })`
Disable strict requirement checks (tests)	`new ObjectKernel({ skipSystemValidation: true })`
Which services start	Each service is registered with `kernel.use(serviceFactory(...))`; not started → not running
Schema sync on boot	Driver-specific (`SqlDriver` calls `syncSchema()` on `init()`)

A typical bootstrap looks like this — no defineConfig, no metadata-side boot options:

import { ObjectKernel } from '@objectstack/core';
import { AppPlugin } from '@objectstack/runtime';
import { SqlDriver } from '@objectstack/driver-sql';
import stack from './objectstack.config';

const kernel = new ObjectKernel({
  defaultStartupTimeout: 60_000,
  gracefulShutdown: true,
  shutdownTimeout: 30_000,
});

kernel.use(new SqlDriver({ /* ... */ }));
kernel.use(new AppPlugin(stack));
await kernel.bootstrap();

Plugin Ordering Contract

kernel.use() registration order is not a contract. The kernel resolves both init and start order from the plugin dependency graph (topological sort; insertion order is kept only between plugins with no edges), and Phase 1 (every init()) completes before Phase 2 (any start()) begins. A plugin that needs something another plugin sets up declares it (ADR-0116):

Declaration	Semantics
`dependencies: string[]`	Hard — hoisted ahead of the declarant; a name not composed on the kernel fails the boot.
`optionalDependencies: string[]`	Order-if-present — hoisted exactly like `dependencies` when composed, silently skipped when absent. For plugins that degrade without the dependency but must never init before it.
`requiresServices: string[]`	Services the plugin resolves synchronously during `init()` with no fallback. Validated before Phase 1 (a required service whose only declared provider inits later is a named boot error, before any init side effects) and again immediately before the plugin's own init.
`providesServices: string[]`	Services the plugin's `init()` unconditionally registers. Powers the validation above and lets ordering errors name the provider. Never declare option-gated registrations.

Start-time needs require no declaration — the Phase 1/2 split already guarantees every init-registered service is visible to every start().

// AppPlugin: degrades on engine-less kernels, but when the engine is
// composed it must never init first — and a non-empty bundle cannot
// register at all without the manifest service.
class AppPlugin implements Plugin {
  optionalDependencies = ['com.objectstack.engine.objectql'];
  requiresServices = ['manifest'];
  // ...
}

Boot Logs (Example)

[2024-01-15T10:23:01.234Z] INFO  ObjectStack starting...
[2024-01-15T10:23:01.250Z] INFO  Phase 1: Initialize
[2024-01-15T10:23:01.251Z] INFO    ✓ Node.js v20.10.0
[2024-01-15T10:23:01.252Z] INFO    ✓ Memory: 2048 MB available
[2024-01-15T10:23:01.300Z] INFO  Phase 2: Configure
[2024-01-15T10:23:01.301Z] INFO    ✓ Loaded objectstack.config.yml
[2024-01-15T10:23:01.302Z] INFO    ✓ Merged 3 config sources
[2024-01-15T10:23:01.400Z] INFO  Phase 3: Connect
[2024-01-15T10:23:01.450Z] INFO    ✓ PostgreSQL connected (10 pool size)
[2024-01-15T10:23:01.460Z] INFO    ✓ Redis connected
[2024-01-15T10:23:01.500Z] INFO  Phase 4: Load Plugins
[2024-01-15T10:23:01.501Z] INFO    → @objectstack/core@2.0.0
[2024-01-15T10:23:01.550Z] INFO    → @mycompany/crm@1.5.0
[2024-01-15T10:23:01.600Z] INFO    → @vendor/salesforce@3.2.1
[2024-01-15T10:23:01.650Z] INFO    ✓ 3 plugins loaded
[2024-01-15T10:23:01.700Z] INFO  Phase 5: Register Metadata
[2024-01-15T10:23:01.701Z] INFO    ✓ 15 objects registered
[2024-01-15T10:23:01.702Z] INFO    ✓ 42 views registered
[2024-01-15T10:23:01.800Z] INFO  Phase 6: Start Services
[2024-01-15T10:23:01.850Z] INFO    ✓ Event bus started
[2024-01-15T10:23:01.900Z] INFO    ✓ Job scheduler started (5 jobs loaded)
[2024-01-15T10:23:01.950Z] INFO    ✓ HTTP server listening on :3000
[2024-01-15T10:23:02.000Z] INFO  Phase 7: Ready
[2024-01-15T10:23:02.001Z] INFO  ObjectStack ready in 766ms

Error Handling During Boot

Scenario: Plugin fails to load

[2024-01-15T10:23:01.500Z] ERROR Phase 4: Load Plugins
[2024-01-15T10:23:01.501Z] ERROR   ✗ @vendor/broken-plugin@1.0.0
[2024-01-15T10:23:01.502Z] ERROR   Dependency @objectstack/core@^3.0.0 not satisfied
[2024-01-15T10:23:01.503Z] ERROR   (Installed version: 2.0.0)
[2024-01-15T10:23:01.504Z] FATAL Boot failed. Exiting.

Resolution Strategy:

rollbackOnFailure: true (default): Boot fails, already-started plugins roll back, process exits with code 1
rollbackOnFailure: false: Boot continues, the failed plugin is skipped and logged

Plugin Installation

Installing a plugin is a multi-step transaction. If any step fails, the entire installation rolls back.

Installation Flow

// Command: objectstack plugin install @vendor/salesforce@3.2.1

async function installPlugin(packageName: string, version: string) {
  const transaction = await db.beginTransaction();
  
  try {
    // Step 1: Download and validate
    const manifest = await registry.download(packageName, version);
    await validateManifest(manifest);
    
    // Step 2: Dependency resolution
    await resolveDependencies(manifest.dependencies);
    
    // Step 3: Backup current state
    const backup = await createBackup();
    
    // Step 4: Run pre-install hook
    await manifest.lifecycle.preInstall?.({ context, transaction });
    
    // Step 5: Apply schema changes (ObjectQL)
    for (const object of manifest.objects) {
      await ObjectQL.createOrUpdateObject(object, { transaction });
    }
    
    // Step 6: Register UI metadata (ObjectUI)
    for (const view of manifest.views) {
      await ObjectUI.registerView(view, { transaction });
    }
    
    // Step 7: Apply configuration defaults
    await ConfigStore.merge(manifest.defaultConfig, { transaction });
    
    // Step 8: Run post-install hook
    await manifest.lifecycle.postInstall?.({ context, transaction });
    
    // Step 9: Mark plugin as installed
    await PluginRegistry.markInstalled(packageName, version, { transaction });
    
    // Step 10: Commit transaction
    await transaction.commit();
    
    logger.info(`✓ Installed ${packageName}@${version}`);
    
  } catch (error) {
    // Rollback on any error
    await transaction.rollback();
    logger.error(`✗ Installation failed: ${error.message}`);
    throw error;
  }
}

Installation States

A plugin progresses through these states:

stateDiagram-v2
    [*] --> NOT_INSTALLED
    NOT_INSTALLED --> DOWNLOADING
    DOWNLOADING --> VALIDATING
    VALIDATING --> INSTALLING
    INSTALLING --> INSTALLED
    INSTALLED --> ENABLED
    DOWNLOADING --> FAILED_DOWNLOAD: error
    VALIDATING --> FAILED_VALIDATION: error
    INSTALLING --> FAILED_INSTALL: error

Dependency Resolution

Example Dependency Graph:

# @mycompany/sales-cloud depends on:
dependencies:
  '@objectstack/core': '^2.0.0'
  '@mycompany/crm-base': '^1.0.0'
  '@vendor/email': '>=2.5.0 <3.0.0'

Resolution Algorithm:

async function resolveDependencies(
  deps: Record<string, string>
): Promise<void> {
  for (const [pkg, versionRange] of Object.entries(deps)) {
    const installed = await PluginRegistry.getInstalled(pkg);
    
    if (!installed) {
      throw new Error(
        `Dependency ${pkg} is not installed. ` +
        `Install it first: objectstack plugin install ${pkg}`
      );
    }
    
    if (!semver.satisfies(installed.version, versionRange)) {
      throw new Error(
        `Dependency ${pkg}@${installed.version} does not satisfy ` +
        `required version ${versionRange}`
      );
    }
  }
}

Plugin Runtime Contract

Installation registers metadata — no plugin code runs at install time. Executable behaviour belongs to the runtime contract, which has exactly three methods (packages/core/src/types.ts), invoked by the kernel:

// plugin.ts — a runtime plugin implements Plugin: `name` plus
// init / start / destroy. `init` is required; the other two are optional.
export class SalesforceSyncPlugin implements Plugin {
  name = 'plugin.salesforce-sync';
  version = '3.2.1';

  // Phase 1 — runs for every plugin, sequentially, in registration order.
  // Register services, schemas and routes here; other plugins' services
  // may not exist yet.
  async init(ctx: PluginContext) {
    if (!process.env.SALESFORCE_API_KEY) {
      throw new Error('Salesforce API key not configured');
    }
  }

  // Phase 2 — runs after EVERY plugin's init has completed, so every
  // registered service is resolvable. Begin actual work here.
  async start(ctx: PluginContext) {
    ctx.logger.info('Salesforce sync started');
  }

  // Shutdown — invoked in reverse registration order, and as rollback for
  // plugins already started when a later plugin's start() fails.
  async destroy() {
    /* stop timers, close connections */
  }
}

An authored app bundle (objectstack.config.ts) is not a kernel plugin and has its own single code seam: a module-level export const onEnable, which AppPlugin invokes at its own start() with the host context. That is the place apps register action handlers — see the worked examples in examples/app-todo and examples/app-showcase.

Earlier revisions of this page documented an onInstall / onEnable / onDisable / onUninstall / onUpgrade hook family on plugins. The kernel never called any of them — code written against them silently never ran — and the family was retired from the protocol (#4212, ADR-0049 enforce-or-remove). Install/uninstall/upgrade are package-registry state transitions (this page, above), not code hooks; boot-time code goes in init/start.

Installation CLI

# Install latest version
objectstack plugin install @vendor/salesforce

# Install specific version
objectstack plugin install @vendor/salesforce@3.2.1

# Install from local directory (development)
objectstack plugin install ./plugins/my-plugin

# Install with options
objectstack plugin install @vendor/salesforce \
  --enable \                    # Auto-enable after install
  --config salesforce.apiKey=abc123  # Set config during install
  
# Dry run (validate without installing)
objectstack plugin install @vendor/salesforce --dry-run

# Force reinstall (remove + install)
objectstack plugin install @vendor/salesforce --force

Installation Output

Installing @vendor/salesforce@3.2.1...

[1/8] Downloading package...         ✓ 1.2 MB in 0.5s
[2/8] Validating manifest...         ✓ 
[3/8] Checking dependencies...       ✓ 
  → @objectstack/core@2.0.0          ✓ (satisfied)
  → @vendor/http@1.5.0               ✓ (satisfied)
[4/8] Creating backup...             ✓ backup-20240115-102301.tar.gz
[5/8] Applying schema changes...     ✓ 3 objects created
[6/8] Registering UI metadata...     ✓ 7 views registered
[7/8] Running post-install hook...   ✓ 
[8/8] Finalizing installation...     ✓ 

✓ Successfully installed @vendor/salesforce@3.2.1

Next steps:
  1. Configure API credentials:
     objectstack config set salesforce.apiKey <YOUR_KEY>
     
  2. Enable the plugin:
     objectstack plugin enable @vendor/salesforce
     
  3. Test connection:
     objectstack plugin test @vendor/salesforce

Upgrades

ObjectStack supports zero-downtime upgrades with automatic rollback on failure.

Upgrade Strategies

1. In-Place Upgrade (Default)

Upgrade the current instance without creating a new one.

objectstack upgrade @vendor/salesforce --to 3.3.0

Process:

Download new version
Create backup of current state
Stop services gracefully (wait for in-flight requests)
Apply schema migrations
Update plugin files
Restart services
Validate health checks
If validation fails → automatic rollback

Downtime: 5-15 seconds (during service restart)

2. Blue-Green Deployment

Run two versions simultaneously, switch traffic after validation.

objectstack upgrade @vendor/salesforce --to 3.3.0 --strategy blue-green

Process:

Provision "green" instance with new version
Apply schema migrations to green database
Run smoke tests on green instance
If tests pass → switch load balancer to green
If tests fail → destroy green, keep blue
After validation period → destroy blue

Downtime: 0 seconds

3. Rolling Upgrade (Multi-Instance)

Upgrade instances one at a time in a cluster.

objectstack upgrade @vendor/salesforce --to 3.3.0 --strategy rolling

Process:

Take instance 1 out of load balancer
Upgrade instance 1
Add instance 1 back to load balancer
Repeat for instances 2, 3, ...N

Downtime: 0 seconds (requires N ≥ 2 instances)

Schema Evolution

ObjectStack treats schema as metadata, not migrations. The canonical source of truth is your object definitions; the driver's syncSchema() reconciles the physical database to match. Generated migration files exist as an escape hatch for explicit DDL when automatic sync isn't enough (e.g. data backfill, non-trivial column renames).

Declarative schema (the default)

// src/objects/salesforce_account.object.ts
import { ObjectSchema, Field } from '@objectstack/spec/data';

export default ObjectSchema.create({
  name: 'salesforce_account',
  sharingModel: 'private',
  label: 'Salesforce Account',
  fields: {
    salesforce_id: Field.text({ required: true, unique: 'global' }),
    account_name: Field.text(),
    last_sync: Field.datetime(),
  },
});

On boot, the driver's syncSchema(object, schema) creates / alters the table to match. Add a field → edit the file → restart (or hot-reload in dev). No imperative createObject / addField calls.

Generated migration files (escape hatch)

For environments where you want explicit, reviewable DDL — e.g. production deployments behind change control — generate a migration file from your current config:

os generate migration               # → migrations/<timestamp>_migration.ts
os generate migration --format sql  # → migrations/<timestamp>_migration.sql
os generate migration --dry-run     # Preview without writing

The generated TypeScript file uses plain Knex-style up(db) / down(db) functions — no custom migration DSL:

// migrations/20260101000000_migration.ts — auto-generated
export async function up(db: any): Promise<void> {
  await db.schema.createTable('salesforce_account', (t: any) => {
    t.string('id').primary();
    t.string('salesforce_id').notNullable().unique();
    t.string('account_name');
    t.datetime('last_sync');
  });
}

export async function down(db: any): Promise<void> {
  await db.schema.dropTable('salesforce_account');
}

Run the generated files through your existing Knex / driver tooling — ObjectStack does not execute these generated migration files for you. (To apply metadata-driven schema changes directly, without generating files, ObjectStack does ship first-party os migrate plan / os migrate apply commands that reconcile the physical database to your object definitions.) The intent of os generate migration is to give you a hand-off file you can commit, review, and execute via the database tools your team already uses.

Schema safety

The same rules apply whether you rely on syncSchema() or hand-written migrations:

✅ Adding optional fields is safe — old code ignores them.
❌ Adding required fields without a default breaks running clients.
✅ Adding required fields with defaultValue keeps old code happy.
⚠️ Renaming or dropping columns always needs a planned migration window.

Upgrade Rollback

If upgrade fails, ObjectStack automatically rolls back to previous version.

Rollback Scenarios

1. Schema Migration Fails:

[2024-01-15T10:30:00.000Z] INFO  Starting upgrade: 3.2.1 → 3.3.0
[2024-01-15T10:30:01.000Z] INFO  [1/3] Running migration 005_add_column...
[2024-01-15T10:30:01.500Z] ERROR Migration failed: column "account_name" already exists
[2024-01-15T10:30:01.501Z] WARN  Rolling back migration 005...
[2024-01-15T10:30:02.000Z] INFO  ✓ Rollback complete
[2024-01-15T10:30:02.001Z] INFO  Restoring previous version from backup...
[2024-01-15T10:30:03.000Z] INFO  ✓ Restored to version 3.2.1

2. Health Check Fails:

[2024-01-15T10:30:00.000Z] INFO  Upgrade complete, validating...
[2024-01-15T10:30:01.000Z] INFO  Running health checks...
[2024-01-15T10:30:02.000Z] ERROR Health check failed: /api/health returned 500
[2024-01-15T10:30:02.001Z] WARN  Automatic rollback initiated
[2024-01-15T10:30:05.000Z] INFO  ✓ Rolled back to version 3.2.1

Manual Rollback

To switch an environment back to a previous build, install the prior package version into it (via the Cloud control plane / Marketplace, or os package publish <older-artifact> --env <env-id> --install). This is the cloud equivalent of "switch back to yesterday's build"; it does not run DDL or undo schema migrations.

The legacy revision-activate os rollback CLI was removed (#2237); environment version management now goes through package install.

For schema rollback, run the down() of your generated migration through whatever Knex / driver tooling your team uses to apply the up().

Upgrade Configuration

The behaviour around backups, rollback, and health checks during an objectstack upgrade is controlled by the host runtime / hosting platform — not by a top-level defineStack key today. Hosted ObjectStack and self-hosted ObjectStack expose this through environment-specific configuration:

Concern	Hosted control plane	Self-hosted
Pre-upgrade backups	enabled by default, retention configurable per project	run your own snapshotter (e.g. `pg_dump`, LiteFS snapshot)
Health-check timeout after upgrade	dashboard setting	`OS_HEALTH_TIMEOUT_MS` env var
Automatic rollback on failed health check	dashboard setting	`OS_AUTO_ROLLBACK=true` env var
Maintenance-mode banner	dashboard setting	`OS_MAINTENANCE_MESSAGE` env var consumed by `plugin-hono-server`

This intentionally lives outside the metadata layer: it's an operational policy, not a property of the application.

Health Checks

ObjectStack includes built-in health monitoring to validate system state.

Health Check Endpoints

GET /health
  → 200 { success: true, data: { status: "ok", timestamp, version, uptime } }
        if the process is alive (liveness)
  → never fails on a dependency — see below

GET /ready
  → 200 { success: true, data: { status: "ready", state } } when the kernel is
        running AND every data driver answers (readiness)
  → 503 while still booting or shutting down
  → 503 carrying { state, drivers: [...] } in error.details when a data driver
        stops answering

The two probes answer deliberately different questions (#3756). /health checks nothing but the process, because a failing liveness probe makes the orchestrator restart the pod — which cannot fix an unreachable database, but would put every replica into a restart storm for the length of the outage. /ready pings the data drivers (bounded, memoized ~1s) because its failure mode — leave the load-balancer rotation — is the one that helps a replica that would otherwise fail 100% of its requests. The readiness check fails open: a kernel with no data engine, or a probe that itself errors, still reads as ready rather than black-holing a working deployment.

GET /health returns a compact liveness body (status, timestamp, version, uptime) and GET /ready returns readiness. The per-plugin health report and the plugin-declared custom checks below describe the internal health-monitor model (PluginHealthMonitor), which covers plugins, not driver connections — they are not yet exposed as a dedicated HTTP endpoint or as a declarative plugin field.

Health Status Response

{
  "success": true,
  "data": {
    "status": "ok",
    "timestamp": "2024-01-15T11:00:00.000Z",
    "version": "<the serving artifact's version>",
    "uptime": 3600
  }
}

status is the fixed string "ok": the probe reports that the process is executing code and has nothing else to report. version is the serving artifact's version — the OS_RUNTIME_VERSION stamp when a deployment injects one, otherwise the resolved @objectstack/runtime version (#10993) — never a hardcoded literal. uptime is process uptime in seconds, and is omitted entirely outside Node-like runtimes, where there is no process to read it from. There is no checks key: per-subsystem results belong to the internal health-monitor model below, not to this body.

Custom Health Checks

A plugin can expose one custom health check, and it takes two halves: the plugin defines an ordinary method, and whoever runs the monitor names that method in the plugin's PluginHealthCheck config. There is no healthChecks field to declare — the monitor resolves the method dynamically off the plugin object (plugin[checkMethod]), so it is not a member of the Plugin interface either.

import { PluginHealthMonitor } from '@objectstack/core';
import { PluginHealthCheckSchema } from '@objectstack/spec/kernel';

// 1. The plugin side — a plain method, invoked with NO arguments.
export const salesforcePlugin = {
  name: '@vendor/salesforce',

  // The name is free; `checkMethod` below is what points at it.
  async healthCheck() {
    try {
      await salesforce.query('SELECT Id FROM Account LIMIT 1');
      return true;
    } catch (error) {
      return { status: 'unhealthy', message: error.message };
    }
  },
};

// 2. The host side — the embedding application owns the monitor.
const monitor = new PluginHealthMonitor(kernel.logger);

// `registerPlugin` takes the PARSED config, so parse it: the schema fills in
// interval 30000, timeout 5000, failureThreshold 3 and successThreshold 1.
monitor.registerPlugin(
  salesforcePlugin.name,
  PluginHealthCheckSchema.parse({ checkMethod: 'healthCheck' }),
);

monitor.startMonitoring(salesforcePlugin.name, salesforcePlugin);

startMonitoring runs one check immediately, then repeats every interval milliseconds; each run is raced against timeout, and the method may be synchronous or return a promise.

Only two returned shapes count as a failure: false, and an object whose status is exactly 'unhealthy' (whose message, if any, becomes the report's). Everything else passes — true, undefined, { status: 'healthy' } and a rich object of metrics alike, so a check has nowhere to publish latency or row counts: no key beyond status and message is read. Consecutive returned failures move the plugin to degraded first, and to unhealthy only once failureThreshold of them accumulate. A check that throws — including one that exceeds timeout — is the separate failed status, applied immediately with no threshold.

Recovery is the mirror of that half, and successThreshold is its counter: the number of consecutive passing rounds a plugin needs before the monitor reports it healthy again. The count is consulted from every status that records an observed failure — degraded, unhealthy, failed and recovering alike — and while it accumulates the plugin sits in recovering, which is therefore a reported status and not merely a vocabulary entry. healthy and unknown are the two statuses the count is not consulted from: neither records a failure to recover from, so a passing round promotes straight to healthy. unknown is what registerPlugin writes before any check has run, which is why a freshly registered plugin reads healthy on its first passing round however high successThreshold is declared.

"Consecutive" is strict, and it is the failing round that enforces it: any failure resets the success count to zero — both routes included — so a throw part-way through a recovery starts the next attempt at one rather than resuming where it left off. The symmetry holds the other way too: a passing round resets the failure count, so failureThreshold likewise counts only an unbroken run.

The monitor reports; it does not act

Nothing above does anything to the plugin. A failing plugin is labelled degraded, unhealthy or failed and left running; the monitor never calls destroy(), and acting on what it reports is the host's job — this is a host-driven library, and the host is the only party that owns the plugin's lifetime.

It used to claim otherwise. PluginHealthCheck carried autoRestart, maxRestartAttempts and restartBackoff, and a plugin that crossed failureThreshold with autoRestart: true got plugin.destroy() called on it — and nothing else. init() was never called, because the monitor has no PluginContext to call it with and no way to obtain one. The plugin was then logged as Plugin restarted, marked recovering, and kept under periodic checks it went on passing from the grave: the default check when no checkMethod resolves is plugin-loaded, which a destroyed object satisfies forever. So the terminal report on a torn-down plugin was healthy. The three keys were removed in @objectstack/spec 18 under ADR-0049 enforce-or-remove; PluginHealthMonitor.registerPlugin refuses a config that still carries one.

At the default successThreshold: 1 none of this is observable: the first passing round satisfies the count from every status, and recovering is never the status a check leaves behind. The distinction appears only once a config declares a value above 1.

The monitor keeps one report per plugin rather than one aggregate document. Each round of checks builds a PluginHealthReport (@objectstack/spec/kernel, constructed in packages/core/src/health-monitor.ts) and stores it under the plugin's name, where getHealthReport(pluginName) reads it back. Nothing serves this shape over HTTP — it is an in-process model, not a wire body.

{
  "status": "healthy",
  "timestamp": "2024-01-15T11:00:00.000Z",
  "metrics": {
    "uptime": 3600000
  },
  "checks": [
    { "name": "healthCheck", "status": "passed" }
  ]
}

Entry `name`	Pushed when
the plugin's configured `checkMethod`	the custom check ran and returned — `"passed"`, or `"failed"` for the two failing shapes above
`"plugin-loaded"`	no `checkMethod` is configured, or the configured name does not resolve to a function on the plugin
`"health-check"`	the check threw — a `timeout` overrun included, since the race surfaces it as a rejection. A fixed name, neither the method's nor the default's, and always `status: "failed"`

metrics.uptime is in milliseconds (Date.now() - startTime), unlike the seconds-valued uptime of GET /health above, and the report carries no version field — it identifies its plugin by the key it is stored under. The optional message is set only when a check fails; the schema's remaining metrics fields (memoryUsage, cpuUsage, activeConnections, errorRate, responseTime) and its dependencies array are declared but left unset by the monitor today.

Shutdown Sequence

Graceful shutdown ensures in-flight requests complete before process exits.

Shutdown Phases

SIGTERM received
  ↓
[1] Stop accepting new requests
  ↓
[2] Finish in-flight requests (timeout: 30s)
  ↓
[3] Stop background jobs
  ↓
[4] Close database connections
  ↓
[5] Flush audit logs
  ↓
[6] Exit process

Shutdown Configuration

Shutdown is configured at kernel construction, not via metadata:

import { ObjectKernel } from '@objectstack/core';

const kernel = new ObjectKernel({
  // Enable graceful shutdown on SIGTERM / SIGINT
  gracefulShutdown: true,
  // Wait this long for in-flight work to drain before force-stopping
  shutdownTimeout: 30_000,
});

The host process is responsible for signal handling. ObjectStack ships the listeners only when gracefulShutdown: true; otherwise it stays out of the process-management business so adapters (Express, Fastify, Hono, Vercel, …) can install their own.

Best Practices

1. Always Use Transactions for Installations

Every installation step should be atomic. If step 5/8 fails, steps 1-4 must rollback.

// ✓ GOOD: Use transaction
const tx = await db.beginTransaction();
try {
  await step1(tx);
  await step2(tx);
  await tx.commit();
} catch (error) {
  await tx.rollback();
}

// ✗ BAD: No transaction
await step1();
await step2(); // If this fails, step1 is not reverted!

2. Version Migrations, Don't Modify Them

Once a migration file is applied in production, never modify it. Generate a new one instead.

// ✗ BAD: Modifying an existing migration file
// migrations/20260101000000_migration.ts (ALREADY APPLIED)
// Editing it here changes history that databases have already seen.

// ✓ GOOD: Generate a new migration after updating the object metadata
//   1. Edit src/objects/account.object.ts — add the new field
//   2. Run:  os generate migration
//   3. Commit  migrations/20260201000000_migration.ts

3. Test Upgrades in Staging First

Always validate upgrades in a staging environment that mirrors production.

# Staging
objectstack upgrade --dry-run  # Preview changes
objectstack upgrade            # Apply upgrade
objectstack test               # Run integration tests

# If tests pass → Production
objectstack upgrade --production

4. Monitor Health After Upgrades

Don't assume success. Monitor health checks for 5-10 minutes after upgrade.

# Automated monitoring
objectstack upgrade @vendor/salesforce --monitor --duration 300
# Watches /health for 5 minutes, auto-rollback if unhealthy

5. Document Breaking Changes

Plugin authors must document breaking changes in CHANGELOG.md.

## v4.0.0 (Breaking Changes)

### Removed
- `salesforce.sync()` method (use `salesforce.syncAccounts()` instead)

### Changed
- `salesforce_account.name` field renamed to `account_name`

### Migration Guide
1. Update code: `sync()` → `syncAccounts()`
2. Edit the relevant object metadata files
3. Run `os generate migration` and review the generated DDL
4. Apply via your usual Knex / driver pipeline

Summary

ObjectStack lifecycle management provides:

Deterministic Boot: 7-phase boot sequence with clear error handling
Atomic Installations: Transactions ensure all-or-nothing plugin installs
Zero-Downtime Upgrades: Blue-green and rolling strategies for production
Automatic Rollback: Failed upgrades auto-revert to previous version
Health Monitoring: Built-in health checks validate system state

Next: Learn how to define plugin manifests in Plugin Specification.

System Lifecycle

On this page