Metadata Service

The Metadata Service is the backbone of ObjectStack configuration management. It decouples the definition of metadata (Objects, Views, Flows) from its runtime delivery path: local development reads files or dist/objectstack.json, while production ObjectStack reads an immutable artifact from the control plane. Project databases store business rows only.

Architecture

The Metadata Service uses a Provider/Loader architecture. The core MetadataManager acts as a registry and orchestrator, delegating actual I/O operations to registered Loaders.

Core Concepts

1. The Manager

The MetadataManager is the single entry point for all metadata operations. It handles caching, serialization, and correct routing of requests to loaders.

import { NodeMetadataManager } from '@objectstack/metadata/node';

// NodeMetadataManager wires up the FilesystemLoader and file watcher.
// The browser-safe base `MetadataManager` (from '@objectstack/metadata')
// registers no loaders on its own — callers must provide them.
const manager = new NodeMetadataManager({
  rootDir: './src',
  watch: true // Enable hot reload
});

// Load an Object Definition
const accountObj = await manager.load('object', 'account');

2. Loaders (Data Sources)

Loaders adapt different storage backends to a common interface.

3. Granular Capabilities

Each Loader declares its capabilities via a Contract:

export interface MetadataLoaderContract {
  name: string;
  protocol: 'file:' | 'http:' | 's3:' | 'datasource:' | 'memory:';
  capabilities: {
    read: boolean;  // Can read items?
    write: boolean; // Can save modifications?
    watch: boolean; // Can push real-time updates?
    list: boolean;  // Can enumerate items?
  };
  supportedFormats: ('json' | 'yaml' | 'typescript' | 'javascript')[];
}

Internal Persistence API

The Metadata Service supports full CRUD operations for authoring tools and control-plane services. Runtime ObjectStack treats metadata as read-only after artifact/file hydration; MetadataPlugin does not automatically bridge ObjectQL to DatabaseLoader.

Saving Metadata (Node.js)

// 1. Simple Save (Auto-detects loader)
await manager.save('object', 'customer_profile', data);

// 2. Targeted Save (Specific Source)
await manager.save('object', 'customer_profile', data, {
  loader: 'filesystem',
  format: 'yaml', // Convert to YAML on save
  atomic: true    // Use write-rename strategy for safety
});

// 3. Create Backup
await manager.save('object', 'workflow_config', data, {
  backup: true // Creates 'workflow_config.json.bak' before writing
});

Loading Strategy

When load() is called, the Manager iterates through registered loaders. In ObjectStack runtime boot this means "artifact/file first"; database-backed metadata is reserved for explicitly configured control-plane services.

graph LR
    A[Request: load('object', 'user')] --> B{Metadata Manager}
    B --> C[Check Artifact / Memory Loader]
    C -- Not Found --> D[Check Filesystem Loader]
    D -- Found --> E[Return & Cache]
    C -- Found --> E

Watching & Hot Reload

The Service emits events when metadata changes, allowing the runtime to reconfigure itself without restarting.

const unsubscribe = manager.subscribe('object', (event) => {
  console.log(`Object ${event.name} changed!`);
  console.log(`New Schema:`, event.data);
  
  // Trigger system update
  SchemaEngine.refresh(event.name);
});

Remote API Access

The system provides standard REST and TypeScript APIs for interacting with metadata, including full read/write support.

REST API

• GET /api/v1/meta: List all metadata types.
• GET /api/v1/meta/:type: List items of a type.
• GET /api/v1/meta/:type/:name: Get the definition of an item.
• PUT /api/v1/meta/:type/:name: Save or update a metadata item.

Client SDK

// Save metadata from the browser
const result = await client.meta.saveItem('object', 'customer', {
  label: 'Customer',
  fields: {
    name: { type: 'text', required: true }
  }
});

Server-Side Protocol

// Access from Protocol Service (Server-side)
const protocol = kernel.getService('protocol');
await protocol.saveMetaItem({ 
  type: 'object', 
  name: 'lead', 
  item: { ... } 
});

Runtime Controls (v3.1+)

Four orthogonal levers shape how MetadataManager and MetadataPlugin behave at boot and at request time. All four default to "dev-friendly" so existing code keeps working.

1. Bootstrap Modes

MetadataPluginConfigSchema.bootstrap decides what the plugin does on start():

new MetadataPlugin({
  config: { bootstrap: 'artifact-only' },
  artifactSource: { mode: 'local-file', path: './dist/objectstack.json' },
});

Artifact source

artifactSource: { mode: 'local-file', path, fetchTimeoutMs? } is the single artifact source, honored by every bootstrap mode above. path is a filesystem path or an http(s) URL fetched verbatim — the control plane's public artifact route (/pub/v1/environments/:id/artifact[?commit=<id>]) serves exactly such URLs, so a sealed runtime can boot straight off a published revision:

new MetadataPlugin({
  config: { bootstrap: 'artifact-only' },
  artifactSource: {
    mode: 'local-file',
    path: 'https://cloud.example.com/pub/v1/environments/env_42/artifact?commit=cmt_1a2b',
  },
});

Only non-URL paths get the artifact-file HMR watcher (artifactWatch); remote reads honor fetchTimeoutMs (and OS_ARTIFACT_FETCH_TIMEOUT_MS, default 60 s). Outside artifact-only, an absent local file is "nothing compiled yet", not a fault.

A second artifact-api mode (a Bearer-authenticated control-plane pull) was removed while resolving #4246: it had no consumers — the cloud runtime uses its own ArtifactApiClient, and package distribution into a running OSS instance goes through @objectstack/cloud-connection (os package install). A still-configured artifact-api source fails loudly at start() rather than silently falling back to the filesystem scan.

2. Persistence Write Gates

MetadataManagerConfigSchema.persistence is a two-axis runtime freeze. Both flags default to true.

new MetadataManager({
  persistence: { writable: false, overlayWritable: false },
});

3. DatabaseLoader Read-Through Cache

DatabaseLoader wraps load / loadMany / list / stat results in a generic LRU cache (lazy TTL, promote-on-get, write invalidation). Reads always observe writes performed through the same loader instance; out-of-band SQL writes are honored within ttl milliseconds.

new MetadataManager({
  datasource: 'default',
  cache: {
    enabled: true,
    databaseLoader: { enabled: true, maxSize: 500, ttl: 60_000 },
  },
});

The cache exposes LRUCache.stats() (size / hits / misses / hitRate) for metrics.

4. Single-Source Schema Discipline

The canonical MetadataManagerConfigSchema and MetadataFallbackStrategySchema live in kernel/metadata-loader.zod.ts. The persistence-side module system/metadata-persistence.zod.ts re-exports them, so consumers observe a single TypeScript type regardless of import path. Drift between kernel-side runtime config and persistence-side typing is no longer possible.

CLI & Tooling Integration

The Metadata Service is designed to power the CLI.

• os compile: Validates TypeScript metadata and emits dist/objectstack.json.
• os package publish: Uploads the compiled dist/objectstack.json artifact to the control plane as a versioned package.
• os dev: Boots ObjectStack from local files or the compiled artifact without a control-plane connection.