ObjectStackObjectStack

IDataEngine Contract

Reference for the Data Engine contract — the core data persistence layer for CRUD operations, queries, aggregations, and transactions

The Data Engine is the core persistence layer of ObjectStack. Every data operation — inserts, finds, updates, deletes, counts, and aggregations — flows through this contract. The Kernel delegates to the Data Engine after applying security, validation, and hooks.

Source: packages/spec/src/contracts/data-engine.ts Schema: packages/spec/src/data/data-engine.zod.ts Service name: data (registered via CoreServiceName)


Interface Definition

The canonical IDataEngine interface uses QueryAST-aligned parameter names (where, fields, orderBy, limit, offset, expand) — no mechanical translation is needed between the Engine and Driver layers.

import type {
  BaseEngineOptions,
  EngineQueryOptions,
  DataEngineInsertOptions,
  EngineUpdateOptions,
  EngineDeleteOptions,
  EngineAggregateOptions,
  EngineCountOptions,
  DataEngineRequest,
} from '@objectstack/spec/data';

export interface IDataEngine {
  // Query (reads take the execution context in a TRAILING options argument —
  // the same position the write methods take theirs)
  find(objectName: string, query?: EngineQueryOptions, options?: BaseEngineOptions): Promise<any[]>;
  findOne(objectName: string, query?: EngineQueryOptions, options?: BaseEngineOptions): Promise<any>;
  count(objectName: string, query?: EngineCountOptions, options?: BaseEngineOptions): Promise<number>;
  aggregate(objectName: string, query: EngineAggregateOptions, options?: BaseEngineOptions): Promise<any[]>;

  // Mutation (write ops also accept in-process WriteObservabilityOptions — see `update`)
  insert(objectName: string, data: any | any[], options?: DataEngineInsertOptions & WriteObservabilityOptions): Promise<any>;
  update(objectName: string, data: any, options?: EngineUpdateOptions & WriteObservabilityOptions): Promise<any>;
  delete(objectName: string, options?: EngineDeleteOptions): Promise<any>;

  // AI / Vector Search (optional)
  vectorFind?(objectName: string, vector: number[], options?: {
    where?: any; limit?: number; fields?: string[]; threshold?: number;
  }): Promise<any[]>;

  // Batch Operations (optional, transactional)
  batch?(requests: DataEngineRequest[], options?: { transaction?: boolean }): Promise<any[]>;

  // Raw Command Escape Hatch (optional)
  execute?(command: any, options?: Record<string, any>): Promise<any>;

  // Driver Registry (optional — engines that own named drivers)
  getDefaultDriverName?(): string | undefined;
  getDriverByName?(name: string): IDataDriver | undefined;
}

Query Operations

All query methods use canonical QueryAST parameter names: where, fields, orderBy, limit, offset, expand.

Reads take the execution context in the trailing options argument, the same position the write methods take theirs — find, findOne, count and aggregate all accept options?: BaseEngineOptions.

This matters because the mistake it prevents is silent. The same { context } object is correct as the third argument to insert, and passing it as the third argument to find used to be dropped without error — so an intended isSystem bypass simply vanished, and control-plane reads started coming back empty once org-scoping hooks landed (#4251).

query.context remains supported. When both are given, options.context wins.

find

Executes a structured query with filtering, sorting, pagination, and field selection. Returns an array of records.

const tasks = await engine.find('task', {
  where: {
    status: 'open',
    priority: { $in: ['high', 'critical'] },
  },
  orderBy: [{ field: 'due_date', order: 'asc' }],
  limit: 20,
  offset: 0,
  fields: ['id', 'title', 'status', 'priority', 'due_date'],
});

EngineQueryOptions

Defined by EngineQueryOptionsSchema in @objectstack/spec:

interface EngineQueryOptions {
  where?: FilterCondition;                   // WHERE clause — MongoDB-style $op
  fields?: FieldNode[];                      // SELECT — the object's OWN column names; related
                                             // data comes from `expand`, not a dotted path (#7532)
  orderBy?: SortNode[];                      // ORDER BY
  limit?: number;                            // LIMIT
  offset?: number;                           // OFFSET
  top?: number;                              // Alias for limit (OData compat)
  search?: string | FullTextSearch;          // Full-text search — the bare query text
                                             // is canonical (ADR-0061 D1); the object
                                             // form carries the Tier-2 knobs (#7178)
  searchFields?: string[];                   // Fields the `search` expansion may match
                                             // against — a validated override, intersected
                                             // with the object's declared/derived
                                             // searchable set (ADR-0061 D1)
  expand?: Record<string, QueryAST>;         // Recursive relation loading
  context?: ExecutionContext;            // Identity, tenant, transaction — any subset
}

ExecutionContext is the author state — every field optional (the parse result, with defaults applied, is ExecutionContextParsed). Supply what you have, the engine reads what it needs. A system read passes { isSystem: true }; an automation run with no resolvable identity passes only its run id ({ flowRunId }), a context that deliberately carries no principal.

Removed in protocol 17: cursor and distinct

Both keys were removed from EngineQueryOptions in protocol 17 (#4286, ADR-0049), alongside the identically-named keys on Query. They are not merely absent: the schema keeps a tombstone for each, so a query still carrying one is rejected by name with the migration prose below rather than silently ignored.

  • cursor?: Record<string, unknown> (keyset pagination) — no driver ever implemented it, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). QueryBuilder.cursor() was removed with it. Express the keyset as an ordinary where predicate on your sort key — where: { created_at: { $gt: last.created_at } } with the matching orderBy — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record.
  • distinct?: boolean (SELECT DISTINCT) — no driver ever rendered it, and the flag's only observable effect was MIS-WIRED: the REST list path treated a distinct query as not countable and silently degraded total/hasMore to a page-local estimate while still returning duplicate rows. QueryBuilder.distinct() was removed with it, and the count suppression is gone (total is truthful again). For unique values of one column use the SQL/memory drivers' distinct(object, field) door; for unique combinations, groupBy; for a deduplicated count, the count_distinct aggregation.

FilterCondition (where)

Filters use the canonical where + MongoDB-style $op object syntax from FilterConditionSchema:

// Implicit equality
where: { status: 'active' }

// Explicit operators
where: { amount: { $gt: 50000 } }

// Logical combinations
where: {
  $and: [
    { status: 'open' },
    { priority: { $in: ['high', 'critical'] } },
  ],
}

// Logical OR
where: {
  $or: [
    { role: 'admin' },
    { email: { $contains: '@company.com' } },
  ],
}

// Nested relation filter
where: {
  account: { industry: 'tech' },
}

Supported operators: $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $between, $contains, $notContains, $startsWith, $endsWith, $null, $exists

Logical operators: $and, $or, $not

findOne

Returns the ONE record the query selects, or null.

const task = await engine.findOne('task', {
  where: { title: 'Implement login page' },
  fields: ['id', 'title', 'status'],
});

The query must say which record it wants — a where (or a search that expands to one), or an orderBy meaning "the first record in this order". A query with neither is rejected (#4419):

await engine.findOne('task', {});                    // throws
await engine.findOne('task', {                       // the newest task
  orderBy: [{ field: 'created_at', order: 'desc' }],
});
await engine.find('task', { limit: 1 });             // any task will do

findOne reads a single row, so a missing predicate does not come back as null — it comes back as the object's first row, a real record unrelated to the request that no if (!task) check can catch. No ordering is imposed when you supply none: findOne promises a matching record, never a position in a sequence.

count

Returns the number of records matching a filter without fetching data.

const openTasks = await engine.count('task', {
  where: { status: 'open' },
});
// 23

Mutation Operations

insert

Creates one or more records. Returns the inserted record(s) with system-generated fields (id, created_at, etc.).

const task = await engine.insert('task', {
  title: 'Implement login page',
  status: 'open',
  priority: 'high',
  assigned_to: 'usr_01HQ3V5K8N2M4P6R7T9W',
});

console.log(task.id);         // "tsk_01HQ4A7B9D3F5G8J2K4L"
console.log(task.created_at); // "2025-01-20T10:30:00.000Z"

Bulk insert:

const tasks = await engine.insert('task', [
  { title: 'Task 1', status: 'open', priority: 'high' },
  { title: 'Task 2', status: 'open', priority: 'medium' },
  { title: 'Task 3', status: 'open', priority: 'low' },
]);

DataEngineInsertOptions

interface DataEngineInsertOptions {
  returning?: boolean;          // Return inserted record(s)? Default: true
  context?: ExecutionContext;  // Identity, tenant, transaction — any subset
}

update

Updates specific fields on matched record(s). The where clause in options identifies target records.

const updated = await engine.update('task', 
  { status: 'in_progress', estimated_hours: 12 },
  { where: { id: 'tsk_01HQ4A7B9D3F5G8J2K4L' } }
);

Partial Updates: Only fields included in the data parameter are modified. Omitted fields retain their current values.

EngineUpdateOptions

interface EngineUpdateOptions {
  where?: FilterCondition;      // Filter to identify records (WHERE)
  upsert?: boolean;             // Insert if not found? Default: false
  multi?: boolean;              // Update multiple records? Default: false
  returning?: boolean;          // Return updated record(s)? Default: false
  context?: ExecutionContext;
}

WriteObservabilityOptions

The write methods (insert / update) additionally accept two in-process options that govern what happens when caller-supplied write fields are legally stripped from the payload before the driver write: observe the strip (onFieldsDropped) or refuse the write outright (strictReadonlyWrites).

The strips these two options cover are the engine's legal ones:

StripreasonVerbsWriters it skips
Static readonly: true (#2948)readonlyupdateisSystem
A TRUE readonlyWhen predicate (#3042)readonly_whenupdatenone at the API boundary — every caller, isSystem included; a value a beforeUpdate hook derived or overwrote is not a caller write and is never stripped (#9107)
Implicitly-readonly runtime-owned type (#5503 — RUNTIME_OWNED_FIELD_TYPES, today autonumber)readonlyinsert and updateisSystem, preserveAudit (#3493)
Primary-key strip of a payload id the update dispatch already ruled is not an identifier (#6437)primary_keyupdatenone

The two AUTHOR-DECLARED strips are insert-exempt at this seam by design (#3413) — see On insert below.

interface WriteObservabilityOptions {
  onFieldsDropped?: (event: DroppedFieldsEvent) => void;
  strictReadonlyWrites?: boolean;  // refuse instead of stripping. Default: false
}

interface DroppedFieldsEvent {
  object: string;   // resolved object name
  fields: string[]; // caller-supplied fields that were dropped
  // why they were dropped — an OPEN vocabulary that grows with the write
  // path's legal strips; branch on it exhaustively, never with a binary test
  reason: 'readonly' | 'readonly_when' | 'primary_key';
}

onFieldsDropped — quiet and observable

The engine invokes the listener once per strip pass that dropped at least one caller-supplied field. The write still succeeds and commits without those fields; the listener exists so callers that report per-field success (e.g. the flow engine's update_record step) can surface a warning instead of a silent success. Branch on reason exhaustively — it is an OPEN vocabulary that grows with the write path's legal strips, never a binary test.

strictReadonlyWrites — loud instead (#5126)

Default false. When true, a write whose payload WOULD have caller-supplied fields stripped throws before the driver is touched instead of committing the remainder. Nothing is written: not the stripped fields, not the fields that would have survived. The strip passes still run — that is how the engine learns WHICH fields would go — but their result is discarded.

Its coverage is DERIVED from what onFieldsDropped reports, not an enumeration frozen at #5126: every strip in the table above is refused, and a new reason adds a new refusal by construction. Covering only the static arm would leave a trusted caller — the very caller this option exists for, one that already passes { context: { isSystem: true } } and is therefore exempt from the static strip — still losing readonlyWhen fields in silence. The flag's NAME is narrower than its coverage and stays that way on purpose; the coverage sentence, not the name, is the contract.

The refusal is ReadonlyFieldRejectedError, code ERR_READONLY_FIELD_REJECTED (registered in ERROR_CODE_LEDGER under @objectstack/objectql), carrying the FULL list of rejected fields accumulated across every strip pass the operation runs — one error naming everything, so a caller fixes its payload once instead of one round-trip per field. Catch it by code, not instanceof, and read drops for the per-reason breakdown; the code is stable across reasons deliberately, so adding a reason never adds an error code.

onFieldsDropped does not fire on a write this option refuses. The two are alternative outputs of one seam, not a sequence: DroppedFieldsEvent means "fields dropped and the write completed without them", and under strict the write does not complete. Quiet-and-observable or loud — pick one per call.

On insert. The two AUTHOR-DECLARED strips are deliberately insert-exempt at this seam (#3413: an in-process create may seed a readonly: true field's initial value, and readonlyWhen cannot lock anything on a create at all), so an insert refusal can only ever be about a runtime-owned value — a caller-supplied record number. With the option true that insert throws (operation: 'insert') and nothing is written; without it the value is stripped, the write completes, and onFieldsDropped fires with reason: 'readonly'. The engine-level writers exempt from that strip — and therefore never refused — are the two the error message itself names: isSystem, and the preserveAudit historical import reinstating legacy record numbers (#3493).

Layering — this is the engine seam. The exemption pair above is this in-process seam's. The DataProtocol ingress enforces its own author-declared readonly policy on create (#3043), where preserveAudit is UPDATE-only (#6640) and runtime-owned types are left to the engine strip — see FieldSchema.readonly. Nothing on this page widens or narrows that ingress policy.

WriteObservabilityOptions is a TS-contract-level, in-process-only bag — both members. It is deliberately not part of the serializable Zod options schemas: a function is unrepresentable in JSON Schema and cannot cross the RPC (Virtual Data Engine) boundary, so remote callers never receive these events; and putting strictReadonlyWrites in the serializable bag would let any client toggle write-refusal on a security-adjacent path (#5126 ruling). A remote caller can set neither and gets NEITHER behaviour: its write is stripped and committed, silently from its side — a 200 whose read-only columns kept their stored values. Widening strict to the wire is a SEPARATE decision. A listener that throws never breaks the write — the engine catches and logs.

delete

Deletes record(s) matching the where condition.

await engine.delete('task', {
  where: { id: 'tsk_01HQ4A7B9D3F5G8J2K4L' },
});

EngineDeleteOptions

interface EngineDeleteOptions {
  where?: FilterCondition;      // Filter to identify records (WHERE)
  multi?: boolean;              // Delete multiple records? Default: false
  context?: ExecutionContext;
}

Aggregation

aggregate

Runs aggregation queries with grouping for analytics and reporting.

const result = await engine.aggregate('task', {
  where: { project: 'prj_01HQ3V5K8N' },
  groupBy: ['status'],
  aggregations: [
    { function: 'count', alias: 'task_count' },
    { function: 'sum', field: 'estimated_hours', alias: 'total_hours' },
    { function: 'avg', field: 'estimated_hours', alias: 'avg_hours' },
  ],
});

// result:
// [
//   { status: 'open', task_count: 15, total_hours: 120, avg_hours: 8 },
//   { status: 'in_progress', task_count: 8, total_hours: 96, avg_hours: 12 },
//   { status: 'done', task_count: 24, total_hours: 192, avg_hours: 8 },
// ]

EngineAggregateOptions

interface EngineAggregateOptions {
  where?: FilterCondition;                 // Pre-aggregation filter (WHERE)
  groupBy?: string[];                      // GROUP BY fields
  aggregations?: AggregationNode[];        // Aggregation definitions
  context?: ExecutionContext;
}

interface AggregationNode {
  function: 'count' | 'sum' | 'avg' | 'min' | 'max' | 'count_distinct';
  field?: string;           // Field to aggregate (optional for COUNT(*))
  alias: string;            // Result column alias
  filter?: FilterCondition; // Per-aggregation FILTER WHERE
}

distinct?: boolean was removed from AggregationNode in protocol 17 (#6815, ADR-0049). Only the engine's in-memory fallback ever honoured it — every SQL face ignored it — so the same query answered a deduplicated sum or an ordinary one depending on which backend served it. For a deduplicated count use the count_distinct function, which every face computes.


Optional Capabilities

vectorFind (AI/RAG)

Perform similarity search using vector embeddings:

const results = await engine.vectorFind?.('document', embeddingVector, {
  where: { category: 'technical' },
  fields: ['title', 'content'],
  limit: 5,
  threshold: 0.8,
});

batch (Transactional)

Execute multiple operations in a single transaction:

const results = await engine.batch?.([
  { method: 'insert', object: 'project', data: { name: 'Website Redesign', status: 'active' } },
  { method: 'insert', object: 'task', data: { title: 'Design mockups', status: 'open' } },
  { method: 'insert', object: 'task', data: { title: 'Implement frontend', status: 'open' } },
], { transaction: true });

execute (Raw Command)

Escape hatch for raw driver-specific commands:

const result = await engine.execute?.(
  'SELECT * FROM tasks WHERE status = $1',
  { params: ['open'] }
);

getDefaultDriverName / getDriverByName (Driver Registry)

Look up the engine's registered drivers. Only engines that own a named-driver registry implement these (ObjectQL does; test fakes and remote/virtual engines need not) — always probe with ?.:

const driverName = engine.getDefaultDriverName?.();
const driver = driverName ? engine.getDriverByName?.(driverName) : undefined;

This is the surface the runtime uses to re-register the default driver as a driver.<name> kernel service, which is where os migrate and serve's storage detection locate drivers.


Error Codes

CodeHTTPDescription
RECORD_NOT_FOUND404No record exists with the given ID
VALIDATION_FAILED400Data does not match the object schema
DELETE_RESTRICTED409Cannot delete; dependent child records reference it via a restrict delete behavior

Legacy Compatibility

Deprecated Parameter Names

The following legacy parameter names are accepted by the RPC layer for backward compatibility but should not be used in new code. The protocol normalizer resolves conflicts with canonical names taking precedence.

Legacy (Deprecated)CanonicalNotes
filterwhereFilterCondition object
selectfieldsArray of FieldNode (field-name strings)
sortorderByArray of { field, order }
skipoffsetNumber
populateexpandRecord of field → QueryAST

The deprecated DataEngineQueryOptionsSchema, DataEngineUpdateOptionsSchema, DataEngineDeleteOptionsSchema, and DataEngineAggregateOptionsSchema are maintained in @objectstack/spec for backward compatibility but will be removed in a future major version. Migrate to the QueryAST-aligned equivalents: EngineQueryOptionsSchema, EngineUpdateOptionsSchema, EngineDeleteOptionsSchema, EngineAggregateOptionsSchema.

On this page