IDataEngine Contract
Reference for the Data Engine contract — the core data persistence layer for CRUD operations, queries, aggregations, and transactions
The Data Engine is the core persistence layer of ObjectStack. Every data operation — inserts, finds, updates, deletes, counts, and aggregations — flows through this contract. The Kernel delegates to the Data Engine after applying security, validation, and hooks.
Source: packages/spec/src/contracts/data-engine.ts
Schema: packages/spec/src/data/data-engine.zod.ts
Service name: data (registered via CoreServiceName)
Interface Definition
The canonical IDataEngine interface uses QueryAST-aligned parameter names (where, fields, orderBy, limit, offset, expand) — no mechanical translation is needed between the Engine and Driver layers.
import type {
BaseEngineOptions,
EngineQueryOptions,
DataEngineInsertOptions,
EngineUpdateOptions,
EngineDeleteOptions,
EngineAggregateOptions,
EngineCountOptions,
DataEngineRequest,
} from '@objectstack/spec/data';
export interface IDataEngine {
// Query (reads take the execution context in a TRAILING options argument —
// the same position the write methods take theirs)
find(objectName: string, query?: EngineQueryOptions, options?: BaseEngineOptions): Promise<any[]>;
findOne(objectName: string, query?: EngineQueryOptions, options?: BaseEngineOptions): Promise<any>;
count(objectName: string, query?: EngineCountOptions, options?: BaseEngineOptions): Promise<number>;
aggregate(objectName: string, query: EngineAggregateOptions, options?: BaseEngineOptions): Promise<any[]>;
// Mutation (write ops also accept in-process WriteObservabilityOptions — see `update`)
insert(objectName: string, data: any | any[], options?: DataEngineInsertOptions & WriteObservabilityOptions): Promise<any>;
update(objectName: string, data: any, options?: EngineUpdateOptions & WriteObservabilityOptions): Promise<any>;
delete(objectName: string, options?: EngineDeleteOptions): Promise<any>;
// AI / Vector Search (optional)
vectorFind?(objectName: string, vector: number[], options?: {
where?: any; limit?: number; fields?: string[]; threshold?: number;
}): Promise<any[]>;
// Batch Operations (optional, transactional)
batch?(requests: DataEngineRequest[], options?: { transaction?: boolean }): Promise<any[]>;
// Raw Command Escape Hatch (optional)
execute?(command: any, options?: Record<string, any>): Promise<any>;
// Driver Registry (optional — engines that own named drivers)
getDefaultDriverName?(): string | undefined;
getDriverByName?(name: string): IDataDriver | undefined;
}Query Operations
All query methods use canonical QueryAST parameter names: where, fields, orderBy, limit, offset, expand.
Reads take the execution context in the trailing options argument, the same
position the write methods take theirs — find, findOne, count and aggregate
all accept options?: BaseEngineOptions.
This matters because the mistake it prevents is silent. The same { context } object
is correct as the third argument to insert, and passing it as the third argument to
find used to be dropped without error — so an intended isSystem bypass simply
vanished, and control-plane reads started coming back empty once org-scoping hooks
landed (#4251).
query.context remains supported. When both are given, options.context wins.
find
Executes a structured query with filtering, sorting, pagination, and field selection. Returns an array of records.
const tasks = await engine.find('task', {
where: {
status: 'open',
priority: { $in: ['high', 'critical'] },
},
orderBy: [{ field: 'due_date', order: 'asc' }],
limit: 20,
offset: 0,
fields: ['id', 'title', 'status', 'priority', 'due_date'],
});EngineQueryOptions
Defined by EngineQueryOptionsSchema in @objectstack/spec:
interface EngineQueryOptions {
where?: FilterCondition; // WHERE clause — MongoDB-style $op
fields?: FieldNode[]; // SELECT — the object's OWN column names; related
// data comes from `expand`, not a dotted path (#7532)
orderBy?: SortNode[]; // ORDER BY
limit?: number; // LIMIT
offset?: number; // OFFSET
top?: number; // Alias for limit (OData compat)
search?: string | FullTextSearch; // Full-text search — the bare query text
// is canonical (ADR-0061 D1); the object
// form carries the Tier-2 knobs (#7178)
searchFields?: string[]; // Fields the `search` expansion may match
// against — a validated override, intersected
// with the object's declared/derived
// searchable set (ADR-0061 D1)
expand?: Record<string, QueryAST>; // Recursive relation loading
context?: ExecutionContext; // Identity, tenant, transaction — any subset
}ExecutionContext is the author state — every field optional (the parse result,
with defaults applied, is ExecutionContextParsed). Supply what you have, the engine reads what it needs. A system read passes
{ isSystem: true }; an automation run with no resolvable identity passes only
its run id ({ flowRunId }), a context that deliberately carries no principal.
Removed in protocol 17: cursor and distinct
Both keys were removed from EngineQueryOptions in protocol 17 (#4286,
ADR-0049), alongside the identically-named keys on Query. They are not merely
absent: the schema keeps a tombstone for each, so a query still carrying one is
rejected by name with the migration prose below rather than silently ignored.
cursor?: Record<string, unknown>(keyset pagination) — no driver ever implemented it, so the cursor was accepted and ignored and every page came back identical (a caller looping "untilhasMoreis false" never terminates).QueryBuilder.cursor()was removed with it. Express the keyset as an ordinarywherepredicate on your sort key —where: { created_at: { $gt: last.created_at } }with the matchingorderBy— which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record.distinct?: boolean(SELECT DISTINCT) — no driver ever rendered it, and the flag's only observable effect was MIS-WIRED: the REST list path treated a distinct query as not countable and silently degradedtotal/hasMoreto a page-local estimate while still returning duplicate rows.QueryBuilder.distinct()was removed with it, and the count suppression is gone (totalis truthful again). For unique values of one column use the SQL/memory drivers'distinct(object, field)door; for unique combinations,groupBy; for a deduplicated count, thecount_distinctaggregation.
FilterCondition (where)
Filters use the canonical where + MongoDB-style $op object syntax from FilterConditionSchema:
// Implicit equality
where: { status: 'active' }
// Explicit operators
where: { amount: { $gt: 50000 } }
// Logical combinations
where: {
$and: [
{ status: 'open' },
{ priority: { $in: ['high', 'critical'] } },
],
}
// Logical OR
where: {
$or: [
{ role: 'admin' },
{ email: { $contains: '@company.com' } },
],
}
// Nested relation filter
where: {
account: { industry: 'tech' },
}Supported operators: $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $between, $contains, $notContains, $startsWith, $endsWith, $null, $exists
Logical operators: $and, $or, $not
findOne
Returns the ONE record the query selects, or null.
const task = await engine.findOne('task', {
where: { title: 'Implement login page' },
fields: ['id', 'title', 'status'],
});The query must say which record it wants — a where (or a search that
expands to one), or an orderBy meaning "the first record in this order". A
query with neither is rejected (#4419):
await engine.findOne('task', {}); // throws
await engine.findOne('task', { // the newest task
orderBy: [{ field: 'created_at', order: 'desc' }],
});
await engine.find('task', { limit: 1 }); // any task will dofindOne reads a single row, so a missing predicate does not come back as
null — it comes back as the object's first row, a real record unrelated to
the request that no if (!task) check can catch. No ordering is imposed when
you supply none: findOne promises a matching record, never a position in a
sequence.
count
Returns the number of records matching a filter without fetching data.
const openTasks = await engine.count('task', {
where: { status: 'open' },
});
// 23Mutation Operations
insert
Creates one or more records. Returns the inserted record(s) with system-generated fields (id, created_at, etc.).
const task = await engine.insert('task', {
title: 'Implement login page',
status: 'open',
priority: 'high',
assigned_to: 'usr_01HQ3V5K8N2M4P6R7T9W',
});
console.log(task.id); // "tsk_01HQ4A7B9D3F5G8J2K4L"
console.log(task.created_at); // "2025-01-20T10:30:00.000Z"Bulk insert:
const tasks = await engine.insert('task', [
{ title: 'Task 1', status: 'open', priority: 'high' },
{ title: 'Task 2', status: 'open', priority: 'medium' },
{ title: 'Task 3', status: 'open', priority: 'low' },
]);DataEngineInsertOptions
interface DataEngineInsertOptions {
returning?: boolean; // Return inserted record(s)? Default: true
context?: ExecutionContext; // Identity, tenant, transaction — any subset
}update
Updates specific fields on matched record(s). The where clause in options identifies target records.
const updated = await engine.update('task',
{ status: 'in_progress', estimated_hours: 12 },
{ where: { id: 'tsk_01HQ4A7B9D3F5G8J2K4L' } }
);Partial Updates: Only fields included in the data parameter are modified. Omitted fields retain their current values.
EngineUpdateOptions
interface EngineUpdateOptions {
where?: FilterCondition; // Filter to identify records (WHERE)
upsert?: boolean; // Insert if not found? Default: false
multi?: boolean; // Update multiple records? Default: false
returning?: boolean; // Return updated record(s)? Default: false
context?: ExecutionContext;
}WriteObservabilityOptions
The write methods (insert / update) additionally accept two in-process
options that govern what happens when caller-supplied write fields are legally
stripped from the payload before the driver write: observe the strip
(onFieldsDropped) or refuse the write outright (strictReadonlyWrites).
The strips these two options cover are the engine's legal ones:
| Strip | reason | Verbs | Writers it skips |
|---|---|---|---|
Static readonly: true (#2948) | readonly | update | isSystem |
A TRUE readonlyWhen predicate (#3042) | readonly_when | update | none at the API boundary — every caller, isSystem included; a value a beforeUpdate hook derived or overwrote is not a caller write and is never stripped (#9107) |
Implicitly-readonly runtime-owned type (#5503 — RUNTIME_OWNED_FIELD_TYPES, today autonumber) | readonly | insert and update | isSystem, preserveAudit (#3493) |
Primary-key strip of a payload id the update dispatch already ruled is not an identifier (#6437) | primary_key | update | none |
The two AUTHOR-DECLARED strips are insert-exempt at this seam by design (#3413) —
see On insert below.
interface WriteObservabilityOptions {
onFieldsDropped?: (event: DroppedFieldsEvent) => void;
strictReadonlyWrites?: boolean; // refuse instead of stripping. Default: false
}
interface DroppedFieldsEvent {
object: string; // resolved object name
fields: string[]; // caller-supplied fields that were dropped
// why they were dropped — an OPEN vocabulary that grows with the write
// path's legal strips; branch on it exhaustively, never with a binary test
reason: 'readonly' | 'readonly_when' | 'primary_key';
}onFieldsDropped — quiet and observable
The engine invokes the listener once per strip pass that dropped at least one
caller-supplied field. The write still succeeds and commits without those
fields; the listener exists so callers that report per-field success (e.g. the
flow engine's update_record step) can surface a warning instead of a silent
success. Branch on reason exhaustively — it is an OPEN vocabulary that grows
with the write path's legal strips, never a binary test.
strictReadonlyWrites — loud instead (#5126)
Default false. When true, a write whose payload WOULD have caller-supplied
fields stripped throws before the driver is touched instead of committing the
remainder. Nothing is written: not the stripped fields, not the fields that would
have survived. The strip passes still run — that is how the engine learns WHICH
fields would go — but their result is discarded.
Its coverage is DERIVED from what onFieldsDropped reports, not an enumeration
frozen at #5126: every strip in the table above is refused, and a new reason
adds a new refusal by construction. Covering only the static arm would leave a
trusted caller — the very caller this option exists for, one that already passes
{ context: { isSystem: true } } and is therefore exempt from the static strip —
still losing readonlyWhen fields in silence. The flag's NAME is narrower than
its coverage and stays that way on purpose; the coverage sentence, not the name,
is the contract.
The refusal is ReadonlyFieldRejectedError, code ERR_READONLY_FIELD_REJECTED
(registered in ERROR_CODE_LEDGER under @objectstack/objectql), carrying the
FULL list of rejected fields accumulated across every strip pass the operation
runs — one error naming everything, so a caller fixes its payload once instead of
one round-trip per field. Catch it by code, not instanceof, and read drops
for the per-reason breakdown; the code is stable across reasons deliberately, so
adding a reason never adds an error code.
onFieldsDropped does not fire on a write this option refuses. The two are
alternative outputs of one seam, not a sequence: DroppedFieldsEvent means
"fields dropped and the write completed without them", and under strict the write
does not complete. Quiet-and-observable or loud — pick one per call.
On insert. The two AUTHOR-DECLARED strips are deliberately insert-exempt at
this seam (#3413: an in-process create may seed a readonly: true field's initial
value, and readonlyWhen cannot lock anything on a create at all), so an insert
refusal can only ever be about a runtime-owned value — a caller-supplied record
number. With the option true that insert throws (operation: 'insert') and
nothing is written; without it the value is stripped, the write completes, and
onFieldsDropped fires with reason: 'readonly'. The engine-level writers exempt
from that strip — and therefore never refused — are the two the error message
itself names: isSystem, and the preserveAudit historical import reinstating
legacy record numbers (#3493).
Layering — this is the engine seam. The exemption pair above is this
in-process seam's. The DataProtocol ingress enforces its own author-declared
readonly policy on create (#3043), where preserveAudit is UPDATE-only (#6640)
and runtime-owned types are left to the engine strip — see FieldSchema.readonly.
Nothing on this page widens or narrows that ingress policy.
WriteObservabilityOptions is a TS-contract-level, in-process-only bag —
both members. It is deliberately not part of the serializable Zod options schemas:
a function is unrepresentable in JSON Schema and cannot cross the RPC (Virtual
Data Engine) boundary, so remote callers never receive these events; and putting
strictReadonlyWrites in the serializable bag would let any client toggle
write-refusal on a security-adjacent path (#5126 ruling). A remote caller can set
neither and gets NEITHER behaviour: its write is stripped and committed, silently
from its side — a 200 whose read-only columns kept their stored values. Widening
strict to the wire is a SEPARATE decision. A listener that throws never breaks the
write — the engine catches and logs.
delete
Deletes record(s) matching the where condition.
await engine.delete('task', {
where: { id: 'tsk_01HQ4A7B9D3F5G8J2K4L' },
});EngineDeleteOptions
interface EngineDeleteOptions {
where?: FilterCondition; // Filter to identify records (WHERE)
multi?: boolean; // Delete multiple records? Default: false
context?: ExecutionContext;
}Aggregation
aggregate
Runs aggregation queries with grouping for analytics and reporting.
const result = await engine.aggregate('task', {
where: { project: 'prj_01HQ3V5K8N' },
groupBy: ['status'],
aggregations: [
{ function: 'count', alias: 'task_count' },
{ function: 'sum', field: 'estimated_hours', alias: 'total_hours' },
{ function: 'avg', field: 'estimated_hours', alias: 'avg_hours' },
],
});
// result:
// [
// { status: 'open', task_count: 15, total_hours: 120, avg_hours: 8 },
// { status: 'in_progress', task_count: 8, total_hours: 96, avg_hours: 12 },
// { status: 'done', task_count: 24, total_hours: 192, avg_hours: 8 },
// ]EngineAggregateOptions
interface EngineAggregateOptions {
where?: FilterCondition; // Pre-aggregation filter (WHERE)
groupBy?: string[]; // GROUP BY fields
aggregations?: AggregationNode[]; // Aggregation definitions
context?: ExecutionContext;
}
interface AggregationNode {
function: 'count' | 'sum' | 'avg' | 'min' | 'max' | 'count_distinct';
field?: string; // Field to aggregate (optional for COUNT(*))
alias: string; // Result column alias
filter?: FilterCondition; // Per-aggregation FILTER WHERE
}distinct?: boolean was removed from AggregationNode in protocol 17 (#6815,
ADR-0049). Only the engine's in-memory fallback ever honoured it — every SQL face
ignored it — so the same query answered a deduplicated sum or an ordinary one
depending on which backend served it. For a deduplicated count use the
count_distinct function, which every face computes.
Optional Capabilities
vectorFind (AI/RAG)
Perform similarity search using vector embeddings:
const results = await engine.vectorFind?.('document', embeddingVector, {
where: { category: 'technical' },
fields: ['title', 'content'],
limit: 5,
threshold: 0.8,
});batch (Transactional)
Execute multiple operations in a single transaction:
const results = await engine.batch?.([
{ method: 'insert', object: 'project', data: { name: 'Website Redesign', status: 'active' } },
{ method: 'insert', object: 'task', data: { title: 'Design mockups', status: 'open' } },
{ method: 'insert', object: 'task', data: { title: 'Implement frontend', status: 'open' } },
], { transaction: true });execute (Raw Command)
Escape hatch for raw driver-specific commands:
const result = await engine.execute?.(
'SELECT * FROM tasks WHERE status = $1',
{ params: ['open'] }
);getDefaultDriverName / getDriverByName (Driver Registry)
Look up the engine's registered drivers. Only engines that own a named-driver
registry implement these (ObjectQL does; test fakes and remote/virtual engines
need not) — always probe with ?.:
const driverName = engine.getDefaultDriverName?.();
const driver = driverName ? engine.getDriverByName?.(driverName) : undefined;This is the surface the runtime uses to re-register the default driver as a
driver.<name> kernel service, which is where os migrate and serve's storage
detection locate drivers.
Error Codes
| Code | HTTP | Description |
|---|---|---|
RECORD_NOT_FOUND | 404 | No record exists with the given ID |
VALIDATION_FAILED | 400 | Data does not match the object schema |
DELETE_RESTRICTED | 409 | Cannot delete; dependent child records reference it via a restrict delete behavior |
Legacy Compatibility
Deprecated Parameter Names
The following legacy parameter names are accepted by the RPC layer for backward compatibility but should not be used in new code. The protocol normalizer resolves conflicts with canonical names taking precedence.
| Legacy (Deprecated) | Canonical | Notes |
|---|---|---|
filter | where | FilterCondition object |
select | fields | Array of FieldNode (field-name strings) |
sort | orderBy | Array of { field, order } |
skip | offset | Number |
populate | expand | Record of field → QueryAST |
The deprecated DataEngineQueryOptionsSchema, DataEngineUpdateOptionsSchema, DataEngineDeleteOptionsSchema, and DataEngineAggregateOptionsSchema are maintained in @objectstack/spec for backward compatibility but will be removed in a future major version. Migrate to the QueryAST-aligned equivalents: EngineQueryOptionsSchema, EngineUpdateOptionsSchema, EngineDeleteOptionsSchema, EngineAggregateOptionsSchema.