Skip to content

Commit 3ae0b77

Browse files
fix(metadata-protocol): a seed reference written blank ('' or whitespace) is no reference, by the engine's isMissing (#24502)
Fixes #24459 Clause-②: no (no published accept set or surface moves: the row was already written, and only the false "unresolved" report and its `success: false` go away; owes the seat's `TIER_DEFAULT` review of record before the queue) Dispatched under claim 6261694593 (`domain:engine` seat 1, session `session_01JtSQQ3EWsbNPzyYMug1tXy`); dev session `session_01KxmveGHL9EpPEGVggMSzDB`. ## What changed The seed loader decided "this reference has a value" with `undefined` / `null` only, while the engine's blank rule, `isMissing` (`@objectstack/core`, `packages/core/src/utils/required-emptiness.ts`), also calls `''` and whitespace-only strings missing. Every site in `packages/metadata-protocol/src/seed-loader.ts` that judges a reference value now asks `isMissing` (imported, not re-spelled): | Site (symbol) | Before | After | |---|---|---| | single reference, `recordRefs` loop (`if (isMissing(fieldValue)) continue;`) | skipped `undefined` / `null` | skips every `isMissing` value; left **as written** for the engine | | each element of a `multiple: true` reference, pass 1 | dropped a `null` / `undefined` hole | drops every `isMissing` hole | | each element of a deferred value, pass 2 (`resolveDeferredUpdates`) | same as pass 1 | same rule as pass 1 (the deferred value is the whole authored array, holes included) | | `referenceVia` pointer id half | skipped `undefined` / `null` / `''` | skips every `isMissing` value (whitespace-only joins `''`) | Other "has a value" sites in the file were read and left alone, on purpose: `externalIdKey` judges the record's own natural-key components (not a reference); the organization stamps compare `== null` on `organization_id`, which is resolved by its own path; `formatAttempted` is display only. No change to `isMissing`, the engine's required check, or any error code. ### A blank element inside a multi-valued reference: dropped (decided by the four axes) - **Real business need:** measured on `main`, one blank element made the whole field defer and fail, so the row landed with **none** of its associations (`books: null` for `['Dune', '', ' ', 'Emma']`). Dropping the hole lands `[Dune, Emma]`, which is what the author wrote. - **Long-term soundness:** one hole rule. `null` holes were already dropped, and `isMissing` is the platform's single definition of "no value", so blank holes follow the same rule. Leaving the whole value as written for the engine is not possible, because the other elements are natural keys that must be resolved first. - **Guarding against AI-authored metadata errors:** the drop is not a lenient fallback for off-spec input. A blank element names no row under the engine's own rule. A key that names a missing row is still reported loudly, as before (control pin). - **No scope creep:** no new key, option or error code. ## Measured (real `ObjectQL` + real `SqlDriver` on better-sqlite3, `SeedLoaderService`, `multiPass: true`) Seed: `blank_ref_book` (`Dune`, `Emma`) and `blank_ref_loan` (`loan-real` → `'Dune'`, `loan-empty` → `''`, `loan-whitespace` → `' '`, `loan-multi` → `books: ['Dune', '', ' ', 'Emma']`). Plus a required lookup written `''`. | Reading | `main` (`01fafdee`, before) | this branch (after) | |---|---|---| | first load | `success: false`, inserted 6, `errored: 3`: three `Deferred reference unresolved after pass 2: …book = '' / ' ' / …books = '' → blank_ref_book.title not found` | `success: true`, inserted 6, `errored: 0`, no errors | | replay | `success: false`, skipped 6, `errored: 3` (same three) | `success: true`, skipped 6, updated 0, `errored: 0` | | `loan-empty` / `loan-whitespace` `.book` | `null` (the deferral deleted the key) | `''` / `' '`: left as written; `isMissing` true (see Acceptance notes) | | `loan-multi` `.books` | `null`: the whole association was lost | `[Dune.id, Emma.id]` | | `loan-real` `.book` (control) | `Dune.id` | `Dune.id` | | required lookup written `''` | `errored: 2`: `Failed to write … record #0 (name=loan-required-empty): book is required` **and** `Deferred reference unresolved after pass 2: …book = '' → …` | `errored: 1`: `Failed to write … record #0 (name=loan-required-empty): book is required` only; row not written | ## Pins - `packages/runtime/src/seed-blank-reference-real-driver.integration.test.ts` (new): the table above, on the real driver. It lives in `runtime` because that package already has the real-driver seed harness and the `driver-sql` dependency (`seed-multi-value-lookup-real-driver.integration.test.ts` is its sibling). `metadata-protocol` has neither. This is outside the claim's "pins beside it" fence, and the reason is the measurement the dispatch asked for. - `packages/metadata-protocol/src/seed-loader-multi-value-reference.test.ts` (6 new cases in a new `describe`): single `''` and whitespace skipped and left as written (not deferred, not dropped); a blank element dropped in pass 1 (no back-fill `update`); a blank element dropped in pass 2 (a forward self-reference, so it is proven by the back-fill `update`); a whitespace pointer skipped like `''`, including the un-addressable branch, plus a live-pair control; a control showing that a real key resolves and an unknown key still errors. These cases reuse that file's already-pinned engine double, so `check:engine-double-contract` needs no ledger edit. A new file with its own double was tried first, and the gate reported it. - Pin sweep: `git grep` for `Deferred reference unresolved` / `UNRESOLVED` finds no test that pins the old blank behaviour (the only hits are the loader, `CHANGELOG.md`, and these new pins). Existing `''` seed values in tests are a field-definition `reference: ''` (`seed-loader-reference-carrier.test.ts`) and an unrelated ownership claim (`claim-seed-ownership.test.ts`). No pin needed flipping. ## Ablations (committed fix first, `scripts/ablation-replace.mjs`, trap restore, restore proven by blob == HEAD and an empty porcelain) - Unit, per site, `src/` resolution (19-case file): `:1464` single makes the 2 single-blank cases red; pass-1 element makes the pass-1 element case red; pass-2 element makes the pass-2 case red; pointer makes the pointer case red. Each restores to 19/19 green. - Real driver, `dist/` resolution (`runtime` reads `@objectstack/metadata-protocol` through `exports`): all three reference skips set back to `undefined`/`null`, rebuilt, `ablation-dist-preflight` shows the markers present in `dist/`, and both real-driver cases go **red**. After restore and rebuild, `--absent` passes and both cases are green. - First attempt declared: ablating `:1464` alone was a void reading, in two ways. (a) The first marker did not reach `dist/`, because the build re-spells `undefined` as `void 0`; the preflight reported this, so a string-literal marker was used. (b) With the marker live, the real-driver file stayed green, because the pass-1 element skip still drops a scalar `''` (it is wrapped as a one-element list) and the insert lands NULL. The single-site difference (left as written vs dropped) is pinned by the unit cases above. The faithful "remove the skip" is all three sites together, and that run is the red reading above. ## Verification (final head `3e5746dd6`, after `git merge origin/main` at `ce2e08ee3` and a full `turbo build`, 76/76) - `pnpm --filter @objectstack/metadata-protocol test`: exit 0, 247 files passed (3 skipped), 20698 tests passed. - `pnpm --filter @objectstack/metadata-protocol typecheck`: exit 0. - `pnpm --filter @objectstack/runtime typecheck`: exit 0 (`check:test-typecheck` OK). - `pnpm --filter @objectstack/runtime test`: exit 0, 422 files, 6637 tests passed (pre-merge head `f749e65d1`; post-merge, the two real-driver seed files were re-run: 5/5). - `pnpm --filter @objectstack/core test:repo`: exit 0, 6 files, 82 tests. - `node scripts/pm/dispatch-gates.mjs --commands` (no paths, 4 changed paths vs merge base `ce2e08ee3`): 64 commands, **all exit 0**. `--ran` reconciliation: 64 derived, 64 run, 0 NOT-MEASURED (a derived zero: every line carries its exit code). `check:dual-build-cjs-loads` first exited 3 (PREREQUISITE NOT MET, 39 packages without `dist/`); it was re-run after the full build and exited 0. ## Acceptance notes - **What "the field is empty" stores.** The loader leaves a blank single reference as written, so the engine decides it. The engine's insert door (`normalizeBlankTypedValues`) deliberately does not null a blank string-stored value: its doc lists "lookup ids" among the columns where `''` is a value, and calls what "empty" means there "a separate question, not answered here". So an optional lookup seeded `''` stores `''` (and whitespace stores `' '`), exactly as a direct `engine.insert` of the same payload does. Every reader that asks `isMissing` sees it as empty, and the required check refuses it. Before this change the deferral deleted the key, so the row stored NULL. Carrier: none. This records the engine's own declared open question; it is not filed here. --- _Generated by [Claude Code](https://claude.ai/code/session_01KxmveGHL9EpPEGVggMSzDB)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 6b42b6b commit 3ae0b77

4 files changed

Lines changed: 378 additions & 7 deletions

File tree

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
"@objectstack/metadata-protocol": patch
3+
---
4+
5+
fix(metadata-protocol): a seed reference written blank (`''` or whitespace-only) is no reference, by the engine's own `isMissing`
6+
7+
Clause-②: no (no published accept set or surface moves: the row was already written, and only the false "unresolved" report and its `success: false` go away; owes the seat's `TIER_DEFAULT` review of record before the queue)
8+
9+
- **What changed.** The seed loader skipped a reference only when its value was `undefined` or `null`, so a lookup written `''` was resolved as a natural key, missed, deferred, and reported after pass 2 as `Deferred reference unresolved … = '' → … not found`: `errored` counted it and the load answered `success: false` for a row that had landed. A replay reported the same error again. Every site that decides whether a reference has a value now asks `isMissing` from `@objectstack/core`, the rule the engine's insert door and required check already use: a single reference, each element of a `multiple: true` reference in pass 1 and in pass 2, and the id half of a `referenceVia` pointer pair (which already skipped `''` and now also skips whitespace-only).
10+
- **A blank single reference** is not resolved and not reported. Its value is left as written, so the engine decides it: an optional lookup's row lands with the field empty and a replay skips it clean; a required lookup is refused by the engine's own required check (`book is required`), and no longer also reported as an unresolved reference.
11+
- **A blank element of a `multiple: true` reference** is dropped like a `null` hole, and the other elements resolve. Before, the blank element made the whole field defer and fail, so the row landed with none of its associations.
12+
- **What stays the same.** A real natural key resolves as before, and one that names no row is still reported. No export, error code, configuration key or result shape changed. Nothing to do for a consumer.

‎packages/metadata-protocol/src/seed-loader-multi-value-reference.test.ts‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -378,3 +378,167 @@ describe('seed reference resolution — multi-value lookup (multiple: true)', ()
378378
expect(bookNode.references.find((r) => r.field === 'reviewer')!.multiple).toBeUndefined();
379379
});
380380
});
381+
382+
/**
383+
* [#24459] A blank seed reference is NO reference.
384+
*
385+
* "Blank" is the engine's own rule, `isMissing` (`@objectstack/core`,
386+
* `required-emptiness.ts`): `undefined`, `null`, and a string that is empty or
387+
* only whitespace — the rule the insert door and the required check use. The
388+
* loader used to skip a reference only on `undefined` / `null`, so `''`
389+
* resolved as a natural key, missed, deferred, and was reported after pass 2 as
390+
* "Deferred reference unresolved" for a row that had landed.
391+
*
392+
* Every site that decides "this reference has a value" asks that one rule:
393+
* the single reference, each element of a multi-value one (pass 1 and pass 2)
394+
* and the pointer pair's id half. A blank single value is left AS WRITTEN for
395+
* the engine (its required check refuses it on a required field); a blank
396+
* element is dropped like a `null` hole, so the field's other elements land.
397+
*
398+
* Same engine double as the suite above, on its own schema map. What the
399+
* real engine stores and refuses is pinned on a real `SqlDriver` in
400+
* `packages/runtime/src/seed-blank-reference-real-driver.integration.test.ts`;
401+
* this double does not validate, so nothing here claims it.
402+
*/
403+
const BLANK_SCHEMAS: Record<string, any> = {
404+
author: {
405+
name: 'author',
406+
fields: { name: { type: 'text', required: true } },
407+
},
408+
book: {
409+
name: 'book',
410+
fields: {
411+
name: { type: 'text', required: true },
412+
authors: { type: 'lookup', reference: 'author', multiple: true },
413+
reviewer: { type: 'lookup', reference: 'author' },
414+
// A self-reference to a row later in the same dataset: not inserted yet
415+
// in pass 1, so the field defers its WHOLE authored array to pass 2.
416+
related: { type: 'lookup', reference: 'book', multiple: true },
417+
},
418+
},
419+
activity: {
420+
name: 'activity',
421+
fields: {
422+
summary: { type: 'text', required: true },
423+
object_name: { type: 'text' },
424+
record_id: { type: 'text', referenceVia: 'object_name' },
425+
},
426+
},
427+
};
428+
429+
const ENV = ['prod', 'dev', 'test'];
430+
431+
const blankAuthorSeed = (records: any[]) => ({ object: 'author', externalId: 'name', mode: 'upsert', env: ENV, records });
432+
const blankBookSeed = (records: any[]) => ({ object: 'book', externalId: 'name', mode: 'upsert', env: ENV, records });
433+
434+
const DEFERRED_UNRESOLVED = /Deferred reference unresolved/;
435+
436+
async function loadBlank(seeds: any[]) {
437+
const { engine, store } = createEngine(BLANK_SCHEMAS);
438+
const logger = createLogger();
439+
const result = await new SeedLoaderService(engine, createEmptyMetadata(), logger).load({ seeds, config: CONFIG });
440+
return { result, store, logger, engine };
441+
}
442+
443+
describe('a blank seed reference is no reference (isMissing)', () => {
444+
it.each([
445+
['empty', ''],
446+
['whitespace-only', ' '],
447+
])('skips a single reference written %s: not resolved, not deferred, left as written', async (_label, blank) => {
448+
const { result, store, logger } = await loadBlank([
449+
blankAuthorSeed([{ name: 'Alice' }]),
450+
blankBookSeed([{ name: 'Refactoring', reviewer: blank }]),
451+
]);
452+
453+
expect(result.errors.map((e) => e.message)).toEqual([]);
454+
expect(result.success).toBe(true);
455+
expect(result.summary.totalErrored).toBe(0);
456+
expect(result.summary.totalReferencesDeferred).toBe(0);
457+
expect(result.summary.totalReferencesDropped).toBe(0);
458+
expect(logger.error).not.toHaveBeenCalled();
459+
// Left for the engine's insert door and required check, exactly as authored.
460+
expect(store.book[0].reviewer).toBe(blank);
461+
});
462+
463+
it('drops a blank ELEMENT of a multi-value reference like a null hole; the rest resolve in order', async () => {
464+
const { result, store, engine } = await loadBlank([
465+
blankAuthorSeed([{ name: 'Alice' }, { name: 'Bob' }]),
466+
blankBookSeed([{ name: 'Refactoring', authors: ['Alice', '', null, ' ', 'Bob'] }]),
467+
]);
468+
469+
expect(result.errors.map((e) => e.message)).toEqual([]);
470+
expect(result.success).toBe(true);
471+
const alice = store.author.find((r) => r.name === 'Alice')!;
472+
const bob = store.author.find((r) => r.name === 'Bob')!;
473+
expect(store.book[0].authors).toEqual([alice.id, bob.id]);
474+
// Resolved in PASS 1: nothing deferred the array to a back-fill
475+
// (where pass 2 would re-resolve it, blanks included).
476+
expect((engine.update as any).mock.calls.filter(([o, d]: [string, any]) => o === 'book' && 'authors' in d)).toEqual([]);
477+
});
478+
479+
it('drops a blank element in PASS 2 too, where the deferred value is the whole authored array', async () => {
480+
const { result, store, engine } = await loadBlank([
481+
blankBookSeed([
482+
{ name: 'Refactoring', related: ['Dune', '', ' ', 'Emma'] },
483+
{ name: 'Dune' },
484+
{ name: 'Emma' },
485+
]),
486+
]);
487+
488+
expect(result.errors.map((e) => e.message)).toEqual([]);
489+
expect(result.success).toBe(true);
490+
expect(result.summary.totalErrored).toBe(0);
491+
// The array really went through pass 2 — the path this case exists for:
492+
// the insert carried no `related`, and the back-fill wrote it. (A pass-2
493+
// success un-books the deferral, so the summary counter reads 0 here.)
494+
const id = (name: string) => store.book.find((r) => r.name === name)!.id;
495+
expect(store.book.find((r) => r.name === 'Refactoring')!.related).toEqual([id('Dune'), id('Emma')]);
496+
const backFills = (engine.update as any).mock.calls.filter(([o, d]: [string, any]) => o === 'book' && 'related' in d);
497+
expect(backFills).toHaveLength(1);
498+
expect(backFills[0][1].related).toEqual([id('Dune'), id('Emma')]);
499+
});
500+
501+
it('skips a whitespace-only pointer-pair id the way it skips an empty one', async () => {
502+
const { result, store } = await loadBlank([
503+
blankAuthorSeed([{ name: 'Alice' }]),
504+
{
505+
object: 'activity', externalId: 'summary', mode: 'upsert', env: ENV,
506+
records: [
507+
{ summary: 'empty pointer', object_name: 'author', record_id: '' },
508+
{ summary: 'whitespace pointer', object_name: 'author', record_id: ' ' },
509+
// The pair's type half empty: an AUTHORED id here is refused as
510+
// un-addressable; a blank one is no pointer, so there is nothing to
511+
// address — the same answer `''` always got.
512+
{ summary: 'empty pointer, no type', record_id: '' },
513+
{ summary: 'whitespace pointer, no type', record_id: ' ' },
514+
// Control: the pair is live on this object — a real key resolves.
515+
{ summary: 'real pointer', object_name: 'author', record_id: 'Alice' },
516+
],
517+
},
518+
]);
519+
520+
expect(result.errors.map((e) => e.message)).toEqual([]);
521+
expect(result.success).toBe(true);
522+
expect(result.summary.totalErrored).toBe(0);
523+
expect(result.summary.totalReferencesDeferred).toBe(0);
524+
const alice = store.author.find((r) => r.name === 'Alice')!;
525+
expect(store.activity.find((r) => r.summary === 'real pointer')!.record_id).toBe(alice.id);
526+
expect(store.activity).toHaveLength(5);
527+
});
528+
529+
it('control: a real natural key still resolves, and an unresolvable one still errors', async () => {
530+
const { result, store } = await loadBlank([
531+
blankAuthorSeed([{ name: 'Alice' }]),
532+
blankBookSeed([
533+
{ name: 'Refactoring', reviewer: 'Alice' },
534+
{ name: 'Dune', reviewer: 'Nobody' },
535+
]),
536+
]);
537+
538+
const alice = store.author.find((r) => r.name === 'Alice')!;
539+
expect(store.book.find((r) => r.name === 'Refactoring')!.reviewer).toBe(alice.id);
540+
expect(result.success).toBe(false);
541+
expect(result.errors.filter((e) => DEFERRED_UNRESOLVED.test(e.message))).toHaveLength(1);
542+
expect(result.errors[0].message).toContain("'Nobody'");
543+
});
544+
});

‎packages/metadata-protocol/src/seed-loader.ts‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ import { SEED_WRITE_EXECUTION_CONTEXT } from '@objectstack/spec/kernel';
2525
import { resolveSeedRecord } from '@objectstack/formula';
2626
import {
2727
bulkWrite, withTransientRetry, defaultIsTransientError, type BulkWriteRowResult, runWithAdvisoryAggregation, type AdvisoryGroup,
28-
temporalComparandKind, temporalStorageForm,
28+
temporalComparandKind, temporalStorageForm, isMissing,
2929
} from '@objectstack/core';
3030
// [#8442] The repo's ONE recogniser for "this throw is a record-validation
3131
// failure" — duck-typed on `code`/`name`, the same predicate `mapDataError` and
@@ -1403,7 +1403,9 @@ export class SeedLoaderService implements ISeedLoaderService {
14031403
for (const pair of pointerPairs) {
14041404
const pointerValue = record[pair.field];
14051405
// Empty pointer = no pointer: nothing to resolve, nothing to refuse.
1406-
if (pointerValue === undefined || pointerValue === null || pointerValue === '') continue;
1406+
// "Empty" is the engine's own blank rule (`isMissing`), the one every
1407+
// reference below is skipped by.
1408+
if (isMissing(pointerValue)) continue;
14071409
const targetName = record[pair.objectField];
14081410
if (typeof targetName !== 'string' || targetName === '') {
14091411
// UN-ADDRESSABLE: the id half is authored but the type half names no
@@ -1461,7 +1463,13 @@ export class SeedLoaderService implements ISeedLoaderService {
14611463
// so naming an org by its natural key keeps working.
14621464
if (stampedTenantOrg && ref.field === 'organization_id') continue;
14631465
const fieldValue = record[ref.field];
1464-
if (fieldValue === undefined || fieldValue === null) continue;
1466+
// A blank reference is NO reference: the engine's blank rule
1467+
// (`isMissing` — `undefined`, `null`, `''`, whitespace-only) is the one
1468+
// the insert door stores as empty and the required check refuses by,
1469+
// so the value is left as written for those two. Resolving `''` as a
1470+
// natural key could only miss, and reported "Deferred reference
1471+
// unresolved" for a row that landed with the field empty.
1472+
if (isMissing(fieldValue)) continue;
14651473

14661474
const pushError = (message: string, attemptedValue: unknown): ReferenceResolutionError => {
14671475
const error: ReferenceResolutionError = {
@@ -1533,9 +1541,10 @@ export class SeedLoaderService implements ISeedLoaderService {
15331541
let sawUnresolved = false;
15341542

15351543
for (const item of items) {
1536-
// A null/undefined hole carries no reference — drop it rather than
1537-
// writing it into the stored array.
1538-
if (item === undefined || item === null) continue;
1544+
// A blank hole (`isMissing`: null, undefined, `''`, whitespace-only)
1545+
// carries no reference — drop it rather than writing it into the
1546+
// stored array, so the field's other elements still land.
1547+
if (isMissing(item)) continue;
15391548

15401549
const outcome = await this.resolveReferenceItem(ref, item, insertedRecords, config);
15411550
if (outcome.status === 'invalid') {
@@ -2211,7 +2220,9 @@ export class SeedLoaderService implements ISeedLoaderService {
22112220
let stillUnresolved = false;
22122221

22132222
for (const item of items) {
2214-
if (item === undefined || item === null) continue;
2223+
// The same blank rule pass 1 drops a hole by: the deferred value is the
2224+
// WHOLE authored array, holes included.
2225+
if (isMissing(item)) continue;
22152226
const outcome = await this.resolveReferenceItem(
22162227
deferred, item, insertedRecords, { organizationId },
22172228
);

0 commit comments

Comments
 (0)