Skip to content

Commit bd68248

Browse files
fix(objectql): a cleared optional single-value select or lookup stores null, so a second unique clear is no longer refused 409 (#29277) (#29464)
domain:server#3 · os-steve · session_01ADHkmQJdgarUjPLNgFBY67 <!-- claude-projects-attribution: {"github_login":"os-steve"} --> _Requested by **Steve** · [project thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9CGzC168tQ8WTbgReWZVSo4)_ Fixes #29277 Clause-②: yes (widening) Metadata-protocol: none ## 维护者速读 - **改了什么**:可选的单值选项、引用、文件字段(`select` / `radio` / `lookup` / `master_detail` / `user` / `tree` / `image` / `file` / `avatar` / `video` / `audio`)被清空(送来 `''`)时,服务端写入口改存 `null`,唯一与否都一样。哪些类型算在内由 `$empty` 规则表的 `null_only` 行加「存一个字符串」推出,不靠手写名单。唯一字段上第二条清空的记录不再被 `409 UNIQUE_VIOLATION` 拒绝,清空的记录也能被「为空」筛选到。 - **种子重放**:服务端把空白存成 `null` 的规则(三条:#20308 类型列、#29176 唯一文本、本卡单值列)移到 `@objectstack/core` 一处(`blankAsNullColumns` / `blankStorageForm`),写入口和种子加载器的重放比较都读它。种子里写 `''` 的行重放时不再每次被当作变更重写(主干上三条都会这样空转,已实测)。core 新增导出,所以声明改为 `Clause-②: yes (widening)`。 - **为什么改**:维护者裁决 #29346 选 B(记录 6579813296)。`''` 既不是选项值,也不是记录 id,本来就不该算一个值;`$empty` 规则表在这些类型上只认 `null` 为空,规则表不动。 - **风险与回滚**:这些字段清空后读回的是 `null` 而不是 `''`,按 `= ''` 查询的条件对新清空的记录不再命中。必填字段照旧拒收 `''`,`multiple: true` 字段不动,已存的 `''` 不迁移(迁移另立卡)。回滚 = revert 本 PR。 - **先看**:`packages/core/src/utils/blank-storage-form.ts`(规则本体);`packages/metadata-protocol/src/seed-loader.ts` 的 `ReplayComparands.blankAsNull` 与 `isNoOpReplay`;`packages/objectql/src/validation/record-validator.ts` 的 `normalizeBlankTypedValues`;`packages/rest/src/rest-data-single-value-blank.test.ts`;`.changeset/29277-single-value-blank-null.md`。 - **两分钟验证**:`pnpm --filter @objectstack/rest exec vitest run src/rest-data-single-value-blank.test.ts`,预期 11 个测试全绿(主干上第二次清空是 409,`$empty` 查不到清空的记录);`pnpm --filter @objectstack/runtime exec vitest run src/seed-blank-as-null-replay.integration.test.ts`,预期 2 个全绿(主干上重放 `totalUpdated` 为 4)。 - **需要你做的**:无,席位落地。 ## What changes - **The ruling.** Maintainer ruling B on #29346 (record 6579813296): the write door stores `''` as `null` on every optional column whose type sits on the `$empty` table's `null_only` row and is string-stored single-value. The `$empty` table is unchanged; `required` keeps refusing `''`; a blank on insert takes a declared `defaultValue`; cleaning stored `''` is a separate migrate-lane card. - **The seam.** `normalizeBlankTypedValues` (`packages/objectql/src/validation/record-validator.ts`), the engine's existing blank write door: `ObjectQL.insert()` / `insertMany()` / `update()` (by id and by predicate) and the dry run `validate()` apply it before anything else reads the payload, and every REST, batch, upsert and import door reaches the engine through those methods. `engine.ts` is not touched. - **The new arm.** The door's blank-as-null rule is now ONE function in `@objectstack/core` (`packages/core/src/utils/blank-storage-form.ts`): `blankAsNullColumns(objectSchema)` maps each column to its reach, `'blank'` (any `isMissing` blank: the #20308 typed arm, `NON_TEXT_STORED_VALUE_TYPES`, unchanged) or `'empty'` (exactly `''`), and `blankStorageForm(value, reach)` gives the stored form. An `'empty'` column is not `required: true`, not a lifecycle column the engine owns (`ENGINE_LIFECYCLE_COLUMNS`, which the validator's `SKIP_FIELDS` now reads), and EITHER text-like and unique-covered (#29176, unchanged) OR on the `$empty` table's `null_only` row (`expandEmptyOperator(def).arm`, read first, so a `multiple: true` column, the `multi_value` row of #29343, is out) AND storing one string (`ONE_STRING_STORED_TYPES`). `normalizeBlankTypedValues` consumes it and keeps no arm logic of its own. - **The seed replay (round 3).** `isNoOpReplay` (`packages/metadata-protocol/src/seed-loader.ts`) compared a seed `''` literally with the stored `null`, so every seed row writing `''` on a blank-as-null column read as changed and was rewritten on each replay. A new replay comparand `ReplayComparands.blankAsNull` (built once per dataset from the definition by `blankAsNullColumns`, beside `temporal` and `masked`) puts the SEED side in its stored form before `seedValueEquals`. There is no blanket `'' == null`: on a column the door keeps `''` (plain text), a seed `''` against a stored `null` is still a change (pinned). **Pre-existing on main:** measured on head `071872b7` (where the #20308 and #29176 arm code and the seed loader are byte-identical to `origin/main`), a replay of one `''` per arm rewrote 4 of 4 rows: the number, date and unique-text rows churned before this PR, and the select row is this PR's arm. All three arms are now pinned. - **The storage-shape set.** The spec has no predicate for "stores one string", so `ONE_STRING_STORED_TYPES` is stated once, in core, from the spec's classes: `SINGLE_OPTION_TYPES` (an option code), `REFERENCE_VALUE_TYPES` (a record id), `FILE_REFERENCE_TYPES` (an opaque `sys_file` id). A pin ties it to the table: every `FieldType` whose expansion is `null_only` and that the typed arm does not own must sit in exactly one of that set, `STRUCTURED_JSON_TYPES` or `COMPUTED_VALUE_TYPES`, so a type that joins the row unclassified turns the pin red. - **Left out, measured on SQLite through the engine** (non-blank value, then `''` twice, on a unique column): `json`, `composite`, `record`, `location`, `address`, `repeater`, `vector` store an object or an array (`{"a":1}`, `[0.1,0.2]`), so `''` is not a blank of their shape; `formula` is never stored; `autonumber` is issued by the engine on insert whatever the caller sent (`A-1`, `A-2`, `A-3` for a value, `''`, `''`). The file references stored one string id (JSON-encoded on SQL: `"img_1"`) and a second `''` was refused `DUPLICATE_RECORD`, so they join. - **Edges.** Exactly `''`: whitespace stays a value (on a reference column the referential-integrity check refuses it, before and after). Lifecycle columns (`created_by`, `updated_by`) are never rewritten. `owner_id` (an injected optional lookup) is in the set; the security middleware already reads `owner_id` `''` and `null` as one (`security-plugin-authz-middleware.ts`, the `row.owner_id == null || row.owner_id === ''` checks), so its answer does not move. - **New export (Clause-② widening).** `@objectstack/core` exports `blankAsNullColumns`, `blankStorageForm`, `ENGINE_LIFECYCLE_COLUMNS` and the types `BlankAsNullReach` / `BlankAsNullFieldDef`. The changeset adds `@objectstack/core` and `@objectstack/metadata-protocol` patch entries. No spec edit. - **The fence.** Neither the `BOUNDED_STRING_FIELD_TYPES` string arm (#29241) nor the select value-shape check (#29244) is touched. - **Pending changeset corrected.** `.changeset/29176-optional-unique-blank-null.md` (unreleased) said `select` / `lookup` columns keep storing `''`. This PR makes that false in the same release, so the sentence now says text columns keep `''` and points at this change. That is a deliberate correction of another PR's pending changeset, so `check-empty-changeset` reports it (see Verification). ## Why `''` is neither an option code nor a record id. Stored as sent, it was a value to SQL `UNIQUE` (the second cleared record on a unique column was refused 409) and not empty to `$empty`, whose ruled table reads only `null` as empty on these types. The formula matcher, `having` and CEL `isBlank` already read `''` as empty, so the SQL face now agrees with them for every newly cleared row. ## Tests New files: - `packages/objectql/src/validation/record-validator.single-value-blank.test.ts` (8 tests): the population derived from `FieldType` and the table, with every untyped `null_only` type classified into exactly one storage class; the joined set (11 types) and the left-out set (9 types) pinned; the left-out types keep `''`; `''` becomes `null` on every optional member, unique or not; `required` (with and without a `defaultValue`) keeps `''`; `multiple: true` is the `multi_value` row and keeps `''`; lifecycle columns untouched; whitespace and non-blank values untouched; purity. - `packages/core/src/utils/blank-storage-form.test.ts` (7 tests): each arm's reach (typed incl. required and lifecycle `created_at`; unique text by field and by index, plain text and `password` out; one-string `select` / `lookup` / `image` in, required / `multiple: true` / `json` / `autonumber` / `created_by` out); the lifecycle list; an unresolvable definition declares none; `blankStorageForm` per reach. - `packages/runtime/src/seed-blank-as-null-replay.integration.test.ts` (2 tests), real `ObjectQL` + SQLite `SqlDriver` + `SeedLoaderService`: one seed `''` per arm (number, date, unique text, select) lands `null` and a replay skips all four (`totalUpdated: 0`, `totalSkipped: 4`); control: a plain text column stored `null` and reseeded `''` is still updated and stores `''`. - `packages/rest/src/rest-data-single-value-blank.test.ts` (11 tests), real `ObjectQL` + SQLite `SqlDriver` + `RestServer` routes, physical column read past the engine: POST `''` on all eleven joined types stores NULL; PATCH clear on all eleven; unique `select` + `lookup` created `''` twice and cleared by PATCH twice both land; a unique `image` created `''` twice both land; a declared unique-index member; batch create and `createMany`; a real duplicate still `409 UNIQUE_VIOLATION`; required `select` / `lookup` still `400 VALIDATION_FAILED` / `required` (one with a `defaultValue`); a blank on insert takes the declared `defaultValue` exactly as `null` does; the `$empty` pin: on a row written `''` past the engine, `$empty: true` answers only the newly cleared row, `$empty: false` answers the legacy `''` row and the valued row, `= ''` answers only the legacy row. Forced edits to existing tests (each pinned that a `select` / `lookup` `''` stays `''`, which the ruling reverses): - `packages/objectql/src/validation/record-validator.blank-typed-value.test.ts`: the string-stored control list drops the option and reference types (they keep their "not typed" assertion). - `packages/objectql/src/validation/record-validator.optional-unique-blank.test.ts`: the "other string-stored" control is now `autonumber` only. - `packages/objectql/src/engine-optional-unique-blank-door.test.ts`: the unique `pick` (select) / `ref` (lookup) columns move from the kept-`''` set to the covered-`null` set. - `packages/objectql/src/engine-blank-typed-value-door.test.ts` and `packages/rest/src/rest-data-blank-typed-value.test.ts`: the `c_lookup` / `c_select` controls now expect `null`; `c_text` still `''`. Round 3 forced edits (no expectation changed): - `packages/runtime/src/seed-blank-reference-real-driver.integration.test.ts`: the comment at the `loan-empty` assertion said the insert door "does not null" a blank lookup; it now says an optional lookup's `''` is stored as `null` since #29277 and the replay compares it in that form. Its expectations (`totalUpdated: 0`, `totalSkipped: 6`) are unchanged and now pass. - `packages/objectql/src/federated-injected-column-readers.test.ts`: four VOCABULARY rows (`validation/record-validator.ts#<module> :: created_at / created_by / updated_at / updated_by`) deleted, as the census asks when a seam use leaves the source: the lifecycle names moved to core's `ENGINE_LIFECYCLE_COLUMNS`. <details><summary>Red on main, green on head</summary> The two new files were committed first (`0acdf485`) and run against `origin/main` `96fb7b5d`'s objectql build: rest `Tests 9 failed | 2 passed (11)` (POST stored `''`; unique `[201, 409]` on create, `[200, 409]` on PATCH and the index member; batch `[true, false]`; defaults not applied; `$empty: true` answered `[]`), unit `Tests 2 failed | 5 passed (7)`. Two of those rest reds were my own fixture faults (a `tree` value naming no record of its object, a whitespace lookup id the referential-integrity check refuses), fixed in `a71af048`; the whitespace case moved to the unit file. On head: rest 11/11, unit 8/8. </details> <details><summary>Ablations</summary> Made with `scripts/ablation-replace.mjs` (anchor hit once), objectql rebuilt, marker proven in dist by `scripts/ablation-dist-preflight.mjs` ("marker present in 4 built files"), each restored with `git checkout HEAD`, blob hash equal to HEAD, then a final rebuild with the marker absent from all 8 built files and the rest file green (10/10 round 1, 11/11 round 2). | Ablation | Red | |---|---| | L1: the new arm off (`isSingleValueCodeColumn` never true) | rest `Tests 8 failed \| 2 passed (10)` (every positive case, including the `$empty` pin), unit 2 failed | | L2: the table-row check removed (a `multiple: true` column joins) | unit `a multiple: true column is the multi_value row, not this arm` | | L3 (round 2): `FILE_REFERENCE_TYPES` dropped from the one-string set | rest `Tests 3 failed \| 8 passed (11)` (POST, PATCH, unique image), unit `"" on every optional member becomes null` | | L4 (round 2): `STRUCTURED_JSON_TYPES` added to the one-string set | unit `the left-out types keep ""` | | L5 (round 3): the replay comparand off (`isNoOpReplay` compares the raw seed value), metadata-protocol rebuilt | runtime `Tests 2 failed \| 2 passed (4)`: `seed-blank-as-null-replay` (replay `totalUpdated` 4) and `seed-blank-reference-real-driver` `:161` (1), the CI red; the plain-text control and the required case stay green. Restored from a copy, blob hash equal, rebuilt, both green. | The required / lifecycle / whitespace exclusions are the #29176 lines, ablated on PR #29271; the required and default pins here pass on base and head, as they must. </details> ## Acceptance notes - **Clause-②: yes (widening).** `@objectstack/core` gains exports (round 3, the seat's ruling 6582252856). Measured: no door refuses anything it accepted. `''` on every ruled type was accepted (201 / 200) on base and is on head; required refusals are identical on both. A second unique clear moves from 409 to 201, the same direction #29176 declared `no`. - **Read shape.** A field cleared from now on reads `null`, not `''`, and matches `$empty: true`, `$null: true` and `= null`; it no longer matches `= ''` (the changeset says so). Stored `''` rows are not migrated and keep their old answers; the migrate-lane card cleans them. - **File reference types join** (seat 3's round-2 reading of ruling B, comment 6581612131: the rule is stated by the table plus the storage shape). A `multiple: true` file or image column is the `multi_value` row and stays out (#29343). - **Not measured:** PostgreSQL, MySQL (same knex path), MongoDB (the `mongod` download is 403 in this container). ## Verification Head `22a77935` (`origin/main` `96fb7b5d` merged in at `e607df78`; a full-declaration rebuild of the runtime and rest closures). CI runs the rest: see the PR's check-runs. <details><summary>Suites, typecheck, lint, gates</summary> | Run | Result | |---|---| | core full suite + typecheck | Test Files 138 passed (138) · Tests 3021 passed · typecheck exit 0 | | metadata-protocol full suite + typecheck | Test Files 305 passed, 4 skipped (309) · Tests 22161 passed, 26 skipped · typecheck exit 0 | | objectql full suite + typecheck | Test Files 533 passed, 1 failed (534) on the first run: the injected-column census (re-keyed above, then `5 passed (5)`) · Tests 10320 · typecheck exit 0 | | runtime full suite + typecheck | Test Files 527 passed (527) · Tests 9678 passed, 19 skipped · typecheck exit 0 | | rest full suite + typecheck | Test Files 376 passed (376) · Tests 7329 passed, 352 skipped · typecheck exit 0 | | `pnpm lint` | exit 0 | | `dispatch-gates --commands` (63 derived), reconciled with `--ran` | 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN; 62 exit 0 (`check:type-check-debt` first queue-timed out behind the suites, exit 3, then rerun under the lock: exit 0) | | `node scripts/check-empty-changeset.mjs --base origin/main` | exit 1, by design: the DELIBERATE CORRECTION class for `.changeset/29176-optional-unique-blank-null.md` (see What changes). Not restored, no `skip-changeset`; the seat confirmed. `Check Changeset` is not a required context. | | `check-changeset-no-major` against this body (offline `--event`) | exit 0: level axis `Clause-②: yes (widening)` / `Metadata-protocol: none`, patch | The plugin-teardown self-test needed `git fetch origin 621a487…` and `check:dual-build-cjs-loads` a full `./packages/**` build first; both then exit 0. Full dogfood not run: no `!` in the title and no `(narrowing)` arm. </details> --- _Generated by [Claude Code](https://claude.ai/code/session_01ADHkmRPC4f86r9bWGuYWJu)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 1b2043e commit bd68248

17 files changed

Lines changed: 903 additions & 106 deletions

‎.changeset/29176-optional-unique-blank-null.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,6 @@ Metadata-protocol: none
1010
A cleared text input is sent as `''`, and the write path stored it as sent. Every unique constraint the drivers build is NULL-distinct, so `null` never collides, but `''` is a value: the first record whose SKU (tax id, employee number, …) was cleared saved, and the second was refused "A record with this value already exists" although neither held a value. Measured on SQLite and the memory driver: two `''` rows collide, two `null` rows do not.
1111

1212
- **Now.** The engine's write door, which `insert`, `insertMany`, `update` (by id and by predicate) and the dry run (`validate`) apply before anything else reads the payload, stores exactly `''` as `null` on a column that is text-like (`text`, `textarea`, `email`, `url`, `phone`, `markdown`, `html`, `richtext`, `code`, `color`, `signature`, `qrcode`), not `required`, and covered by a unique constraint: the field's own `unique`, or membership in a declared `indexes[]` entry carrying `unique`. Every REST, batch, upsert and import door reaches the engine through those methods.
13-
- **Unchanged.** A `required` field given `''` is refused `required` exactly as before, also when it declares a `defaultValue`. A real duplicate is still `409 UNIQUE_VIOLATION`. Whitespace is a value and is stored as sent. `password` and `secret` keep their `''` refusal. A column no unique constraint covers, and `select` / `lookup` columns, keep storing `''`.
13+
- **Unchanged.** A `required` field given `''` is refused `required` exactly as before, also when it declares a `defaultValue`. A real duplicate is still `409 UNIQUE_VIOLATION`. Whitespace is a value and is stored as sent. `password` and `secret` keep their `''` refusal. A text column no unique constraint covers keeps storing `''`. (Optional single-value `select` / `lookup` columns store a cleared `''` as `null` too, unique or not: a separate change in this release.)
1414
- **Defaults.** An optional unique field that declares a `defaultValue` now takes it on insert when sent `''`, exactly as when sent `null`.
1515
- **Rows already stored.** Existing `''` values are not migrated. They keep matching a `field = ''` filter; a newly cleared row matches `$null: true` instead. The `is_empty` filter (`$empty`) matches both, as before. A newly cleared row no longer collides with a stored `''`.
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
'@objectstack/objectql': patch
3+
'@objectstack/core': patch
4+
'@objectstack/metadata-protocol': patch
5+
---
6+
7+
fix(objectql): a cleared optional single-value `select` or `lookup` stores `null`, so a second record cleared on a unique one is no longer refused `409 UNIQUE_VIOLATION`
8+
9+
Clause-②: yes (widening)
10+
Metadata-protocol: none
11+
12+
A cleared `select`, `lookup` or single file field sent as `''` was stored as sent. `''` is neither an option code, a record id nor a file id, so it held no value, yet it was one to the store and to the `is_empty` filter: on a unique column the second record cleared was refused "A record with this value already exists", and on every column the cleared record did not match `is_empty` (`$empty`), whose ruled table reads only `null` as empty on these types.
13+
14+
- **Now.** The engine's write door (`insert`, `insertMany`, `update` by id and by predicate, and the `validate` dry run; every REST, batch, upsert and import door reaches it) stores exactly `''` as `null` on every optional column the `$empty` table puts on its `null_only` row whose value is one string: the option types (`select`, `radio`), the reference types (`lookup`, `master_detail`, `user`, `tree`) and the file references (`image`, `file`, `avatar`, `video`, `audio`). Unique or not. Object- and array-valued types (`json`, `composite`, `location`, …) and the engine-owned `formula` / `autonumber` keep what they store.
15+
- **Read shape.** Such a field cleared from now on reads back `null`, not `''`. It matches `is_empty` / `$empty: true`, `$null: true` and `field = null`, and no longer matches `field = ''`.
16+
- **Unchanged.** The `$empty` table itself. A `required` field given `''` is refused `required` as before, also when it declares a `defaultValue`. Whitespace is a value. A `multiple: true` option or reference column keeps what it stores. A real duplicate is still `409 UNIQUE_VIOLATION`.
17+
- **Defaults.** An optional field that declares a `defaultValue` now takes it on insert when sent `''`, exactly as when sent `null`.
18+
- **Rows already stored.** Existing `''` values are not migrated by this change; they keep answering as a value (`$empty: false`, `field = ''`) until a data step cleans them.
19+
- **Seed replay.** A seed that writes `''` on such a column, or on any column whose blank the door already stored as `null` (an optional unique text column, a number / boolean / date column), is compared in that stored form, so replaying it is no longer a change on every boot. Before, the replay compared the seed's `''` with the stored `null` and rewrote the row each time (bumping `updated_at`).
20+
- **New export.** `@objectstack/core` exports the rule itself, `blankAsNullColumns` and `blankStorageForm` (with `ENGINE_LIFECYCLE_COLUMNS`), which the write door and the seed loader both read.

‎packages/core/src/index.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,14 @@ export * from './utils/multi-value-storage-form.js';
169169
// engine at runtime.
170170
export * from './utils/required-emptiness.js';
171171

172+
// [#29277] …and the blank-as-null rule: which columns of an object store a
173+
// blank string as `null`, and which blanks (the #20308, #29176 and #29277
174+
// arms). `@objectstack/objectql`'s write door applies it, and
175+
// `@objectstack/metadata-protocol`'s seed replay compares a seed value in the
176+
// form it stores — one rule, here because those two do not depend on each
177+
// other.
178+
export * from './utils/blank-storage-form.js';
179+
172180
// [#21007] …and the refusal a scalar comparison gets on a field stored as a
173181
// JSON column: the operator set and the words. `driver-sql` refuses it on
174182
// `where`, and `@objectstack/objectql` on the per-aggregation `filter` it
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
import { describe, it, expect } from 'vitest';
4+
import { blankAsNullColumns, blankStorageForm, ENGINE_LIFECYCLE_COLUMNS } from './blank-storage-form.js';
5+
6+
const schema = {
7+
fields: {
8+
qty: { type: 'number', required: true },
9+
due: { type: 'date' },
10+
sku: { type: 'text', unique: true },
11+
code: { type: 'text' },
12+
idx_a: { type: 'text' },
13+
pw: { type: 'password', unique: true },
14+
stage: { type: 'select', options: [{ value: 'a' }] },
15+
stage_req: { type: 'select', required: true, options: [{ value: 'a' }] },
16+
tags: { type: 'select', multiple: true, options: [{ value: 'a' }] },
17+
ref: { type: 'lookup' },
18+
pic: { type: 'image' },
19+
blob: { type: 'json' },
20+
num: { type: 'autonumber' },
21+
created_by: { type: 'user' },
22+
created_at: { type: 'datetime' },
23+
},
24+
indexes: [{ fields: ['idx_a'], unique: true }, { fields: ['code'] }],
25+
};
26+
27+
describe('blankAsNullColumns: the three arms of the write door\'s blank-as-null rule', () => {
28+
const cols = blankAsNullColumns(schema);
29+
30+
it('[#20308] a typed column takes any blank, required and lifecycle included', () => {
31+
expect(cols.get('qty')).toBe('blank');
32+
expect(cols.get('due')).toBe('blank');
33+
expect(cols.get('created_at')).toBe('blank');
34+
});
35+
36+
it('[#29176] exactly "" on an optional text column a unique constraint covers', () => {
37+
expect(cols.get('sku')).toBe('empty');
38+
expect(cols.get('idx_a')).toBe('empty');
39+
expect(cols.has('code')).toBe(false);
40+
expect(cols.has('pw')).toBe(false);
41+
});
42+
43+
it('[#29277] exactly "" on an optional one-string null_only column, unique or not', () => {
44+
expect(cols.get('stage')).toBe('empty');
45+
expect(cols.get('ref')).toBe('empty');
46+
expect(cols.get('pic')).toBe('empty');
47+
for (const out of ['stage_req', 'tags', 'blob', 'num', 'created_by']) expect(cols.has(out), out).toBe(false);
48+
});
49+
50+
it('lifecycle names are the five engine-owned columns', () => {
51+
expect([...ENGINE_LIFECYCLE_COLUMNS]).toEqual(['id', 'created_at', 'created_by', 'updated_at', 'updated_by']);
52+
});
53+
54+
it('an unresolvable definition declares none', () => {
55+
expect(blankAsNullColumns(undefined).size).toBe(0);
56+
expect(blankAsNullColumns({}).size).toBe(0);
57+
});
58+
});
59+
60+
describe('blankStorageForm', () => {
61+
it('"blank" nulls "" and whitespace; "empty" nulls only ""; no reach keeps everything', () => {
62+
expect(blankStorageForm('', 'blank')).toBeNull();
63+
expect(blankStorageForm(' ', 'blank')).toBeNull();
64+
expect(blankStorageForm('', 'empty')).toBeNull();
65+
expect(blankStorageForm(' ', 'empty')).toBe(' ');
66+
expect(blankStorageForm('', undefined)).toBe('');
67+
});
68+
69+
it('a non-blank value or a non-string comes back as it is', () => {
70+
const obj = { a: 1 };
71+
for (const v of ['a', 0, false, null, undefined, obj]) {
72+
expect(blankStorageForm(v, 'blank')).toBe(v);
73+
expect(blankStorageForm(v, 'empty')).toBe(v);
74+
}
75+
});
76+
});
Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* THE blank-as-null rule — which columns of an object store a blank string as
5+
* `null`, and which blanks — read by the write door that stores it and by the
6+
* seed loader's replay, which must compare a seed value in the form the door
7+
* stores it.
8+
*
9+
* ## Two faces, one rule
10+
*
11+
* - `@objectstack/objectql`'s write door (`normalizeBlankTypedValues`) applies
12+
* it to every insert and update payload before anything else reads it.
13+
* - `@objectstack/metadata-protocol`'s seed loader asks it in `isNoOpReplay`
14+
* (the `blankAsNull` replay comparand): a seed writing `''` on such a column
15+
* lands as `null`, and compared as written it read the row as changed and
16+
* rewrote it on every boot (#29277; the #20308 and #29176 arms churned the
17+
* same way before it).
18+
*
19+
* The two packages do not depend on each other, and a copy each is how a
20+
* replay comes to disagree with the store. So the rule lives here, beside
21+
* `multiValueStorageForm` and `temporalStorageForm`, which are shared for the
22+
* same reason. The type classes and the `$empty` table it reads are the
23+
* spec's, never re-listed.
24+
*
25+
* ## The rule — three arms
26+
*
27+
* - [#20308] `'blank'`: a column whose type is non-string-typed
28+
* (`NON_TEXT_STORED_VALUE_TYPES`: numeric, boolean, `date`, `datetime`,
29+
* `time`) stores any blank (`isMissing`: `''` or whitespace) as `null` —
30+
* required or not, lifecycle or not: a blank is never a value of its type.
31+
* - [#29176] `'empty'`: exactly `''` on an optional text-like column
32+
* (`STRING_VALUE_TYPES` minus `password` / `secret`) a unique constraint
33+
* covers — its own `unique`, or a unique `indexes[]` entry, both read through
34+
* `isUniqueDeclared`, the drivers' own predicate.
35+
* - [#29277] `'empty'`: exactly `''` on an optional column on the `$empty`
36+
* table's `null_only` row (`expandEmptyOperator`) whose value is ONE string
37+
* ({@link ONE_STRING_STORED_TYPES}), unique or not (maintainer ruling B on
38+
* #29346, record 6579813296).
39+
*
40+
* The two `'empty'` arms never reach a `required: true` column (there `''` is
41+
* refused `required`, and must not first become a `null` a `defaultValue`
42+
* would fill) nor a lifecycle column the engine owns
43+
* ({@link ENGINE_LIFECYCLE_COLUMNS}), and never whitespace (a value on both
44+
* rows of the `$empty` table they touch).
45+
*/
46+
47+
import {
48+
NON_TEXT_STORED_VALUE_TYPES,
49+
STRING_VALUE_TYPES,
50+
SINGLE_OPTION_TYPES,
51+
REFERENCE_VALUE_TYPES,
52+
FILE_REFERENCE_TYPES,
53+
expandEmptyOperator,
54+
isUniqueDeclared,
55+
type ValueShapeFieldDef,
56+
} from '@objectstack/spec/data';
57+
import { isMissing } from './required-emptiness.js';
58+
59+
/**
60+
* Lifecycle columns the engine always owns and the client never supplies,
61+
* skipped by NAME because they are not author-declared business fields. The
62+
* write door's validator skips the same names (its `SKIP_FIELDS` is this list).
63+
*/
64+
export const ENGINE_LIFECYCLE_COLUMNS: readonly string[] = Object.freeze([
65+
'id', 'created_at', 'created_by', 'updated_at', 'updated_by',
66+
]);
67+
const LIFECYCLE: ReadonlySet<string> = new Set(ENGINE_LIFECYCLE_COLUMNS);
68+
69+
/**
70+
* [#29176] The text-like types of the unique arm: the spec's
71+
* `STRING_VALUE_TYPES` — exactly the "text-like" row of the `$empty` table,
72+
* where `''` and `null` already answer as one — minus the two credential
73+
* types. ⛔ `password` and `secret` stay out: a maintainer ruling refuses `''`
74+
* on them (`EmptyCredentialWriteError`) while `null` means "clear", so
75+
* rewriting the blank would turn that refusal into a silent clear.
76+
*/
77+
const UNIQUE_BLANK_TEXT_TYPES: ReadonlySet<string> = new Set(
78+
[...STRING_VALUE_TYPES].filter((t) => t !== 'password' && t !== 'secret'),
79+
);
80+
81+
/**
82+
* [#29277] The storage-shape half of the `null_only` arm: the spec's value
83+
* classes whose single value IS one string — an option code
84+
* (`SINGLE_OPTION_TYPES`), a record id (`REFERENCE_VALUE_TYPES`), an opaque
85+
* `sys_file` id (`FILE_REFERENCE_TYPES`; SQL stores it JSON-encoded, still one
86+
* string to every reader). Read as the spec's constants, never re-listed. The
87+
* spec has no predicate for "stores one string", so this set is stated here,
88+
* and `record-validator.single-value-blank.test.ts` (`@objectstack/objectql`)
89+
* ties it to the table: every type on the `null_only` row that is not typed
90+
* must be in this set or in one of the two classes left out below, so a type
91+
* that joins the row unclassified turns that pin red.
92+
*
93+
* Left out, each measured on SQLite through the engine:
94+
*
95+
* - `STRUCTURED_JSON_TYPES` (`json`, `composite`, `record`, `location`,
96+
* `address`, `repeater`, `vector`): the value is an object or an array, so
97+
* `''` is not a blank of its shape. What empty means for them is a
98+
* separate question.
99+
* - `COMPUTED_VALUE_TYPES` (`formula`, `autonumber`; `summary` is typed): the
100+
* engine owns the value — a formula is never stored, an autonumber is
101+
* issued on insert whatever the caller sent, `''` included.
102+
*/
103+
const ONE_STRING_STORED_TYPES: ReadonlySet<string> = new Set([
104+
...SINGLE_OPTION_TYPES,
105+
...REFERENCE_VALUE_TYPES,
106+
...FILE_REFERENCE_TYPES,
107+
]);
108+
109+
/**
110+
* Which blanks a column stores as `null`: `'blank'` — any blank string
111+
* (`isMissing`); `'empty'` — exactly `''`.
112+
*/
113+
export type BlankAsNullReach = 'blank' | 'empty';
114+
115+
/** The field shape the rule reads. */
116+
export interface BlankAsNullFieldDef extends ValueShapeFieldDef {
117+
required?: boolean;
118+
unique?: unknown;
119+
}
120+
121+
/**
122+
* The columns of one object whose blank the write door stores as `null`, each
123+
* with its reach. A column not in the map keeps every blank as written.
124+
*/
125+
export function blankAsNullColumns(
126+
objectSchema: { fields?: Record<string, BlankAsNullFieldDef | undefined>; indexes?: unknown } | undefined | null,
127+
): ReadonlyMap<string, BlankAsNullReach> {
128+
const out = new Map<string, BlankAsNullReach>();
129+
const fields = objectSchema?.fields;
130+
if (!fields || typeof fields !== 'object') return out;
131+
let covered: Set<string> | undefined;
132+
const isCovered = (name: string) => (covered ??= uniqueCoveredColumns(fields, objectSchema?.indexes)).has(name);
133+
for (const [name, def] of Object.entries(fields)) {
134+
if (!def || typeof def.type !== 'string') continue;
135+
if (NON_TEXT_STORED_VALUE_TYPES.has(def.type)) {
136+
out.set(name, 'blank');
137+
continue;
138+
}
139+
if (LIFECYCLE.has(name) || def.required === true) continue;
140+
const empty = UNIQUE_BLANK_TEXT_TYPES.has(def.type)
141+
? isCovered(name)
142+
: expandEmptyOperator(def).arm === 'null_only' && ONE_STRING_STORED_TYPES.has(def.type);
143+
if (empty) out.set(name, 'empty');
144+
}
145+
return out;
146+
}
147+
148+
/**
149+
* The form a value written to a column of the given reach is stored in:
150+
* `null` when the value is a blank that reach takes, else the value itself
151+
* (the same reference). `reach` `undefined` — a column not in
152+
* {@link blankAsNullColumns} — keeps every value.
153+
*/
154+
export function blankStorageForm(value: unknown, reach: BlankAsNullReach | undefined): unknown {
155+
if (reach === undefined || typeof value !== 'string') return value;
156+
if (reach === 'empty') return value === '' ? null : value;
157+
return isMissing(value) ? null : value;
158+
}
159+
160+
function uniqueCoveredColumns(fields: Record<string, BlankAsNullFieldDef | undefined>, indexes: unknown): Set<string> {
161+
const covered = new Set<string>();
162+
for (const [name, def] of Object.entries(fields)) {
163+
if (def && isUniqueDeclared(def.unique)) covered.add(name);
164+
}
165+
if (Array.isArray(indexes)) {
166+
for (const idx of indexes) {
167+
if (!idx || typeof idx !== 'object' || Array.isArray(idx)) continue;
168+
const { unique, fields: members } = idx as { unique?: unknown; fields?: unknown };
169+
if (!isUniqueDeclared(unique) || !Array.isArray(members)) continue;
170+
for (const name of members) if (typeof name === 'string') covered.add(name);
171+
}
172+
}
173+
return covered;
174+
}

0 commit comments

Comments
 (0)