Skip to content

Commit 5924e50

Browse files
fix(driver-mongodb): two records with a null unique field are both admitted, as on driver-sql and driver-memory (#29339)
<!-- claude-projects-attribution: {"github_login":"os-steve"} --> _Requested by **Steve** · [project thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9JDHkRvsqpBwWHeNYZk2JHR)_ Fixes #29278 Clause-②: yes (widening) Metadata-protocol: none ## 维护者速读 - **改了什么**:MongoDB 驱动的唯一索引不再把 `null` 当成一个值:两条唯一字段为空(`null` 或缺失)的记录都能保存,和 SQL、内存驱动一致;真正的重复值(含 `''`)照旧报 `E11000`。 - **为什么改**:旧索引是 `sparse`,只跳过“没有这个字段”的文档,显式 `null` 仍进索引,所以第二条清空唯一字段的记录被拒。已用真实 mongod 8.3.7 在主干上复现。 - **风险与回滚**:已部署的集合在下次 schema 同步时会把旧的 sparse 唯一索引删掉并按新形态重建(新索引约束更少,必然建成功);中间有毫秒级无唯一约束窗口,若期间写入重复值则同步失败并报错,不会静默。每个唯一索引多一个普通读索引,写入成本略增。回滚 = revert 本 PR(按源码推断、未实测:旧代码遇到同名的新形索引会在同步时报 86,需先手工删掉该索引)。 - **先看**:`packages/drivers/driver-mongodb/src/mongodb-schema.ts` 的 `nullDistinctUniqueOps` 与 `rebuildLegacySparseUnique`;`src/mongodb-unique-null-distinct.test.ts`。 - **两分钟验证**:`pnpm --filter @objectstack/driver-mongodb exec vitest run src/mongodb-unique-null-distinct.test.ts`,默认 7 个 DDL 测试绿、8 个真实 mongod 测试跳过;加 `OS_TEST_MONGODB_MEMORY_SERVER_ENABLED=1` 且能拿到 mongod 时 15 个全绿(主干上 8 红)。 - **需要你做的**:无,席位落地。 ## What changes - **The unique index is NULL-distinct.** `syncCollectionSchema` built a field-level `unique` and a declared `indexes[]` entry carrying `unique` as `{ unique: true, sparse: true }`. Both are now `{ unique: true, partialFilterExpression: { FIELD: { $type: [every non-null BSON type] } } }`, one entry per key part, with no `sparse` (MongoDB refuses `sparse` beside a partial filter). A document with a `null` or missing value in any key part is not constrained; every other value is constrained exactly as before. - Why a type list: MongoDB refuses `$ne` / `$not` inside a partial filter (`{ $exists: true, $ne: null }` → `CannotCreateIndex`, code 67, measured). - Why the full list and not the field's declared type: a stored value of an unexpected type stays constrained, as under the sparse index. - **A plain read index beside each unique index.** The planner cannot use a partial index for an equality lookup (measured: `find({ code: 'X' })` → `COLLSCAN` on the partial index, `EXPRESS_IXSCAN` on the old sparse one; a single `$type: 'string'` filter is not used either). So each unique index also gets `idx_FIELDS`, a plain index over the same keys (the name a declared plain index over those columns already gets, so they converge). A plain index already present under another name (a lookup's `idx_FIELD_lookup`) answers `IndexOptionsConflict` (85), which the loop already skips. An author-named unique index that already takes the `idx_FIELDS` name gets no read index. - **Declared `indexes[]` unique entries are in this fix, not left on the old shape.** A single-column declaration and a field-level `unique` converge on one index name (`idx_FIELD_unique`, pinned in `mongodb-schema-declared-indexes.test.ts`); leaving the declared route sparse would ask for one name with two option sets. Multi-column: a `null` in ANY key part exempts the row, the rule `memory-unique-constraint.ts` states for driver-sql and driver-memory. - **Existing sparse indexes on deployed collections are rebuilt at the next schema sync (measured).** Asking for the new options under the old name is answered `IndexKeySpecsConflict` (86) on 8.3.7, and `collMod` cannot change `sparse` / `partialFilterExpression`. On that conflict the loop reads the collection's indexes; when the same-name index is exactly the legacy shape this driver built (`unique: true`, `sparse: true`, same keys in the same order, no partial filter) it is dropped and created again under its own name. Measured on a real server: a collection seeded with the legacy index, a `null` row and an `'X'` row refuses a second `null` (`E11000`) before the sync; after `syncSchema` the index carries the partial filter and no `sparse`, a second `null` is admitted, a second `'X'` is still refused, and a second sync is a no-op. - The rebuild always succeeds on existing data: the new index constrains a subset of what the old one did. - Between the drop and the create there is a moment with no unique index; a non-null duplicate written in that moment makes the create fail `E11000`, which is thrown (the sync fails loudly, the field is not left unconstrained). The read index is created first, so lookups stay indexed. A concurrent sync's drop of an already-dropped index (`IndexNotFound`, 27) is ignored. - Any other same-name index (an option this driver never wrote, or other keys) is left alone and answers as before: 85 skipped, 86 thrown. - **What the driver writes is unchanged.** Not changed: driver-sql, driver-memory, `packages/spec`, any error code. A non-null duplicate is refused with the server's own `E11000 duplicate key error` (code 11000), as before. ## Tests No existing test was edited. One new file, `packages/drivers/driver-mongodb/src/mongodb-unique-null-distinct.test.ts`, two suites: - **Recording fake `Db` (7 tests, runs on every lane):** the field-level index options (partial filter, no `sparse`); the read index created before the unique one; the declared two-column filter over both parts; the author-named collision; the legacy rebuild sequence `createIndex → dropIndex → createIndex`; and two controls where a same-name index the driver did not build (extra `collation`; other keys) is not dropped and the `86` still throws. - **Real mongod (8 tests, opt-in `OS_TEST_MONGODB_MEMORY_SERVER_ENABLED=1`, #5517):** two `null` creates admitted; two updates clearing to `null` admitted; two missing admitted; a non-null duplicate refused `11000` with message starting `E11000 duplicate key error`; two `''` refused; declared `(region, tag)`: `(eu, null)` twice admitted, `(eu, t)` twice refused; a lookup by the unique field is not a `COLLSCAN`; the legacy-index migration. <details><summary>Red on main, green on head (real mongod 8.3.7)</summary> `mongodb-schema.ts` restored to `276ca307` in the tree only, the new file run, then restored to HEAD (`git diff HEAD` empty, status clean). The suites import `src/` directly (no `dist/` in the path). | Reading | Result | |---|---| | Base `276ca307` | `Tests 8 failed, 7 passed (15)`. `admits two records whose unique field is null`: `expected 11000 to be 'ok'`; the migration test: `expected true to be undefined` (`sparse` still set). The 7 that pass on base are the ones that must: missing ×2 admitted, non-null and `''` duplicates refused, the lookup index, the author-named case, the two leave-alone controls. | | Head `84fb196d` | `Tests 15 passed (15)`, no suite skipped | Premise probe on base, raw client against the same server (`{ f: 1 }` index): sparse unique: two `null` → second `E11000`; two missing → both `ok`; two `''`, two `'x'`, `1` and `1.0`, two equal dates, two `true` → second `E11000`. The partial index answers the same row for every case except two `null` (both `ok`). </details> <details><summary>Ablations of the negative pins</summary> Each mutation was made with `scripts/ablation-replace.mjs` (anchor 1 → 0, blob changed, then restored with blob equal to HEAD and `git diff HEAD` empty). No build leg: the suite reads `src/`. | Ablation | Red | |---|---| | A1: the unique index built `unique: false` | 6 failed / 15, incl. `still refuses a non-null duplicate`: `expected undefined to be 11000`, `''` refused, `(eu, t)` refused, the migration's `'X'` refusal | | A2: any same-name index read as the legacy shape | 2 failed / 15: both leave-alone controls | | A3: no read index | 3 failed / 15, incl. `serves a lookup … not a collection scan`: `expected '{"isCached":false,"stage":"COLLSCAN",…' not to contain 'COLLSCAN'` | </details> ## Acceptance notes - **The same case on all three drivers, measured** (head `84fb196d`, built `dist`, each driver's own `syncSchema` + `create`, field `code` `unique: 'global'`, declared `(region, tag)` unique): | case | memory | sql (SQLite) | mongo | |---|---|---|---| | `code` null ×2 | ok, ok | ok, ok | ok, ok | | `code` missing ×2 | ok, ok | ok, ok | ok, ok | | `code` `''` ×2 | ok, refused | ok, refused | ok, refused | | `code` `'X'` ×2 | ok, refused | ok, refused | ok, refused | | `(eu, null)` ×2 | ok, ok | ok, ok | ok, ok | | `(eu, t)` ×2 | ok, refused | ok, refused | ok, refused | On base, mongo answered `ok, refused` for both `null` rows. - **No shared conformance case-set was added.** The case-sets live in `packages/spec/src/data/*-conformance.ts`, which this lane does not edit. The table above is the cross-driver reading; a shared `unique` case-set is an open question for the spec seat. `check:driver-conformance`: 66 covered, 0 DEBT, 0 exempt, on base and head (unchanged, as expected with no new case-set). - **The real-server suite runs on no CI lane** (opt-in since #5517). The 7 DDL tests are the ones CI runs. This PR's real-server readings were taken with a MongoDB 8.3.7 `mongod` from conda-forge (`mongodb-8.3.7-hc064ac0_0.conda`, sha256 matching conda-forge's index) supplied to `mongodb-memory-server` through `MONGOMS_SYSTEM_BINARY`, because `fastdl.mongodb.org` is refused in the card container. - **Write cost.** Each unique index now has a plain read index beside it, so a write to a unique field updates two indexes instead of one. - **`''` still collides**, as on driver-sql and driver-memory: `''` is a value. PR #29271 (open) makes the engine store a cleared optional unique text field as `null`; with this PR, Mongo admits those too. ## Verification <details><summary>Suites, typecheck, lint, gates</summary> | Run | Head | Result | |---|---|---| | `pnpm --filter @objectstack/driver-mongodb test` with the real mongod | `84fb196d` | Test Files 38 passed (38) · Tests 911 passed (911), no skip | | the same, default lane (no opt-in) | `84fb196d` | Test Files 33 passed, 5 skipped (38) · Tests 721 passed, 190 skipped (911) | | `pnpm --filter @objectstack/driver-mongodb typecheck` | `e38ab227` | exit 0; `check:test-typecheck` 0 debt | | `pnpm exec eslint --no-inline-config packages/drivers/driver-mongodb` | `e38ab227` | exit 0 | | `turbo run build --filter='./packages/**'` (gate prerequisite) | `e38ab227` | VERDICT command-exit 0 | | `dispatch-gates.mjs --commands` (60 families), each run | `e38ab227` | 60 × exit 0; `--ran`: 60 derived, 60 run, 0 NOT-MEASURED, 0 UNRUN | One family first exited 3 (prerequisite, nothing measured): `check-plugin-teardown-shape --self-test` needed its pinned fixture commit fetched; after `git fetch --depth=1 origin 621a487…` it exited 0. `origin/main` moved 27 commits past the base `276ca307` while this ran; none touch `packages/drivers` or `packages/spec/src/data`, so the branch was not merged. Not run locally: the full dogfood battery (no `!` in the title, no `(narrowing)` arm). </details> --- _Generated by [Claude Code](https://claude.ai/code/session_01ADHkmRc2jLWPKqPfVHStXw)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent eef116e commit 5924e50

3 files changed

Lines changed: 449 additions & 12 deletions

File tree

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
'@objectstack/driver-mongodb': patch
3+
---
4+
5+
fix(driver-mongodb): a unique field holding `null` in two records is admitted, as on driver-sql and driver-memory
6+
7+
Clause-②: yes (widening)
8+
Metadata-protocol: none
9+
10+
A field-level `unique` index and a declared `indexes[]` entry carrying `unique` were built as `{ unique: true, sparse: true }`. A sparse index skips only documents that lack the field and still indexes an explicit `null`, so the second record whose optional unique field was cleared to `null` was refused with a duplicate-key error. driver-sql and driver-memory treat a `null` in any key part as exempt.
11+
12+
- **The index.** A unique index is now partial over every non-null BSON type (`partialFilterExpression: { FIELD: { $type: [...] } }`, one entry per key part), with no `sparse`. Two `null` values, two missing values, and in a multi-column index any row with a `null` part, are admitted. Every non-null duplicate is still refused with the server's `E11000` duplicate-key error, `''` included.
13+
- **A read index beside it.** MongoDB's planner cannot use a partial index for an equality lookup on the field, so each unique index also gets a plain index over the same keys, named `idx_FIELDS` (the name a declared plain index over those columns already gets). Lookups by a unique field stay indexed.
14+
- **Existing collections are migrated at the next schema sync.** A unique index this driver built before, matched exactly (`unique` and `sparse` over the same keys, no partial filter, under the driver's name for it), is dropped and created again in the new form. The new index constrains fewer documents than the old one, so the rebuild always succeeds on existing data. Any other index holding that name is left alone and the sync answers as before.

‎packages/drivers/driver-mongodb/src/mongodb-schema.ts‎

Lines changed: 154 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,145 @@
77
* JSON Schema validation for ObjectStack object definitions.
88
*/
99

10-
import type { Db, CreateIndexesOptions, IndexSpecification } from 'mongodb';
10+
import type { Collection, Db, CreateIndexesOptions, Document, IndexSpecification } from 'mongodb';
1111

1212
import { StandardErrorCode } from '@objectstack/spec/api';
1313

14+
/**
15+
* Every BSON type a unique key part is constrained under: all of them except
16+
* `null` and `undefined`.
17+
*
18+
* ## Why a unique index here carries a partial filter and no `sparse` (#29278)
19+
*
20+
* The NULL rule every other driver keeps is SQL's: a row whose unique column is
21+
* NULL (or, in a multi-column key, a row with a NULL in ANY key part) is exempt
22+
* from the constraint, so two such rows never collide. `driver-sql` gets it from
23+
* the database's NULL-distinct UNIQUE; `driver-memory` reproduces it (see the
24+
* `memory-unique-constraint.ts` docblock). This driver used to build
25+
* `{ unique: true, sparse: true }`, and a sparse index skips only documents that
26+
* LACK the field — it still indexes an explicit `null`. Measured against a real
27+
* `mongod` (8.3.7): two documents storing `f: null` → the second refused
28+
* `E11000`; two documents without `f` → both admitted. So two records whose
29+
* optional unique field was cleared collided here and nowhere else.
30+
*
31+
* A `partialFilterExpression` that admits only these types excludes both
32+
* shapes — missing and `null` — and constrains every other value exactly as the
33+
* sparse index did (measured: two `''`, two equal strings, `1` and `1.0`, two
34+
* equal dates and two equal booleans are each still refused `E11000`). It is
35+
* spelled as a type list rather than "not null" because MongoDB refuses `$ne`
36+
* (and `$not`) inside a partial filter: `{ $exists: true, $ne: null }` is
37+
* answered `CannotCreateIndex` (code 67). It is the full non-null list rather
38+
* than the field's declared type so a stored value of an unexpected type is
39+
* still constrained, as it was under the sparse index.
40+
*
41+
* MongoDB does not allow `sparse` and `partialFilterExpression` on one index,
42+
* and the filter already implies what `sparse` said, so `sparse` is gone.
43+
*/
44+
const NON_NULL_BSON_TYPES = [
45+
'double', 'string', 'object', 'array', 'binData', 'objectId', 'bool', 'date',
46+
'regex', 'javascript', 'int', 'timestamp', 'long', 'decimal', 'minKey', 'maxKey',
47+
] as const;
48+
49+
/**
50+
* The options of a NULL-distinct unique index over `fields` — see
51+
* {@link NON_NULL_BSON_TYPES}. Every listed field must hold a non-null value
52+
* for a document to be constrained, which is the "a NULL in ANY key part exempts
53+
* the row" rule of the other drivers.
54+
*/
55+
function nullDistinctUniqueOptions(fields: readonly string[]): Pick<CreateIndexesOptions, 'unique' | 'partialFilterExpression'> {
56+
const partialFilterExpression: Document = {};
57+
for (const f of fields) partialFilterExpression[f] = { $type: [...NON_NULL_BSON_TYPES] };
58+
return { unique: true, partialFilterExpression };
59+
}
60+
61+
/**
62+
* The index ops for one NULL-distinct unique constraint: a plain READ index
63+
* over the same keys, then the unique index itself.
64+
*
65+
* Why the read index: the query planner uses a partial index only for a query
66+
* whose predicate it can prove lies inside the filter, and it cannot prove that
67+
* for an equality on a value. Measured on 8.3.7: `find({ f: 'x' })` is answered
68+
* by `EXPRESS_IXSCAN` on the old sparse unique index and by `COLLSCAN` on the
69+
* partial one (a single `$type: 'string'` filter too). The sparse index was the
70+
* one serving lookups by a unique field, so dropping it without a replacement
71+
* would trade a wrong refusal for a full collection scan on every such lookup.
72+
*
73+
* The read index is named `idx_<fields>` — the name a declared plain
74+
* `indexes[]` entry over the same columns gets — so the two converge on one
75+
* index. A plain index over the same keys that already exists under another name
76+
* (a lookup field's `idx_FIELD_lookup`) answers `IndexOptionsConflict`, which
77+
* the create loop skips: that index serves the reads. When an author-named
78+
* unique index already takes the `idx_<fields>` name, no read index is added.
79+
*/
80+
function nullDistinctUniqueOps(
81+
fields: readonly string[],
82+
uniqueName: string,
83+
): Array<{ spec: IndexSpecification; options: CreateIndexesOptions }> {
84+
const spec: Record<string, 1> = {};
85+
for (const f of fields) spec[f] = 1;
86+
const readName = `idx_${fields.join('_')}`;
87+
const ops: Array<{ spec: IndexSpecification; options: CreateIndexesOptions }> = [];
88+
if (readName !== uniqueName) ops.push({ spec: spec as IndexSpecification, options: { name: readName } });
89+
ops.push({ spec: spec as IndexSpecification, options: { ...nullDistinctUniqueOptions(fields), name: uniqueName } });
90+
return ops;
91+
}
92+
93+
/**
94+
* Is `existing` the unique index this driver built before #29278 for `spec` —
95+
* `{ unique: true, sparse: true }` over the same keys in the same order, with no
96+
* partial filter?
97+
*
98+
* Deliberately exact: only an index whose every option this driver itself used
99+
* to write is ever rebuilt. Any other same-name index (one an operator made by
100+
* hand, say) is left alone and keeps the create loop's old answer.
101+
*/
102+
function isLegacySparseUnique(existing: Document, spec: IndexSpecification): boolean {
103+
if (existing.unique !== true || existing.sparse !== true) return false;
104+
if (existing.partialFilterExpression !== undefined) return false;
105+
const want = Object.entries(spec as Record<string, unknown>);
106+
const have = Object.entries((existing.key ?? {}) as Record<string, unknown>);
107+
return want.length === have.length && want.every(([k, v], i) => have[i][0] === k && have[i][1] === v);
108+
}
109+
110+
/**
111+
* Rebuild a legacy sparse unique index as the NULL-distinct one, in place —
112+
* the migration of a collection that was synced before #29278.
113+
*
114+
* Asking for the new options under the old name is answered
115+
* `IndexKeySpecsConflict` (code 86, measured on 8.3.7), and MongoDB cannot
116+
* change an index's `sparse` / `partialFilterExpression` in place (`collMod`
117+
* does not take them), so the index is dropped and created again under its own
118+
* name. That always succeeds on the existing data: the new index constrains a
119+
* subset of the documents the old one did, and the old one was holding.
120+
*
121+
* Between the drop and the create no unique index exists for a moment; a
122+
* concurrent write that lands a non-null duplicate in that moment makes the
123+
* create fail `E11000`, and that error is thrown, failing the sync loudly
124+
* rather than leaving the field unconstrained. The read index is created
125+
* before this runs, so lookups stay indexed throughout. A second process
126+
* syncing at the same time is harmless: a drop that finds the index already
127+
* gone (`IndexNotFound`, 27) is ignored, and creating an index that already
128+
* exists with the same options is a no-op.
129+
*
130+
* @returns `true` when the index was the legacy shape and is now rebuilt;
131+
* `false` when it was something else, which the caller handles as before.
132+
*/
133+
async function rebuildLegacySparseUnique(
134+
collection: Collection,
135+
spec: IndexSpecification,
136+
options: CreateIndexesOptions,
137+
): Promise<boolean> {
138+
const existing = (await collection.indexes()).find((i) => i.name === options.name);
139+
if (!existing || !isLegacySparseUnique(existing, spec)) return false;
140+
try {
141+
await collection.dropIndex(options.name as string);
142+
} catch (error: any) {
143+
if (error?.code !== 27 && error?.codeName !== 'IndexNotFound') throw error;
144+
}
145+
await collection.createIndex(spec, options);
146+
return true;
147+
}
148+
14149
/**
15150
* ObjectStack field definition (subset needed for schema sync).
16151
*/
@@ -264,10 +399,8 @@ export async function syncCollectionSchema(
264399
if (schema.fields) {
265400
for (const [fieldName, field] of Object.entries(schema.fields)) {
266401
if (field.unique) {
267-
indexOps.push({
268-
spec: { [fieldName]: 1 },
269-
options: { unique: true, sparse: true, name: `idx_${fieldName}_unique` },
270-
});
402+
// NULL-distinct (#29278) — see {@link NON_NULL_BSON_TYPES}.
403+
indexOps.push(...nullDistinctUniqueOps([fieldName], `idx_${fieldName}_unique`));
271404
}
272405

273406
// Lookup + user (a lookup specialized to sys_user) fields get an index for
@@ -315,22 +448,31 @@ export async function syncCollectionSchema(
315448
const fields = (idx.fields ?? []).filter((f) => typeof f === 'string' && f.length > 0);
316449
if (fields.length === 0) continue;
317450
const unique = Boolean(idx.unique);
451+
const name = idx.name ?? `idx_${fields.join('_')}${unique ? '_unique' : ''}`;
452+
if (unique) {
453+
// NULL-distinct on every key part (#29278) — a single-column declaration
454+
// must also take the field-level shape, or the two routes would ask for
455+
// `idx_FIELD_unique` with different options.
456+
indexOps.push(...nullDistinctUniqueOps(fields, name));
457+
continue;
458+
}
318459
const spec: Record<string, 1> = {};
319460
for (const f of fields) spec[f] = 1;
320-
indexOps.push({
321-
spec: spec as IndexSpecification,
322-
options: {
323-
...(unique ? { unique: true, sparse: true } : {}),
324-
name: idx.name ?? `idx_${fields.join('_')}${unique ? '_unique' : ''}`,
325-
},
326-
});
461+
indexOps.push({ spec: spec as IndexSpecification, options: { name } });
327462
}
328463

329464
// Create indexes (idempotent — MongoDB ignores duplicates)
330465
for (const { spec, options } of indexOps) {
331466
try {
332467
await collection.createIndex(spec, options);
333468
} catch (error: any) {
469+
// A unique index synced before #29278 under this name — rebuild it.
470+
const conflict = error?.code === 85 || error?.code === 86
471+
|| error?.codeName === 'IndexOptionsConflict' || error?.codeName === 'IndexKeySpecsConflict';
472+
if (conflict && options.partialFilterExpression !== undefined
473+
&& await rebuildLegacySparseUnique(collection, spec, options)) {
474+
continue;
475+
}
334476
// Index already exists with different options — skip silently
335477
if (error.codeName === 'IndexOptionsConflict' || error.code === 85) {
336478
continue;

0 commit comments

Comments
 (0)