Repository navigation
Commit cd5f554
fix(objectql,cli): os migrate stored-expressions fills a stored computed field on rows stored before it was declared, and the boot names it (#30227)
<!-- claude-projects-attribution: {"github_login":"os-bill"} -->
_Requested by **Bill** · [project
thread](https://claude.ai/code/project/chan_016iMXWSZDzqWRWvWzdNo8wg?thread=cmsg_016iMXWSZDzqWRWvWzdNo8wg1cdCRrn63ZpnD6M1busUdm)_
domain:server#2 · os-bill · session_01Utp6Cb2FL7H75wB1wA6PpZ
## 维护者速读
- **改了什么:** 新增 `os migrate
stored-expressions`:给"在已有数据的对象上新加的存储型计算字段"补算存量行(默认只补 NULL,`--changed`
再重算表达式改过后不一致的值),先报告、`--apply` 才写;另在启动时抽查每个声明了存储型计算字段的对象的前 100 行,发现这种
NULL 就告警一次并点名命令。
- **为什么改:** 存储型计算字段只在写入时计算,升级后的库里新字段在所有旧行上一直是 NULL(hotclm 121/121
行),分析看板把 NULL 当 0 天,静默多算超期。
- **风险与回滚:** 命令默认只读;写入走 `engine.update`(保留
`updated_at`),钩子、汇总、审计照常触发;表达式在某行算不出而该行已有值时整行跳过并列出,不会用 NULL
覆盖。启动抽查只读、有上限、出错不影响启动。回滚即撤 PR。
- **先看:**
`packages/objectql/src/stored-expression-backfill.ts`(分类与写入规则)、`packages/objectql/src/engine.ts`
新方法 `computeStoredExpressionsOver`、`packages/objectql/src/plugin.ts` 的
`kernel:bootstrapped` 一行。
- **两分钟验证:** `pnpm --filter @objectstack/objectql exec vitest run
src/stored-expression-backfill.test.ts`,预期 29 个用例全过(含"加字段后旧行 NULL →
--apply 补齐 → 再跑为 0")。
- **需要你做的:** 无,席位落地。
Fixes #29961
Clause-②: yes (widening)
Metadata-protocol: none
## Why
See the first two bullets of 维护者速读 and #29961: a stored computed field
added to an object that already has rows stays NULL on every existing
row until each row is next written, and an analytics view reading it
treats the NULL as 0.
## What changes
**The producer.** A stored computed field (a non-`formula` field
declaring `expression`) is computed by `computeStoredExpressions` on
insert and update only. Schema sync adds its column NULL on every
existing row, and nothing computes it until each row is next written.
The fix leaves the write path alone and adds the missing producer for
existing rows, in the shape `os migrate summary-nulls` and `os migrate
recompute-home-amounts` already have: a report-first command, plus a
bounded boot advisory.
- **`ObjectQL.computeStoredExpressionsOver(object, rows, context?,
onFault?)`** (`engine.ts`, one method): the write's own plan
(`planStoredExpressions`) and serving (declared option labels, the one
hop, the `can()` map) over rows already stored. Evaluates over copies,
writes nothing.
- **`stored-expression-backfill.ts`** (new): `backfillStoredExpressions`
walks every row by keyset (`id`), computes each page through the method
above, and classifies each (row, field):
- stored NULL, expression gives a value → **hole**: written by default;
- stored NULL, expression gives NULL (an empty reference, a null input)
→ already the write's value, left alone;
- stored value differs from the expression → **stale**: counted on every
run, written only with `changed`;
- the expression does not evaluate on the row → **faulted**: listed;
when the row holds a value there, the whole row is skipped, because the
write would store NULL over it.
A written row goes through `engine.update` with the computed values and
the row's `updated_at` under `preserveAudit`, as
`recompute-home-amounts` writes. The write computes every stored field
of the row again, so a stale value on a filled row is rewritten with it
(counted under `written`). Refused before any row is read: an unknown
object (`OBJECT_NOT_FOUND`), a named object or `field` declaring no
stored computed field, and `field` without exactly one object
(`INVALID_FIELD`, 400). Idempotent; no `sys_migration` flag (nothing
gates on it).
- **Boot advisory** (`announceStoredExpressionHoles`, called from the
objectql plugin's `kernel:bootstrapped` hook beside
`announceOpenMigrationGates`): one `find` of at most 100 rows per object
that declares a stored computed field, nothing for objects declaring
none. When the sample holds a hole, it warns once, naming each
`object.field (n of the first m rows)` and the command. It writes
nothing, and a read or compute failure is logged at debug and never
fails the boot. A column added to an object with rows is NULL on every
row, so the first page finds it.
- **`os migrate stored-expressions`** (`packages/cli`, new, discovered
by oclif's pattern strategy, so no registration file): `--object`
(repeatable), `--field` (needs `--object`), `--changed`, `--apply`,
`--yes`, `--json`, `--force`, `--max-records`, `--database-url`; the
occupancy gate, confirmation, read-only dry-run boot and SQLite
journal-mode note of its siblings. Exits 1 while a row is skipped, a
write failed, or `--max-records` truncated the walk.
- **Docs:** one row in `content/docs/deployment/cli.mdx`'s
data-migration table.
## Tests
Measured at `c82c1c7f` (branch merged with `origin/main` once before
this PR). Head `72809e52` adds only the changeset's `Clause-②` line on
top of it.
- `packages/objectql/src/stored-expression-backfill.test.ts` (new, 29
tests): a real `ObjectQL` over an in-file driver. Dry run reports and
writes nothing; `apply` fills holes, keeps `updated_at`, and a re-run
finds 0; stale values are counted by default and written with `changed`;
a faulted row that holds a value is skipped and listed; a refused write
is isolated to its row; `field` narrows the walk; the three refusals; an
unreadable object; the boot advisory (holes found, none found, a read
failure that never throws); a `sameStoredValue` table.
- `packages/cli/src/commands/migrate/stored-expressions.test.ts` (new,
10 tests, integration project): flags reach the walk, `--field` needs
`--object`, `--apply` without `--yes` writes nothing and exits 1, an
undeclared object is refused, exit 1 on skipped / failed / truncated,
and the dry-run face closes with the report's line.
- Pin and ledger rows: `migrate-confirmation-exit.pin.test.ts` (the new
command in the confirmation roster) and
`federated-injected-column-readers.test.ts` (the new `updated_at` reader
skips an unprovisioned federated column).
- Ablation: replacing the `if (!apply) continue;` guard turns 5 backfill
tests red; restore verified blob == HEAD.
## Acceptance notes
- **Boot backfill vs command, measured choice.** An automatic boot
backfill was not taken: each fill is a write (hooks, record-triggered
flows, audit rows, roll-up recomputes) nobody ordered, a large table
would hold the boot for a full walk, a remote driver pays a round trip
per row, and an expression reading a lookup reads another object per
page. The boot advisory is bounded to one 100-row read per object
declaring a stored computed field.
- **Roll-ups have no boot notice.** The triage said "as roll-ups have";
on `main` the roll-up backfill (`summary-backfill.ts`) has no boot
notice (the lint's `isAlwaysValuedSummary` docblock says "Nothing
backfills a stored NULL automatically"). The in-tree boot-advisory shape
is `announceOpenMigrationGates` at `kernel:bootstrapped`, which this
follows.
- **A changed expression.** Covered by `--changed`. No boot can tell
that an expression changed (the stored value is not a function of the
declaration alone, and a `now()`-reading expression always differs), so
the advisory does not report stale values; every run of the command
counts them and its report says to re-run with `--changed`.
- **Clause-② corrected to `yes (widening)`.** The dispatch order
proposed `no`. Measured: new published `@objectstack/objectql` exports
(the stored-expression backfill functions and types), one new public
method `ObjectQL.computeStoredExpressionsOver`, and a new CLI command
`os migrate stored-expressions`; nothing narrows. Declared as #30183
declared the same shape; the changeset carries the same line from
`72809e52`. `Metadata-protocol` stays `none`.
- **Advisory false negatives.** Holes only past the first 100 rows by
`id` are not seen by the advisory; the command walks every row.
## Verification
<details><summary>Reproduction on SQLite and memory, before and
after</summary>
An object with 4 rows, booted; stored computed fields then declared (a
doubled number, a date offset, a boolean, a text, a scale-2 decimal) and
booted again over the same file. Before: 4/4 rows NULL in each new
column on both drivers. `os migrate stored-expressions` dry run reported
the holes and wrote nothing; `--apply` wrote the expected values
(`qty_double` 2/4/6/8, `offset` -5/-5/-5/null for a row with an empty
input); a re-run found 0 holes and 0 stale. Changing an expression to
`qty*3` then showed 4 stale in default mode, `--changed --apply` rewrote
them, and a re-run found 0.
</details>
<details><summary>Suites, typechecks, lint and gates</summary>
```
pnpm --filter @objectstack/objectql test :: exit 0 (549 files, 10892 tests)
pnpm --filter @objectstack/objectql typecheck :: exit 0
pnpm --filter @objectstack/cli typecheck :: exit 0
pnpm --filter @objectstack/cli exec vitest run --project unit :: exit 0 (382 files, 5384 tests)
cli integration: stored-expressions.test.ts :: exit 0 (10 tests)
pnpm lint :: exit 0
node scripts/pm/dispatch-gates.mjs --ran :: 96 derived, 96 run, 0 NOT-MEASURED
```
</details>
## Human Minimum
- [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic.
(`packages/spec` untouched.)
- [x] 2. Every name I added uses the casing for its kind.
- [x] 3. No formatter ran; `pnpm lint` passes.
- [x] 4. One issue, one branch off an up-to-date `main`, one pull
request.
- [x] 5. A change that publishes carries a changeset, and both
declaration lines are filled in here and in the changeset.
- [x] 6. `packages/spec` touched: not touched.
- [x] 7. No hand edits to `content/docs/references/`,
`packages/*/CHANGELOG.md` or `content/docs/releases/`.
- [x] 8. Title, body and comments are in English (the 维护者速读 block is the
maintainer's Chinese summary).
- [x] The *Before Submitting* list in CONTRIBUTING.md is done.
<!-- signed: domain:server#2 · os-bill ·
session_01Utp6Cb2FL7H75wB1wA6PpZ -->
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 171b6dd commit cd5f554
14 files changed
Lines changed: 1411 additions & 3 deletions
File tree
- .changeset
- content/docs/deployment
- packages
- cli
- src
- commands/migrate
- utils
- test
- objectql/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1303 | 1303 | | |
1304 | 1304 | | |
1305 | 1305 | | |
| 1306 | + | |
1306 | 1307 | | |
1307 | 1308 | | |
1308 | 1309 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
0 commit comments