Repository navigation
Commit d7a7193
fix(cli): os migrate apply takes a database lock so concurrent migrators run one at a time (#31714)
<!-- claude-projects-attribution: {"github_login":"os-steve"} -->
_Requested by **Steve** · [project
thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9EtLz4Q3mytsuVUjBvqeWPK)_
Fixes #31598
Clause-②: no
Metadata-protocol: none
## 维护者速读
- **改了什么**:`os migrate apply` 在真正读取待办表结构变更之前,先在主库上拿一把数据库级的会话锁(Postgres
`pg_advisory_lock`、MySQL `GET_LOCK`,锁名固定为
`objectstack:schema-migrate`),命令结束时释放;SQLite 不加锁。
- **为什么改**:多副本部署时每个 Pod 的 init 容器都会跑一次迁移。实测 3 个迁移同时打同一个
Postgres:三个都报告"建了表"(82 + 86 + 37 张,库里实际 86 张),第三个还在别人建到一半的表上做了 drift
变更,而且全部退出码 0。
- **风险与回滚**:单机路径只多一条无竞争的加锁语句,输出逐字节不变(实测)。有竞争时多打印一行"正在等待",最多等 10
分钟,超时则拒绝、不做任何变更并以非零退出。回滚即 revert 本 PR。
- **先看**:① `packages/cli/src/utils/schema-migration-lock.ts` 的
`acquireSchemaMigrationLock`;②
`packages/cli/src/utils/schema-migrate.ts` 中 `bootSchemaStack` 里
`migrationLock` 的取锁点与 `shutdown()` 的释放;③
`schema-migration-lock.integration.test.ts`。
- **两分钟验证**:有本地 Postgres 时运行 `OS_TEST_POSTGRES_URL=postgres://… pnpm
--filter @objectstack/cli exec vitest run
src/utils/schema-migration-lock.integration.test.ts`,预期 4/4
通过:两个迁移排队等锁、只有一个建表、另一个报告 `in_sync`。
- **需要你做的**:无,席位落地
## The one exception to R3 this card accepts
R3 on #3814 (「同意:每张卡都只在"声明多副本并接上 Redis 驱动"时才生效」) keeps the single-node
default path unchanged. This card names the one exception it accepts, in
its item 2:
> 2. The lock is harmless on a single node (uncontended); keep its
output identical apart from no new lines when uncontended. Note: this
changes the single-node code path by one uncontended lock call; say so
in the PR.
So, stated plainly: **on a single node, every applying `os migrate
apply` now runs one extra statement** (`select pg_try_advisory_lock(…)`
on Postgres, `select GET_LOCK(…, 0)` on MySQL) on a connection of its
own, plus the matching unlock at the end. It is not gated behind
declared replicas or a Redis driver, because the collision happens
before any replica count is known to the migrator. Uncontended it prints
nothing. Measured: one `os migrate apply --yes` on a fresh Postgres
database, human face, gives byte-identical stdout and stderr on
`origin/main` and on this diff over the same base once ISO timestamps
and durations are normalized (stdout sha256 `8980fba0…1eda0`, 41,920
bytes, both sides; stderr empty, both sides). SQLite (and the Turso and
WASM drivers on it) runs no statement at all.
## What changes
- **`packages/cli/src/utils/schema-migration-lock.ts` (new).**
`acquireSchemaMigrationLock(driver, { database, warn, waitMs? })`. It
reads the dialect from the driver's own knex client: `postgresql` takes
`pg_advisory_lock`, `mysql` takes `GET_LOCK`, and every other dialect
answers `null` and runs nothing. The flow is a no-wait try first (silent
when it succeeds); on contention it prints ONE line naming the database
and waits, bounded (Postgres `lock_timeout`, MySQL `GET_LOCK`'s seconds
argument). If the wait runs out it throws a refusal naming the database,
and the run migrates nothing. `release()` unlocks on the same session,
then closes it. It is idempotent and never throws. Its own module,
importing only `node:crypto`, so its pins load the lock and not the
migration stack (`check:test-source-alias` asked for exactly that).
- **The key.** `SCHEMA_MIGRATION_LOCK_NAME =
'objectstack:schema-migrate'`, named once. The Postgres key is derived
from it: the first 8 bytes of SHA-256 as a signed 64-bit integer,
`255402229896085448`. The unit file pins the value, because a changed
key would let an old and a new CLI migrate side by side during a rolling
upgrade. It is not per-database: a Postgres advisory lock is already
scoped to its database. MySQL's `GET_LOCK` is server-wide, see
Acceptance notes.
- **The bound.** `SCHEMA_MIGRATION_LOCK_WAIT_MS = 10 * 60_000`, a
constant. No environment override exists for migration timeouts, so none
is added.
- **`packages/cli/src/utils/schema-migrate.ts`, `bootSchemaStack`.** A
boot that can apply (a deferred boot that is not a read-only probe)
takes the lock on the primary right after `runtime.start()`, before the
coverage pass and before the first `previewDeferredSchemaWork()`. That
boot held all its DDL back, so nothing has been written yet. A migrator
that waited then reads only the work the first one left, which is none.
`shutdown()` releases the lock first, in a `finally`. If one of the
three post-lock reads (coverage, preview, package-table plan) throws,
`underLock` releases it before the throw leaves. Which commands take it:
`os migrate apply`, plus the `--apply` modes that boot deferred, `os
migrate organization-ownership --apply` and `os migrate
multi-value-columns --apply`. `os migrate plan` and every other
read-only probe take none.
- **The session.** The lock rides a connection opened through the
driver's own client (its configuration, credentials and TLS) but outside
its pool, via knex `acquireRawConnection` / `destroyRawConnection`. See
Acceptance note 1 for why not a pooled connection.
- **Changeset.** `@objectstack/cli` patch, with the claim's two lines at
line start.
## Tests
- `packages/cli/src/utils/schema-migration-lock.test.ts` (new, unit, 16
cases, fake knex recording every statement and the session it ran on).
The key value is pinned. SQLite / MSSQL / Oracle / no dialect / no knex
answer `null` with no connection opened and no line. Postgres and MySQL
each cover: uncontended, one statement and no line; release on the same
session, once; contended, exactly one line naming the database, then the
bounded wait (`lock_timeout` in ms, `GET_LOCK` in whole seconds); a wait
that runs out refuses and closes the session. MySQL `GET_LOCK` answering
`0` or `NULL` refuses. Any other wait failure is handed on unchanged.
- `packages/cli/src/utils/schema-migration-lock.integration.test.ts`
(new, integration tier, live Postgres via `OS_TEST_POSTGRES_URL`, named
skip without it). The contention is staged, not raced. The test holds
the lock on its own session, spawns two `os migrate apply --yes --json`
(`bin/run-dev.js`), waits until `pg_locks` shows both queued (an
advisory lock not granted), then releases. Assertions: both queued (2);
both exit 0 and each printed the wait line exactly once, on stderr, with
stdout still one JSON document; exactly one created the schema and the
other answered `in_sync` with nothing created; the table exists. A
third, uncontended run is the control: no wait line, `in_sync`.
- No forced test edits.
## Acceptance notes
1. **Seat ruling 1 said "on the SAME connection the migration uses (a
pooled second connection would not serialize)".** Measured otherwise,
and taken by its intent. Both locks are session-scoped and exclude every
other session on the database, the migration's own pooled ones included.
The integration pin shows a lock held on a session outside both
migrators' pools serializing them. What a pool breaks is the release:
knex's pool gives no promise that the unlock runs on the session that
locked, its idle reaper can close a held connection (silently releasing
the lock), and an `OS_DATABASE_POOL_MAX=1` pool would have no connection
left for the migration itself, so it would deadlock until the acquire
timeout. So the lock takes its own session through the driver's own
client and configuration. The seat may re-rule.
2. **Where the init container's command comes from.** `helm/` lives in
the cloud repository, which this session cannot read. The card names
`schema-migrate.ts`, whose `bootSchemaStack` is the `os migrate apply`
path, so that is what is locked. If the chart's init container runs a
SERVING boot with `OS_MIGRATE_AND_EXIT=1` (`os serve`,
`packages/cli/src/commands/serve.ts`) instead, its boot-time schema sync
does not go through `bootSchemaStack` and this lock does not cover it.
The same holds for any replica's own serving boot, whose additive sync
runs at start. carrier: the cloud C2 card (the Helm hook Job), which
decides which command the Job runs; noted, not filed.
3. **The non-deferred `--apply` data commands** (`os migrate meta
--stored --apply`, `value-shapes --apply` and the others that boot
plain) run their schema sync inside `runtime.start()`, before any point
a lock could be taken without adding a kernel plugin. A plugin would add
three INFO lines to every boot and break the byte-identical control.
They are not locked. They are data repairs, not the schema migration
this card names.
4. **MySQL.** `GET_LOCK` is server-wide, not per-database: two
migrations of different databases on one MySQL server now run one after
the other. That is harmless, only slower, and the seat ruled out
per-database-name keys. NOT MEASURED live: the container has no MySQL
server. The statements and result shapes are pinned in the unit file
against mysql2's `[rows, fields]` shape.
5. **MSSQL / Oracle are out of scope, measured.** The CLI's primary
database never is one: `resolveStandaloneDatabase` refuses their URL
schemes ("Supported schemes: postgres://, pg://, mysql://, mysql2://,
mongodb://, mongodb+srv://, libsql://, file:"). A declared datasource
can be MSSQL, but the lock lives on the primary, which every migrator of
the deployment shares.
6. **The refusal carries no error code.** It is a plain `Error` whose
message names the database and the remedy, and `os migrate apply` prints
it and exits 1 the way it handles any boot failure. A new ADR-0112 code
would be a new public name, so it is listed as an open question for the
seat rather than coined.
## Verification
Head `53aea71cf3cae9820396e0642854ee88cf3694db`, a clean merge with
`origin/main` at `b126c1306`. Every reading below the ablations is on
that head.
<details>
<summary>Red on main, green after: three concurrent os migrate apply
against one fresh Postgres 16 database</summary>
```text
origin/main 5d1382a, run 1: migrator 0 created 82 · migrator 1 created 86 · migrator 2 created 37, applied 3 drift · all exit 0 · 86 tables
origin/main 5d1382a, run 2: created 12 · 86 · 41 · all exit 0 · 86 tables
origin/main 5d1382a, 2 migrators: created 86 · 85 · both exit 0
fix c8de894, run 1: created 86 · 0 (in_sync) · 0 (in_sync) · all exit 0 · 86 tables
fix c8de894, run 2: created 0 (in_sync) · 86 · 0 (in_sync) · one printed "Another schema migration is running on postgres://127.0.0.1:5432/…; waiting up to 600 s for it to finish…"
```
</details>
<details>
<summary>CONTROL: one uncontended migration, byte-identical to
main</summary>
```text
origin/main 5d1382a os migrate apply --yes (human face), fresh PG db: exit 0 · stdout 41,920 bytes sha256 8980fba0d8e1fb6a50c5511b88ede883c1d3f3465fadafca069c730fb0b1eda0 · stderr 0 bytes
ba38138 (this diff on that base) same command, same fixture: exit 0 · stdout 41,920 bytes sha256 8980fba0d8e1fb6a50c5511b88ede883c1d3f3465fadafca069c730fb0b1eda0 · stderr 0 bytes
(normalized: ISO timestamps -> TS, durations -> N ms; main ran twice first and was self-identical)
53aea71 (after the second main merge): one line differs from the 5d1382a reading, and it is main's, not this diff's:
"[platform-objects] contributed platform translations (20 locale entries)" -> "(22 locale entries)",
from b126c13 (#31588, the platform objects in Korean). No wait line and no other change.
```
</details>
<details>
<summary>Ablations (scripts/ablation-replace.mjs, each restored to the
HEAD blob, git diff HEAD empty) on ba38138</summary>
```text
A1 lock never taken (`false as boolean && deferral …` in bootSchemaStack) -> integration 3 failed | 1 passed:
"both migrators queued" expected +0 to be 2; "applied ONCE" expected 2 creators to have a length of 1
A2 Postgres 55P03 proceeds instead of refusing -> unit 1 failed | 15 passed ("a wait that runs out REFUSES")
A3 MySQL GET_LOCK 0/NULL proceeds -> unit 2 failed | 14 passed (both GET_LOCK refusal cases)
A4 the wait line printed even uncontended -> unit 4 failed | 12 passed (both "NOT ONE line printed", both "ONE line")
A5 dialect gate removed (SQLite etc. take a lock) -> unit 4 failed | 12 passed (sqlite3 / mssql / oracle / undefined)
```
</details>
<details>
<summary>Tests and gates on 53aea71 (after pnpm install
--frozen-lockfile and a full pnpm build, 119/119)</summary>
```text
pnpm --filter @objectstack/cli typecheck tsc green · check:test-typecheck OK (3 files / 28 errors / 6 pinned signatures held, unchanged)
cli unit tier (vitest --project unit) Test Files 406 passed (406) · Tests 5740 passed (5740)
schema-migration-lock.integration.test.ts (live PG) Tests 4 passed (4) (integration tier run locally because this diff adds an integration file)
eslint packages/cli --no-inline-config exit 0, no findings
dispatch-gates --commands 67 derived families, each run: 67 exit 0
dispatch-gates --ran 67 derived, 67 run, 0 NOT-MEASURED, 0 UNRUN
first round on ba38138 (before the second merge): 67/67 exit 0 after deepening for the teardown self-test fixture (exit 3 on the shallow clone, a container prerequisite)
earlier red, fixed in this PR: check:test-source-alias flagged both new test files for reaching the migration stack's dynamic imports; the lock moved to its own module (ba38138)
```
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_01ADHkmNs71AKtAebnmL7SxF)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 7200f54 commit d7a7193
5 files changed
Lines changed: 660 additions & 4 deletions
File tree
- .changeset
- packages/cli/src/utils
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| 69 | + | |
69 | 70 | | |
70 | 71 | | |
71 | 72 | | |
| |||
1469 | 1470 | | |
1470 | 1471 | | |
1471 | 1472 | | |
| 1473 | + | |
| 1474 | + | |
| 1475 | + | |
| 1476 | + | |
| 1477 | + | |
| 1478 | + | |
| 1479 | + | |
| 1480 | + | |
| 1481 | + | |
| 1482 | + | |
| 1483 | + | |
| 1484 | + | |
| 1485 | + | |
| 1486 | + | |
| 1487 | + | |
1472 | 1488 | | |
1473 | 1489 | | |
1474 | 1490 | | |
| |||
1488 | 1504 | | |
1489 | 1505 | | |
1490 | 1506 | | |
1491 | | - | |
| 1507 | + | |
1492 | 1508 | | |
1493 | 1509 | | |
1494 | 1510 | | |
| |||
1501 | 1517 | | |
1502 | 1518 | | |
1503 | 1519 | | |
1504 | | - | |
| 1520 | + | |
1505 | 1521 | | |
1506 | 1522 | | |
1507 | 1523 | | |
| |||
1510 | 1526 | | |
1511 | 1527 | | |
1512 | 1528 | | |
1513 | | - | |
| 1529 | + | |
1514 | 1530 | | |
1515 | 1531 | | |
1516 | 1532 | | |
| |||
1650 | 1666 | | |
1651 | 1667 | | |
1652 | 1668 | | |
1653 | | - | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
1654 | 1676 | | |
1655 | 1677 | | |
1656 | 1678 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
0 commit comments