You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f647430
Browse filesBrowse the repository at this point in the historyBrowse files
fix(objectql): a server boot's schema sync takes the schema-migration lock, so replicas starting together create each table once (#31747)
<!-- claude-projects-attribution: {"github_login":"os-steve"} -->
_Requested by **Steve** · [project
thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9XahPEEET2Vy9ZB5xcVcgHE)_Fixes#31715
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:服务器启动时的表结构同步(含 `OS_MIGRATE_AND_EXIT=1`)现在和 `os migrate apply`
拿同一把数据库迁移锁,只在建表期间持有,同步完立即释放。
- 为什么改:多个副本同时启动会重复建同一张表,Postgres 报 `pg_type` 冲突,那个启动记下 `Schema sync
FAILED` 却仍以 0 退出;实测可复现。
- 风险与回滚:单机无争用时只多一条加锁语句、输出逐字节不变;SQLite 不加锁。回滚即还原本 PR,锁模块挪回 cli。
- 先看:`packages/objectql/src/plugin.ts`(`acquireBootSchemaMigrationLock`
与 Phase 1–3 的
try/finally)、`packages/objectql/src/schema-migration-lock.ts`(从 cli
挪来)、`packages/drivers/driver-sql/src/sql-driver.ts`(新只读 getter
`schemaDdlDeferred`)。
- 两分钟验证:`OS_TEST_POSTGRES_URL=postgres://… pnpm --filter
@objectstack/cli exec vitest run --project integration
src/utils/schema-migration-lock.integration.test.ts`,预期 8/8
通过,三个启动各打印一行等待、每张表只建一次。
- 需要你做的:无,席位落地。
## #31598 card item 2 (quoted)
> 2. The lock is harmless on a single node (uncontended); keep its
output identical apart from no new lines when uncontended. Note: this
changes the single-node code path by one uncontended lock call; say so
in the PR.
As there: this PR changes the single-node server boot by one uncontended
lock call (`pg_try_advisory_lock` / `GET_LOCK` on a connection of its
own, then the unlock), on Postgres and MySQL only. Nothing is printed
when it is uncontended; the single-boot output was diffed byte for byte
below.
## What changes
- **Where the boot syncs, measured.** A server boot's DDL is
`ObjectQLPlugin.start()` Phase 1 (`installRegisteredSchemas`), Phase 2
(hydrate from `sys_metadata`) and Phase 3 (sync what Phase 2 added).
`OS_MIGRATE_AND_EXIT=1` is that same boot followed by an exit in
`serve.ts`, so the lock goes into the engine's boot, not into
`serve.ts`. **`serve.ts` is untouched.**
- **The lock.** `start()` takes it right before Phase 1 and releases it
in `finally` after Phase 3 (a boot that fails between the phases
releases it too). It is never held while the server runs. It is taken
even when nothing turns out to be pending, so the "read what is missing,
then create it" race is closed. It is taken on the default datasource's
driver, the database the tables live in (as #31714 locks the primary).
- **No lock where this boot runs no DDL:** `skipSchemaSync` /
`OS_SKIP_SCHEMA_SYNC=1`, and a default driver whose DDL `os migrate
plan` / `apply` holds back. That command takes the lock itself before it
reads what is pending; taking it in the boot too would make one migrator
wait twice and spend a second uncontended statement. The engine reads
this from a new read-only getter, `SqlDriver#schemaDdlDeferred`.
- **One lock, one module (seat ruling 1).** `schema-migration-lock.ts`
and its unit test moved from `packages/cli/src/utils/` to
`packages/objectql/src/`: `@objectstack/objectql` cannot import
`@objectstack/cli` (the CLI depends on it), and objectql is the lowest
package that holds a caller of the lock (`@objectstack/core` / `types`
are driver-agnostic and call nothing here). It is exported from the
objectql barrel (not the lean `./core` entry).
`packages/cli/src/utils/schema-migration-lock.ts` is now a re-export, so
`bootSchemaStack` and the CLI tests import exactly what they did. Same
name `objectstack:schema-migrate`, same Postgres key (its unit pin moved
with it, unchanged).
- **Dialect from the driver, before its Knex (CI fix, `ab5e5910`).** The
lock reads `SqlDriver#dialectName` (the driver's one public "which SQL
do I speak") and calls `getKnex()` only for `postgres` / `mysql`. At
`86ea712b` it asked every driver for its Knex first, and the Turso
REMOTE transport refuses `getKnex()` by throwing, so a remote Turso boot
failed in `ObjectQLPlugin.start()` (CI `Test Core (3/8)`,
`organization-ownership-boot-guard.turso-remote.kernel.test.ts`). A
Postgres or MySQL driver whose Knex cannot be handed out still throws;
nothing is swallowed. `bootSchemaStack` passes the same `SqlDriver`, so
the CLI path reads the same answer.
- **Wait bound.** A server boot waits at most half of
`ObjectQLPlugin.startupTimeout` (60 s by default), so a boot queued
behind a long migration refuses with the lock's own message (it names
the lock and says nothing was changed) instead of the kernel's generic
start timeout, and keeps the other half for its own sync. `os migrate
apply` keeps its 10-minute bound.
- **Forced edit outside the named surface:**
`packages/drivers/driver-turso/src/turso-driver.ts` gains one row in
`REMOTE_FACE_ANSWERS` (`schemaDdlDeferred: 'inherited'`). That ledger is
typed `Record<keyof SqlDriver, …>`, so the new getter fails the Turso
DTS build until it is classified. It is `inherited` because the remote
face refuses arming, so it always reads `false`.
## Tests
- `packages/objectql/src/plugin-boot-schema-migration-lock.test.ts`
(new, 6 pins): a real `ObjectKernel` boot with a stub default driver
whose knex and `syncSchema` write into one ordered log. The lock comes
before the first DDL and the unlock after the last; uncontended it is
one lock statement and no line logged; contended it is one wait line,
then DDL; a boot failing between the phases still unlocks; no lock for
`skipSchemaSync` or a deferring driver; control: SQLite and memory
drivers run no statement and sync as before.
- `packages/cli/src/utils/schema-migration-lock.integration.test.ts`: a
second top-level describe in #31714's live-Postgres harness (ruling
record 6583487846: one harness file per component). It holds the lock,
starts three `os serve` boots with `OS_MIGRATE_AND_EXIT=1`, waits until
Postgres shows all three queued, then lets go. Asserted: all three
queued; each exits 0 and prints exactly one wait line; none logs `Schema
sync FAILED`; an event trigger sees each table created once; control:
one more boot prints no wait line, creates nothing, exits 0. A named
skip without `OS_TEST_POSTGRES_URL`, as the apply cell; the event
trigger needs a superuser URL.
- Moved: `schema-migration-lock.test.ts` (header text updated for its
second caller). Its fake drivers now carry `dialectName`; the dialect
pins assert `getKnex()` is never called for `sqlite` / `mssql` /
`oracle` / `unknown` / none; new pins: a SQLite face whose `getKnex()`
throws answers `null` without calling it, and a `postgres` / `mysql`
driver whose `getKnex()` throws rejects with that error.
- `plugin-boot-schema-migration-lock.test.ts` gains a 7th pin: a
SQLite-dialect default driver whose `getKnex()` throws (Turso remote)
boots, is never asked for Knex, takes no lock and syncs.
## Acceptance notes
- Postgres and MySQL only, as #31714. MySQL is NOT MEASURED live (no
MySQL server in the container); the MySQL statements are the module's
own, pinned in its unit test.
- **The lock covers the default datasource only; DDL on any other
datasource is unchanged by this PR.** Only the default datasource's
database is locked (as #31714 locks the primary); objects bound to
another SQL datasource on another database sync exactly as before,
outside the lock.
- The `kernel:ready` platform-table repairs (`metadata-protocol`
identity-index migrations, `IF [NOT] EXISTS` index DDL) run after the
lock is released. In the 5 barrier-aligned runs and 2 staged integration
runs with the fix, no boot logged a failure from them. The runtime
resyncs (`metadata:reloaded`, `ObjectQL.syncSchemas` on install) are not
boot DDL and are not touched.
- The red on main needs the boots aligned. Started together without
help, tsx start jitter puts one boot about 100 ms ahead and it leads the
whole sync: 4 unaided runs on main showed no collision. Held together at
their first database round trip by a TCP barrier in front of Postgres (a
measurement tool, not committed), main collided every valid time.
## Verification
Concurrency measurements at `3fa9769b` (main `4013e308` merged in),
Postgres 16 in the container. `86ea712b` moved a JSDoc (no code change).
Head `ab5e5910` fixes the Turso remote boot:
```text
runtime organization-ownership-boot-guard.turso-remote.kernel.test.ts before (objectql dist of 86ea712): 2 failed, "Handing out the Knex instance (`getKnex()`) is not supported by the Turso REMOTE transport" / after (rebuilt): 2 passed
runtime turso files 2/2 (9 tests) · rest remote-tables-twin 10/10 · cli schema-migrate.remote-posture 11/11
objectql typecheck ✓ · vitest 577 files / 11736 tests passed (lock pins 26/26)
driver-turso typecheck ✓ · vitest 97 files, 2522 passed | 34 skipped
cli typecheck ✓ · live-PG schema-migration-lock.integration.test.ts 8/8 (OS_TEST_POSTGRES_URL set)
eslint --no-inline-config on the 4 changed .ts files: exit 0
```
<details><summary>Red on main (b126c13), green with the fix: concurrent
boots on one fresh database</summary>
DDL counted from the Postgres log (`log_statement = ddl`); boots aligned
by the barrier.
```text
main 3 x OS_MIGRATE_AND_EXIT run A CREATE TABLE 85 / distinct 84 / created twice: 1 (sys_migration) / pg_type_typname_nsp_index duplicate key x1 / all exit 0, one boot logged "Schema sync FAILED for object 'sys_migration'"
main 3 x OS_MIGRATE_AND_EXIT run B CREATE TABLE 85 / distinct 84 / created twice: 1 / pg_type collision x1 / all exit 0
main 2 x ordinary boot run A created twice: 1 / pg_type collision x1
main 2 x ordinary boot run B created twice: 1 / pg_type collision x1
(a third main run was void: the barrier's own port was taken; both boots failed to connect)
fix 3 x OS_MIGRATE_AND_EXIT 2 runs CREATE TABLE 84 / distinct 84 / created twice: 0 / no pg_type error / all exit 0 / 2 wait lines per run (one per waiter)
fix 2 x ordinary boot 3 runs CREATE TABLE 84 / created twice: 0 / no pg_type error / both listening / 1 wait line per run
```
</details>
<details><summary>Control: single-boot output unchanged</summary>
Timestamps, durations, ports, temp paths and database names normalized;
nothing else.
```text
single OS_MIGRATE_AND_EXIT boot, main vs fix: stdout IDENTICAL (0 lines), stderr IDENTICAL (11 lines)
single ordinary boot, main vs fix: stdout IDENTICAL (0 lines), stderr IDENTICAL (32 lines)
```
</details>
<details><summary>Ablation: the lock call removed from
ObjectQLPlugin.start()</summary>
`scripts/ablation-replace.mjs` replaced `const migrationLock = await
this.acquireBootSchemaMigrationLock(ctx);` with `await
Promise.resolve(null)` (anchor 1 → 0, planted 1). Rebuilt objectql: the
build exits 1 because the DTS step rejects the now-unused private
method, but the JS was written: `ablation-dist-preflight.mjs … --absent`
shows the call absent from all 8 built files.
```text
unit × locks before the first DDL… × uncontended: ONE lock statement… × contended: says ONCE… × a boot that fails… (4 failed | 2 passed)
live × all three boots queued behind the held lock (expected +0 to be 3) × each boot … says ONCE that it waited (2 failed | 6 passed)
restore: blob == HEAD, git diff HEAD empty; rebuilt; preflight finds the call in dist/index.js and dist/index.mjs; live file 8/8 passed
```
</details>
<details><summary>Packages, lint, gates</summary>
```text
objectql typecheck ✓ · vitest 577 files / 11732 tests passed (pre-merge head e02afaa); after the main merge: typecheck ✓, the two lock files 22/22
driver-sql typecheck ✓ · vitest 246 passed | 13 skipped files, 4170 passed | 223 skipped tests
driver-turso typecheck ✓ · vitest 97 files, 2522 passed | 34 skipped
cli typecheck ✓ (both heads) · unit tier 406 files / 5755 tests · live-PG file 8/8 (both heads; OS_TEST_POSTGRES_URL set)
eslint --no-inline-config on all 9 changed .ts files: exit 0
dispatch-gates --commands (79 derived on 3fa9769): 79 run, 79 exit 0; --ran reconciliation: 79 derived, 79 run, 0 NOT-MEASURED, 0 UNRUN
(first round: teardown self-test, dual-build-cjs-loads and i18n-coverage exited 3 on prerequisites; green after fetching the teardown fixture and a full `pnpm build`)
```
</details>
<!-- signed: domain:server#2 · os-steve ·
session_01ADHkmRuoKNTbmDhQPaG2vW -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
fix(objectql): a server boot's schema sync takes the same schema-migration lock as `os migrate apply`, so replicas starting together create each table once
8
+
9
+
Clause-②: no
10
+
Metadata-protocol: none
11
+
12
+
Several servers booting against one Postgres or MySQL database at the same time (replicas of one deployment, or several `OS_MIGRATE_AND_EXIT=1` jobs) each read the same missing tables and each created them. Postgres then refused one of the duplicate `CREATE TABLE` statements with a `pg_type` unique-key collision, and the boot that lost logged `Schema sync FAILED` for that object while still exiting 0. The boot's schema sync now takes the schema-migration lock that `os migrate apply` takes (one name, `objectstack:schema-migrate`, and one Postgres key), on the default datasource's database, and releases it as soon as the sync is done. It is never held while the server runs. A boot that finds another migration running prints one line saying it waits, then finds the tables already there.
13
+
14
+
On a single node nothing changes: the lock is uncontended and prints nothing, and SQLite (with Turso and WASM SQLite) takes no lock at all: the lock reads the dialect from the driver's `dialectName` and never asks a SQLite-dialect driver for its Knex, so a remote Turso transport, which refuses to hand its Knex out, boots as before. A boot with `skipSchemaSync` / `OS_SKIP_SCHEMA_SYNC=1` takes none either, since it runs no DDL. A waiting boot gives up after half of the engine plugin's start budget (60 s by default), changes nothing and fails the boot with a message naming the lock; start it again once the other migration has finished.
15
+
16
+
The lock now lives in `@objectstack/objectql` (`acquireSchemaMigrationLock`, `SCHEMA_MIGRATION_LOCK_NAME`), so the engine's boot can take it; `@objectstack/cli` keeps importing it from there. `SqlDriver` gains a read-only `schemaDdlDeferred` getter, which tells the engine's boot that `os migrate plan` / `apply` is holding the driver's DDL back and will take the lock itself.
0 commit comments