Skip to content

Commit d7a7193

Browse files
fix(cli): os migrate apply takes a database lock so concurrent migrators run one at a time (#31714)
<!-- claude-projects-attribution: {"github_login":"os-steve"} --> _Requested by **Steve** · [project thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9EtLz4Q3mytsuVUjBvqeWPK)_ Fixes #31598 Clause-②: no Metadata-protocol: none ## 维护者速读 - **改了什么**:`os migrate apply` 在真正读取待办表结构变更之前,先在主库上拿一把数据库级的会话锁(Postgres `pg_advisory_lock`、MySQL `GET_LOCK`,锁名固定为 `objectstack:schema-migrate`),命令结束时释放;SQLite 不加锁。 - **为什么改**:多副本部署时每个 Pod 的 init 容器都会跑一次迁移。实测 3 个迁移同时打同一个 Postgres:三个都报告"建了表"(82 + 86 + 37 张,库里实际 86 张),第三个还在别人建到一半的表上做了 drift 变更,而且全部退出码 0。 - **风险与回滚**:单机路径只多一条无竞争的加锁语句,输出逐字节不变(实测)。有竞争时多打印一行"正在等待",最多等 10 分钟,超时则拒绝、不做任何变更并以非零退出。回滚即 revert 本 PR。 - **先看**:① `packages/cli/src/utils/schema-migration-lock.ts` 的 `acquireSchemaMigrationLock`;② `packages/cli/src/utils/schema-migrate.ts` 中 `bootSchemaStack` 里 `migrationLock` 的取锁点与 `shutdown()` 的释放;③ `schema-migration-lock.integration.test.ts`。 - **两分钟验证**:有本地 Postgres 时运行 `OS_TEST_POSTGRES_URL=postgres://… pnpm --filter @objectstack/cli exec vitest run src/utils/schema-migration-lock.integration.test.ts`,预期 4/4 通过:两个迁移排队等锁、只有一个建表、另一个报告 `in_sync`。 - **需要你做的**:无,席位落地 ## The one exception to R3 this card accepts R3 on #3814 (「同意:每张卡都只在"声明多副本并接上 Redis 驱动"时才生效」) keeps the single-node default path unchanged. This card names the one exception it accepts, in its item 2: > 2. The lock is harmless on a single node (uncontended); keep its output identical apart from no new lines when uncontended. Note: this changes the single-node code path by one uncontended lock call; say so in the PR. So, stated plainly: **on a single node, every applying `os migrate apply` now runs one extra statement** (`select pg_try_advisory_lock(…)` on Postgres, `select GET_LOCK(…, 0)` on MySQL) on a connection of its own, plus the matching unlock at the end. It is not gated behind declared replicas or a Redis driver, because the collision happens before any replica count is known to the migrator. Uncontended it prints nothing. Measured: one `os migrate apply --yes` on a fresh Postgres database, human face, gives byte-identical stdout and stderr on `origin/main` and on this diff over the same base once ISO timestamps and durations are normalized (stdout sha256 `8980fba0…1eda0`, 41,920 bytes, both sides; stderr empty, both sides). SQLite (and the Turso and WASM drivers on it) runs no statement at all. ## What changes - **`packages/cli/src/utils/schema-migration-lock.ts` (new).** `acquireSchemaMigrationLock(driver, { database, warn, waitMs? })`. It reads the dialect from the driver's own knex client: `postgresql` takes `pg_advisory_lock`, `mysql` takes `GET_LOCK`, and every other dialect answers `null` and runs nothing. The flow is a no-wait try first (silent when it succeeds); on contention it prints ONE line naming the database and waits, bounded (Postgres `lock_timeout`, MySQL `GET_LOCK`'s seconds argument). If the wait runs out it throws a refusal naming the database, and the run migrates nothing. `release()` unlocks on the same session, then closes it. It is idempotent and never throws. Its own module, importing only `node:crypto`, so its pins load the lock and not the migration stack (`check:test-source-alias` asked for exactly that). - **The key.** `SCHEMA_MIGRATION_LOCK_NAME = 'objectstack:schema-migrate'`, named once. The Postgres key is derived from it: the first 8 bytes of SHA-256 as a signed 64-bit integer, `255402229896085448`. The unit file pins the value, because a changed key would let an old and a new CLI migrate side by side during a rolling upgrade. It is not per-database: a Postgres advisory lock is already scoped to its database. MySQL's `GET_LOCK` is server-wide, see Acceptance notes. - **The bound.** `SCHEMA_MIGRATION_LOCK_WAIT_MS = 10 * 60_000`, a constant. No environment override exists for migration timeouts, so none is added. - **`packages/cli/src/utils/schema-migrate.ts`, `bootSchemaStack`.** A boot that can apply (a deferred boot that is not a read-only probe) takes the lock on the primary right after `runtime.start()`, before the coverage pass and before the first `previewDeferredSchemaWork()`. That boot held all its DDL back, so nothing has been written yet. A migrator that waited then reads only the work the first one left, which is none. `shutdown()` releases the lock first, in a `finally`. If one of the three post-lock reads (coverage, preview, package-table plan) throws, `underLock` releases it before the throw leaves. Which commands take it: `os migrate apply`, plus the `--apply` modes that boot deferred, `os migrate organization-ownership --apply` and `os migrate multi-value-columns --apply`. `os migrate plan` and every other read-only probe take none. - **The session.** The lock rides a connection opened through the driver's own client (its configuration, credentials and TLS) but outside its pool, via knex `acquireRawConnection` / `destroyRawConnection`. See Acceptance note 1 for why not a pooled connection. - **Changeset.** `@objectstack/cli` patch, with the claim's two lines at line start. ## Tests - `packages/cli/src/utils/schema-migration-lock.test.ts` (new, unit, 16 cases, fake knex recording every statement and the session it ran on). The key value is pinned. SQLite / MSSQL / Oracle / no dialect / no knex answer `null` with no connection opened and no line. Postgres and MySQL each cover: uncontended, one statement and no line; release on the same session, once; contended, exactly one line naming the database, then the bounded wait (`lock_timeout` in ms, `GET_LOCK` in whole seconds); a wait that runs out refuses and closes the session. MySQL `GET_LOCK` answering `0` or `NULL` refuses. Any other wait failure is handed on unchanged. - `packages/cli/src/utils/schema-migration-lock.integration.test.ts` (new, integration tier, live Postgres via `OS_TEST_POSTGRES_URL`, named skip without it). The contention is staged, not raced. The test holds the lock on its own session, spawns two `os migrate apply --yes --json` (`bin/run-dev.js`), waits until `pg_locks` shows both queued (an advisory lock not granted), then releases. Assertions: both queued (2); both exit 0 and each printed the wait line exactly once, on stderr, with stdout still one JSON document; exactly one created the schema and the other answered `in_sync` with nothing created; the table exists. A third, uncontended run is the control: no wait line, `in_sync`. - No forced test edits. ## Acceptance notes 1. **Seat ruling 1 said "on the SAME connection the migration uses (a pooled second connection would not serialize)".** Measured otherwise, and taken by its intent. Both locks are session-scoped and exclude every other session on the database, the migration's own pooled ones included. The integration pin shows a lock held on a session outside both migrators' pools serializing them. What a pool breaks is the release: knex's pool gives no promise that the unlock runs on the session that locked, its idle reaper can close a held connection (silently releasing the lock), and an `OS_DATABASE_POOL_MAX=1` pool would have no connection left for the migration itself, so it would deadlock until the acquire timeout. So the lock takes its own session through the driver's own client and configuration. The seat may re-rule. 2. **Where the init container's command comes from.** `helm/` lives in the cloud repository, which this session cannot read. The card names `schema-migrate.ts`, whose `bootSchemaStack` is the `os migrate apply` path, so that is what is locked. If the chart's init container runs a SERVING boot with `OS_MIGRATE_AND_EXIT=1` (`os serve`, `packages/cli/src/commands/serve.ts`) instead, its boot-time schema sync does not go through `bootSchemaStack` and this lock does not cover it. The same holds for any replica's own serving boot, whose additive sync runs at start. carrier: the cloud C2 card (the Helm hook Job), which decides which command the Job runs; noted, not filed. 3. **The non-deferred `--apply` data commands** (`os migrate meta --stored --apply`, `value-shapes --apply` and the others that boot plain) run their schema sync inside `runtime.start()`, before any point a lock could be taken without adding a kernel plugin. A plugin would add three INFO lines to every boot and break the byte-identical control. They are not locked. They are data repairs, not the schema migration this card names. 4. **MySQL.** `GET_LOCK` is server-wide, not per-database: two migrations of different databases on one MySQL server now run one after the other. That is harmless, only slower, and the seat ruled out per-database-name keys. NOT MEASURED live: the container has no MySQL server. The statements and result shapes are pinned in the unit file against mysql2's `[rows, fields]` shape. 5. **MSSQL / Oracle are out of scope, measured.** The CLI's primary database never is one: `resolveStandaloneDatabase` refuses their URL schemes ("Supported schemes: postgres://, pg://, mysql://, mysql2://, mongodb://, mongodb+srv://, libsql://, file:"). A declared datasource can be MSSQL, but the lock lives on the primary, which every migrator of the deployment shares. 6. **The refusal carries no error code.** It is a plain `Error` whose message names the database and the remedy, and `os migrate apply` prints it and exits 1 the way it handles any boot failure. A new ADR-0112 code would be a new public name, so it is listed as an open question for the seat rather than coined. ## Verification Head `53aea71cf3cae9820396e0642854ee88cf3694db`, a clean merge with `origin/main` at `b126c1306`. Every reading below the ablations is on that head. <details> <summary>Red on main, green after: three concurrent os migrate apply against one fresh Postgres 16 database</summary> ```text origin/main 5d1382a, run 1: migrator 0 created 82 · migrator 1 created 86 · migrator 2 created 37, applied 3 drift · all exit 0 · 86 tables origin/main 5d1382a, run 2: created 12 · 86 · 41 · all exit 0 · 86 tables origin/main 5d1382a, 2 migrators: created 86 · 85 · both exit 0 fix c8de894, run 1: created 86 · 0 (in_sync) · 0 (in_sync) · all exit 0 · 86 tables fix c8de894, run 2: created 0 (in_sync) · 86 · 0 (in_sync) · one printed "Another schema migration is running on postgres://127.0.0.1:5432/…; waiting up to 600 s for it to finish…" ``` </details> <details> <summary>CONTROL: one uncontended migration, byte-identical to main</summary> ```text origin/main 5d1382a os migrate apply --yes (human face), fresh PG db: exit 0 · stdout 41,920 bytes sha256 8980fba0d8e1fb6a50c5511b88ede883c1d3f3465fadafca069c730fb0b1eda0 · stderr 0 bytes ba38138 (this diff on that base) same command, same fixture: exit 0 · stdout 41,920 bytes sha256 8980fba0d8e1fb6a50c5511b88ede883c1d3f3465fadafca069c730fb0b1eda0 · stderr 0 bytes (normalized: ISO timestamps -> TS, durations -> N ms; main ran twice first and was self-identical) 53aea71 (after the second main merge): one line differs from the 5d1382a reading, and it is main's, not this diff's: "[platform-objects] contributed platform translations (20 locale entries)" -> "(22 locale entries)", from b126c13 (#31588, the platform objects in Korean). No wait line and no other change. ``` </details> <details> <summary>Ablations (scripts/ablation-replace.mjs, each restored to the HEAD blob, git diff HEAD empty) on ba38138</summary> ```text A1 lock never taken (`false as boolean && deferral …` in bootSchemaStack) -> integration 3 failed | 1 passed: "both migrators queued" expected +0 to be 2; "applied ONCE" expected 2 creators to have a length of 1 A2 Postgres 55P03 proceeds instead of refusing -> unit 1 failed | 15 passed ("a wait that runs out REFUSES") A3 MySQL GET_LOCK 0/NULL proceeds -> unit 2 failed | 14 passed (both GET_LOCK refusal cases) A4 the wait line printed even uncontended -> unit 4 failed | 12 passed (both "NOT ONE line printed", both "ONE line") A5 dialect gate removed (SQLite etc. take a lock) -> unit 4 failed | 12 passed (sqlite3 / mssql / oracle / undefined) ``` </details> <details> <summary>Tests and gates on 53aea71 (after pnpm install --frozen-lockfile and a full pnpm build, 119/119)</summary> ```text pnpm --filter @objectstack/cli typecheck tsc green · check:test-typecheck OK (3 files / 28 errors / 6 pinned signatures held, unchanged) cli unit tier (vitest --project unit) Test Files 406 passed (406) · Tests 5740 passed (5740) schema-migration-lock.integration.test.ts (live PG) Tests 4 passed (4) (integration tier run locally because this diff adds an integration file) eslint packages/cli --no-inline-config exit 0, no findings dispatch-gates --commands 67 derived families, each run: 67 exit 0 dispatch-gates --ran 67 derived, 67 run, 0 NOT-MEASURED, 0 UNRUN first round on ba38138 (before the second merge): 67/67 exit 0 after deepening for the teardown self-test fixture (exit 3 on the shallow clone, a container prerequisite) earlier red, fixed in this PR: check:test-source-alias flagged both new test files for reaching the migration stack's dynamic imports; the lock moved to its own module (ba38138) ``` </details> --- _Generated by [Claude Code](https://claude.ai/code/session_01ADHkmNs71AKtAebnmL7SxF)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 7200f54 commit d7a7193

5 files changed

Lines changed: 660 additions & 4 deletions

File tree

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
'@objectstack/cli': patch
3+
---
4+
5+
fix(cli): `os migrate apply` takes a database lock, so concurrent migrators against one database run one at a time
6+
7+
Clause-②: no
8+
Metadata-protocol: none
9+
10+
Two `os migrate apply` runs against the same Postgres or MySQL database (for example one per replica, from an init container on each pod) both read the same pending schema work and both performed it. Each applying run now takes a session-level advisory lock on the primary database (Postgres `pg_advisory_lock`, MySQL `GET_LOCK`, named `objectstack:schema-migrate`) after its DDL-free boot and before it reads what is pending, and releases it when the command ends. A second migrator waits for the first, then finds the schema in sync and changes nothing.
11+
12+
On a single node nothing changes: the lock is uncontended and prints nothing, and SQLite (with Turso and WASM SQLite) takes no lock at all. A migrator that has to wait prints one line saying so. If the other migration still holds the lock after 10 minutes, the waiting run refuses, changes nothing and exits non-zero; run it again once the other migration has finished.

‎packages/cli/src/utils/schema-migrate.ts‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ import {
6666
} from '../commands/doctor.js';
6767
import { describeDriverConnection, describeDriverSqliteFile } from './connection-display.js';
6868
import { reserveStdoutForJson } from './json-stdout.js';
69+
import { acquireSchemaMigrationLock } from './schema-migration-lock.js';
6970
import {
7071
buildSchemaMigrationPlugins,
7172
measureComposedCoverage,
@@ -1469,6 +1470,21 @@ export async function bootSchemaStack(
14691470
: [];
14701471
const plannedDrivers: SqlDriverLike[] = [...servedDrivers, ...declaredDrivers.map((d) => d.driver)];
14711472

1473+
// [#31598] One migrator per database at a time. A boot that can apply — a
1474+
// deferred boot that is not a read-only probe, as `os migrate apply` is —
1475+
// takes the schema-migration lock on the primary HERE: after a boot that held
1476+
// its DDL back, before the first read of what is pending, so a migrator that
1477+
// waited reads only the work the other one left. Released by `shutdown()`,
1478+
// and by {@link underLock} when this boot throws after taking it.
1479+
const migrationLock = deferral && opts.readOnlyProbe !== true
1480+
? await acquireSchemaMigrationLock(driver, { database: describeDb(driver), warn: (line) => console.warn(line) })
1481+
: null;
1482+
const underLock = <T>(work: Promise<T>): Promise<T> =>
1483+
work.catch(async (error: unknown) => {
1484+
await migrationLock?.release();
1485+
throw error;
1486+
});
1487+
14721488
// #13028 — the composed host declared its objects in `init()`; the pass that
14731489
// hands them to their driver lives in `ObjectQLPlugin.start()`, which the
14741490
// declaration-phase composition suppressed (and which a host bringing its
@@ -1488,7 +1504,7 @@ export async function bootSchemaStack(
14881504
);
14891505
}
14901506
if (composedHost) {
1491-
const measured = await measureComposedCoverage(kernel, plannedDrivers, defer, { ddlFree: skipSchemaSync });
1507+
const measured = await underLock(measureComposedCoverage(kernel, plannedDrivers, defer, { ddlFree: skipSchemaSync }));
14921508
composition.coverage = measured.coverage;
14931509
composition.notes.push(...measured.notes);
14941510
}
@@ -1501,7 +1517,7 @@ export async function bootSchemaStack(
15011517
// [#21391] Every datasource the deferral armed, not only the default's.
15021518
const pendingSchemaWork: PendingSchemaWork[] = [];
15031519
for (const d of deferral?.drivers ?? []) {
1504-
if (d.previewDeferredSchemaWork) pendingSchemaWork.push(...(await d.previewDeferredSchemaWork()));
1520+
if (d.previewDeferredSchemaWork) pendingSchemaWork.push(...(await underLock(d.previewDeferredSchemaWork())));
15051521
}
15061522
// [#23022] …and the one table outside the object set, `sys_packages`, which
15071523
// no driver deferral can see. Read-only here; `flushSchemaDdl` creates it.
@@ -1510,7 +1526,7 @@ export async function bootSchemaStack(
15101526
// included: `nonObjectTables` below names the table from it whether or not
15111527
// anything is planned for it.
15121528
const packageStorage = findPackageStorage(kernel);
1513-
let packageTable = defer && packageStorage ? await planPackageTable(packageStorage, composition.notes) : null;
1529+
let packageTable = defer && packageStorage ? await underLock(planPackageTable(packageStorage, composition.notes)) : null;
15141530
if (packageTable) pendingSchemaWork.push(packageTable.work);
15151531
sortPendingSchemaWork(pendingSchemaWork);
15161532
// [#21529] The tables the held-back sync would create: measured absent.
@@ -1650,7 +1666,13 @@ export async function bootSchemaStack(
16501666
* `destroy()` closes the ones it owns); a second disconnect is a no-op.
16511667
*/
16521668
shutdown: async () => {
1653-
deferral?.release();
1669+
// [#31598] The migration lock first, in `finally`: whatever the rest of
1670+
// the teardown does, the next migrator waiting on it may go.
1671+
try {
1672+
deferral?.release();
1673+
} finally {
1674+
await migrationLock?.release();
1675+
}
16541676
try { await kernel.shutdown(); } catch { /* teardown is best-effort */ }
16551677
// [#22579] The sibling is a `DriverPlugin` driver, which owns no teardown
16561678
// of its own: the same backstop as the primary's.
Lines changed: 266 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* Two `os migrate apply` runs against ONE Postgres database serialize on the
5+
* schema-migration lock (`acquireSchemaMigrationLock`, taken by
6+
* `bootSchemaStack` on an applying boot): one waits, both succeed, the schema
7+
* is applied once and the second finds nothing to do.
8+
*
9+
* ## What this inverts, measured before the lock
10+
*
11+
* Three concurrent applies against one fresh Postgres database each reported
12+
* creating tables — 82, 86 and 37 for the 86 the database ended with — and the
13+
* third applied drift on tables the others were still creating. Every run
14+
* exited 0, so no exit status carried the collision.
15+
*
16+
* ## Why the contention is staged, not raced
17+
*
18+
* Two migrators started together contend only if their boots overlap, which
19+
* is timing. So this file takes the lock FIRST, on a session of its own, waits
20+
* until Postgres reports both migrators queued behind it (`pg_locks`, an
21+
* advisory lock not yet granted), and only then lets go. Both then wait by
22+
* construction, and the one that gets the lock second must read the database
23+
* the first one left. The third run is the control: uncontended, it prints
24+
* no wait line and finds the schema in sync.
25+
*
26+
* ## The driver axis
27+
*
28+
* Postgres only, and only when `OS_TEST_POSTGRES_URL` is set, in a database
29+
* named from this file's workspace-relative path (the `os_lv_` derivation every
30+
* live suite shares). Without the URL the cell is a NAMED skip, and a FAILURE
31+
* under `OS_EXPECT_LIVE_DIALECT_MATRIX=1`. ⚠️ A named skip is a report, not
32+
* coverage: no CI leg supplies the URL to this package today. SQLite takes no
33+
* lock, which the unit file pins.
34+
*/
35+
36+
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
37+
import { execFile } from 'node:child_process';
38+
import { createHash } from 'node:crypto';
39+
import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
40+
import { tmpdir } from 'node:os';
41+
import { basename, dirname, join, resolve } from 'node:path';
42+
import { fileURLToPath } from 'node:url';
43+
import { SqlDriver } from '@objectstack/driver-sql';
44+
import { SCHEMA_MIGRATION_PG_LOCK_KEY } from './schema-migration-lock.js';
45+
46+
const HERE = dirname(fileURLToPath(import.meta.url));
47+
/** The source entry, as `test/helpers/serve-process.ts` spawns it; `src/` cannot import that helper. */
48+
const CLI = resolve(HERE, '../../bin/run-dev.js');
49+
50+
/** Env that would point the child's boot somewhere other than this file's database. */
51+
const OVERRIDING_ENV = ['OS_DATABASE_URL', 'DATABASE_URL', 'TURSO_DATABASE_URL', 'OS_DATABASE_DRIVER', 'OS_HOME'] as const;
52+
53+
/**
54+
* This process's environment for a child, minus the families
55+
* `test/helpers/serve-process.ts` `childEnv()` strips (its header says why).
56+
*/
57+
function childEnv(): Record<string, string | undefined> {
58+
const env: Record<string, string | undefined> = {};
59+
for (const [key, value] of Object.entries(process.env)) {
60+
if (key === 'TEST' || key === 'VITEST' || key.startsWith('VITEST_') || key === 'NODE_PATH') continue;
61+
if ((OVERRIDING_ENV as readonly string[]).includes(key)) continue;
62+
env[key] = value;
63+
}
64+
// The fixed key `serve-process.ts` hands its children, so no boot mints and
65+
// persists a crypto key into this runner's home directory.
66+
return { ...env, OS_SECRET_KEY: '0e2e'.repeat(16), NO_COLOR: '1' };
67+
}
68+
69+
const PG_URL = process.env.OS_TEST_POSTGRES_URL;
70+
const EXPECT_LIVE_DIALECTS = process.env.OS_EXPECT_LIVE_DIALECT_MATRIX === '1';
71+
/** Three CLI boots, two of them queued behind the staged lock. */
72+
const CELL_TIMEOUT_MS = 240_000;
73+
const LIVE_DB_PREFIX = 'os_lv_';
74+
75+
/**
76+
* This file's live database: `os_lv_<slug>_<12 hex of sha256(path)>`, the
77+
* derivation `packages/drivers/driver-sql/src/live-dialect-matrix.testkit.ts`
78+
* uses. A copy rather than an import: that testkit is not on driver-sql's
79+
* public surface.
80+
*/
81+
function liveDatabaseNameFor(testFileKey: string): string {
82+
const key = testFileKey.replace(/\\/g, '/');
83+
const slug = basename(key)
84+
.replace(/\.test\.tsx?$/, '')
85+
.toLowerCase()
86+
.replace(/[^a-z0-9]+/g, '_')
87+
.replace(/^_+|_+$/g, '')
88+
.slice(0, 34);
89+
const hash = createHash('sha256').update(key).digest('hex').slice(0, 12);
90+
const name = `${LIVE_DB_PREFIX}${slug}_${hash}`;
91+
if (!/^[a-z][a-z0-9_]*$/.test(name) || name.length > 63) {
92+
throw new Error(`live-dialect isolation: derived an unusable database name ${JSON.stringify(name)}`);
93+
}
94+
return name;
95+
}
96+
97+
/** The test path relative to the workspace root, so a worktree and CI derive the same name. */
98+
function repoRelativeTestPath(absolutePath: string): string {
99+
let dir = dirname(absolutePath);
100+
for (;;) {
101+
if (existsSync(join(dir, 'pnpm-workspace.yaml'))) return absolutePath.slice(dir.length + 1);
102+
const parent = dirname(dir);
103+
if (parent === dir) return basename(absolutePath);
104+
dir = parent;
105+
}
106+
}
107+
108+
function liveDatabase(): string {
109+
const testPath = expect.getState().testPath;
110+
if (!testPath) {
111+
throw new Error('live-dialect isolation: vitest reported no testPath, so this cell cannot be given a database of its own.');
112+
}
113+
return liveDatabaseNameFor(repoRelativeTestPath(testPath));
114+
}
115+
116+
const LIVE_DATABASE = liveDatabase();
117+
118+
function liveUrl(): string {
119+
const url = new URL(PG_URL!);
120+
url.pathname = `/${LIVE_DATABASE}`;
121+
return url.toString();
122+
}
123+
124+
async function adminExecute(sql: string): Promise<void> {
125+
const admin = new SqlDriver({ client: 'pg', connection: PG_URL });
126+
try {
127+
await admin.execute(sql);
128+
} finally {
129+
await admin.disconnect();
130+
}
131+
}
132+
133+
interface Run {
134+
code: number;
135+
stdout: string;
136+
stderr: string;
137+
}
138+
139+
function migrate(cwd: string): Promise<Run> {
140+
return new Promise((resolvePromise) => {
141+
execFile(
142+
process.execPath,
143+
[CLI, 'migrate', 'apply', '--yes', '--json', '--database-url', liveUrl()],
144+
{ cwd, maxBuffer: 64 * 1024 * 1024, env: childEnv() },
145+
(err, stdout, stderr) => {
146+
resolvePromise({
147+
code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0,
148+
stdout: String(stdout),
149+
stderr: String(stderr),
150+
});
151+
},
152+
);
153+
});
154+
}
155+
156+
interface ApplyPayload {
157+
message?: string;
158+
created?: unknown[];
159+
}
160+
161+
const WAIT_LINE = /^.*Another schema migration is running on .*$/gm;
162+
const CONFIG = [
163+
'export default {',
164+
" name: 'migration_lock_31598',",
165+
" label: 'Migration lock',",
166+
" objects: [{ name: 'ml_ticket', label: 'Ticket', fields: { title: { type: 'text', label: 'Title' } } }],",
167+
'};',
168+
'',
169+
].join('\n');
170+
171+
if (!PG_URL) {
172+
describe('os migrate apply serializes on the schema-migration lock (live postgres)', () => {
173+
it.skipIf(!EXPECT_LIVE_DIALECTS)('is provisioned — set OS_TEST_POSTGRES_URL to run this cell', () => {
174+
expect.fail(
175+
'OS_TEST_POSTGRES_URL is unset while OS_EXPECT_LIVE_DIALECT_MATRIX=1: this runner declared it provisions a live '
176+
+ 'server, so the concurrent-migration cell must not be skipped.',
177+
);
178+
});
179+
});
180+
} else {
181+
describe('os migrate apply serializes on the schema-migration lock (live postgres)', () => {
182+
let dir: string;
183+
let first: Run;
184+
let second: Run;
185+
let control: Run;
186+
let queuedBehindHolder = 0;
187+
let tables: string[] = [];
188+
189+
beforeAll(async () => {
190+
await adminExecute(`DROP DATABASE IF EXISTS "${LIVE_DATABASE}" WITH (FORCE)`);
191+
await adminExecute(`CREATE DATABASE "${LIVE_DATABASE}"`);
192+
dir = mkdtempSync(join(tmpdir(), 'os-migration-lock-'));
193+
writeFileSync(join(dir, 'objectstack.config.ts'), CONFIG);
194+
195+
// The holder: the lock, on a session of this file's own.
196+
const holder = new SqlDriver({ client: 'pg', connection: liveUrl() });
197+
const knex = holder.getKnex();
198+
const session = await knex.client.acquireRawConnection();
199+
try {
200+
await knex.raw('select pg_advisory_lock(?::bigint)', [SCHEMA_MIGRATION_PG_LOCK_KEY]).connection(session);
201+
const runs = [migrate(dir), migrate(dir)];
202+
// Until both migrators are queued behind the holder — never a sleep.
203+
const deadline = Date.now() + CELL_TIMEOUT_MS / 2;
204+
while (Date.now() < deadline) {
205+
const r = await knex.raw(
206+
"select count(*)::int as n from pg_locks l join pg_database d on d.oid = l.database "
207+
+ "where l.locktype = 'advisory' and not l.granted and d.datname = ?",
208+
[LIVE_DATABASE],
209+
).connection(session) as { rows: Array<{ n: number }> };
210+
queuedBehindHolder = r.rows[0].n;
211+
if (queuedBehindHolder >= 2) break;
212+
await new Promise((r2) => setTimeout(r2, 250));
213+
}
214+
await knex.raw('select pg_advisory_unlock(?::bigint)', [SCHEMA_MIGRATION_PG_LOCK_KEY]).connection(session);
215+
[first, second] = await Promise.all(runs);
216+
} finally {
217+
await knex.client.destroyRawConnection(session);
218+
await holder.disconnect();
219+
}
220+
control = await migrate(dir);
221+
222+
const probe = new SqlDriver({ client: 'pg', connection: liveUrl() });
223+
try {
224+
const r = await probe.getKnex().raw(
225+
"select table_name from information_schema.tables where table_schema = 'public' order by table_name",
226+
) as { rows: Array<{ table_name: string }> };
227+
tables = r.rows.map((row) => row.table_name);
228+
} finally {
229+
await probe.disconnect();
230+
}
231+
}, CELL_TIMEOUT_MS);
232+
233+
afterAll(async () => {
234+
if (dir) rmSync(dir, { recursive: true, force: true });
235+
await adminExecute(`DROP DATABASE IF EXISTS "${LIVE_DATABASE}" WITH (FORCE)`);
236+
}, CELL_TIMEOUT_MS);
237+
238+
it('both migrators queued behind the held lock — the contention is real, not a race that happened to miss', () => {
239+
expect(queuedBehindHolder).toBe(2);
240+
});
241+
242+
it('both succeed, and each said ONCE that it waited — on stderr, stdout still one JSON document', () => {
243+
for (const run of [first, second]) {
244+
expect(run.code).toBe(0);
245+
expect(run.stderr.match(WAIT_LINE)).toHaveLength(1);
246+
expect(() => JSON.parse(run.stdout)).not.toThrow();
247+
}
248+
});
249+
250+
it('⭐ the schema is applied ONCE: one migrator creates it, the other finds nothing to do', () => {
251+
const payloads = [first, second].map((run) => JSON.parse(run.stdout) as ApplyPayload);
252+
const creators = payloads.filter((p) => (p.created?.length ?? 0) > 0);
253+
expect(creators).toHaveLength(1);
254+
const other = payloads.find((p) => p !== creators[0])!;
255+
expect(other.created ?? []).toEqual([]);
256+
expect(other.message).toBe('in_sync');
257+
expect(tables).toContain('ml_ticket');
258+
});
259+
260+
it('control — an uncontended migrator prints no wait line and finds the schema in sync', () => {
261+
expect(control.code).toBe(0);
262+
expect(control.stderr.match(WAIT_LINE)).toBeNull();
263+
expect((JSON.parse(control.stdout) as ApplyPayload).message).toBe('in_sync');
264+
});
265+
});
266+
}

0 commit comments

Comments
 (0)