fix(database): serialize the tests that race on the global pool - #1180
Merged
Conversation
`pooled_connection_is_reused_after_drop` fails intermittently in CI with `assertion failed: has_temp_table(&conn, "pool_marker")`. The connection pool is a process-global `OnceLock<Mutex<ConnectionPool>>`, and `reset_connection_pool()` clears `idle` for **every** path, not just its own. The reuse test drops a connection (parking it as idle, carrying a TEMP table that proves identity) and then reopens; if `pool_reset_retires_idle_and_checked_out_connections` — or either `register_*_init` helper — resets the pool in that window, the marked connection is discarded and a fresh one comes back. Unique temp paths do not help: the raced state is global. Adds a crate-local `POOL_TEST_LOCK` next to the pool it guards and takes it in all five tests that assert on pool contents, mirroring the existing `ORGII_HOME_TEST_LOCK` idiom in `session-persistence`, poison-tolerant so one failure cannot cascade.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
db::connection::tests::pooled_connection_is_reused_after_dropfailsintermittently in CI:
Observed on the
Rust (clippy)job of #1177 — a PR that changes no file incrates/database.developis green, and the test passes 25/25 locally, so itis load-dependent rather than a real regression.
The cause is shared global state, not the test's own inputs. The connection
pool is a process-wide
OnceLock<Mutex<ConnectionPool>>, andreset_connection_pool()bumps the generation and clearspool.idleforevery path:
pooled_connection_is_reused_after_dropproves connection identity by creatinga
TEMPtable, dropping the guard so the connection parks as idle, thenreopening and asserting the marker survived. Five tests in that binary touch the
same global pool and
cargo testruns them in parallel — so between the dropand the reopen,
pool_reset_retires_idle_and_checked_out_connections(or eitherregister_sessions_init/register_projects_init, which also reset on firstregistration) can clear the idle entry. The reopen then gets a fresh
connection, which has no
TEMPtable, and the assertion fails.The per-test unique temp paths (
temp_db_pathuses pid + nanos) do not help:the state being raced on is the pool, which is global.
Solution
Add a crate-local
POOL_TEST_LOCKimmediately next toconnection_pool()—the state it guards — and take it in all five tests that assert on pool
contents:
pooled_connection_is_reused_after_drop,pool_reset_retires_idle_and_checked_out_connections,connection_left_in_a_transaction_is_not_pooled,replaced_database_file_is_not_served_from_a_stale_connection, andconcurrent_first_connections_wait_for_schema_completion.This mirrors the idiom already in this workspace:
session-persistencehasORGII_HOME_TEST_LOCKfor exactly the same class of problem, documented thereas "One crate-wide lock is required: module-local locks still let those tests
race." The guard helper is poison-tolerant (
Err(poisoned) => poisoned .into_inner()) so a single failing test cannot cascade into the other four.Both items are
#[cfg(test)]; production code paths are untouched.Resulting invariant: at most one test at a time observes the global pool, so a
test's assertion about pool contents depends only on what that test did.
Potential risks
parallel now run one at a time. Measured: 0.09s → 0.25s for the whole
databaselib suite. Negligible.itself. All five are sub-100ms and do no I/O beyond a temp SQLite file.
reset_connection_pool()clearing every path is still a global operation; afuture non-test caller could surprise someone the same way. Making the reset
path-scoped would be the deeper fix, but it changes production semantics
(
register_*_initdeliberately invalidates connections opened before theinitializer existed) and does not belong in a flake fix.
Nothing enforces that mechanically; the doc comment on
POOL_TEST_LOCKsaysso explicitly.
#[cfg(test)].Verification
The race was reproduced deterministically before fixing it. On unmodified
develop, inserting a 150ms sleep intopooled_connection_is_reused_after_dropbetween the drop and the reopen — widening the existing window, changing no
logic — makes it fail every run with the exact CI assertion:
The same widened window passes with this fix applied:
test db::connection::tests::pooled_connection_is_reused_after_drop ... ok,test result: ok. 9 passed; 0 failed. That is what shows the lock closes therace rather than the sleep merely reshuffling the schedule. The experimental
sleep is not part of this diff (
grep -c EXPERIMENT→ 0).cargo test -p database --lib— 9 passed, 0 failed.cargo clippy -p database --all-targets -- -D warnings— exit 0.Run with a private
CARGO_TARGET_DIR: two ORGII worktrees resolve thisworkspace to the same build-script
OUT_DIR, so a shared target dir canverify the wrong worktree.
Not run: no TypeScript changes, so no
pnpm typecheck/vitest. No UIchange, so no screenshots. Full-workspace
cargo testwas not re-run — thisdiff is
#[cfg(test)]-only inside one crate.Pre-commit hook trailer is absent. Committed from a
git worktree, where.husky/_/husky.shis gitignored and therefore never created bygit worktree add. That trailer is structurally always missing in a worktreeand carries no signal; the check it would have run for a Rust diff is
cargo clippy, run manually above.