Skip to content

fix(database): serialize the tests that race on the global pool - #1180

Merged
Harry19081 merged 1 commit into
developfrom
fix/database-pool-test-race
Sep 1, 2026
Merged

fix(database): serialize the tests that race on the global pool#1180
Harry19081 merged 1 commit into
developfrom
fix/database-pool-test-race

Conversation

@Chloe-JY

@Chloe-JY Chloe-JY commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Problem

db::connection::tests::pooled_connection_is_reused_after_drop fails
intermittently in CI:

thread '...pooled_connection_is_reused_after_drop' panicked at
crates/database/src/db/connection.rs:594:9:
assertion failed: has_temp_table(&conn, "pool_marker")

Observed on the Rust (clippy) job of #1177 — a PR that changes no file in
crates/database. develop is green, and the test passes 25/25 locally, so it
is load-dependent rather than a real regression.

The cause is shared global state, not the test's own inputs. The connection
pool is a process-wide OnceLock<Mutex<ConnectionPool>>, and
reset_connection_pool() bumps the generation and clears pool.idle for
every path:

pub fn reset_connection_pool() {
    let mut pool = connection_pool().lock()...;
    pool.generation += 1;
    pool.idle.clear();          // <- every path, not just the caller's
}

pooled_connection_is_reused_after_drop proves connection identity by creating
a TEMP table, dropping the guard so the connection parks as idle, then
reopening and asserting the marker survived. Five tests in that binary touch the
same global pool and cargo test runs them in parallel — so between the drop
and the reopen, pool_reset_retires_idle_and_checked_out_connections (or either
register_sessions_init / register_projects_init, which also reset on first
registration) can clear the idle entry. The reopen then gets a fresh
connection, which has no TEMP table, and the assertion fails.

The per-test unique temp paths (temp_db_path uses pid + nanos) do not help:
the state being raced on is the pool, which is global.

Solution

Add a crate-local POOL_TEST_LOCK immediately next to connection_pool()
the state it guards — and take it in all five tests that assert on pool
contents: pooled_connection_is_reused_after_drop,
pool_reset_retires_idle_and_checked_out_connections,
connection_left_in_a_transaction_is_not_pooled,
replaced_database_file_is_not_served_from_a_stale_connection, and
concurrent_first_connections_wait_for_schema_completion.

This mirrors the idiom already in this workspace: session-persistence has
ORGII_HOME_TEST_LOCK for exactly the same class of problem, documented there
as "One crate-wide lock is required: module-local locks still let those tests
race." The guard helper is poison-tolerant (Err(poisoned) => poisoned .into_inner()) so a single failing test cannot cascade into the other four.

Both items are #[cfg(test)]; production code paths are untouched.

Resulting invariant: at most one test at a time observes the global pool, so a
test's assertion about pool contents depends only on what that test did.

Potential risks

  • Serialization costs wall-clock time. Five tests that used to run in
    parallel now run one at a time. Measured: 0.09s → 0.25s for the whole
    database lib suite. Negligible.
  • A hang in one of the five now blocks the other four rather than only
    itself. All five are sub-100ms and do no I/O beyond a temp SQLite file.
  • The lock is test-only and does not fix the underlying sharpness.
    reset_connection_pool() clearing every path is still a global operation; a
    future non-test caller could surprise someone the same way. Making the reset
    path-scoped would be the deeper fix, but it changes production semantics
    (register_*_init deliberately invalidates connections opened before the
    initializer existed) and does not belong in a flake fix.
  • A sixth test added later that touches the pool must take the guard too.
    Nothing enforces that mechanically; the doc comment on POOL_TEST_LOCK says
    so explicitly.
  • Rollback: revert this commit. It is 25 added lines in one file, all
    #[cfg(test)].

Verification

  • The race was reproduced deterministically before fixing it. On unmodified
    develop, inserting a 150ms sleep into pooled_connection_is_reused_after_drop
    between the drop and the reopen — widening the existing window, changing no
    logic — makes it fail every run with the exact CI assertion:

    test db::connection::tests::pooled_connection_is_reused_after_drop ... FAILED
    panicked at crates/database/src/db/connection.rs:595:9:
    assertion failed: has_temp_table(&conn, "pool_marker")
    
  • The same widened window passes with this fix applied:
    test db::connection::tests::pooled_connection_is_reused_after_drop ... ok,
    test result: ok. 9 passed; 0 failed. That is what shows the lock closes the
    race rather than the sleep merely reshuffling the schedule. The experimental
    sleep is not part of this diff (grep -c EXPERIMENT → 0).

  • cargo test -p database --lib — 9 passed, 0 failed.

  • cargo clippy -p database --all-targets -- -D warnings — exit 0.

  • Run with a private CARGO_TARGET_DIR: two ORGII worktrees resolve this
    workspace to the same build-script OUT_DIR, so a shared target dir can
    verify the wrong worktree.

  • Not run: no TypeScript changes, so no pnpm typecheck / vitest. No UI
    change, so no screenshots. Full-workspace cargo test was not re-run — this
    diff is #[cfg(test)]-only inside one crate.

  • Pre-commit hook trailer is absent. Committed from a git worktree, where
    .husky/_/husky.sh is gitignored and therefore never created by
    git worktree add. That trailer is structurally always missing in a worktree
    and carries no signal; the check it would have run for a Rust diff is
    cargo clippy, run manually above.

`pooled_connection_is_reused_after_drop` fails intermittently in CI with
`assertion failed: has_temp_table(&conn, "pool_marker")`.

The connection pool is a process-global `OnceLock<Mutex<ConnectionPool>>`, and
`reset_connection_pool()` clears `idle` for **every** path, not just its own.
The reuse test drops a connection (parking it as idle, carrying a TEMP table
that proves identity) and then reopens; if
`pool_reset_retires_idle_and_checked_out_connections` — or either
`register_*_init` helper — resets the pool in that window, the marked
connection is discarded and a fresh one comes back. Unique temp paths do not
help: the raced state is global.

Adds a crate-local `POOL_TEST_LOCK` next to the pool it guards and takes it in
all five tests that assert on pool contents, mirroring the existing
`ORGII_HOME_TEST_LOCK` idiom in `session-persistence`, poison-tolerant so one
failure cannot cascade.
@Harry19081
Harry19081 merged commit 3221063 into develop Sep 1, 2026
6 checks passed
@Harry19081 Harry19081 added the tests Test coverage or test infrastructure work label Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Test coverage or test infrastructure work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants