Repository navigation
fix(pool): bound the PoolInner::close() join and idle drain - #4455
Open
daniel-pinder wants to merge 1 commit into
Open
daniel-pinder wants to merge 1 commit into
daniel-pinder wants to merge 1 commit into
Conversation
Carries the close-join half of the fork commit ecbfc25d alone, retargeted onto upstream main (8b65c2f). The rest of that commit is already upstream or in flight separately, so it is deliberately not carried here: - the pop_idle saturating decrement is already on main via transact-rs#4289, so this branch leaves that code untouched; - the bounded release ping (sqlx-core/src/pool/connection.rs) is covered by the open PRs transact-rs#4407, transact-rs#4350 and transact-rs#4363. What this branch does: PoolInner::close() joins every checked-out connection before draining the idle set, so a caller that never returns its PoolConnection blocks shutdown indefinitely. Bound the join with a new CLOSE_JOIN_TIMEOUT (5s): on expiry, carry straight on to the drain, because the pool holds no handle with which to force-close a checked-out connection. The serial per-idle closes share that one deadline instead of each taking a fresh timeout, and a deadline-exhausted drain `continue`s rather than `break`s so the remaining idles are popped and dropped — no waiting is re-introduced (a zero rt::timeout still polls its future once before elapsing). Once the join is abandoned, size is stored as zero while a still-checked-out connection holds a live DecrementSizeGuard. That decrement now saturates: a plain fetch_sub wrapped the counter to u32::MAX and Pool::size() reported a lie. No behaviour change while the count is non-zero. Pool::close()'s contract is amended to the bounded, best-effort semantics. Adds two tests that need no database server, driving in-memory SQLite: close() gives up after the bound when a permit is held forever, and the saturating accounting is pinned (size_saturates_when_an_abandoned_connection_is_dropped). Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Does your PR solve an issue?
Partially, and deliberately not through a
fixes/closeskeyword: therelease-path permit leak in #4349 (
return_to_poolparking against a silentlydead peer) is addressed by the open PRs #4407, #4350 and #4363, not here.
This PR is the complementary shutdown-side fix:
Pool::close()can itselfhang forever, and the accounting a bounded close() exposes wraps.
Is this a breaking change?
No API signature changes. It is a documented behavior change:
Pool::close()'scontract is amended from "wait for all connections" to a bounded, best-effort
close (5 s, one shared deadline), because the previous behavior meant "hang
forever if any caller never returns its connection". The docs on
Pool::closeare updated to say exactly what is now promised.
What this does
PoolInner::close()has two unbounded waits:semaphore.acquire(permits_to_acquire))before draining the idle set. A caller that never returns its
PoolConnectionblocks shutdown indefinitely — the pool holds no handlewith which to force-close it.
idle.live.raw.close().await),an un-timed round trip each to a possibly-wedged server.
The join is now bounded by a new
CLOSE_JOIN_TIMEOUT(5 s, sibling of theexisting
CLOSE_ON_DROP_TIMEOUT); on expiry,close()proceeds to the drain(
mark_closed()has already made that safe). The per-idle closes share onedeadline rather than each taking a fresh timeout, and a deadline-exhausted
drain
continues rather thanbreaks, so the remaining idle entries arepopped and dropped — no waiting is re-introduced (a zero
rt::timeoutstillpolls its future once before elapsing).
Once the join is abandoned,
sizeis stored as zero while a still-checked-outconnection holds a live
DecrementSizeGuard; that decrement now saturates. Aplain
fetch_subwrapped the counter tou32::MAXandPool::size()reporteda lie. No behaviour change while the count is non-zero.
Relationship to other open work (disjoint — races nothing)
sqlx-core/src/pool/connection.rs(the
return_to_poolrelease path). This PR touches onlysqlx-core/src/pool/inner.rsandmod.rs— it can merge in any order withany of them, and neither substitutes for the other. fix(pool): bound the entire connection return operation #4407's body notes
"shutdown stalls aren't separately tested" there; this is that half.
pop_idlesaturating decrement from the same original work is alreadyon main via fix(pool): prevent num_idle underflow that wedges maintenance in a CPU spin #4289 ("prevent num_idle underflow...", fixes 100% CPU in sqlx_core::pool::inner::spawn_maintenance_tasks #3645), so this PR
deliberately leaves that code untouched. Only the
DecrementSizeGuardsaturation (needed because a bounded close() stores
size = 0) is addedhere.
sqlx::Poolarchitecture #3582 (new pool architecture, 0.10+) subsumes this; a 0.9.x-aimed minimalfix, this is trivially droppable when breaking: new
sqlx::Poolarchitecture #3582 lands. (Steered by themaintainer's comment on fix(pool): bound return_to_pool so a dead connection can't leak its permit #4350: "I already have this fix in breaking: new
sqlx::Poolarchitecture #3582 but thatwon't be merged until 0.10 at the earliest.")
Tests
cargo test -p sqlx-core --all-features— green locally (13 unit + 30 doctests). Two new tests, no database server needed (in-memory SQLite), in
inner.rs:close_gives_up_on_a_permit_that_is_never_released— a permit is held forthe life of the test with
std::mem::forget; assertsclose()returnsafter the bound but not before it. It cannot be shown red-first: without
the fix the call never returns (that is the bug), so a pre-fix run is a
hang, not a failure.
size_saturates_when_an_abandoned_connection_is_dropped— red-firstverified: with the saturating decrement reverted to
fetch_sub, the testfails with
left: 4294967295, right: 0; restored, it passes.