Skip to content

docs(test): record why TestFastPathDoesNotOvertakeAQueue must park rather than release - #1159

Closed
Dumbris wants to merge 2 commits into
mainfrom
claude/silly-kowalevski-a1a7a8
Closed

Dumbris wants to merge 2 commits into
mainfrom
claude/silly-kowalevski-a1a7a8

Conversation

@Dumbris

@Dumbris Dumbris commented Sep 1, 2026 •

Copy link
Copy Markdown
Member

Pull Request

Description

Superseded in substance by main. This PR opened as a de-flake of TestFastPathDoesNotOvertakeAQueue (internal/upstream/limiter). While it was open, main landed its own fix for the same flake, with the same diagnosis and a stronger implementation. After merging main in, the net diff is 8 lines of comment. Retained only if that comment is judged worth keeping; otherwise close it — the flake is fixed either way.

The flake blocked at least three unrelated PRs — #1037 (telemetry-only diff) and #1153 twice in a row (a bench/-only diff) — in both Unit Tests and End-to-End Tests on ubuntu-latest.

The previously recorded root cause was wrong. It was characterised as "the assertion demands strict FIFO from a fast path that only usually loses the race", i.e. a real ordering bug in limiter.go. It is not, and limiter.go is unchanged here.

Strict FIFO is a genuine invariant the limiter promises (package doc, FR-004) and it holds deterministically in this test. By the time the test calls held(), both callers are already parked in the queue — Queued == 2 is asserted beforehand — so the fast path is not in play at all. Grant order is then fixed under one mutex: grantLocked grants the front waiter only (Max: 1), and the second waiter can only be granted from inside the first waiter's release().

The flake was in the test harness, which released before reporting:

if aerr == nil { release() }   // <- this is what wakes the second caller
first <- aerr                  // <- reported only afterwards

release() unblocks the second goroutine, so both channel sends become concurrently ready, and a select over two ready cases picks uniformly at random (Go spec). The test read a scheduling artifact as a FIFO violation. On an idle machine the first goroutine almost always sends before the second is even scheduled — which is why it never reproduced locally.

What changed

Main's fix (startParkedCall) is the stronger one and is kept: each admitted call parks in the slot it wins instead of releasing, so with a cap of 1 only one of the two can be past admission at a time. The concurrent report becomes impossible rather than merely unlikely, and occupancy is asserted between the two grants. This branch's atomic sequence stamping and its awaitAdmission helper are dropped as superseded.

What survives from this branch is the doc comment recording why the earlier version failed, so the release-before-report trap is documented next to the helper that avoids it. That is the entire remaining diff.

Testing

  • I have tested these changes locally
  • I have added/updated tests that prove my fix is effective or my feature works
  • All existing tests pass

Verified on the merge result, so these cover main's implementation as shipped:

Check Result
TestFastPathDoesNotOvertakeAQueue, 12 CPU load-generators, -count=800 ok
Full package -race -count=5 ok
Mutant: grantLocked grants from Back() (LIFO) caught
Mutant: fast path drops the waiters.Len() == 0 guard caught
go build ./... ok

For reference, before either fix: the pre-fix test failed under that identical 12-load-generator -count=800 stress.

Notes for the reviewer

  • The generalisable lesson, and the reason the comment is proposed for keeping: a select over two result channels can never prove ordering. When goroutine A's completion is what unblocks goroutine B, both sends land in the same instant. Worth suspecting the harness first for any Go test claiming "X happened before Y" by racing two channels.
  • Repro technique, since -count=N under load is unreliable: copy the test verbatim into a scratch file in the same package and insert for i := 0; i < 50; i++ { runtime.Gosched() } at the suspected preemption point. That failed 20/20 instantly where -count=600 under load had not reproduced at all.

…mission order

The test raced two result channels in a select to decide which caller was
served first. That measures goroutine reporting order, not admission order:
the first waiter's goroutine calls release() -- the very thing that wakes
the second waiter -- BEFORE reporting on its channel, so both sends become
concurrently ready and select picks uniformly at random. On an idle machine
the first goroutine almost always reaches its send before the second is
scheduled, which is why it only failed on loaded CI runners.

Strict FIFO is a genuine limiter invariant and it holds deterministically
here: both callers are already queued when the test releases (Queued == 2 is
asserted first), so the fast path is not in play, and grant order is fixed
under one mutex. limiter.go is unchanged.

Record admission order directly instead: each caller stamps a shared
atomic.Int64 the moment Acquire returns and before it releases -- the same
trick TestRaisedCapGoesToTheWaiterNotANewcomer already uses. The second
waiter can only be granted from inside the first one's release, so a stamp
taken before that release is a faithful record of grant order. Results are
then collected from both buffered channels in a fixed order. The assertion
stays strict FIFO rather than being weakened.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 1, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 231d6c9
Status: ✅  Deploy successful!
Preview URL: https://eb98d655.mcpproxy-docs.pages.dev
Branch Preview URL: https://claude-silly-kowalevski-a1a7.mcpproxy-docs.pages.dev

View logs

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: claude/silly-kowalevski-a1a7a8

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (26 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (21 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 33492599381 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

Both sides de-flake TestFastPathDoesNotOvertakeAQueue and agree on the
diagnosis: the test was measuring goroutine reporting order rather than
admission order, because the first waiter released -- the very thing that
wakes the second -- before reporting, leaving both result channels ready
at once for a select that picks at random.

main's fix is the stronger one and is kept: startParkedCall has each
admitted call PARK in the slot it wins instead of releasing, so with a cap
of 1 only one of the two can be past admission at a time and the concurrent
report is impossible rather than merely unlikely. It also adds occupancy
assertions between the two grants. This branch's atomic sequence stamping
and the awaitAdmission helper are dropped as superseded.

Kept from this branch: the doc comment explaining WHY the previous version
failed, so the release-before-report trap is recorded next to the helper
that avoids it.

Verified on the merge result: mutating grantLocked to grant from Back()
(LIFO) fails the test, and dropping the waiters.Len() == 0 fast-path guard
fails it too, so the merged test still catches the regression it exists for.
@Dumbris Dumbris changed the title fix(test): de-flake TestFastPathDoesNotOvertakeAQueue by observing admission order docs(test): record why TestFastPathDoesNotOvertakeAQueue must park rather than release Sep 1, 2026
@Dumbris

Dumbris commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

Closing: main landed its own fix for this flake (startParkedCall) while this PR was open — same diagnosis, stronger implementation. After merging main in, the net diff was 8 lines of comment, which isn't worth a PR on its own. The flake is fixed on main either way.

@Dumbris Dumbris closed this Sep 1, 2026
@Dumbris
Dumbris deleted the claude/silly-kowalevski-a1a7a8 branch September 1, 2026 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants