Skip to content

test(server): fix two startup races that fail internal/server tests on macOS runners - #1890

Merged
Dumbris merged 4 commits into
mainfrom
fix/flaky-server-tests-macos
Nov 16, 2026
Merged

Dumbris merged 4 commits into
mainfrom
fix/flaky-server-tests-macos

Conversation

@Dumbris

@Dumbris Dumbris commented Nov 11, 2026

Copy link
Copy Markdown
Member

Summary

Two internal/server tests failed on macOS runners on unrelated PRs (#1885, #1887). Both are races in the tests, not in production code. This PR fixes them. Test-only.

Causes and fixes

TestSearchTools_WithholdsApprovalLockedHits returned only cedar:pending_item when orchid:read_item was also expected.

  • After PhaseReady the runtime starts background tool indexing, whose first step deletes index entries for servers the upstream manager does not know.
  • The test seeds two hand-made servers into the index right after PhaseReady, so the cleanup could delete a seeded document mid-seeding.
  • Reproduced locally with GOMAXPROCS=1 go test -race -count=40: failed within the first two runs.
  • Fix: a newSettledConsistencyServer fixture waits for PhaseReady and for the cleanup's completion log line before the test seeds anything. TestSearchTools_AnnotationsSurviveToTier seeds the same way and now uses it too.

TestTPAFetchLifecycle_ActivatesReprotectsAndRefusesBadReleases failed with "exactly one cycle reported activated: expected 1, actual 0".

  • The test waited for the bundle source to flip, which happens inside the fetch leg. The telemetry counter is incremented later, by the observer that runs after the cycle is recorded.
  • Reproduced deterministically by delaying that observer by 400 ms (not part of this PR).
  • Fix: poll until the counter reaches 1, then keep the exact == 1 assertion.

Not fixed here

The test timed out after 40m0s failure on #1886 was not a hang. The whole internal/server package ran 40 minutes on a slow runner; the same job passed in 20 minutes on retry. A healthy macOS run already uses about half the 40-minute budget. Raising the timeout or trimming the fixed 12-second waits in the upstream-add tests is a separate decision.

Testing

  • GOMAXPROCS=1 go test -race -count=100 -run TestSearchTools_WithholdsApprovalLockedHits ./internal/server/: pass.
  • Both search tests together, -race -count=15: pass.
  • The fetch lifecycle test, GOMAXPROCS=1 -race: 15 of 15 runs passed.
  • golangci-lint v2 on ./internal/server/...: 0 issues.

…g the search gate test

backgroundToolIndexing, spawned right after PhaseReady, begins with
cleanupOrphanedIndexEntries, which deletes the index documents of every
indexed server the upstream manager does not know. TestSearchTools_WithholdsApprovalLockedHits
seeds hand-made servers into the index after PhaseReady, so on a loaded runner
the cleanup removed orchid:read_item mid-seeding and the approved hit went
missing from the 'gadget' query (about 1 in 40 runs at GOMAXPROCS=1 -race).
The fixture now also waits for the pass's completion log line.
…ore asserting telemetry

BundleStatus().Source flips to fetch inside the cycle's fetch leg, but the
cycle reports its outcome to the telemetry observer only afterwards
(recordCycle, logOutcome, notifyRefreshObserver). TestTPAFetchLifecycle read
TPABundleRefreshActivations as soon as the source flipped, so on a slow runner
it saw 0 instead of 1. Poll for the counter, then assert it is exactly 1.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: f2cf81a
Status: ✅  Deploy successful!
Preview URL: https://25c7a8d4.mcpproxy-docs.pages.dev
Branch Preview URL: https://fix-flaky-server-tests-macos.mcpproxy-docs.pages.dev

View logs

@Dumbris
Dumbris enabled auto-merge (squash) November 11, 2026 19:18
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: fix/flaky-server-tests-macos

Available Artifacts

  • archive-darwin-amd64 (32 MB)
  • archive-darwin-arm64 (29 MB)
  • archive-linux-amd64 (20 MB)
  • archive-linux-arm64 (18 MB)
  • archive-windows-amd64 (32 MB)
  • archive-windows-arm64 (29 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (28 MB)
  • installer-dmg-darwin-arm64 (26 MB)
  • smart-mcp-proxymcpproxy-goZGNY6F.dockerbuild (0 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 43858099492 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Dumbris
Dumbris merged commit ad1b9b9 into main Nov 16, 2026
53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants