Skip to content

fix: Run nested expect_* tests in 3 lanes instead of exclusive - #1939

Draft
bor-p-s wants to merge 7 commits into
masterfrom
bor-p-s/nested-test-lanes-stacked
Draft

fix: Run nested expect_* tests in 3 lanes instead of exclusive#1939
bor-p-s wants to merge 7 commits into
masterfrom
bor-p-s/nested-test-lanes-stacked

Conversation

@bor-p-s

@bor-p-s bor-p-s commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Description

Splits the shared nested output base expect_build_failure_test (and its sibling macros) use into 3 lanes (hash(name) % 3, picked via --lane in nested_bazel_setup), replacing the exclusive tag from #1894 with per-lane isolation: tests in different lanes run concurrently.

Also switches these tests' default tags from local to no-sandbox + no-remote-exec. Both keep the nested bazel outside the sandbox, which is what it actually needs; local additionally blocks the test action itself from remote cache, while no-sandbox leaves that path open.

How many same-lane tests can run at once is set per platform in .bazelrc: 1 on Linux/Windows, so they queue behind each other there; 1000 on macOS, to dodge a Bazel scheduler bug (bazelbuild/bazel#18153) where a scarce resource delays every unrelated test scheduled alongside it. On macOS a same-lane pair can genuinely run together, which matters for a test asserting on freshly-printed build output (--clean-before-build in expect_build_failure.sh): nested_bazel.sh adds its own lock (_nested_bazel_acquire_lane_lock) around exactly that pair of nested bazel calls to close the gap.

Motivation

#1894 traded speed for correctness: forcing every nested test to run one at a time removed the shared-lock timeout risk (#1884, #1885), at a disclosed cost on the critical path (+8 min on the steps that run them). This restores concurrency without reintroducing full lock contention.

The local -> no-sandbox switch matters beyond lanes: #1932 adds remote cache flags to test_rules_scala.sh's bazel invocations, but a local-tagged test stays uncached regardless of the outer invocation's flags. Stacked together, the two PRs measured a 66% drop on Linux and roughly 50% on macOS/Windows for that CI step (numbers in #1932).

Alternatives considered

Parallel output bases at K=4 were probed before (#1890/#1891) and rejected for disk cost (~1.7GB per lane) and a shared --disk_cache across lanes. This reuses the same idea at K=3; lanes here only share --repository_cache (external-repo downloads), already shared regardless of lane count, so the added disk cost is 2 extra ~1.7GB output bases, while --disk_cache contention stays exactly what it already was.

Capping the macOS lane lock at 1 too, matching Linux/Windows, was tried: it removes the same-lane race outright, at the cost of the bazel#18153 stall on unrelated tests. Kept the higher macOS capacity plus the script-level lock instead -- same safety, without that throughput cost.

Test plan

  • bazel test -- //test/strict_dependency/... //test/missing_direct_deps/...: 22/22 pass, including the 7 targets that take the new lock (--clean-before-build callers).
  • Verified the lock itself outside Bazel: two concurrent shell processes on the same lane, one holding the lock for a few seconds -- the second waited the whole time and acquired it right after release. A process that exits mid-lock (simulated exit 1) still releases it, via the EXIT trap in nested_bazel_setup.
  • Full local suite (bazel test -- //... -//test_expect_failure/...): 344/346 pass, 2 skipped.

@bor-p-s
bor-p-s force-pushed the bor-p-s/nested-test-lanes-stacked branch 4 times, most recently from 7effcdc to c329bd7 Compare August 25, 2026 09:10
@bor-p-s
bor-p-s force-pushed the bor-p-s/nested-test-lanes-stacked branch 2 times, most recently from 5b7214d to f8e49ed Compare August 26, 2026 13:53
Base automatically changed from bor-p-s/nested-cacheable to master August 26, 2026 15:36
@bor-p-s
bor-p-s force-pushed the bor-p-s/nested-test-lanes-stacked branch 4 times, most recently from c08253c to 8dbd3fa Compare August 31, 2026 12:10
@bor-p-s
bor-p-s marked this pull request as ready for review August 31, 2026 17:38
@bor-p-s
bor-p-s requested a review from PawelLipski August 31, 2026 17:42
@bor-p-s
bor-p-s force-pushed the bor-p-s/nested-test-lanes-stacked branch from 43d7f92 to a987473 Compare August 31, 2026 17:42
@bor-p-s
bor-p-s force-pushed the bor-p-s/nested-test-lanes-stacked branch from 60333d8 to a04d547 Compare August 31, 2026 20:08
@bor-p-s
bor-p-s marked this pull request as draft August 31, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant