fix: Run nested expect_* tests in 3 lanes instead of exclusive - #1939
Draft
bor-p-s wants to merge 7 commits into
Draft
fix: Run nested expect_* tests in 3 lanes instead of exclusive#1939bor-p-s wants to merge 7 commits into
bor-p-s wants to merge 7 commits into
Conversation
bor-p-s
force-pushed
the
bor-p-s/nested-test-lanes-stacked
branch
4 times, most recently
from
August 25, 2026 09:10
7effcdc to
c329bd7
Compare
bor-p-s
force-pushed
the
bor-p-s/nested-test-lanes-stacked
branch
2 times, most recently
from
August 26, 2026 13:53
5b7214d to
f8e49ed
Compare
bor-p-s
force-pushed
the
bor-p-s/nested-test-lanes-stacked
branch
4 times, most recently
from
August 31, 2026 12:10
c08253c to
8dbd3fa
Compare
bor-p-s
marked this pull request as ready for review
August 31, 2026 17:38
bor-p-s
requested review from
WojciechMazur,
mbland and
simuons
as code owners
August 31, 2026 17:38
bor-p-s
force-pushed
the
bor-p-s/nested-test-lanes-stacked
branch
from
August 31, 2026 17:42
43d7f92 to
a987473
Compare
…ing expect_* tests exclusive
bor-p-s
force-pushed
the
bor-p-s/nested-test-lanes-stacked
branch
from
August 31, 2026 20:08
60333d8 to
a04d547
Compare
bor-p-s
marked this pull request as draft
August 31, 2026 20:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Splits the shared nested output base
expect_build_failure_test(and its sibling macros) use into 3 lanes (hash(name) % 3, picked via--laneinnested_bazel_setup), replacing theexclusivetag from #1894 with per-lane isolation: tests in different lanes run concurrently.Also switches these tests' default
tagsfromlocaltono-sandbox+no-remote-exec. Both keep the nestedbazeloutside the sandbox, which is what it actually needs;localadditionally blocks the test action itself from remote cache, whileno-sandboxleaves that path open.How many same-lane tests can run at once is set per platform in
.bazelrc: 1 on Linux/Windows, so they queue behind each other there; 1000 on macOS, to dodge a Bazel scheduler bug (bazelbuild/bazel#18153) where a scarce resource delays every unrelated test scheduled alongside it. On macOS a same-lane pair can genuinely run together, which matters for a test asserting on freshly-printed build output (--clean-before-buildinexpect_build_failure.sh):nested_bazel.shadds its own lock (_nested_bazel_acquire_lane_lock) around exactly that pair of nestedbazelcalls to close the gap.Motivation
#1894 traded speed for correctness: forcing every nested test to run one at a time removed the shared-lock timeout risk (#1884, #1885), at a disclosed cost on the critical path (+8 min on the steps that run them). This restores concurrency without reintroducing full lock contention.
The
local->no-sandboxswitch matters beyond lanes: #1932 adds remote cache flags totest_rules_scala.sh's bazel invocations, but alocal-tagged test stays uncached regardless of the outer invocation's flags. Stacked together, the two PRs measured a 66% drop on Linux and roughly 50% on macOS/Windows for that CI step (numbers in #1932).Alternatives considered
Parallel output bases at K=4 were probed before (#1890/#1891) and rejected for disk cost (~1.7GB per lane) and a shared
--disk_cacheacross lanes. This reuses the same idea at K=3; lanes here only share--repository_cache(external-repo downloads), already shared regardless of lane count, so the added disk cost is 2 extra ~1.7GB output bases, while--disk_cachecontention stays exactly what it already was.Capping the macOS lane lock at 1 too, matching Linux/Windows, was tried: it removes the same-lane race outright, at the cost of the bazel#18153 stall on unrelated tests. Kept the higher macOS capacity plus the script-level lock instead -- same safety, without that throughput cost.
Test plan
bazel test -- //test/strict_dependency/... //test/missing_direct_deps/...: 22/22 pass, including the 7 targets that take the new lock (--clean-before-buildcallers).exit 1) still releases it, via theEXITtrap innested_bazel_setup.bazel test -- //... -//test_expect_failure/...): 344/346 pass, 2 skipped.