Skip to content

Commit 5444da4

Browse files
committed
fix(ci): isolate local-CI runs per run id, and tidy what each run creates
Concurrent local-CI legs clobbered each other. Container names were already unique per `compose run --rm`, but the mutable state was not: - Docker: `cbm-build` is ONE named volume shared by every service and every concurrent run, so two legs wrote the same /src/build - objects and test-logs included. The parallel scheduler then dies reading a suite log another run replaced ('cannot read suite log .../extraction.log'). - Windows VM: one checkout (/c/cbm) plus a FIXED log path (/tmp/win-test.log) and a shared build dir; -PruneStale could also delete a live run's temp root. Each run now carries a unique id (pid+epoch, overridable via CBM_CI_RUN_ID): - docker-compose: the build volume takes its name from CBM_CI_BUILD_VOLUME, defaulting to today's `cbm-build` so the single-run path is unchanged. - run.sh: derives the id, points the build volume at it, and removes that volume on success. CBM_CI_SHARED_BUILD=1 opts back into the shared volume. - vm-run-tests.sh: per-run log; and when a CALLER sets CBM_CI_RUN_ID (i.e. declares concurrency) a per-run BUILD_DIR too, removed on success. ccache and the fixture cache stay SHARED deliberately - ccache is concurrency-safe and content-verified, and sharing them is what keeps an isolated run fast instead of cold. Cleanup follows one rule: a run tidies what it created, EXCEPT when it failed - then the artifacts are the post-mortem and the run prints how to inspect and drop them. CBM_CI_KEEP=1 keeps them regardless. Verified: bash -n on both scripts; `docker compose config` valid, and with CBM_CI_BUILD_VOLUME set the volume resolves to the per-run name. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent eb3e571 commit 5444da4

3 files changed

Lines changed: 81 additions & 3 deletions

File tree

‎test-infrastructure/docker-compose.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -319,5 +319,13 @@ volumes:
319319
# native filesystem: object writes over the virtiofs bind mount are the
320320
# container legs' largest avoidable I/O cost, and the fixture cache makes
321321
# the network clone once-per-volume instead of once-per-container.
322+
# The build volume is the ONLY per-run-mutable one: concurrent legs writing
323+
# the same build dir clobber each other's objects and test-logs (a scheduler
324+
# then fails reading a suite log that another run replaced). run.sh gives each
325+
# run its own volume via CBM_CI_BUILD_VOLUME and removes it afterwards; the
326+
# default name preserves the single-run fast path. ccache and the fixture
327+
# cache stay SHARED on purpose — ccache is concurrency-safe and
328+
# content-verified, and sharing them is what keeps isolated runs fast.
322329
cbm-build:
330+
name: ${CBM_CI_BUILD_VOLUME:-cbm-build}
323331
cbm-fixture-cache:

‎test-infrastructure/run.sh‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,46 @@
5656
set -euo pipefail
5757

5858
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
59+
60+
# ── Per-run isolation (concurrent agents / multiple worktrees) ───────────────
61+
# Container names are already unique per `compose run --rm`, but the BUILD
62+
# volume was not: two legs running at once wrote the same /src/build, so one
63+
# run's objects and test-logs replaced the other's — after which the parallel
64+
# scheduler dies reading a suite log that another run had removed. Each run now
65+
# gets its own build volume, keyed by a unique run id. ccache and the fixture
66+
# cache stay SHARED on purpose: ccache is concurrency-safe and content-verified,
67+
# and sharing them is what keeps an isolated run fast rather than cold.
68+
#
69+
# CBM_CI_RUN_ID=<id> name/reuse a run (default: pid + epoch, unique)
70+
# CBM_CI_SHARED_BUILD=1 opt back into the single shared `cbm-build` volume
71+
# CBM_CI_KEEP=1 keep this run's build volume even on success
72+
RUN_ID="${CBM_CI_RUN_ID:-$$-$(date +%s)}"
73+
if [ "${CBM_CI_SHARED_BUILD:-0}" = "1" ]; then
74+
CBM_CI_BUILD_VOLUME="cbm-build"
75+
else
76+
CBM_CI_BUILD_VOLUME="cbm-build-${RUN_ID}"
77+
fi
78+
export CBM_CI_BUILD_VOLUME
79+
80+
# Tidy what this run generated. A FAILED run KEEPS its volume — those artifacts
81+
# are the post-mortem — and prints how to inspect and drop it. Successful runs
82+
# leave nothing behind. Shared-volume mode never removes anything.
83+
ci_cleanup() {
84+
local rc=$?
85+
if [ "${CBM_CI_BUILD_VOLUME}" = "cbm-build" ] || [ "${CBM_CI_KEEP:-0}" = "1" ]; then
86+
return $rc
87+
fi
88+
if [ $rc -eq 0 ]; then
89+
docker volume rm -f "${CBM_CI_BUILD_VOLUME}" >/dev/null 2>&1 || true
90+
else
91+
echo "run.sh: kept ${CBM_CI_BUILD_VOLUME} for post-mortem (run failed)" >&2
92+
echo " inspect: docker run --rm -v ${CBM_CI_BUILD_VOLUME}:/b alpine ls -R /b" >&2
93+
echo " drop: docker volume rm ${CBM_CI_BUILD_VOLUME}" >&2
94+
fi
95+
return $rc
96+
}
97+
trap ci_cleanup EXIT
98+
5999
COMPOSE="docker compose -f $ROOT/test-infrastructure/docker-compose.yml"
60100

61101
usage() {

‎test-infrastructure/vm/vm-run-tests.sh‎

Lines changed: 33 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,25 @@ case "${1:-}" in
4646
esac
4747

4848
RUNNER="${CBM_VM_RUNNER:-}"
49-
LOG="${CBM_VM_TEST_LOG:-/tmp/win-test.log}"
49+
50+
# Per-run identity. The VM holds ONE checkout (/c/cbm), so two concurrent runs
51+
# shared a FIXED log path and the same build dir: the later run's output
52+
# replaced the earlier one's, and -PruneStale could delete a live run's temp
53+
# root out from under it. The run id namespaces the log and the build dir; the
54+
# protected temp root is already unique per run.
55+
CALLER_RUN_ID="${CBM_CI_RUN_ID:-}"
56+
RUN_ID="${CBM_CI_RUN_ID:-$$-$(date +%s)}"
57+
export CBM_CI_RUN_ID="$RUN_ID"
58+
LOG="${CBM_VM_TEST_LOG:-/tmp/win-test-${RUN_ID}.log}"
59+
60+
# A caller that sets CBM_CI_RUN_ID is declaring concurrency, so give that run
61+
# its own BUILD_DIR; the default single-run path keeps the shared one and its
62+
# incremental reuse (a per-run build dir on the VM costs a full rebuild).
63+
if [ -n "$CALLER_RUN_ID" ]; then
64+
BUILD_ARGS=("BUILD_DIR=build/vm-${RUN_ID}")
65+
else
66+
BUILD_ARGS=()
67+
fi
5068

5169
[ $# -ge 1 ] || { echo "vm-run-tests: missing arguments. Please consult --help." >&2; exit 2; }
5270
if [ "$1" = "--soak" ]; then
@@ -130,13 +148,25 @@ if [ -n "$RUNNER" ]; then
130148
rc="${PIPESTATUS[0]}"
131149
fi
132150
elif [ "$1" = "--par" ]; then
133-
scripts/test.sh CC=clang CXX=clang++ 2>&1 | tee "$LOG"
151+
scripts/test.sh CC=clang CXX=clang++ \
152+
${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"} 2>&1 | tee "$LOG"
134153
rc="${PIPESTATUS[0]}"
135154
else
136-
scripts/test.sh --suites "$*" CC=clang CXX=clang++ 2>&1 | tee "$LOG"
155+
scripts/test.sh --suites "$*" CC=clang CXX=clang++ \
156+
${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"} 2>&1 | tee "$LOG"
137157
rc="${PIPESTATUS[0]}"
138158
fi
139159

160+
# Tidy this run's own build dir on success; keep it on failure as the
161+
# post-mortem. Only ever touches a dir this run created (concurrent mode).
162+
if [ -n "$CALLER_RUN_ID" ] && [ "${CBM_CI_KEEP:-0}" != "1" ]; then
163+
if [ "${rc:-1}" -eq 0 ]; then
164+
rm -rf "build/vm-${RUN_ID}"
165+
else
166+
echo "vm-run-tests: kept build/vm-${RUN_ID} and $LOG for post-mortem" >&2
167+
fi
168+
fi
169+
140170
if ! grep -Eq '[0-9]+ passed' "$LOG"; then
141171
echo "GUARD: test runner produced no completion summary — the suites did" \
142172
"not validly run; treating as failure (runner rc=$rc)" >&2

0 commit comments

Comments
 (0)