Skip to content

Commit 568e613

Browse files
committed
test-infra: overlap the local ladder, open the serial tail, keep container I/O off virtiofs
Local runs left most of the machine idle: the serial tail ran sixteen suites one at a time on otherwise-idle cores, the three platform legs were launched by hand (usually sequentially), and every container-leg object write crossed the virtiofs bind mount. - run-tests-parallel.sh tail scheduling in two phases: the FLEX suites (timing-shaped but free of the shared per-account daemon runtime namespace) run CBM_TAIL_JOBS-wide (default 2), then the EXCL group — daemon-family plus the suites that drive daemon one-shots or supervisor rendezvous — runs strictly sequentially on a machine exactly as quiet as the old fully-serial tail gave it. The wave was already fed longest-first by the shard dealing order, so the drain-out no longer ends on a heavy straggler. - ladder.sh: one maintained entry point for the full local push gate with the legs overlapped — lint, the Linux container suite, and the Windows VM suite in the background, the macOS suite in the foreground, one verdict per leg, logs kept per leg. A missing prerequisite fails its leg loudly instead of silently skipping. - win.sh test-par now runs through vm-run-tests.sh (--par mode): the full parallel harness under the CI-shaped protected temp root with complete output. It previously ran under the MSYS-shared /tmp and piped through `tail -25` — the same truncated-blindness class that hid 40 Windows failures from the `test` command. - docker-compose: build artifacts and the incremental fixture cache move to named volumes on the container VM's native filesystem. Object writes over the virtiofs bind mount are the container legs' largest avoidable I/O cost, and the fixture cache now survives across container runs. - test_mem(win): the first valid full-parallel VM run proved working-set trimming beats the re-touch mitigation (19 MB resident of a 256 MB double-touch at 18 parallel suites). The RSS probe now VirtualLocks a 64 MB span — locked pages are exempt from trimming, making the measurement pressure-immune — with bounded touch-and-sample retries when the lock is unavailable. Red-to-green under the same 18-job load. - cli: the portable install's staging error now appends the activation refusal note (predicate, SID, object) like the managed path already does — a bare "activation transaction I/O failed" on a CI-only failure is undiagnosable without it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent d7096e7 commit 568e613

7 files changed

Lines changed: 150 additions & 11 deletions

File tree

‎scripts/run-tests-parallel.sh‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -186,10 +186,34 @@ run_one() {
186186
export -f run_one
187187

188188
xargs -P "$JOBS" -I{} bash -c 'run_one "$@"' _ {} < "$PAR_FILE"
189-
# Serial tail: quiet machine for the deadline-sensitive suites.
189+
190+
# Tail scheduling in two phases. The FLEX suites are timing-shaped but do
191+
# not rendezvous through the shared per-account daemon runtime namespace,
192+
# so a small fixed overlap (CBM_TAIL_JOBS, default 2) is safe and converts
193+
# idle cores into wall time — the old fully-serial tail ran them one at a
194+
# time on an idle machine. The EXCL group (daemon-family plus the suites
195+
# that drive daemon one-shots or supervisor rendezvous) then runs strictly
196+
# sequentially on a machine exactly as quiet as the old tail gave it.
197+
TAIL_EXCL="cli mcp index_supervisor daemon_application daemon_runtime \
198+
daemon_frontend daemon_bootstrap daemon_ipc"
199+
is_tail_excl() {
200+
case " $TAIL_EXCL " in *" $1 "*) return 0 ;; *) return 1 ;; esac
201+
}
202+
FLEX_FILE="$LOGDIR/suites-tail-flex.txt"
203+
EXCL_FILE="$LOGDIR/suites-tail-excl.txt"
204+
: > "$FLEX_FILE"
205+
: > "$EXCL_FILE"
190206
while IFS= read -r sname; do
191-
run_one "$sname"
207+
if is_tail_excl "$sname"; then
208+
echo "$sname" >> "$EXCL_FILE"
209+
else
210+
echo "$sname" >> "$FLEX_FILE"
211+
fi
192212
done < "$SER_FILE"
213+
xargs -P "${CBM_TAIL_JOBS:-2}" -I{} bash -c 'run_one "$@"' _ {} < "$FLEX_FILE"
214+
while IFS= read -r sname; do
215+
run_one "$sname"
216+
done < "$EXCL_FILE"
193217

194218
# Machine-checkable manifest for CI's cross-shard completeness job: it
195219
# proves at runtime that the shards of one leg agree on N and on the full

‎src/cli/cli.c‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10627,8 +10627,10 @@ int cbm_cmd_install(int argc, char **argv) {
1062710627
cli_binary_validator_t staged_validator = {{0}};
1062810628
if (stage_status != CBM_ACTIVATION_TRANSACTION_OK || !binary_transaction ||
1062910629
!cli_activation_transaction_expected_build(binary_transaction, &staged_validator)) {
10630-
(void)fprintf(stderr, "error: failed to stage install candidate: %s\n",
10631-
cbm_activation_transaction_status_message(stage_status));
10630+
const char *stage_refusal = cbm_activation_transaction_refusal_note();
10631+
(void)fprintf(stderr, "error: failed to stage install candidate: %s%s%s\n",
10632+
cbm_activation_transaction_status_message(stage_status),
10633+
stage_refusal[0] ? ": " : "", stage_refusal);
1063210634
(void)cli_activation_transaction_abort(&binary_transaction);
1063310635
if (prepared_dir[0]) {
1063410636
(void)cbm_rmdir(prepared_dir);

‎test-infrastructure/docker-compose.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ services:
2020
platform: linux/arm64
2121
volumes:
2222
- ..:/src
23+
- cbm-build:/src/build
24+
- cbm-fixture-cache:/root/.cache/cbm-test-fixtures
2325
- cbm-ccache-arm64:/root/.ccache
2426
environment:
2527
CCACHE_DIR: /root/.ccache
@@ -33,6 +35,8 @@ services:
3335
platform: linux/amd64
3436
volumes:
3537
- ..:/src
38+
- cbm-build:/src/build
39+
- cbm-fixture-cache:/root/.cache/cbm-test-fixtures
3640
- cbm-ccache-amd64:/root/.ccache
3741
environment:
3842
CCACHE_DIR: /root/.ccache
@@ -50,6 +54,8 @@ services:
5054
platform: linux/arm64
5155
volumes:
5256
- ..:/src
57+
- cbm-build:/src/build
58+
- cbm-fixture-cache:/root/.cache/cbm-test-fixtures
5359
- cbm-ccache-arm64:/root/.ccache
5460
environment:
5561
CCACHE_DIR: /root/.ccache
@@ -71,6 +77,8 @@ services:
7177
platform: linux/amd64
7278
volumes:
7379
- ..:/src
80+
- cbm-build:/src/build
81+
- cbm-fixture-cache:/root/.cache/cbm-test-fixtures
7482
- cbm-ccache-amd64:/root/.ccache
7583
environment:
7684
CCACHE_DIR: /root/.ccache
@@ -236,3 +244,9 @@ services:
236244
volumes:
237245
cbm-ccache-arm64:
238246
cbm-ccache-amd64:
247+
# Build artifacts and the incremental-suite fixture cache stay on the VM's
248+
# native filesystem: object writes over the virtiofs bind mount are the
249+
# container legs' largest avoidable I/O cost, and the fixture cache makes
250+
# the network clone once-per-volume instead of once-per-container.
251+
cbm-build:
252+
cbm-fixture-cache:

‎test-infrastructure/ladder.sh‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
#!/usr/bin/env bash
2+
# ladder.sh — the full local push gate, with all platform legs OVERLAPPED.
3+
#
4+
# The three platforms are separate scheduling domains (macOS native, the
5+
# Colima container VM, the Windows UTM VM), so running their legs
6+
# sequentially leaves most of the machine idle most of the time. This
7+
# launches lint + the Linux container suite + the Windows VM suite in the
8+
# background, runs the macOS full suite in the foreground, then joins and
9+
# prints one verdict per leg. Each leg streams to its own log so a red leg
10+
# can be inspected without rerunning anything.
11+
#
12+
# test-infrastructure/ladder.sh
13+
#
14+
# Prerequisites: Colima running (Linux/lint legs), the Windows VM
15+
# provisioned and reachable (vm/provision-windows.sh). A missing
16+
# prerequisite fails that leg loudly — never silently skips it.
17+
set -uo pipefail
18+
19+
cd "$(dirname "$0")/.."
20+
LOGS="${CBM_LADDER_LOGS:-${TMPDIR:-/tmp}/cbm-ladder-$$}"
21+
mkdir -p "$LOGS"
22+
23+
echo "=== local ladder: lint + linux + windows in background, mac in foreground ==="
24+
echo "=== logs: $LOGS ==="
25+
26+
./test-infrastructure/run.sh lint > "$LOGS/lint.log" 2>&1 &
27+
LINT_PID=$!
28+
./test-infrastructure/run.sh test > "$LOGS/linux.log" 2>&1 &
29+
LINUX_PID=$!
30+
(./test-infrastructure/vm/win.sh sync &&
31+
./test-infrastructure/vm/win.sh test-par) > "$LOGS/windows.log" 2>&1 &
32+
WIN_PID=$!
33+
34+
scripts/test.sh 2>&1 | tee "$LOGS/mac.log"
35+
MAC_RC=${PIPESTATUS[0]}
36+
37+
wait "$LINT_PID"
38+
LINT_RC=$?
39+
wait "$LINUX_PID"
40+
LINUX_RC=$?
41+
wait "$WIN_PID"
42+
WIN_RC=$?
43+
44+
echo ""
45+
echo "=== ladder verdicts ==="
46+
overall=0
47+
report() {
48+
local leg="$1" rc="$2" log="$3"
49+
if [ "$rc" -eq 0 ]; then
50+
echo " PASS $leg"
51+
else
52+
echo " FAIL $leg (rc=$rc) — $log"
53+
echo " ── last 15 lines ──"
54+
tail -15 "$log" | sed 's/^/ /'
55+
overall=1
56+
fi
57+
}
58+
report "macOS " "$MAC_RC" "$LOGS/mac.log"
59+
report "lint " "$LINT_RC" "$LOGS/lint.log"
60+
report "linux " "$LINUX_RC" "$LOGS/linux.log"
61+
report "windows " "$WIN_RC" "$LOGS/windows.log"
62+
exit "$overall"

‎test-infrastructure/vm/vm-run-tests.sh‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ set -uo pipefail
2020
RUNNER="${CBM_VM_RUNNER:-build/c/test-runner}"
2121
LOG="${CBM_VM_TEST_LOG:-/tmp/win-test.log}"
2222

23-
[ $# -ge 1 ] || { echo "usage: vm-run-tests.sh <suite...>" >&2; exit 2; }
23+
[ $# -ge 1 ] || { echo "usage: vm-run-tests.sh <suite...> | --par" >&2; exit 2; }
2424
[ -x "$RUNNER" ] || { echo "ERROR: runner '$RUNNER' missing — build first" >&2; exit 2; }
2525

2626
# Stale roots from earlier runs are removed up front; the current root is kept
@@ -70,8 +70,16 @@ MSYS2_ARG_CONV_EXCL='*' icacls "${runner_dir_w}\\*" /reset /T /C /Q >/dev/null 2
7070

7171
echo "=== vm-run-tests: runner=$RUNNER temp=$TEMP suites: $* ==="
7272

73-
"$RUNNER" "$@" 2>&1 | tee "$LOG"
74-
rc="${PIPESTATUS[0]}"
73+
# --par runs the repo's full parallel harness (wave + tail scheduling,
74+
# per-shard union guard, manifest) under this same protected environment;
75+
# explicit suite names keep the direct single-runner invocation.
76+
if [ "$1" = "--par" ]; then
77+
bash scripts/run-tests-parallel.sh "$RUNNER" 2>&1 | tee "$LOG"
78+
rc="${PIPESTATUS[0]}"
79+
else
80+
"$RUNNER" "$@" 2>&1 | tee "$LOG"
81+
rc="${PIPESTATUS[0]}"
82+
fi
7583

7684
if ! grep -Eq '[0-9]+ passed' "$LOG"; then
7785
echo "GUARD: test runner produced no completion summary — the suites did" \

‎test-infrastructure/vm/win.sh‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -191,8 +191,11 @@ pageheap)
191191
esac
192192
;;
193193
test-par)
194-
# Full-suite parallel run on all VM cores via the repo's parallel harness.
195-
vm clangarm64 "cd /c/cbm && bash scripts/run-tests-parallel.sh build/c/test-runner 2>&1 | tail -25"
194+
# Full-suite parallel run on all VM cores via the repo's parallel harness,
195+
# under the CI-shaped protected temp root with FULL output (this leg once
196+
# ran under the MSYS-shared /tmp and piped through `tail -25` — the same
197+
# truncated-blindness class that hid 40 Windows failures from `test`).
198+
vm clangarm64 "cd /c/cbm && CBM_VM_TEST_LOG=/tmp/win-test-par.log bash test-infrastructure/vm/vm-run-tests.sh --par"
196199
;;
197200
*)
198201
echo "unknown command: $cmd (see header for usage)" >&2; exit 2

‎tests/test_mem.c‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,11 @@
2121
#include <mimalloc.h>
2222
#ifndef _WIN32
2323
#include <sys/mman.h>
24+
#else
25+
#ifndef WIN32_LEAN_AND_MEAN
26+
#define WIN32_LEAN_AND_MEAN
27+
#endif
28+
#include <windows.h>
2429
#endif
2530

2631
/* ASan detection — mimalloc MI_OVERRIDE=0 under ASan, mi_process_info
@@ -275,11 +280,32 @@ TEST(mem_rss_reflects_external_resident_memory) {
275280
* reporting a broken small counter instead of true resident memory — which
276281
* the Linux #else branch exercises directly against the undercount. */
277282
const size_t threshold = (size_t)32 * 1024 * 1024;
283+
const size_t lock_span = (size_t)64 * 1024 * 1024;
278284
void *big = malloc(region);
279285
ASSERT_NOT_NULL(big);
280286
memset(big, 0x5A, region);
281-
memset(big, 0x5B, region); /* re-touch right before the measurement */
282-
size_t rss = cbm_mem_rss();
287+
/* Trimming can evict even a just-touched region: at 18 parallel suites
288+
* the VM kept 19 MB resident of a 256 MB double-touch, losing the
289+
* re-touch race this test previously relied on. Locked pages are exempt
290+
* from working-set trimming, so lock a span comfortably above the
291+
* threshold and the measurement becomes pressure-immune. When the lock
292+
* is unavailable (working-set quota policy), fall back to bounded
293+
* touch-and-sample retries — those races are transient. */
294+
HANDLE self_process = GetCurrentProcess();
295+
bool locked = SetProcessWorkingSetSize(self_process, lock_span + (size_t)32 * 1024 * 1024,
296+
(size_t)512 * 1024 * 1024) != 0 &&
297+
VirtualLock(big, lock_span) != 0;
298+
size_t rss = 0;
299+
for (int attempt = 0; attempt < 6; attempt++) {
300+
memset(big, 0x5B + attempt, lock_span);
301+
rss = cbm_mem_rss();
302+
if (locked || rss >= threshold) {
303+
break;
304+
}
305+
}
306+
if (locked) {
307+
(void)VirtualUnlock(big, lock_span);
308+
}
283309
ASSERT_GTE(rss, threshold);
284310
free(big);
285311
#else

0 commit comments

Comments
 (0)