Skip to content

Commit 36075da

Browse files
authored
Merge pull request #24 from ngallodev/feat/test-scheduler
Add a test-level scheduler with per-test timeouts and resume (scripts/test.sh --smart)
2 parents 0756d9d + d703c2f commit 36075da

10 files changed

Lines changed: 953 additions & 36 deletions

File tree

‎AGENTS.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@
77
files and the compiler cache.
88
- Prefer `scripts/test.sh --suites <suite>` for local validation. This uses the
99
incremental test runner; choose the smallest relevant suite set.
10+
- For resumable per-test iteration, use `scripts/test.sh --smart` with
11+
`--suites`, `--changed`, `--tests`, or `--failed`; use `--dry-run` on
12+
`scripts/testsched/schedule.py` to inspect its timing based plan.
1013
- Do not use `scripts/build.sh` or the default `scripts/test.sh` for ordinary
1114
iteration. Those are clean, complete venue gates for GitHub, Jenkins, and
1215
release validation.

‎CONTRIBUTING.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,15 @@
22

33
Contributions are welcome. This guide covers setup, testing, and PR guidelines.
44

5+
### Fast test iteration
6+
7+
Use `scripts/test.sh --smart` to schedule individual tests in resumable batches.
8+
Narrow a run with `--suites 'cli daemon_ipc'`, `--tests suite:test,...`,
9+
`--changed [BASE]`, or `--failed`; `--jobs N` controls concurrency. Add
10+
`--tool valgrind` for per-test memcheck runs. `--dry-run` is available on the
11+
scheduler (`python3 scripts/testsched/schedule.py --runner build/c/test-runner --dry-run`).
12+
The regular `scripts/test.sh` remains the full CI venue gate.
13+
514
> **Important**: This project is a **pure C binary** (rewritten from Go in v0.5.0). Please submit C code, not Go. Go PRs may be ported but cannot be merged directly.
615
716
## Build from Source

‎scripts/test.sh‎

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,19 +5,43 @@
55

66
set -euo pipefail
77

8+
SMART=0
9+
SMART_ARGS=()
10+
FILTERED_ARGS=()
11+
while (($#)); do
12+
case "$1" in
13+
--smart) SMART=1; shift ;;
14+
--dry-run) SMART_ARGS+=(--dry-run); shift ;;
15+
--tool|--jobs|--tests) SMART_ARGS+=("$1" "${2:?$1 needs a value}"); shift 2 ;;
16+
--attribute|--no-attribute) SMART_ARGS+=("$1"); shift ;;
17+
--changed)
18+
SMART_ARGS+=(--changed)
19+
if (($# > 1)) && [[ "$2" != -* ]]; then SMART_ARGS+=("$2"); shift 2; else shift; fi ;;
20+
--failed) SMART_ARGS+=(--failed); shift ;;
21+
*) FILTERED_ARGS+=("$1"); shift ;;
22+
esac
23+
done
24+
set -- "${FILTERED_ARGS[@]}"
25+
if [ "$SMART" -eq 0 ] && [ "${#SMART_ARGS[@]}" -gt 0 ]; then
26+
echo "test.sh: scheduler options require --smart" >&2
27+
exit 2
28+
fi
29+
830
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
931
cd "$ROOT"
1032

1133
usage() {
1234
cat <<'EOF'
13-
Usage: scripts/test.sh [--suites LIST] [--phase PHASE] [--arch ARCH] [VAR=VAL ...]
35+
Usage: scripts/test.sh [--smart [--tool valgrind] [--suites LIST|--changed [BASE]|--failed] [--jobs N]] [--suites LIST] [--phase PHASE] [--arch ARCH] [VAR=VAL ...]
1436
1537
The canonical test entry: identical in local CI, PR CI, dry run and release.
1638
DEFAULT (no --suites) is exactly what CI runs: static contract checks
1739
(Step 0a-0t), a CLEAN sanitizer build, every suite via the parallel harness,
1840
then the prod-binary regression guards (Steps 4-6).
1941
2042
Modes:
43+
--smart Test-level resumable iteration scheduler; accepts --suites,
44+
--tests, --changed [BASE], --failed, --tool valgrind, --jobs.
2145
(default) The venue leg. Clean build (scripts/clean.sh) + all suites +
2246
all contract steps. This is the shape every gate runs.
2347
--suites LIST ITERATION mode: comma- or space-separated suite names, e.g.
@@ -269,6 +293,25 @@ if [ "$TSAN" -eq 1 ]; then
269293
exit "$?"
270294
fi
271295

296+
if [ "$SMART" -eq 1 ]; then
297+
BUILD_JOBS="$NPROC"
298+
if [[ " ${SMART_ARGS[*]} " == *" --tool valgrind "* ]] && [ "$SANITIZE_GIVEN" -eq 0 ]; then
299+
if [ "$BUILD_JOBS" -gt 16 ]; then BUILD_JOBS=16; fi
300+
if [ "$BUILD_DIR" = "build/c" ]; then
301+
BUILD_DIR="build/c-valgrind"
302+
MAKE_ARGS+=("BUILD_DIR=$BUILD_DIR")
303+
fi
304+
MAKE_ARGS+=("SANITIZE=")
305+
EXPECTED_SANITIZED=0
306+
fi
307+
make -j"$BUILD_JOBS" -f Makefile.cbm "$BUILD_DIR/test-runner" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
308+
assert_test_runner_build_config "$BUILD_DIR/test-runner" "$EXPECTED_SANITIZED"
309+
MODE=asan; [ "$EXPECTED_SANITIZED" -eq 1 ] || MODE=plain
310+
SCHED_ARGS=(--runner "$BUILD_DIR/test-runner" --mode "$MODE")
311+
if [ -n "$SUITES" ]; then read -r -a SUITE_ARGS <<< "$SUITES"; SCHED_ARGS+=(--suites "${SUITE_ARGS[@]}"); fi
312+
exec python3 "$ROOT/scripts/testsched/schedule.py" "${SCHED_ARGS[@]}" "${SMART_ARGS[@]}"
313+
fi
314+
272315
# ── Iteration mode (--suites): incremental rebuild + subset run ──
273316
# The documented fast path: no clean, no contract steps, no prod-binary
274317
# guards — those all still gate every merge through the default full run.

‎scripts/testsched/planner.py‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
"""Build setup-aware consecutive batches and expose their timing plan."""
2+
from __future__ import annotations
3+
4+
from dataclasses import dataclass
5+
6+
7+
# Add a suite only after a full split run proves that parallel batches break
8+
# its order/state assumptions. Each entry needs a short reason.
9+
SERIAL_SUITES: dict[str, str] = {
10+
"daemon_application": "The 10-second default-capacity admission fixture timed out in the split tail batch under full-run load.",
11+
}
12+
13+
14+
@dataclass(frozen=True)
15+
class Batch:
16+
suite: str
17+
tests: tuple[str, ...]
18+
setup_ms: float
19+
work_ms: float
20+
21+
@property
22+
def estimate_ms(self) -> float:
23+
return self.setup_ms + self.work_ms
24+
25+
26+
def plan_batches(
27+
names: list[str], timing, target_seconds: float, multiplier: float
28+
) -> list[Batch]:
29+
"""Keep registration order within each suite and account for setup cost."""
30+
by_suite: dict[str, list[str]] = {}
31+
for name in names:
32+
suite = name.split(":", 1)[0]
33+
by_suite.setdefault(suite, []).append(name)
34+
35+
batches = []
36+
target_ms = target_seconds * multiplier * 1000
37+
for suite, tests in by_suite.items():
38+
setup_ms = timing.setup_ms(suite)
39+
if suite in SERIAL_SUITES:
40+
batches.append(
41+
Batch(suite, tuple(tests), setup_ms, sum(timing.estimate_ms(n) for n in tests))
42+
)
43+
continue
44+
45+
# Keep setup at most one quarter of the estimate, even when setup cost
46+
# pushes the minimum batch above the requested target.
47+
minimum_work_ms = 3 * setup_ms
48+
target_work_ms = max(target_ms - setup_ms, minimum_work_ms)
49+
current: list[str] = []
50+
work_ms = 0.0
51+
for name in tests:
52+
test_ms = timing.estimate_ms(name)
53+
if current and work_ms >= target_work_ms:
54+
batches.append(Batch(suite, tuple(current), setup_ms, work_ms))
55+
current = []
56+
work_ms = 0.0
57+
current.append(name)
58+
work_ms += test_ms
59+
if current:
60+
if batches and batches[-1].suite == suite and work_ms < minimum_work_ms:
61+
previous = batches.pop()
62+
current = list(previous.tests) + current
63+
work_ms += previous.work_ms
64+
batches.append(Batch(suite, tuple(current), setup_ms, work_ms))
65+
return batches
66+
67+
68+
def ordered_batches(batches: list[Batch]) -> list[tuple[int, Batch]]:
69+
"""Return stable longest-processing-time-first queue priorities."""
70+
return sorted(enumerate(batches), key=lambda item: (-item[1].estimate_ms, item[0]))
71+
72+
73+
def print_plan(names: list[str], batches: list[Batch], jobs: int, mode: str) -> None:
74+
suites = {}
75+
for batch in batches:
76+
suites[batch.suite] = batch.setup_ms
77+
print(f"plan: {len(names)} tests, {len(batches)} batches, {jobs} workers, {mode}")
78+
for suite in sorted(suites):
79+
print(f"setup {suite}: {suites[suite]:.0f}ms")
80+
for index, batch in ordered_batches(batches):
81+
print(
82+
f"batch {index + 1}: {batch.suite}, {len(batch.tests)} tests, "
83+
f"setup {batch.setup_ms:.0f}ms, work {batch.work_ms:.0f}ms, "
84+
f"estimate {batch.estimate_ms:.0f}ms"
85+
)

0 commit comments

Comments
 (0)