Skip to content

Commit 950adc1

Browse files
committed
Add resumable test-level scheduler
Add test listing and opt-in machine-readable per-test timing results. Schedule selected tests into suite-ordered batches with per-test watchdogs, timeout resume, union accounting, timing history, changed/failed selectors, and Valgrind result aggregation. Route opt-in smart runs through scripts/test.sh and document the iteration flow. Validation: build-dev, lint, scheduler syntax and shell checks, listing and five-suite comparison, full smart run, timeout resume proof, same-build parallel baseline, and bounded Valgrind runs. See the acceptance report for results and remaining Valgrind coverage.
1 parent 4f6ede9 commit 950adc1

6 files changed

Lines changed: 559 additions & 54 deletions

File tree

‎AGENTS.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@
77
files and the compiler cache.
88
- Prefer `scripts/test.sh --suites <suite>` for local validation. This uses the
99
incremental test runner; choose the smallest relevant suite set.
10+
- For resumable per-test iteration, use `scripts/test.sh --smart` with
11+
`--suites`, `--changed`, `--tests`, or `--failed`; use `--dry-run` on
12+
`scripts/testsched/schedule.py` to inspect its timing based plan.
1013
- Do not use `scripts/build.sh` or the default `scripts/test.sh` for ordinary
1114
iteration. Those are clean, complete venue gates for GitHub, Jenkins, and
1215
release validation.

‎CONTRIBUTING.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,15 @@
22

33
Contributions are welcome. This guide covers setup, testing, and PR guidelines.
44

5+
### Fast test iteration
6+
7+
Use `scripts/test.sh --smart` to schedule individual tests in resumable batches.
8+
Narrow a run with `--suites 'cli daemon_ipc'`, `--tests suite:test,...`,
9+
`--changed [BASE]`, or `--failed`; `--jobs N` controls concurrency. Add
10+
`--tool valgrind` for per-test memcheck runs. `--dry-run` is available on the
11+
scheduler (`python3 scripts/testsched/schedule.py --runner build/c/test-runner --dry-run`).
12+
The regular `scripts/test.sh` remains the full CI venue gate.
13+
514
> **Important**: This project is a **pure C binary** (rewritten from Go in v0.5.0). Please submit C code, not Go. Go PRs may be ported but cannot be merged directly.
615
716
## Build from Source

‎scripts/test.sh‎

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,19 +5,42 @@
55

66
set -euo pipefail
77

8+
SMART=0
9+
SMART_ARGS=()
10+
FILTERED_ARGS=()
11+
while (($#)); do
12+
case "$1" in
13+
--smart) SMART=1; shift ;;
14+
--dry-run) SMART_ARGS+=(--dry-run); shift ;;
15+
--tool|--jobs|--tests) SMART_ARGS+=("$1" "${2:?$1 needs a value}"); shift 2 ;;
16+
--changed)
17+
SMART_ARGS+=(--changed)
18+
if (($# > 1)) && [[ "$2" != -* ]]; then SMART_ARGS+=("$2"); shift 2; else shift; fi ;;
19+
--failed) SMART_ARGS+=(--failed); shift ;;
20+
*) FILTERED_ARGS+=("$1"); shift ;;
21+
esac
22+
done
23+
set -- "${FILTERED_ARGS[@]}"
24+
if [ "$SMART" -eq 0 ] && [ "${#SMART_ARGS[@]}" -gt 0 ]; then
25+
echo "test.sh: scheduler options require --smart" >&2
26+
exit 2
27+
fi
28+
829
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
930
cd "$ROOT"
1031

1132
usage() {
1233
cat <<'EOF'
13-
Usage: scripts/test.sh [--suites LIST] [--phase PHASE] [--arch ARCH] [VAR=VAL ...]
34+
Usage: scripts/test.sh [--smart [--tool valgrind] [--suites LIST|--changed [BASE]|--failed] [--jobs N]] [--suites LIST] [--phase PHASE] [--arch ARCH] [VAR=VAL ...]
1435
1536
The canonical test entry: identical in local CI, PR CI, dry run and release.
1637
DEFAULT (no --suites) is exactly what CI runs: static contract checks
1738
(Step 0a-0t), a CLEAN sanitizer build, every suite via the parallel harness,
1839
then the prod-binary regression guards (Steps 4-6).
1940
2041
Modes:
42+
--smart Test-level resumable iteration scheduler; accepts --suites,
43+
--tests, --changed [BASE], --failed, --tool valgrind, --jobs.
2144
(default) The venue leg. Clean build (scripts/clean.sh) + all suites +
2245
all contract steps. This is the shape every gate runs.
2346
--suites LIST ITERATION mode: comma- or space-separated suite names, e.g.
@@ -269,6 +292,25 @@ if [ "$TSAN" -eq 1 ]; then
269292
exit "$?"
270293
fi
271294

295+
if [ "$SMART" -eq 1 ]; then
296+
BUILD_JOBS="$NPROC"
297+
if [[ " ${SMART_ARGS[*]} " == *" --tool valgrind "* ]] && [ "$SANITIZE_GIVEN" -eq 0 ]; then
298+
if [ "$BUILD_JOBS" -gt 16 ]; then BUILD_JOBS=16; fi
299+
if [ "$BUILD_DIR" = "build/c" ]; then
300+
BUILD_DIR="build/c-valgrind"
301+
MAKE_ARGS+=("BUILD_DIR=$BUILD_DIR")
302+
fi
303+
MAKE_ARGS+=("SANITIZE=")
304+
EXPECTED_SANITIZED=0
305+
fi
306+
make -j"$BUILD_JOBS" -f Makefile.cbm "$BUILD_DIR/test-runner" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
307+
assert_test_runner_build_config "$BUILD_DIR/test-runner" "$EXPECTED_SANITIZED"
308+
MODE=asan; [ "$EXPECTED_SANITIZED" -eq 1 ] || MODE=plain
309+
SCHED_ARGS=(--runner "$BUILD_DIR/test-runner" --mode "$MODE")
310+
if [ -n "$SUITES" ]; then read -r -a SUITE_ARGS <<< "$SUITES"; SCHED_ARGS+=(--suites "${SUITE_ARGS[@]}"); fi
311+
exec python3 "$ROOT/scripts/testsched/schedule.py" "${SCHED_ARGS[@]}" "${SMART_ARGS[@]}"
312+
fi
313+
272314
# ── Iteration mode (--suites): incremental rebuild + subset run ──
273315
# The documented fast path: no clean, no contract steps, no prod-binary
274316
# guards — those all still gate every merge through the default full run.

0 commit comments

Comments
 (0)