Skip to content

Commit 674a731

Browse files
committed
ci(test-impact): give the engine full history; publish maps with an incomplete suite
The test-impact engine certifies the history it diffs and refuses a shallow clone ("Graft or shallow state prevents history certification"). The shadow job, the gate-mode select-tests job and the main-branch producer all checked out with fetch-depth 2. So every shadow answer since DeusData#2458 was run_all: GIT_UNAVAILABLE, then POLICY_UNAVAILABLE, because the merge base's .codebase-memory.json is read through the same git facts. That was 24 of 24 measured PR runs. All three jobs now check out full history (about 400 MiB, these jobs only). The producer never published a bundle either. coverage-map.py exits 1 when a suite fails under coverage: the map is still written and that suite's tests are incomplete. test-impact-publish.sh ran it under set -e, so the publish aborted. The failing suite was test_impact_git. Its hostile-diff probe re-enters the runner as a child, which inherited CBM_TEST_COVERAGE_DIR, started a coverage run of its own, and refused the per-test LLVM_PROFILE_FILE it inherits. Two changes: - The probe now clears CBM_TEST_COVERAGE_DIR, as it already clears CBM_TEST_ONLY_FILE. The child's profile then goes to the spawning test, like any other child process. - publish treats exit 1 as a written, conservative map (incomplete tests run whole), with a warning that names the suite. It fails only on exit 2. Proof: - Depth-2 clone at f9ce386: test-impact select answers run_all [GIT_UNAVAILABLE, POLICY_UNAVAILABLE] with "Graft or shallow state ...", as in CI. Same commit with full history: graph certified, run_all [RUNNER_REACHED] (the change reaches the runner's main). - Coverage runner (Homebrew LLVM, macOS), test_impact_git: before, 43 passed and 2 failed ("a coverage run starts with LLVM_PROFILE_FILE=/dev/null"); after, 45 passed. coverage-map.py --suite test_impact_git: 45 tests, 0 incomplete, exit 0. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent c202539 commit 674a731

4 files changed

Lines changed: 28 additions & 7 deletions

File tree

‎.github/workflows/pr.yml‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,9 @@ jobs:
6565
steps:
6666
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
6767
with:
68-
# The PR merge commit and its base parent: the change to select for.
69-
fetch-depth: 2
68+
# Full history: the engine certifies the history it diffs and refuses
69+
# a shallow clone, so a depth-2 checkout always answered `all`.
70+
fetch-depth: 0
7071
persist-credentials: false
7172
- name: Build this checkout's engine
7273
run: |
@@ -281,8 +282,11 @@ jobs:
281282
steps:
282283
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
283284
with:
284-
# The PR merge commit and its base parent: the diff to predict for.
285-
fetch-depth: 2
285+
# Full history: the engine (test-impact select) certifies the history
286+
# it diffs and refuses a shallow clone ("Graft or shallow state
287+
# prevents history certification"), which made every shadow answer
288+
# run_all (GIT_UNAVAILABLE). About 400 MiB, this job only.
289+
fetch-depth: 0
286290
persist-credentials: false
287291
- name: Fetch the latest released binary
288292
id: release

‎.github/workflows/test-impact-artifact.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,9 @@ jobs:
3838
steps:
3939
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
4040
with:
41-
fetch-depth: 2
41+
# Full history: `test-impact select/publish` certify the history they
42+
# read and refuse a shallow clone.
43+
fetch-depth: 0
4244
persist-credentials: false
4345

4446
- name: Install build deps and LLVM 21 (coverage toolchain)

‎scripts/ci/test-impact-publish.sh‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,19 @@ fi
5656
map() { # out [suite...]
5757
local out="$1"; shift
5858
local args=()
59+
local rc=0
5960
for suite in "$@"; do args+=(--suite "$suite"); done
6061
python3 "$ROOT/scripts/test-impact/coverage-map.py" --runner "$RUNNER" --out "$out" \
61-
${LLVM_BIN:+--llvm-bin "$LLVM_BIN"} ${args[@]+"${args[@]}"}
62+
${LLVM_BIN:+--llvm-bin "$LLVM_BIN"} ${args[@]+"${args[@]}"} || rc=$?
63+
# 1 = a suite failed or timed out under coverage: the map is still written
64+
# and that suite's tests are `incomplete`, which only ever selects MORE
65+
# tests (an incomplete test runs whole). Publish it, but say which suite.
66+
# 2 = usage or toolchain error: no trustworthy map, fail.
67+
if [ "$rc" -eq 1 ]; then
68+
echo "::warning::coverage map: a suite failed under coverage; its tests stay incomplete ($(python3 -c 'import json,sys; print(", ".join(s["suite"] for s in json.load(open(sys.argv[1]))["suites"] if s["exit"] != 0))' "$out/meta.json"))"
69+
return 0
70+
fi
71+
return "$rc"
6272
}
6373

6474
OBSERVED=""

‎tests/test_test_impact_git.c‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,9 +64,14 @@ int tf_maybe_run_git_facts_diff_probe(int argc, char **argv) {
6464
* Re-enter the existing exact-diff test; the parent process remains untouched.
6565
* An inherited CBM_TEST_ONLY_FILE (a narrowed CI run) is cleared: the runner
6666
* unions it with CBM_TEST_ONLY, and one naming this suite would re-run the
67-
* spawning test, which spawns again. */
67+
* spawning test, which spawns again. An inherited CBM_TEST_COVERAGE_DIR (a
68+
* coverage-map run) is cleared too: with it the child would start a coverage
69+
* run of its own, which refuses the per-test LLVM_PROFILE_FILE it inherits.
70+
* Without it the child's profile goes to that file, credited to the spawning
71+
* test like any other child process. */
6872
if (cbm_setenv("GIT_DIFF_OPTS", "--unified=999", 1) != 0 ||
6973
cbm_setenv("CBM_TEST_ONLY_FILE", "", 1) != 0 ||
74+
cbm_unsetenv("CBM_TEST_COVERAGE_DIR") != 0 ||
7075
cbm_setenv(
7176
"CBM_TEST_ONLY",
7277
"test_impact_git:test_git_facts_diff_uses_merge_base_zero_context_and_nul_metadata",

0 commit comments

Comments
 (0)