Skip to content

Commit 24608fc

Browse files
TMHSDigitalclaude
andcommitted
ci: shard the Blender smoke catalog three ways per series
One job per series ran every catalog row and falsifier serially, a cold Blender launch each: 15-40 minutes per run, growing by two launches per series with every new example toward the 75-minute timeout. run_catalog.py takes --shard K/N (index mod N, the scheme the weekly showcase sweep already uses; the helpers now live once in protocol.py), and the smoke matrix is series x 3 shards. Template, harness-driver and canary steps run on shard 0 only; each shard summarizes its own rows and stays red on zero PASS. Unit tests prove the shards partition the catalog and an empty shard is red. Closes #529 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LpkeXRvikhJZSVGBFM91Dv Signed-off-by: TM Hospitality Strategies <TMHospitalityStrategies@gmail.com>
1 parent 3c21c8c commit 24608fc

6 files changed

Lines changed: 84 additions & 24 deletions

File tree

‎.github/workflows/blender-smoke.yml‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -106,15 +106,21 @@ jobs:
106106
echo "Smoke matrix: $json"
107107
108108
smoke:
109-
name: Blender ${{ matrix.series }} smoke
109+
# The catalog (a cold Blender launch per happy path and per falsifier) is
110+
# split across three parallel jobs per series (#529): one serial job
111+
# took 15-40 minutes and grew two launches per series with every example.
112+
# Shard 0 also runs the template, harness-driver and canary steps, which
113+
# are not catalog rows.
114+
name: Blender ${{ matrix.series }} smoke (${{ matrix.shard }}/3)
110115
needs: resolve-matrix
111116
if: needs.resolve-matrix.result == 'success'
112117
runs-on: ubuntu-latest
113-
timeout-minutes: 75 # happy paths + one falsifier per catalog row (~2x the old run)
118+
timeout-minutes: 45
114119
strategy:
115120
fail-fast: false
116121
matrix:
117122
series: ${{ fromJSON(needs.resolve-matrix.outputs.series) }}
123+
shard: [0, 1, 2] # keep in step with the K/3 below
118124
steps:
119125
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
120126

@@ -203,17 +209,20 @@ jobs:
203209
|| { echo "::error::version does not match series ${{ matrix.series }}"; exit 1; }
204210
205211
- name: Run in-Blender smoke driver
212+
if: matrix.shard == 0
206213
run: |
207214
set -euo pipefail
208215
mkdir -p "$RUNNER_TEMP/out"
209216
xvfb-run -a "$BLENDER" --background --python-exit-code 1 --python tests/smoke/run_smoke.py -- "$RUNNER_TEMP/out"
210217
211218
- name: Build template input scene
219+
if: matrix.shard == 0
212220
run: |
213221
set -euo pipefail
214222
xvfb-run -a "$BLENDER" --background --python-exit-code 1 --python tests/smoke/make_input.py -- "$RUNNER_TEMP/out/input.blend"
215223
216224
- name: Headless glTF template runs (exit 0, .glb produced)
225+
if: matrix.shard == 0
217226
run: |
218227
set -euo pipefail
219228
xvfb-run -a "$BLENDER" --background "$RUNNER_TEMP/out/input.blend" \
@@ -223,6 +232,7 @@ jobs:
223232
head -c4 "$RUNNER_TEMP/out/out.glb" | grep -q "glTF" || { echo "::error::not a glTF binary"; exit 1; }
224233
225234
- name: Headless template no-mesh path returns exit 5
235+
if: matrix.shard == 0
226236
run: |
227237
set +e
228238
xvfb-run -a "$BLENDER" --background "$RUNNER_TEMP/out/empty.blend" \
@@ -233,6 +243,7 @@ jobs:
233243
echo "no-mesh exit code = $code (correct)"
234244
235245
- name: Headless template applies the new modifier after the existing stack (#468)
246+
if: matrix.shard == 0
236247
run: |
237248
set -euo pipefail
238249
# stack.blend: a cube with a live SUBSURF (levels 1). SUBSURF then
@@ -245,18 +256,21 @@ jobs:
245256
"$RUNNER_TEMP/out/stack.glb" 48
246257
247258
- name: Extension template passes Blender's own manifest validator
259+
if: matrix.shard == 0
248260
run: |
249261
set -euo pipefail
250262
"$BLENDER" --command extension validate templates/extension-addon-template
251263
252264
- name: Build pipeline template fixture GLB
265+
if: matrix.shard == 0
253266
run: |
254267
set -euo pipefail
255268
xvfb-run -a "$BLENDER" --background --python-exit-code 1 --python tests/smoke/make_pipeline_glb.py -- \
256269
"$RUNNER_TEMP/out/pipeline_src.glb"
257270
test -s "$RUNNER_TEMP/out/pipeline_src.glb" || { echo "::error::pipeline fixture missing"; exit 1; }
258271
259272
- name: Headless AI asset pipeline template runs (exit 0, LODs produced)
273+
if: matrix.shard == 0
260274
run: |
261275
set -euo pipefail
262276
mkdir -p "$RUNNER_TEMP/out/pipeline"
@@ -271,6 +285,7 @@ jobs:
271285
head -c4 "$RUNNER_TEMP/out/pipeline/lod0.glb" | grep -q "glTF" || { echo "::error::lod0 not a glTF binary"; exit 1; }
272286
273287
- name: Headless pipeline template missing input returns exit 13
288+
if: matrix.shard == 0
274289
run: |
275290
set +e
276291
xvfb-run -a "$BLENDER" --background --python-exit-code 1 --python templates/ai-asset-pipeline-template/pipeline.py -- \
@@ -283,6 +298,7 @@ jobs:
283298
echo "missing-input exit code = $code (correct)"
284299
285300
- name: Pipeline template keeps every part of a shared-mesh, parented GLB (#462, #467)
301+
if: matrix.shard == 0
286302
run: |
287303
set -euo pipefail
288304
# Root rotated 90 deg on X, scaled 2x, at z=3; Body plus two wheels
@@ -301,6 +317,7 @@ jobs:
301317
"$RUNNER_TEMP/out/parented_src.glb" "$RUNNER_TEMP/out/pipeline_parented/lod0.glb"
302318
303319
- name: Falsifier - pipeline template rejects a 2 km asset with exit 8
320+
if: matrix.shard == 0
304321
run: |
305322
set -euo pipefail
306323
xvfb-run -a "$BLENDER" --background --python-exit-code 1 --python tests/smoke/make_pipeline_glb.py -- \
@@ -316,6 +333,7 @@ jobs:
316333
echo "out-of-range extent exit code = $code (correct)"
317334
318335
- name: Headless render template runs (exit 0, PNG produced)
336+
if: matrix.shard == 0
319337
run: |
320338
set -euo pipefail
321339
# Cycles (CPU) so this is reliable on GPU-less runners; the EEVEE-id regression
@@ -329,6 +347,7 @@ jobs:
329347
# file would make summarize.py's "zero PASS is red" guard unreachable
330348
# (#536). Each one asserts its own verdict inline instead.
331349
- name: Canary - expected skip is SKIP not PASS
350+
if: matrix.shard == 0
332351
run: |
333352
set -euo pipefail
334353
python3 tests/smoke/run_example.py --no-record \
@@ -342,6 +361,7 @@ jobs:
342361
|| { echo "::error::expected skip was not classified SKIP"; exit 1; }
343362
344363
- name: Canary - unexpected skip is red
364+
if: matrix.shard == 0
345365
run: |
346366
set +e
347367
python3 tests/smoke/run_example.py --no-record \
@@ -357,6 +377,7 @@ jobs:
357377
echo "unexpected skip exited 1 (correct)"
358378
359379
- name: Canary - uncaught exception is red
380+
if: matrix.shard == 0
360381
run: |
361382
set +e
362383
python3 tests/smoke/run_example.py --no-record \
@@ -371,6 +392,7 @@ jobs:
371392
echo "uncaught exception exited 1 (correct)"
372393
373394
- name: Canary - post-exit sidecar
395+
if: matrix.shard == 0
374396
run: |
375397
set -euo pipefail
376398
python3 tests/smoke/run_example.py --no-record \
@@ -385,6 +407,7 @@ jobs:
385407
|| { echo "::error::post-exit sidecar canary did not PASS"; exit 1; }
386408
387409
- name: Canary - missing sidecar is red
410+
if: matrix.shard == 0
388411
run: |
389412
set +e
390413
python3 tests/smoke/run_example.py --no-record \
@@ -407,6 +430,7 @@ jobs:
407430
python3 tests/smoke/run_catalog.py \
408431
--blender "$BLENDER" \
409432
--series "${{ matrix.series }}" \
433+
--shard "${{ matrix.shard }}/3" \
410434
--status "$BDT_SMOKE_STATUS" \
411435
--out "$RUNNER_TEMP/out" \
412436
--xvfb \

‎AGENTS.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -292,8 +292,8 @@ Should the repository move to an organization, or `release.yml` stop pushing
292292
to `main`, these are the checks that run unconditionally on every PR and are
293293
therefore the required-check set:
294294

295-
- `Blender 4.5 smoke`
296-
- `Blender 5.2 smoke`
295+
- `Blender 4.5 smoke (0/3)`, `(1/3)`, `(2/3)` (the catalog is sharded three ways, #529)
296+
- `Blender 5.2 smoke (0/3)`, `(1/3)`, `(2/3)`
297297
- `Ecosystem drift check`
298298
- `Validate content counts`
299299
- `Validate plugin manifest`
@@ -302,14 +302,14 @@ therefore the required-check set:
302302

303303
Three checks that appear on PRs are deliberately **excluded**:
304304

305-
- **`Blender 5.1 smoke` is label-gated.** It only runs when `needs-5.1` is
305+
- **`Blender 5.1 smoke (K/3)` is label-gated.** It only runs when `needs-5.1` is
306306
applied. As a required check it would block every unlabeled PR forever on
307307
a job that never reports.
308308
- **`Auto-label by path`** — `label-sync.yml` triggers on `opened` and
309309
`synchronize` only. A reopened PR with no new push never reports it.
310310
- **`Resolve smoke matrix`** — skipped on label events other than
311311
`needs-5.1`, and it is plumbing rather than a gate; its failure already
312-
blocks the two smoke jobs that depend on it.
312+
blocks the smoke jobs that depend on it.
313313

314314
The two `Socket Security` checks come from a third-party GitHub App. An
315315
outage or an uninstall would deadlock merges, so they are advisory.

‎tests/check_falsifier_targets.py‎

Lines changed: 1 addition & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@
5454

5555
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
5656
sys.path.insert(0, os.path.join(ROOT, "tests", "smoke"))
57-
from protocol import FORBIDDEN_FALSIFIER_EXITS # noqa: E402
57+
from protocol import FORBIDDEN_FALSIFIER_EXITS, in_shard, parse_shard # noqa: E402,F401
5858

5959
SHOWCASE = os.path.join(ROOT, "showcase")
6060

@@ -236,21 +236,6 @@ def check_static():
236236
RUN_TIMEOUT = 900 # seconds per Blender run, matching tests/smoke/run_example.py
237237

238238

239-
def parse_shard(text):
240-
"""'K/N' -> (K, N) with 0 <= K < N; argparse-friendly."""
241-
try:
242-
k, n = (int(x) for x in text.split("/"))
243-
except ValueError:
244-
raise argparse.ArgumentTypeError(f"--shard wants K/N, got {text!r}")
245-
if n < 1 or not 0 <= k < n:
246-
raise argparse.ArgumentTypeError(f"--shard {text!r}: need 0 <= K < N")
247-
return k, n
248-
249-
250-
def in_shard(index, shard):
251-
return shard is None or index % shard[1] == shard[0]
252-
253-
254239
def check_runtime(blender, only=None, shard=None):
255240
failures, checked = [], 0
256241
for index, (name, readme, script) in enumerate(piece_paths()):

‎tests/smoke/protocol.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
"""
1111
from __future__ import annotations
1212

13+
import argparse
1314
import os
1415
from typing import Optional, Tuple
1516

@@ -215,3 +216,20 @@ def classify_content_falsifier(phase, error_type):
215216
if error_type in BROKEN_CODE_ERRORS:
216217
return FAIL, f"check raised {error_type}: the edit broke the code, not the contract"
217218
return FALSIFIED, ""
219+
220+
221+
# Sharding: run_catalog.py (PR smoke, #529) and check_falsifier_targets.py (the
222+
# weekly showcase sweep, #482) split their rows across parallel jobs the same way.
223+
def parse_shard(text):
224+
"""'K/N' -> (K, N) with 0 <= K < N; argparse-friendly."""
225+
try:
226+
k, n = (int(x) for x in text.split("/"))
227+
except ValueError:
228+
raise argparse.ArgumentTypeError(f"--shard wants K/N, got {text!r}")
229+
if n < 1 or not 0 <= k < n:
230+
raise argparse.ArgumentTypeError(f"--shard {text!r}: need 0 <= K < N")
231+
return k, n
232+
233+
234+
def in_shard(index, shard):
235+
return shard is None or index % shard[1] == shard[0]

‎tests/smoke/run_catalog.py‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@
1313
A row may set ``timeout`` (seconds) for an example that legitimately needs
1414
longer than run_example.py's default; a hung run is killed and reported as a
1515
failure, and the catalog moves on to the next row.
16+
17+
``--shard K/N`` runs only rows whose catalog index is K mod N, so CI can split
18+
the catalog across parallel jobs (#529); the shards partition every row.
1619
"""
1720
from __future__ import annotations
1821

@@ -24,10 +27,15 @@
2427

2528
HERE = os.path.dirname(os.path.abspath(__file__))
2629
sys.path.insert(0, HERE)
27-
from protocol import parse_version # noqa: E402
30+
from protocol import in_shard, parse_shard, parse_version # noqa: E402
2831
from run_example import append_status # noqa: E402
2932

3033

34+
def select_rows(catalog, shard):
35+
"""The rows this shard runs; every row lands in exactly one shard."""
36+
return [row for i, row in enumerate(catalog) if in_shard(i, shard)]
37+
38+
3139
def main(argv=None):
3240
p = argparse.ArgumentParser()
3341
p.add_argument("--blender", required=True)
@@ -39,13 +47,20 @@ def main(argv=None):
3947
p.add_argument("--fail-fast", action="store_true", help="stop at the first failure")
4048
p.add_argument("--no-falsifiers", action="store_true",
4149
help="run only the happy paths (local iteration; CI runs falsifiers)")
50+
p.add_argument("--shard", type=parse_shard, default=None, metavar="K/N",
51+
help="run only rows with index %% N == K (parallel CI jobs)")
4252
args = p.parse_args(argv)
4353

4454
with open(args.catalog, encoding="utf-8") as fh:
4555
catalog = json.load(fh)
4656
if not catalog:
4757
print("ERROR: catalog is empty; nothing would be smoke-tested", file=sys.stderr)
4858
return 1
59+
total = len(catalog)
60+
catalog = select_rows(catalog, args.shard)
61+
if not catalog:
62+
print(f"ERROR: shard {args.shard} selects no row of {total}", file=sys.stderr)
63+
return 1
4964

5065
os.makedirs(args.out, exist_ok=True)
5166
runner = os.path.join(HERE, "run_example.py")
@@ -133,7 +148,8 @@ def row_cmd(item, run_name, extra, falsifier=None):
133148
break
134149
if failures and args.fail_fast:
135150
break
136-
print(f"catalog finished {n} of {len(catalog)} entries, {n_falsifiers} falsifier run(s)",
151+
shard = f" (shard {args.shard[0]}/{args.shard[1]} of {total} rows)" if args.shard else ""
152+
print(f"catalog finished {n} of {len(catalog)} entries{shard}, {n_falsifiers} falsifier run(s)",
137153
flush=True)
138154
if failures:
139155
print(f"{len(failures)} catalog failure(s):", file=sys.stderr)

‎tests/smoke/test_harness.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -346,6 +346,23 @@ def test_two_broken_examples_are_both_reported(self):
346346
self.assertIn("b (exit 1)", err)
347347
self.assertIn("d (exit 3)", err)
348348

349+
def test_shards_partition_the_catalog(self):
350+
# #529: CI splits the catalog across jobs; every row runs exactly once
351+
rows = [{"name": n, "script": f"{n}.py"} for n in "abcdefg"]
352+
seen_all = []
353+
for k in range(3):
354+
code, seen, _ = self._run(rows, {}, ("--shard", f"{k}/3", "--no-falsifiers"))
355+
self.assertEqual(code, 0)
356+
seen_all += seen
357+
self.assertEqual(sorted(seen_all), list("abcdefg"))
358+
359+
def test_empty_shard_is_red(self):
360+
rows = [{"name": "a", "script": "a.py"}]
361+
code, seen, err = self._run(rows, {}, ("--shard", "1/2"))
362+
self.assertEqual(code, 1)
363+
self.assertEqual(seen, [])
364+
self.assertIn("selects no row", err)
365+
349366
def test_fail_fast_is_opt_in(self):
350367
rows = [{"name": n, "script": f"{n}.py"} for n in ("a", "b", "c")]
351368
code, seen, _ = self._run(rows, {"a": 1}, extra=("--fail-fast",))

0 commit comments

Comments
 (0)