Skip to content

Commit 33f26af

Browse files
TMHSDigitalclaude
andcommitted
test: run a SKILL.md block in 12 of 25 skills, and ratchet the rest
Only 9 of 143 skill blocks executed in CI (7 skills); the rest were only compiled, so API drift in the code agents copy most could not turn CI red. Five more blocks are tagged <!-- smoke: run -->: geometry-nodes-python's displace tree, vse-python's COLOR strip, porting-to-blender-5's version branch, headless-batch-scripting's engine id, and slotted-actions-animation's cross-version channelbag helper, which gains a usage tail that writes an F-curve through it and asserts the slot is bound. tests/smoke/skill_blocks.json is a ratchet that check_content_catalog enforces: tagged blocks may not drop below its floor (14), and each of the 13 skills without a runnable block carries a compile_only reason naming the example that proves it (falsified: floor, missing reason, stale entry and unknown skill each fail). Live-run proven: all 14 tagged blocks PASS on Blender 5.2.1 LTS and 4.5.14 LTS via tests/smoke/run_content.py. Closes #525 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LpkeXRvikhJZSVGBFM91Dv Signed-off-by: TM Hospitality Strategies <TMHospitalityStrategies@gmail.com>
1 parent 02323ea commit 33f26af

9 files changed

Lines changed: 80 additions & 5 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ showcase/
105105

106106
3. Aim for 150 to 350 lines covering the canonical pattern, common AI mistakes, version-correctness notes, and one or two worked code examples. Cite Blender API doc URLs where relevant. Avoid encyclopedic API tours.
107107
4. The skill `name` in frontmatter must match the directory name exactly (CI enforces this).
108-
5. Every ```python block must compile (`tests/check_content_catalog.py`). Put `<!-- smoke: run -->` on the line directly above a block that is self-contained, so Content Smoke executes it on 4.5 and 5.2 in a factory-startup scene: no input files, no render, and it ends with `assert`s on what it built. Tag at least one block per skill where the API allows it.
108+
5. Every ```python block must compile (`tests/check_content_catalog.py`). Put `<!-- smoke: run -->` on the line directly above a block that is self-contained, so Content Smoke executes it on 4.5 and 5.2 in a factory-startup scene: no input files, no render, and it ends with `assert`s on what it built. Tag at least one block per skill where the API allows it; Validate requires it unless `tests/smoke/skill_blocks.json` gives the skill a `compile_only` reason naming the example that proves it, and the total tagged count may not drop below that file's `floor` (raise the floor when you tag more).
109109
6. Map at least one example to the skill in `examples/skills.json`; an example mapped to no skill fails Validate.
110110

111111
## Adding a Rule

‎skills/geometry-nodes-python/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ The Python pattern for building a Geometry Nodes tree has four phases:
2424
3. Wire nodes with `tree.links.new`.
2525
4. Add a NODES modifier on the target object that points at the group.
2626

27+
<!-- smoke: run -->
2728
```python
2829
import bpy
2930

‎skills/headless-batch-scripting/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,7 @@ Notes:
211211

212212
## Detecting Blender version in scripts
213213

214+
<!-- smoke: run -->
214215
```python
215216
import bpy
216217

‎skills/porting-to-blender-5/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ This skill is an index. Each row points at the skill, rule or runnable example t
6060

6161
Keep one code path and branch only where the table says *branch*:
6262

63+
<!-- smoke: run -->
6364
```python
6465
import bpy
6566

‎skills/slotted-actions-animation/SKILL.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ On 4.4 / 4.5 LTS, ensure the channelbag yourself with `strip.channelbag(slot, en
4545

4646
Import path (5.0+) verified against the Blender 5.1 API reference: `bpy_extras.anim_utils.action_ensure_channelbag_for_slot(action, slot)`. See [`bpy_extras.anim_utils`](https://docs.blender.org/api/current/bpy_extras.anim_utils.html).
4747

48+
<!-- smoke: run -->
4849
```python
4950
import bpy
5051

@@ -80,6 +81,12 @@ def get_channelbag_for_object(obj):
8081
obj.animation_data.action_slot = slot
8182

8283
return get_channelbag_for_slot(action, slot)
84+
85+
# Usage: every F-curve write goes through the slot's channelbag, on 4.5 and 5.x.
86+
cbag = get_channelbag_for_object(bpy.context.active_object)
87+
fc = cbag.fcurves.new(data_path="location", index=2)
88+
fc.keyframe_points.insert(1, 0.0)
89+
assert bpy.context.active_object.animation_data.action_slot is not None
8390
```
8491

8592
The version boundary that matters is `bpy.app.version >= (5, 0, 0)` (legacy removed / ensure-helper added), **not** `hasattr(action, "slots")` — `action.slots` is present on 4.4 and 4.5 too, so that check does not distinguish 4.5 from 5.x.

‎skills/vse-python/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Use this skill when the user:
1818

1919
## High-level shape
2020

21+
<!-- smoke: run -->
2122
```python
2223
import bpy
2324

‎tests/check_content_catalog.py‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,15 @@
44
with a ``check`` and a ``falsify`` (old, new) whose ``old`` occurs in the
55
snippet, so tests/smoke/run_content.py can never silently skip a snippet.
66
- Every ```python block in skills/*/SKILL.md compiles. Blocks tagged with a
7-
``<!-- smoke: run -->`` line are also executed by run_content.py in Blender;
8-
this reports how many are tagged.
7+
``<!-- smoke: run -->`` line are also executed by run_content.py in Blender.
8+
- Ratchet (#525, tests/smoke/skill_blocks.json): at least ``floor`` blocks are
9+
tagged, and every skill has a tagged block or a ``compile_only`` entry saying
10+
why its blocks cannot run standalone and which example proves it instead.
911
1012
Exit 1 with one ::error:: line per problem.
1113
"""
1214
import glob
15+
import json
1316
import os
1417
import re
1518
import sys
@@ -48,29 +51,56 @@ def check(root=ROOT, cases=None):
4851
errors.append(f"{name}: falsify string {fals[0]!r} is not in the snippet")
4952

5053
blocks = tagged = 0
54+
per_skill = {}
5155
for path in sorted(glob.glob(os.path.join(root, "skills", "*", "SKILL.md"))):
5256
rel = os.path.relpath(path, root).replace(os.sep, "/")
5357
with open(path, encoding="utf-8") as fh:
5458
text = fh.read()
55-
tagged += len(TAG.findall(text))
59+
n_tagged = len(TAG.findall(text))
60+
tagged += n_tagged
61+
per_skill[os.path.basename(os.path.dirname(path))] = n_tagged
5662
for m in FENCE.finditer(text):
5763
blocks += 1
5864
start = text.count("\n", 0, m.start(1)) + 1
5965
try:
6066
compile(m.group(1), rel, "exec")
6167
except SyntaxError as e:
6268
errors.append(f"{rel}:{start + (e.lineno or 1) - 1}: python block does not compile: {e.msg}")
69+
errors += check_ratchet(root, per_skill, tagged)
6370
return errors, blocks, tagged
6471

6572

73+
def check_ratchet(root, per_skill, tagged):
74+
path = os.path.join(root, "tests", "smoke", "skill_blocks.json")
75+
if not os.path.isfile(path):
76+
return []
77+
with open(path, encoding="utf-8") as fh:
78+
cfg = json.load(fh)
79+
floor, allow = cfg["floor"], cfg.get("compile_only", {})
80+
errors = []
81+
if tagged < floor:
82+
errors.append(f"only {tagged} SKILL.md blocks are tagged <!-- smoke: run -->, below the "
83+
f"floor of {floor} in tests/smoke/skill_blocks.json")
84+
for skill, n in sorted(per_skill.items()):
85+
if n == 0 and not allow.get(skill, "").strip():
86+
errors.append(f"skills/{skill}: no block is tagged <!-- smoke: run --> and "
87+
"tests/smoke/skill_blocks.json gives no compile_only reason")
88+
if n and skill in allow:
89+
errors.append(f"skills/{skill} has a tagged block; drop its compile_only entry "
90+
"in tests/smoke/skill_blocks.json")
91+
for skill in sorted(set(allow) - set(per_skill)):
92+
errors.append(f"tests/smoke/skill_blocks.json: compile_only names unknown skill {skill!r}")
93+
return errors
94+
95+
6696
def main():
6797
errors, blocks, tagged = check()
6898
for e in errors:
6999
print(f"::error::{e}")
70100
if errors:
71101
return 1
72102
print(f"{len(CASES)} snippets catalogued with falsifiers; {blocks} skill python blocks compile, "
73-
f"{tagged} tagged for execution")
103+
f"{tagged} tagged for execution (ratchet in tests/smoke/skill_blocks.json)")
74104
return 0
75105

76106

‎tests/smoke/skill_blocks.json‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"_comment": "Ratchet for SKILL.md python blocks that Content Smoke executes (<!-- smoke: run -->, #525). tests/check_content_catalog.py fails when fewer blocks are tagged than `floor` (raise the floor when you tag more), and when a skill has no tagged block and no entry in `compile_only`. A compile_only entry says why the skill's blocks cannot run standalone in a fresh factory scene and which runnable example proves the skill instead; tag a block and drop the entry when one can.",
3+
"floor": 14,
4+
"compile_only": {
5+
"addon-scaffolding": "Blocks are files of an add-on package (relative imports, register()); proven by examples/addon-register-lifecycle.",
6+
"ai-mesh-cleanup": "Blocks are steps over an imported GLB and helpers defined in the first block; proven by examples/coincident-vert-weld, collision-hull-proxy, degenerate-bevel-weld, lod-decimate-chain.",
7+
"bake-high-to-low": "Blocks are steps of one Cycles bake that need a high/low pair and UVs; proven by examples/bake-normal-high-to-low.",
8+
"bl-info-migration": "Blocks are package layouts and import forms, not runnable scripts; proven by examples/addon-register-lifecycle.",
9+
"custom-properties": "Blocks are add-on class and register() fragments; proven by examples/addon-register-lifecycle and cross-version-property-delete.",
10+
"depsgraph-and-evaluated-data": "Blocks define helpers called on caller-supplied objects; proven by examples/armature-bend, boolean-exact-volume, depsgraph-export.",
11+
"drivers-and-app-handlers": "Blocks are register()/handler fragments and steps on a caller-supplied object; proven by examples/driver-wave and exit-pre-sidecar.",
12+
"engine-export-presets": "Blocks are export calls on a prepared selection; proven by examples/export-preset-axis, gltf-export-roundtrip, unapplied-scale-gltf.",
13+
"extension-runtime-and-packaging": "Blocks read __package__ and extension_path_user, which exist only inside an installed extension; proven by examples/extension-package-lifecycle.",
14+
"gpu-drawing-and-overlays": "Blocks need a GPU context (draw handlers, or gpu.init() which exists only on 5.2+); proven by examples/gpu-offscreen-draw.",
15+
"operators": "Blocks are Operator classes and execute() bodies that need registration and an operator context; proven by examples/prop-origin-transform and temp-override-join.",
16+
"timers-modal-and-threading": "Timers never fire under --background, and the blocks call caller-supplied done()/download(); proven by examples/timers-modal-threading in a windowed child Blender.",
17+
"ui-panels": "Blocks are Panel draw() bodies that run only inside a UI redraw; proven by examples/addon-register-lifecycle."
18+
}
19+
}

‎tests/test_gates.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,21 @@ def test_falsify_that_changes_nothing(self):
154154
e = self.run_check({"s.py": {"check": "pass", "falsify": ("x = 1", "x = 1")}})
155155
self.assertTrue(any("changes something" in m for m in e), e)
156156

157+
def test_ratchet_needs_a_run_block_or_a_reason(self):
158+
# #525: a skill with no executed block must say why, and the tagged count cannot drop
159+
case = {"s.py": {"check": "pass", "falsify": ("x = 1", "x = 2")}}
160+
tagged = "# K\n\n<!-- smoke: run -->\n```python\nx = 1\n```\n"
161+
with tempfile.TemporaryDirectory() as td:
162+
root = Path(td)
163+
write(root, "snippets/s.py", "x = 1\n")
164+
write(root, "skills/k/SKILL.md", "# K\n\n```python\nx = 1\n```\n")
165+
write(root, "tests/smoke/skill_blocks.json", json.dumps({"floor": 1, "compile_only": {}}))
166+
errs = ccc.check(str(root), case)[0]
167+
self.assertTrue(any("below the floor" in e for e in errs), errs)
168+
self.assertTrue(any("no compile_only reason" in e for e in errs), errs)
169+
write(root, "skills/k/SKILL.md", tagged)
170+
self.assertEqual(ccc.check(str(root), case)[0], [])
171+
157172
def test_skill_block_that_does_not_compile_reports_its_line(self):
158173
skill = "# K\n\ntext\n\n```python\nok = 1\nif True\n pass\n```\n"
159174
e = self.run_check({"s.py": {"check": "pass", "falsify": ("x = 1", "x = 2")}}, skill=skill)

0 commit comments

Comments
 (0)