Skip to content

Commit f6b5d77

Browse files
committed
test: auto-discover test_*.py modules in the runner (no more hand-kept or-chain)
Track A (N2). The suite aggregator imported ~16 test modules by hand and gated on a 25-term `or` of their return codes; adding a module meant remembering to thread its rc through that chain, and a forgotten term silently dropped a whole module from the gate. The runner now globs `test_*.py`, calls each module's `run()`, and returns `any(rc)` (plus the inline analysis/codegen flags). A new test file is picked up with no edit here; a module missing `run()` fails the suite loudly; discovering zero modules fails loud. `test_codegen_props` keeps its lighter in-suite iteration count (3000) via a one-entry special-case map. Per-module output is unchanged (now ordered by filename). Refs docs/notes/tech-debt-register.md (N2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJSfUpRbRXegrwJNV1rx1X
1 parent 5ceb4b2 commit f6b5d77

2 files changed

Lines changed: 29 additions & 94 deletions

File tree

‎docs/notes/tech-debt-register.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ Python walkers (§2.1).
193193
| # | Item | Where | Why now |
194194
|---|------|-------|---------|
195195
| N1 ✅ | `else: raise` on unknown flow ops | `ownlang/ownir.py` `_lower_flow` | Silent fact-swallowing → fabricated/missed verdicts (§2.1). *Done: raises `OwnIRError`; pinned by test_ownir. The five read-only walkers legitimately ignore ops they don't consume, so the guard is scoped to `_lower_flow` (the only pass that must handle every op or lose facts).* |
196-
| N2 | Auto-discovery in the test runner | `tests/run_tests.py:1144-1149` | 25-term hand-rolled `or`; a forgotten `rc` silently stops gating |
196+
| N2 ✅ | Auto-discovery in the test runner | `tests/run_tests.py` | 25-term hand-rolled `or`; a forgotten `rc` silently stopped gating. *Done: the runner now globs `test_*.py`, calls each module's `run()`, and gates on `any(rc)` — a new test file is auto-included, a module without `run()` fails the suite, and a discovery of zero modules fails loud. `test_codegen_props` keeps its lighter in-suite iteration count via a small special-case map.* |
197197
| N3 | Golden facts snapshots in CI + feed to `test_ownir.py` | `ci.yml` wpf-extractor job | The seam is never diffed at the facts level (§2.2) |
198198
| N4 ◑ | `spec/OwnIR.md` ✅ + JSON Schema (pending) + evolution policy ✅ | `spec/OwnIR.md` | §2.3–2.4. *Done: normative spec incl. the version evolution policy (IR1–IR6), registered in `spec/README.md`. Still to do: `spec/ownir.schema.json` + CI fixture validation.* |
199199
| N5 | DI001–005 + EFF001 into `spec/` + `Diagnostics.md` + `test_spec.py` | `ownlang/di.py`, `effects.py` | Second-largest analyzer has zero normative governance |

‎tests/run_tests.py‎

Lines changed: 28 additions & 93 deletions
Original file line numberDiff line numberDiff line change
@@ -1058,102 +1058,37 @@ def run() -> int:
10581058
print(f"ordering: {'PASS' if not order_fails else 'FAIL'}")
10591059
print(f"helper: {'PASS' if not helper_fails else 'FAIL'}")
10601060

1061-
# Content-level codegen assertions + property fuzzer: these inspect the
1062-
# generated C# itself (release placement/count, declaration order), catching
1063-
# lowerings that are silently wrong rather than ones that merely throw.
1064-
import test_codegen
1065-
cc_rc = test_codegen.run()
1066-
import test_codegen_props
1067-
pf_rc = test_codegen_props.run(iterations=3000, seed=1234)
1068-
1069-
# The "what it catches" gallery: every examples/gallery/ file must still
1070-
# produce exactly the diagnostic it advertises, so the demo can't drift.
1071-
import test_gallery
1072-
gl_rc = test_gallery.run()
1073-
1074-
# Real-world corpus: each case.own (a reduction of a real ArrayPool/Dispose
1075-
# bug) must still produce the diagnostics it documents.
1076-
import test_corpus
1077-
co_rc = test_corpus.run()
1078-
1079-
# WPF lifetime corpus (lifetimes slice #1): subscription/timer leaks caught
1080-
# by the generic ownership checker, carrying the [resource: <kind>] tag.
1081-
import test_wpf
1082-
wpf_rc = test_wpf.run()
1083-
1084-
# Lifetime-region analysis (lifetimes slice #2): the region-escape theorem
1085-
# (a short-lived object captured by a longer-lived source -> OWN014).
1086-
import test_lifetimes
1087-
lt_rc = test_lifetimes.run()
1088-
1089-
# Loop support (P-016 A1): `while` is analysed via a worklist fixpoint over the
1090-
# back-edge — cross-iteration leak/use-after-release/double-release, not OWN020.
1091-
import test_loops
1092-
loops_rc = test_loops.run()
1093-
1094-
# Spec conformance pilot: every normative spec/ rule fires on its example.
1095-
import test_spec
1096-
spec_rc = test_spec.run()
1097-
1098-
# OwnIR fact bridge (P-001): C#-extracted facts route through the core and
1099-
# surface a subscription leak at its C# location.
1100-
import test_ownir
1101-
ownir_rc = test_ownir.run()
1102-
1103-
# Interprocedural ownership summaries (P-005 D5.0): the MOS solver — transfer
1104-
# lattice, depth-capped forward resolution, recursion convergence. Infra only;
1105-
# no findings wired yet (D5.1+).
1106-
import test_ownership
1107-
own5_rc = test_ownership.run()
1108-
1109-
# RID indirection (P-005 D5.4 step 0): the no-op handle->RID layer the alias
1110-
# model (step 1) builds on — 1:1 identity, the join invariant, behaviour
1111-
# unchanged.
1112-
import test_rid
1113-
rid_rc = test_rid.run()
1114-
1115-
# Diagnostic rendering (P-015): the empty-evidence invariant (render()/render_pretty()
1116-
# byte-for-byte unchanged) plus the ordered `note:` slice and the SARIF evidence builders.
1117-
import test_diagnostics
1118-
diag_rc = test_diagnostics.run()
1119-
1120-
# `explain` command: the diagnostic-catalogue CLI surface (text + exit codes +
1121-
# --json code harvest + DI catalogue coverage + dispatch).
1122-
import test_explain
1123-
explain_rc = test_explain.run()
1124-
1125-
# Evidence coverage (execution-surfaces ADR §3/§5): OWN015/OWN016/OWN005 carry
1126-
# a structured acquire->escape / move->use reachability slice, rendered as
1127-
# ordered `note:` lines, with the merge-point move site labelled honestly.
1128-
import test_evidence_coverage
1129-
evid_rc = test_evidence_coverage.run()
1130-
1131-
# Flow-diagnostic SARIF (execution-surfaces ADR §3.1): `check --format sarif`
1132-
# projects each Diagnostic's evidence slice into SARIF relatedLocations /
1133-
# codeFlows via the shared ownlang.evidence builders, for GitHub code scanning.
1134-
import test_diag_sarif
1135-
dsarif_rc = test_diag_sarif.run()
1136-
1137-
# Canonical CFG JSON seam (P-022 step 0): the frozen CFG-layer contract the
1138-
# Rust-port differential oracle diffs — envelope, symbol-table identity,
1139-
# instruction vocabulary, determinism.
1140-
import test_cfg_json
1141-
cfgjson_rc = test_cfg_json.run()
1142-
1143-
# Reactive-effect stability (P-020): the EFF001 effect-storm analysis — the
1144-
# identity lattice, reference propagation, cycle safety, and the OwnIR bridge
1145-
# mapping the optional `effects` block to an EFF001 finding (a new core
1146-
# analysis dimension, like DI001, not the acquire/release leak model).
1147-
import test_effects
1148-
effects_rc = test_effects.run()
1061+
# Every sibling `test_*.py` module contributes a `run() -> int`. AUTO-DISCOVER
1062+
# them instead of importing each by hand and threading its rc through a
1063+
# hand-maintained `or` chain: a forgotten term there would silently drop a whole
1064+
# module from the gate (the exact failure this replaces). Adding a test file now
1065+
# needs no edit here. Each `run()` prints its own PASS/FAIL summary, so
1066+
# per-module output is preserved (the order is now sorted by filename).
1067+
# `test_codegen_props` takes an explicit lighter iteration count for the in-suite
1068+
# pass; every other module runs argless.
1069+
import importlib
1070+
1071+
special = {"test_codegen_props": lambda m: m.run(iterations=3000, seed=1234)}
1072+
here = os.path.dirname(os.path.abspath(__file__))
1073+
module_rcs: list[int] = []
1074+
for fname in sorted(os.listdir(here)):
1075+
if not (fname.startswith("test_") and fname.endswith(".py")):
1076+
continue
1077+
name = fname[:-3]
1078+
mod = importlib.import_module(name)
1079+
runner = special.get(name)
1080+
if runner is None and not hasattr(mod, "run"):
1081+
print(f"run_tests: FAIL — {name} has no run(); every test_*.py must expose one")
1082+
module_rcs.append(1)
1083+
continue
1084+
module_rcs.append(runner(mod) if runner is not None else mod.run())
1085+
if not module_rcs:
1086+
print("run_tests: FAIL — no test_*.py modules discovered")
1087+
return 1
11491088

11501089
return 1 if (failed or cg_fail or golden_fails or buffer_fails
11511090
or escape_fails or branchy_fails or nest_fails
1152-
or order_fails or helper_fails or cc_rc or pf_rc
1153-
or gl_rc or co_rc or wpf_rc or lt_rc or loops_rc
1154-
or spec_rc or ownir_rc or own5_rc or rid_rc or diag_rc
1155-
or explain_rc or effects_rc or evid_rc or dsarif_rc
1156-
or cfgjson_rc) else 0
1091+
or order_fails or helper_fails or any(module_rcs)) else 0
11571092

11581093

11591094
if __name__ == "__main__":

0 commit comments

Comments
 (0)