Found while testing the dynamic eval pipeline.
backend/app/code/eval/dynamic_eval.py:218-232:
passed_match = re.search(r'(\d+)\s+passed', combined)
if passed_match:
passed = int(passed_match.group(1))
failed_match = re.search(r'(\d+)\s+failed', combined)
if failed_match:
failed = int(failed_match.group(1))
skipped_match = re.search(r'(\d+)\s+skipped', combined)
if skipped_match:
skipped = int(skipped_match.group(1))
Two problems:
-
Regex greediness on test names. If pytest verbose output includes a test like test_42_passed_with_warning ... somewhere in stdout, the first match for r'(\d+)\s+passed' could be the 42 in that test name, not the actual summary line 5 passed, 2 failed.
-
No anchor to summary line. Pytest's summary lines look like === 5 passed, 2 failed, 1 skipped in 3.21s ===. The right regex should anchor on the summary:
summary_re = re.compile(
r'={3,}\s*'
r'(?:(\d+)\s+failed,?\s*)?'
r'(?:(\d+)\s+passed,?\s*)?'
r'(?:(\d+)\s+skipped,?\s*)?'
r'(?:(\d+)\s+(?:error|warning),?\s*)?'
r'in\s+[\d.]+s'
r'\s*={3,}',
re.IGNORECASE,
)
-
(Bonus) No handling for xfailed, xpassed, errors. Real pytest output regularly contains these and they're currently treated as nothing.
Concrete demonstration: take any project with test_42_passed_smoke.py in its layout, run pytest -v against it with no actual tests passing; the parser will report passed=42 instead of 0.
Fix
Use the anchored summary regex above and prefer the last match in stdout/stderr (pytest always summarises at the very end). Replace the three independent regexes with the single anchored one.
Found while testing the dynamic eval pipeline.
backend/app/code/eval/dynamic_eval.py:218-232:Two problems:
Regex greediness on test names. If pytest verbose output includes a test like
test_42_passed_with_warning ...somewhere in stdout, the first match forr'(\d+)\s+passed'could be the42in that test name, not the actual summary line5 passed, 2 failed.No anchor to summary line. Pytest's summary lines look like
=== 5 passed, 2 failed, 1 skipped in 3.21s ===. The right regex should anchor on the summary:(Bonus) No handling for
xfailed,xpassed,errors. Real pytest output regularly contains these and they're currently treated as nothing.Concrete demonstration: take any project with
test_42_passed_smoke.pyin its layout, runpytest -vagainst it with no actual tests passing; the parser will reportpassed=42instead of0.Fix
Use the anchored summary regex above and prefer the last match in stdout/stderr (pytest always summarises at the very end). Replace the three independent regexes with the single anchored one.