fix(bench): one source/test convention for LOC and the judge (JS/TS, spec/, mjs/cts, skip .git/)

[zak1201]

What

1. benchmarks/agentic/run.py _is_test now also treats these as tests:

   • *.test.* / *.spec.* filenames for js|jsx|ts|tsx|mjs|cjs|mts|cts
   • files under a __tests__/ or spec/ directory (Jasmine/RSpec layout; a file merely named spec.js stays source)

   --selftest gains an is_test check covering the Python and JS/TS conventions, plus look-alikes that must stay source (latest.ts, contest.tsx, test-utils.js, spec.js, button.js, app.py). Cases for spec/button_spec.js and the .mjs/.cts variants were added too. It also asserts the resulting code_stats split.

2. CODE_EXT gains .mjs, .cjs, .mts and .cts. Before this, a cell that delivered server.mjs or config.cts got 0 LOC for that file.

3. benchmarks/agentic/judge.py drops its own _is_test(name) and imports the one from run.py. source_text now excludes the same files the LOC metrics treat as tests. complete.py picks this up through judge.source_text.

4. judge.source_text also skips any path with a component starting with .. Workspaces of diff: True and fixture tasks are git repos (_git_snapshot), and files inside .git/ (HEAD, config, index, hook samples, objects) don't start with .. They were being sent to the LLM judge as "source" by judge.run, and by complete.submission for diff: True tasks. --selftest gains a judge_source check: it snapshots a workspace and asserts only src/app.js reaches the judge. (This was fix(bench): keep .git/ out of the over-engineering judge's input #1127, folded in here.)

Why

• run.py only knew Python conventions, so JS/TS test files a cell wrote were counted as source LOC. That inflated src_loc / total_loc and kept them out of wrote_tests_rate, which contradicts the README ("Tests are excluded and tracked separately").
• judge.py had a second, even narrower copy: Python filenames only, no tests/ / test/ dirs. So the over-engineering judge was shown JS/TS tests and Python helpers under tests/ as source. Two copies of one rule had already drifted.

Scope

• run.py callers code_stats (surgical/open tasks) and git_diff_stats (fixture/diff tasks) both pick up the fix. Result fields and their names are unchanged. The source/test split moves, and with it the fixture-task correctness gate (see Reviewer notes).
• No deliverable file in tasks.py matches the new rules.
• Apart from the four CODE_EXT additions in point 2, fixture exclusion, the self-check split, Git diff counting and the judge rubric/model are untouched.
• judge.py → run.py is a new import. run.py has no import-time side effects and doesn't import judge, so there is no cycle. complete.py and answer.py import cleanly.

Reviewer notes

• Running --rescore (or re-judging) an existing run will now give a lower src_loc_median, a higher wrote_tests_rate, and possibly different judge scores than the published numbers wherever cells wrote JS/TS tests or a tests/ dir. --rescore overwrites results.json / summary.json in place.
• This also moves correctness, not only LOC. score_workspace scores every fixture task as correct = total_loc > 0 from git_diff_stats. All fixture tasks are also open, so when no source LOC is left it first falls back to fenced code in the chat answer (chat_code_loc). A cell whose diff adds only JS/TS test files (e.g. src/app.test.ts) was correct=1 before; now it is correct=0 unless its chat answer contains a fenced code block, and with no code at all it drops out of wrote_file_rate and the LOC medians. So --rescore can lower correct_rate and raise cost_per_pass for fixture tasks. This is intended: Python tests already behaved this way.
• spec/ is matched as a directory name anywhere in the path. A source tree that uses spec/ for non-test code (e.g. API specs written in TS) would now be counted as test. No task in tasks.py seeds or delivers anything under spec/.
• score_fixture in tasks.py keeps its own .tsx/.ts/.jsx/.js list. I left it alone because score_workspace replaces the score for every fixture task with the git-diff result, so score_fixture is never reached.
• No linked issue.

Verification (offline, no API calls)

• python run.py --selftest → all instruments valid, including the new is_test and judge_source checks.
• On unfixed code the new check fails on exactly the 5 JS/TS cases. Without the spec/ rule it fails on exactly spec/button_spec.js. With the previous CODE_EXT and filename rule it fails on exactly src/button.test.mjs and src/util.spec.cts.
• git_diff_stats with .mjs/.cts files: src/server.mjs and src/cfg.cts count as source (2 files / 4 lines), src/server.test.mjs counts as test.
• Ran git_diff_stats in a throwaway repo: src/app.test.js and src/__tests__/b.ts counted as test, src/new.ts counted as source.
• Fixture correctness gate via score_workspace on tmpl-fe-datepicker (throwaway repo seeded with src/app.ts, diff adding only src/app.test.ts): at 2961b52 total_loc=2, correct=1. At this head: chat answer without code → total_loc=0, test_loc=2, correct=0; chat answer with a fenced code block → total_loc=1 (from the chat), correct=1.
• judge.source_text on a synthetic workspace:
  • Before: also included src/button.test.js, src/Button.spec.tsx, src/__tests__/b.ts and tests/helper.py.
  • After: only app.py, notes.md, src/button.js, src/latest.ts. Non-code artifacts like notes.md are still included, as before.
• node scripts/check-rule-copies.js and node scripts/check-versions.js pass.
• node --test tests/*.test.js: 141/142. The only failure is csv: correct pandas one-liner passes, caused by a broken pandas/numpy install on my machine and unrelated to this change. Because of that failure, npm test stops before the pi-extension tests, so I ran them directly: 30/30 pass.
• Not run: live benchmark cells, the live judge --selftest (needs an API key).

🤖 Generated with Claude Code

[zak1201]

Hi @DietrichGebert, the test workflow for this PR is waiting for maintainer approval (fork PR from a first-time contributor). Could you approve the run when you have a moment? Offline checks pass locally (run.py --selftest, node --test); the only local failure is the pandas csv test, caused by a broken local pandas install. Thanks!