since-cutoff 0.3.0: numbers you can defend, baseline comparisons, repeatable studies

[MohammadHijjawi97]

since-cutoff 0.3.0 is on PyPI (uvx since-cutoff@latest scan). The focus is on making the measurements hold up under scrutiny and easy to repeat.

Numbers you can defend

• The held-out result now keeps every interval next to the number it belongs to: the task-level rates with the number of paired tasks and API changes behind them; their difference with a 95% cluster-bootstrap interval (resampling API changes, not tasks); "changes fixed" with its own Wilson interval and an exact sign test; and the excluded pairs with their reasons.
• A change counts as fixed only when most of its pairs are wrong without the notes and right with them.

Compare against simpler notes

• run --compare template,signatures also answers the held-out tasks with notes built without a model (each change stated from the API diff, or the new signatures and docstrings), on the same pairs, with each block's size in tokens. If the verified notes don't beat plain diff bullets for your project, you'll see it.

Repeatable studies

• run --tasks-out tasks.json saves the exact tasks; --tasks-from tasks.json reruns on them, for another model or another set of notes. Run settings are recorded in the report.

It finds your model

• Without --model, since-cutoff now tests the model your coding agent is set up with: Claude Code, Codex, OpenCode or Aider settings (project before user), and it tells you where it found it.

Also new since 0.2.0: results across 21 models from 8 vendors, Spanish and French documentation, and a logo.

Full changelog: https://github.com/MohammadHijjawi97/since-cutoff/blob/main/CHANGELOG.md

Next (0.3.1): fixes from running the tool on 12 real open-source projects (output on Windows pipes, which dependencies your code imports, less noise from typing stubs and *args changes). If you try it on your project, please share what it finds in Share your results.