Splitting this out from #1128 to track the acceptance-tests piece on its own, now that the unit-test side has a starting point in #1158.
What we want
A test suite that tells us whether matching quality has regressed compared to previous yente versions — ideally something we can run on every release and publish a report for, so we can look back and say "six versions ago, recall on PEPs looked like X."
contrib/validation_report is the seed of this. The gap is:
- it runs against live data, so results drift whenever the underlying datasets change — we can't actually compare across yente versions
- it's not wired into CI in any meaningful way
- the "is this better or worse than last release?" question isn't answered anywhere
Approach
The key constraint is keeping the test fixtures constant for as long as possible, otherwise comparisons across yente versions are meaningless (we'd be measuring data drift, not yente changes).
@pudo's idea from #1128 (which I think is the right one for this, even though I went a different way for unit tests in #1158):
pull statements.csv once and use a duckdb query to extract lets say three or four sanctions lists and a limited subset of columns (just canonical_id, entity_id, schema, dataset, prop, value). This could then be used to freeze out a small-enough CSV fixture to commit. That fixture might then be ftm-ified using an in-memory aggregator in conftest.py and eventually used by a fake manifest that points to it.
The statements-as-source approach is better than freezing FtM directly because it survives FtM schema evolutions — the in-memory aggregation re-derives current-flavor entities from a stable underlying representation.
Some open questions:
- a) what to do about
first_seen / last_seen so date filters are testable — possibly derive from dataset or stamp synthetically
- b) what's the right haystack size — small enough to commit and run quickly, big enough that matching behavior is non-trivial
- c) what's the needle set — could reuse / extend
checks.yml from nomenklatura as a starting point
- d) how to present the report — diff against the previous release? Absolute numbers per release? Both?
Scope
What's in:
- a frozen, committed fixture (statements → in-memory FtM via aggregator → fake manifest)
- a test runner that produces a comparable score/report per yente version
- something publishable alongside releases
What's out (separate concerns):
WDYT about the shape?
🤖 Generated with Claude Code
Splitting this out from #1128 to track the acceptance-tests piece on its own, now that the unit-test side has a starting point in #1158.
What we want
A test suite that tells us whether matching quality has regressed compared to previous yente versions — ideally something we can run on every release and publish a report for, so we can look back and say "six versions ago, recall on PEPs looked like X."
contrib/validation_reportis the seed of this. The gap is:Approach
The key constraint is keeping the test fixtures constant for as long as possible, otherwise comparisons across yente versions are meaningless (we'd be measuring data drift, not yente changes).
@pudo's idea from #1128 (which I think is the right one for this, even though I went a different way for unit tests in #1158):
The statements-as-source approach is better than freezing FtM directly because it survives FtM schema evolutions — the in-memory aggregation re-derives current-flavor entities from a stable underlying representation.
Some open questions:
first_seen/last_seenso date filters are testable — possibly derive fromdatasetor stamp syntheticallychecks.ymlfrom nomenklatura as a starting pointScope
What's in:
What's out (separate concerns):
WDYT about the shape?
🤖 Generated with Claude Code