Formulate and enact testing strategy

[leonhandreke]

Currently, we have unit tests that depend on live data, and that's not idea. What we want is multiple levels:

Unit tests

• should test yente functionality and not depend on OpenSanctions data, most definitely not live data.

Acceptance tests

contrib/validation_report is a start. We want to build tests that test whether matching is still working as well as last version, and to be able to compare and document what the performance was six versions ago (so: release a report with every yente release). We need a moderately complex haystack and needle to test real-world performance, so likely OpenSanctions data. Ideally, we keep the test fixtures constant for as long as possible for comparability to previous versions. One idea for how to do that is to use OS statements as a base and then generate the current flavor of FtM from that for our yente under test to consume.

End-to-end tests

These can test the full stack, from data production to the response, even our Hosted API performance. But they come with the usual issues of end to end tests.

[pudo]

Some notes:

• I think we could pull statements.csv once and use a duckdb query to extract lets say three or four sanctions lists and a limited subset of columns (just canonical_id, entity_id, schema, dataset, prop, value). This could then be used to freeze out a small-enough CSV fixture to commit. That fixture might then be ftm-ified using an in-memory aggregator in conftest.py and eventually used by a fake manifest that points to it. Does that sound sane? We might still have to do something to first_seen, last_seen (perhaps on dataset) so we can check date filters.
• One we have this going for the unit tests, maybe we want to then run on this data a larger suite of integration tests, based on checks.yml in nomenklatura to see if it's doing what we want in the basic direction. Then the validation report comes in as a last QA step, running on live data? Or we could freeze down the validation report to use its own fake manifest pointing to some fixed data URLs?

[pudo]

Not sure this is helpful here, but I was playing with a new name testing file today: https://github.com/opensanctions/rigour/blob/bbc23794f2a3cb3ee7ee110aa1a32edecba8d2ce/contrib/name_comparison/cases.csv

[leonhandreke]

Sorry for the notifications, I fatfingered the buttons.

• I think we could pull statements.csv once and use a duckdb query to extract lets say three or four sanctions lists and a limited subset of columns (just canonical_id, entity_id, schema, dataset, prop, value). This could then be used to freeze out a small-enough CSV fixture to commit. That fixture might then be ftm-ified using an in-memory aggregator in conftest.py and eventually used by a fake manifest that points to it. Does that sound sane? We might still have to do something to first_seen, last_seen (perhaps on dataset) so we can check date filters.

For unit tests in #1158, I chose the route of going with a fixture that contains <100 entities in FtM format, still manually readable, with a little script to regenerate from a live yente. I hadn't seen your comment, and I think your idea is better because it will survive FtM schema evolutions better. I think we should definitely adopt this for the acceptance tests because there, keeping a constant baseline of statements to score against is important.

[leonhandreke]

thoughts mine, words vibes because I cannot english

I think this issue is first and foremost about deciding on a strategy. Let's do that now, open actionable implementation issues for the concrete pieces, and close this one.

As far as I can see, the strategy we've converged on is roughly three tiers:

1. Unit tests — yente logic in isolation, no live data. Started in test: per-test fixture isolation with session-scoped index loading #1158 with per-test fixture isolation and a small committed FtM fixture (zala). Will soon be fully independent of live data once test: drop live-catalog dependency from test suite #1167 goes in.
2. Acceptance tests — matching-quality regression against a frozen fixture, comparable across yente versions. Tracked in Acceptance tests for matching quality #1168.
3. End-to-end tests — black-box tests against a running yente (prod or test instance).

A few open questions before we close this:

• Do we want a separate "integration" tier between unit and end-to-end? As in: real ES, real FastAPI app, fake manifest pointing at fixture data. I don't think this tier is needed — the unit tests already use a live index against fixture data, and they exercise the full machine from HTTP handler to ES/OS index retrieval, so that's pretty integration-testy already.

• End-to-end tests already partially exist in operations/integration/:

  • tests/test_match.py — /match smoke tests with logic-v1 and best algorithms (Falah Taha, Rotenberg, SGM)
  • tests/test_search.py — /search smoke tests (Putin, fuzzy matching on Barack Obama)
  • tests/test_fetch.py — one test, GETs /entities/Q7747 and checks the response ID matches
  • checks/checks.yml + check.py — standalone script (not pytest) that runs ~118 lines of YAML-defined match queries with expected match: / no_match: entity IDs against a live yente and prints OK/FAIL per row. Conceptually overlaps with what we want from Acceptance tests for matching quality #1168, just against live data instead of a frozen fixture.

  I think this is sufficient for now. Also, "end-to-end" is a better name than "integration" since they run against prod/test instances.

If we agree on the tiers, I'd propose:

• close this issue
• finish the work package on unit-test independence from live data Make unit tests independent of the OpenSanctions data collection #1163
• track acceptance tests in Acceptance tests for matching quality #1168

🤖 Generated with Claude Code