Skip to content

Commit 8766968

Browse files
authored
Assertions framework for BabelTest expectations (#102)
Adds `src/babel_validation/assertions`: the engine that turns a named BabelTest assertion plus its parameters into a check evaluated against NodeNorm/NameRes. Nothing consumes it yet — the GitHub issue parser that produces the parameters arrives later in the stack — so this PR is the engine, its generated documentation, and its offline tests. **Stack 2 of 4** splitting #67. Base: `main` (#101 is merged). ### The assertion types `AssertionHandler` is the base class; `NodeNormTest` and `NameResTest` specialize it per service. Each concrete handler declares its parameters and yields `TestResult`s. Registered: `Resolves`, `DoesNotResolve`, `ResolvesWith`, `DoesNotResolveWith`, `HasLabel`, `ResolvesWithType`, `SearchByName`, and `Needed` (a placeholder that always fails, marking an issue as still needing a real test). Both wiki (`{{BabelTest|Resolves|CHEBI:15365}}`) and YAML syntax are supported, and an assertion can carry several independent **params lists**, each evaluated separately so one bad one doesn't sink the rest. ### Shared parameter handling `AssertionHandler.prepare_params_lists()` strips whitespace from every param, rejects params lists whose CURIEs are malformed, and warms the NodeNorm cache for the survivors in a single batched request. Both `test_with_nodenorm()` and `test_with_nameres()` route through it, so the NameRes path gets the same validation and the same one-request warming rather than a NodeNorm round-trip per params list. Two escape hatches keep that uniform treatment from being wrong for particular assertions: - `curie_params()` narrows which params are CURIEs — `HasLabel` to the first, `ResolvesWithType` to everything after the Biolink type, `SearchByName` to the expected CURIE only (its first param is a free-text query). - `VALIDATE_CURIES = False` opts an assertion out of format validation entirely. `DoesNotResolve` sets it: an identifier that isn't even a well-formed CURIE trivially doesn't resolve, which is exactly what that assertion exists to state, so rejecting it up front would leave the assertion unable to express its own purpose. ### A NodeNorm bulk-normalization fix `CachedNodeNorm.normalize_curies()` built its return value from `response.json()`, so a CURIE that NodeNorm silently omitted from its response was absent from the returned dict rather than present with a `None` value. A caller iterating the results would never see it and would report success for a CURIE it never tested. The warm-cache path happened to re-add the missing key, so the hole only opened on a cold lookup. It now builds the result from the requested CURIEs: exactly one entry per request, in request order. The ordering guarantee matters independently — "first CURIE that resolved" logic previously depended on the server's JSON ordering when cold and on set iteration order (per-process string hash randomization) when warm, so *which* CURIE got blamed in a failure message could vary between runs of the same test. ### Types and naming Parameters are named rather than left as nested `list[str]`: - `ParamsList` — one assertion invocation's parameters. Position is significant (`ResolvesWithType` takes its Biolink type first, `HasLabel` is `[curie, label]`), which is why this is a list. - `PreparedParamsList` — a frozen `(params, failure)` record. `prepare_params_lists()` returns a list of these rather than a `(stripped, failures_by_index)` tuple, so callers don't re-zip two structures by hand. Service parameters are annotated with the `NodeNormService` / `NameResService` Protocols that `services/` already defines for the purpose, rather than the concrete `CachedNodeNorm` / `CachedNameRes`, so a future drop-in replacement needs no changes here. ### Guardrails - **Registration.** `_register()` replaces the `{h.NAME: h for h in [...]}` comprehension and raises on a `NAME` that isn't lowercase or one that's already taken. Lowercase is load-bearing — the README promises users that assertion names are matched case-insensitively, which only holds if every registry key is lowercase — and a duplicate `NAME` would otherwise silently drop a handler. Both are mistakes only made while adding an assertion, so they fail loudly at import. - **Missing Biolink types.** When NodeNorm returns a node with no type, messages show `NO TYPE RETURNED`. The earlier placeholder, `unknown type`, had the shape of a real type — the older Biolink vocabulary was lowercase prose like `chemical entity` — so it could be misread as something Babel actually returned. ### Documentation `gen_docs.py` renders `assertions/README.md` from the handler class attributes, grouping handlers by the service they test rather than by their order in `ASSERTION_HANDLERS` — otherwise a handler registered in the wrong place lands under the wrong heading, and the sync test can't catch it because it regenerates the same wrong output. `tests/test_environment/test_assertions_docs.py` asserts the checked-in README stays in sync. The "Adding a New Assertion Type" instructions live in that generated README and nowhere else. There had been a second copy in the package docstring and the two had already drifted; the docstring now points at the README and describes the module layout instead. ### Tests `tests/test_environment/test_assertions.py` stubs `requests.post` rather than the service, so handlers run against the real `CachedNodeNorm` and exercise its contract instead of a fake restating it. The fixture DB drops one CURIE from the response entirely, reproducing what NodeNorm does for some unknown identifiers. 13 unit tests, all offline. Also registers the `unit` pytest marker, first used by these tests. ### Outcomes - The assertion vocabulary is fixed, documented, and enforced, so the issue parser in the next part of the stack has a stable target. - Every check here runs offline — no network, no GitHub API, no Google Sheet. - Deliberately not here: parsing assertions out of issue bodies, and any wiring into the existing pytest suites. ### Notes for review `gh pr diff` shows hunks in `.gitignore`, `services/nameres.py`, `sources/google_sheets/`, and `tests/conftest.py` that came from #101 and are already on `main` — they merge as no-ops. The net change against `main` is the `assertions/` package, the `normalize_curies()` fix, the two test files, and the pytest marker. `result: dict` is left untyped for NodeNorm response entries. A `TypedDict` would be more precise, but the response shape is Babel's to change and a wrong one is worse than an honest `dict`, so the docstrings say what the dict is instead. ### Verify - `uv run pytest -m unit -q` → 13 assertion tests pass offline. - `uv run python -m src.babel_validation.assertions.gen_docs` reproduces the committed README. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents a8c8188 + 347f165 commit 8766968

10 files changed

Lines changed: 1330 additions & 10 deletions

File tree

pyproject.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,6 @@ packages = ["src"]
3232
# (including node_modules) during collection.
3333
testpaths = ["tests"]
3434
timeout = 300
35+
markers = [
36+
"unit: unit tests that do not require network access",
37+
]
Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
<!-- AUTO-GENERATED — do not edit by hand.
2+
Regenerate with: uv run python -m src.babel_validation.assertions.gen_docs -->
3+
4+
# BabelTest Assertion Types
5+
6+
This package defines the assertion types that can be embedded in GitHub issue bodies and evaluated against the NodeNorm and NameRes services.
7+
8+
## Embedding Tests in Issues
9+
10+
Two syntaxes are supported:
11+
12+
**Wiki syntax** (one assertion per line):
13+
```
14+
{{BabelTest|AssertionType|param1|param2|...}}
15+
```
16+
17+
**YAML syntax** (multiple assertions, multiple params lists):
18+
````
19+
```yaml
20+
babel_tests:
21+
AssertionType:
22+
- param1
23+
- [param1, param2]
24+
```
25+
````
26+
27+
Assertion names are case-insensitive, as is the `{{BabelTest|...}}` marker itself.
28+
29+
## Params Lists
30+
31+
Each assertion can be invoked with one or more **params lists** — independent groups of
32+
parameters that are each evaluated separately.
33+
34+
- **Wiki syntax** — each `{{BabelTest|...}}` line is one params list.
35+
- **YAML syntax** — each list entry under an assertion key is one params list; a bare string
36+
is a single-element params list, a YAML list is a multi-element params list.
37+
38+
The meaning of each element in a params list depends on the assertion type (see below).
39+
For most assertions the elements are CURIEs; for `HasLabel` the second element is a
40+
label string; for `ResolvesWithType` the first element is a Biolink type.
41+
42+
---
43+
44+
## NodeNorm Assertions
45+
46+
These assertions test the [NodeNorm](https://nodenorm.transltr.io/docs) service.
47+
48+
### Resolves
49+
50+
**Applies to:** NodeNorm
51+
52+
Each CURIE in each params_list must resolve to a non-null result in NodeNorm.
53+
54+
**Parameters:** One or more CURIEs per params_list.
55+
56+
**Wiki syntax:**
57+
```
58+
{{BabelTest|Resolves|CHEBI:15365}}
59+
{{BabelTest|Resolves|MONDO:0005015|DOID:9351}}
60+
```
61+
62+
**YAML syntax:**
63+
```yaml
64+
babel_tests:
65+
Resolves:
66+
- CHEBI:15365
67+
- [MONDO:0005015, DOID:9351]
68+
```
69+
70+
---
71+
72+
### DoesNotResolve
73+
74+
**Applies to:** NodeNorm
75+
76+
Each CURIE in each params_list must fail to resolve (return null) in NodeNorm. Use this to confirm that an identifier is intentionally not normalizable.
77+
78+
**Parameters:** One or more CURIEs per params_list.
79+
80+
**Wiki syntax:**
81+
```
82+
{{BabelTest|DoesNotResolve|FAKENS:99999}}
83+
```
84+
85+
**YAML syntax:**
86+
```yaml
87+
babel_tests:
88+
DoesNotResolve:
89+
- FAKENS:99999
90+
```
91+
92+
---
93+
94+
### ResolvesWith
95+
96+
**Applies to:** NodeNorm
97+
98+
All CURIEs within each params_list must resolve to the identical normalized result. Use this to assert that two identifiers are equivalent.
99+
100+
**Parameters:** Two or more CURIEs per params_list. All must resolve to the same result.
101+
102+
**Wiki syntax:**
103+
```
104+
{{BabelTest|ResolvesWith|CHEBI:15365|PUBCHEM.COMPOUND:1}}
105+
```
106+
107+
**YAML syntax:**
108+
```yaml
109+
babel_tests:
110+
ResolvesWith:
111+
- [CHEBI:15365, PUBCHEM.COMPOUND:1]
112+
- [MONDO:0005015, DOID:9351]
113+
```
114+
115+
---
116+
117+
### DoesNotResolveWith
118+
119+
**Applies to:** NodeNorm
120+
121+
The CURIEs within each params_list must NOT all resolve to the same normalized result. Use this to assert that two identifiers are intentionally distinct entities.
122+
123+
**Parameters:** Two or more CURIEs per params_list. They must not all resolve to the same result.
124+
125+
**Wiki syntax:**
126+
```
127+
{{BabelTest|DoesNotResolveWith|CHEBI:15365|CHEBI:16856}}
128+
```
129+
130+
**YAML syntax:**
131+
```yaml
132+
babel_tests:
133+
DoesNotResolveWith:
134+
- [CHEBI:15365, CHEBI:16856]
135+
```
136+
137+
---
138+
139+
### HasLabel
140+
141+
**Applies to:** NodeNorm
142+
143+
The CURIE must resolve in NodeNorm and its primary label (id.label) must match the expected label exactly (case-sensitive).
144+
145+
**Parameters:** Exactly two elements per params_list: a CURIE, then the expected label string.
146+
147+
**Wiki syntax:**
148+
```
149+
{{BabelTest|HasLabel|CHEBI:15365|aspirin}}
150+
```
151+
152+
**YAML syntax:**
153+
```yaml
154+
babel_tests:
155+
HasLabel:
156+
- [CHEBI:15365, aspirin]
157+
```
158+
159+
---
160+
161+
### ResolvesWithType
162+
163+
**Applies to:** NodeNorm
164+
165+
Each params_list must have at least two elements: the first is the expected Biolink type (e.g. 'biolink:Gene'), and the remainder are CURIEs that must resolve with that type.
166+
167+
**Parameters:** Each params_list: first element is the expected Biolink type (e.g. `biolink:Gene`), remaining elements are CURIEs.
168+
169+
**Wiki syntax:**
170+
```
171+
{{BabelTest|ResolvesWithType|biolink:Gene|NCBIGene:1}}
172+
```
173+
174+
**YAML syntax:**
175+
```yaml
176+
babel_tests:
177+
ResolvesWithType:
178+
- [biolink:Gene, NCBIGene:1, HGNC:5]
179+
```
180+
181+
---
182+
183+
## NameRes Assertions
184+
185+
These assertions test the [NameRes](https://name-lookup.transltr.io/docs) service.
186+
187+
### SearchByName
188+
189+
**Applies to:** NameRes
190+
191+
Each params_list must have exactly two elements: a search query string and an expected CURIE. The test passes if the CURIE's normalized identifier appears within the top N results (default N=5) when NameRes looks up the search query.
192+
193+
**Parameters:** Each params_list: the **search query string** and the **expected CURIE**. The CURIE is normalized via NodeNorm before matching.
194+
195+
**Wiki syntax:**
196+
```
197+
{{BabelTest|SearchByName|water|CHEBI:15377}}
198+
```
199+
200+
**YAML syntax:**
201+
```yaml
202+
babel_tests:
203+
SearchByName:
204+
- [water, CHEBI:15377]
205+
- [diabetes, MONDO:0005015]
206+
```
207+
208+
---
209+
210+
## Special Assertions
211+
212+
### Needed
213+
214+
**Applies to:** NodeNorm and NameRes
215+
216+
Marks an issue as needing a test — always fails as a reminder to add real assertions.
217+
218+
**Wiki syntax:**
219+
```
220+
{{BabelTest|Needed}}
221+
```
222+
223+
**YAML syntax:**
224+
```yaml
225+
babel_tests:
226+
Needed:
227+
- placeholder
228+
```
229+
230+
---
231+
232+
## Adding a New Assertion Type
233+
234+
1. Choose the right module:
235+
- `nodenorm.py` — for NodeNorm-only assertions (subclass `NodeNormTest`, override `test_params_list`)
236+
- `nameres.py` — for NameRes-only assertions (subclass `NameResTest`, override `test_params_list`)
237+
- `common.py` — for assertions that apply to both services (subclass `AssertionHandler`, override `test_with_nodenorm` and/or `test_with_nameres`)
238+
239+
2. Give the class its five documentation attributes:
240+
- `NAME` — **must be all lowercase.** Assertions are matched case-insensitively by
241+
lowercasing whatever the issue wrote, so a `NAME` containing any uppercase could
242+
never be matched. Registration rejects it rather than letting it fail silently.
243+
- `DESCRIPTION` — one line, shown under the heading here.
244+
- `PARAMETERS` — what each element of a params_list means, and how many are expected.
245+
- `WIKI_EXAMPLES` — complete `{{BabelTest|...}}` lines, reproduced verbatim.
246+
- `YAML_PARAMS` — indented list entries for the YAML example.
247+
248+
These are rendered into this file, so write them for someone reading this README
249+
rather than for someone reading the class.
250+
251+
3. Implement `test_params_list()` (or both `test_with_*` methods for `AssertionHandler`
252+
subclasses). It receives one params_list at a time, already stripped and — unless the
253+
handler sets `VALIDATE_CURIES = False` — with its CURIEs validated and pre-warmed in
254+
the NodeNorm cache. Yield one result per thing checked, usually one per CURIE, so a
255+
failure names the CURIE that failed. Override `curie_params()` if some params are not
256+
CURIEs; see `HasLabel` and `SearchByName`.
257+
258+
4. Import it in `__init__.py` and add an instance to `ASSERTION_HANDLERS`. Order does not
259+
matter — this file groups handlers by the service they test.
260+
261+
5. Run `uv run python -m src.babel_validation.assertions.gen_docs` to regenerate `README.md`,
262+
and `uv run pytest -m unit` to confirm the checked-in copy is in sync.

0 commit comments

Comments
 (0)