Add GitHub-issue-driven test framework, assertion handlers, and `csv-to-babeltests` CLI - #67

Open

gaurav wants to merge 136 commits into

mainfrom

add-github-issue-tests

Commits on Jan 8, 2026

Moved TestRow into its own file.
gaurav
committed
06857d5
Added /.env to .gitignore.
gaurav
committed
a0325b6
Implemented basic GitHub issue testing (but it doesn't work at all).
gaurav
committed
0009dc0

Commits on Jan 12, 2026

Check for a GitHub Authentication Token.
gaurav
committed
f297735
Add support for `--issue` to specify one or more GitHub issues to check.
gaurav
committed
9b96675
Cleaned up some code.
gaurav
committed
97bb7a6
Sped up tests, made things work better.
gaurav
committed
ed8f1db
Fixed some issues.
gaurav
committed
f0342ce
It works! But it shouldn't. Cross that bridge later etc.
gaurav
committed
ebac7ab
Various improvements.
gaurav
committed
05b767e
Various improvements.
gaurav
committed
15b28d1
Got rid of xfail system (too much bother).
gaurav
committed
a6845a3
Incorporated issue state into outputs.
gaurav
committed
54001bb
Better?
gaurav
committed
150f4a6
Nope, still too messy.
gaurav
committed
92821bf

Commits on Jan 15, 2026

Added some NameRes tests.
gaurav
committed
9a4a22c

Commits on Feb 15, 2026

Merge branch 'main' into add-github-issue-tests
gaurav
committed
2a610a4

Commits on Feb 16, 2026

Reorganized test files.
gaurav
committed
0848de4
Don't need __init__.py in top-level src directory.
gaurav
committed
127cca1

Commits on Feb 18, 2026

Merge branch 'main' into add-github-issue-tests
gaurav
committed
b7d2786

Commits on Feb 19, 2026

Moved NodeNorm/NameRes into own files.
gaurav
committed
b91609f
Moved sources into separate directories.
gaurav
committed
8a08e9a
Updated CLAUDE.md in line with updated code.
gaurav
committed
1f84aea
Claude suggested moving blocklist code into src/ so we did.
gaurav
committed
31b4436
Added blocklist.py where a lot of the blocklist code has moved.
gaurav
committed
6b60325
Refactor GitHub issue assertions into a dedicated package.

gaurav
and
claude
committed
ac41e08

Commits on Feb 20, 2026

Use pytest subtests for GitHub issue tests so all assertions run independently.

gaurav
and
claude
committed
ffbc70f
Avoid slow GitHub API calls when building issue IDs.

gaurav
and
claude
committed
1524b27
Cleaned up some Claude code.
gaurav
committed
9a19e45
Refactor assertions into a proper framework for easy authoring

gaurav
and
claude
committed
0ec9e20
Auto-generate assertions/README.md from handler class attributes

gaurav
and
claude
committed
ddaa14f

Commits on Feb 21, 2026

Speed up GitHub issues test collection with pytest_generate_tests hook

gaurav
and
claude
committed
4ed6091
Parallelize GitHub issues tests with pytest-xdist

gaurav
and
claude
committed
c63e2d1
Cache Google Sheet CSV to disk so xdist workers share one download

gaurav
and
claude
committed
75abe00
Fix non-deterministic test ordering under pytest-xdist

gaurav
and
claude
committed
3bf169e
Cleaned up code a bit.
gaurav
committed
93753b0
Improved outputs.
gaurav
committed
2e3b93a
Always download fresh Google Sheet data on each test run

gaurav
and
claude
committed
db7933b
xfail open GitHub issues so failures don't block CI

gaurav
and
claude
committed
768cf91
Wipe GitHub issues cache at the start of every test run

gaurav
and
claude
committed
6ee7441
Add system tests for invalid BabelTest assertion name detection

gaurav
and
claude
committed
74c5e35

Commits on Feb 28, 2026

Report failing subtests of open issues as XFAIL instead of FAIL

gaurav
and
claude
committed
33aa773
Improved outputs.
gaurav
committed
d238f38
Speed up GitHub issues test collection via Search API

gaurav
and
claude
committed
8807972

Commits on Mar 1, 2026

Fail hard on unknown assertion types instead of XFAIL

gaurav
and
claude
committed
056e168
Removed default repo list.
gaurav
committed
79fab58
Add DoesNotResolveWith assertion handler

gaurav
and
claude
committed
fc158e4
Fixed Claude typo.
gaurav
committed
4b77e98
Improve error message.
gaurav
committed
d3fb921
Add HasLabel assertion handler

gaurav
and
claude
committed
4d272f1
Fixed typo in my code.
gaurav
committed
a9c0842
XPASS tests now correctly complain.
gaurav
committed
4b6ea91

Commits on Mar 5, 2026

Merge branch 'main' into add-github-issue-tests
gaurav
committed
91782a2

Commits on Mar 6, 2026

Regenerated BabelTest assertions documentation.
gaurav
committed
8414350
Fixed timeout.
gaurav
committed
1c75b08
Add unit test marker and GitHub Actions workflow for PRs

gaurav
and
claude
committed
3eeeccf

Commits on Mar 15, 2026

Potential fix for pull request finding

gaurav
and
Copilot
authored
bccde07
Potential fix for pull request finding

gaurav
and
Copilot
authored
f733858
Potential fix for pull request finding

gaurav
and
Copilot
authored
79c01f7
Potential fix for pull request finding

gaurav
and
Copilot
authored
9163671
Potential fix for pull request finding

gaurav
and
Copilot
authored
79a0b5c
Potential fix for pull request finding

gaurav
and
Copilot
authored
8e90722
Potential fix for pull request finding

gaurav
and
Copilot
authored
4a399e6
Fix CURIE prewarming to only normalize CURIE parameters

gaurav
and
claude
committed
224cd42
Potential fix for pull request finding

gaurav
and
Copilot
authored
2628f50
Move pytest marker config from pyproject.toml to tests/pytest.ini

gaurav
and
claude
committed
32248b2
Tweaked error message.
gaurav
committed
c175f67
Simplify: remove dead code and minor cleanup

gaurav
and
claude
committed
1db38aa
Handle corrupted GitHub issues cache file gracefully

gaurav
and
claude
committed
2251884
Replace print() with debug logging in SearchByNameHandler

gaurav
and
claude
committed
90bc845

Commits on Mar 16, 2026

Potential fix for pull request finding

gaurav
and
Copilot
authored
d474b18
Fixed comment.
gaurav
committed
14d230c
Move pytest config from tests/pytest.ini to pyproject.toml

gaurav
and
claude
committed
ae9ed8d
Updated pytest to 9.0+ so we can use subtests.
gaurav
committed
dc51c0b
Move GitHub repo list from conftest.py to targets.ini [DEFAULT]

gaurav
and
claude
committed
552d53a
Updated uv.lock.
gaurav
committed
2de6e09
Document param_set/param_sets terminology across assertions package

gaurav
and
claude
committed
870f1f9
Reject extra params in HasLabel and SearchByName assertions

gaurav
and
claude
committed
d6077ea
Add unit test documenting yaml.YAMLError on malformed YAML block

gaurav
and
claude
committed
d7fde85
Add TestEmptyOrNullBabelTests to document current behaviour for empty/null babel test content

gaurav
and
claude
committed
688ef36
Add unit tests for issue_has_tests() and get_issues_with_tests()

gaurav
and
claude
committed
a029093

Commits on Apr 1, 2026

Simplify: remove dead code, fix string parsing, rename fixture

gaurav
and
claude
committed
fd44d66
Handle missing labels in NodeNorm.
gaurav
committed
0c46ce9
Fixed level lookup in NodeNorm result.
gaurav
committed
aa94099

Commits on Apr 23, 2026

Merge branch 'main' into add-github-issue-tests
gaurav
committed
ab72ae4
Reverted NameRes to Solr instead of ElasticSearch.
gaurav
committed
4f74880

Commits on Apr 27, 2026

Reconfigured ci-es.
gaurav
committed
714d4bb

Commits on Apr 30, 2026

Changed Biolink Type param to a list.
gaurav
committed
1e9c42f

Commits on May 4, 2026

Add csv-to-babeltests CLI for converting CSV assets into BabelTests YAML.

gaurav
and
claude
committed
5ee0efa

Commits on May 5, 2026

Add pytest-subtests dependency

gaurav
and
claude
committed
6d5d593
Add request timeouts and fix bulk_lookup error message

gaurav
and
claude
committed
b0a4dc0
Validate Google Sheet HTTP response before caching

gaurav
and
claude
committed
65236d6
Fix wiki syntax parsing: allow assertion-name-only and warn on unresolved IDs

gaurav
and
claude
committed
5150703
Clean up lock files on pytest startup and fix incomplete comment

gaurav
and
claude
committed
da94577
Fix CI unit-test workflow: restrict paths and inject GITHUB_TOKEN

gaurav
and
claude
committed
37d1da6
Move curie_params overrides after class attributes in assertion handlers

gaurav
and
claude
committed
f125cd5

Commits on May 8, 2026

Fix param-aware cache keys in CachedNodeNorm and CachedNameRes

gaurav
and
claude
committed
7eeafde
Improve robustness of GitHub issue parsing and fetching

gaurav
and
claude
committed
e8c4e6f
Fix GitHub issues test harness: lazy dotenv, xfail clarity, missing __init__

gaurav
and
claude
committed
f6088ad
Fix two bugs surfaced by Copilot review

gaurav
and
claude
committed
b7146b8
Fix unbounded logger growth in GitHubIssueTest

gaurav
and
claude
committed
576fa0f
Merge branch 'add-github-issue-tests' of github.com:TranslatorSRI/babel-validation into add-github-issue-tests
gaurav
committed
999240f
Fixed endpoint for NameRes blocklist.
gaurav
committed
be3c09b

Commits on May 11, 2026

Fix four issues from Opus code review of PR #67

gaurav
and
claude
committed
4250f15

Commits on May 12, 2026

Catch expired GitHub token and fail only the GitHub issue tests

gaurav
and
claude
committed
e808947

Commits on May 17, 2026

Fix bugs and clean up code quality from simplify review

gaurav
and
claude
committed
39e4e0e
Fix three bugs identified in PR #67 review

gaurav
and
claude
committed
692418a

Commits on May 18, 2026

Potential fix for pull request finding

gaurav
and
Copilot
authored
09f9a84
Removed unused import.
gaurav
committed
d873d4a
Removed unused import.
gaurav
committed
bbba0d9
Potential fix for pull request finding

gaurav
and
Copilot
authored
27e45ec
Add test documenting YAML scalar assertion param handling

gaurav
and
claude
committed
a329eb0

Commits on Jun 10, 2026

Merge main into add-github-issue-tests, resolving blocklist conflicts

gaurav
and
claude
committed
52ef896
Simplify assertion loop logic and YAML param normalization

gaurav
and
claude
committed
063975d
Simplify test config cleanup and GitHub issue caching

gaurav
and
claude
committed
e36334f
Keep open-issue assertion failures inside the xfail family

gaurav
and
claude
committed
e1ee783
Make the {{BabelTest|...}} wiki marker case-insensitive

gaurav
and
claude
committed
adb7829
Run GitHub-issue unit tests without a GITHUB_TOKEN

gaurav
and
claude
committed
03e0c6b
Validate CURIEs before warming the NodeNorm cache

gaurav
and
claude
committed
e322040
Expire the Google Sheet CSV cache after an hour

gaurav
and
claude
committed
a9ec8e4
Limit bare pytest collection to the tests directory

gaurav
and
claude
committed
7e58ba7
Correct Repositories comment in targets.ini

gaurav
and
claude
committed
af19a1e
Skip GitHub issue fetch when -m deselects the test

gaurav
and
claude
committed
0a20287
Enforce 2+ CURIE arity in ResolvesWith/DoesNotResolveWith

gaurav
and
claude
committed
2a3085a
Avoid KeyError in normalize_curie if NodeNorm omits a CURIE

gaurav
and
claude
committed
f0c6cb8
Wire SearchByName top-N from targets.ini NameResXFailIfInTop

gaurav
and
claude
committed
c7b9da9
Extract deselected_by_markexpr into a shared helper

gaurav
and
claude
committed
24ed4f4
Defer Google Sheet downloads so -m unit stays offline

gaurav
and
claude
committed
e983020

Commits on Jun 24, 2026

Validate babel_tests YAML shape with clear errors

gaurav
and
claude
committed
d71ae23
Guard against empty NodeNorm type list in assertion messages

gaurav
and
claude
committed
add0848
Use a module-local YAML dumper in csv_to_babeltests

gaurav
and
claude
committed
c0ddd7b

Commits on Jun 26, 2026

Document testing/repo gotchas in CLAUDE.md

gaurav
and
claude
committed
1adf194

Commits on Aug 18, 2026