Skip to content

Commit 30373dd

Browse files
JuliaEdomclaude
andcommitted
fix(eval): validate --only-qids before loading the BIRD dataset
eval_baseline loaded the BIRD dump before parsing --only-qids, so on hosts without the local dataset (CI) an invalid qid list died with FileNotFoundError instead of the intended exit-3 usage error the CLI contract (and its test) promises. Parse and validate the flag first; the dataset load stays where the data is actually needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 291bc89 commit 30373dd

1 file changed

Lines changed: 9 additions & 5 deletions

File tree

‎scripts/eval_baseline.py‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -209,18 +209,22 @@ def main(argv: list[str] | None = None) -> int:
209209
)
210210
args = parser.parse_args(argv)
211211

212+
# Validate CLI input before the (filesystem-dependent) dataset load, so a
213+
# typo fails fast with exit 3 instead of a FileNotFoundError on hosts
214+
# without the BIRD dump.
215+
try:
216+
only_qids = [int(x) for x in args.only_qids.split(",") if x.strip()]
217+
except ValueError:
218+
print("[error] invalid --only-qids: expected comma-separated integers", file=sys.stderr)
219+
return 3
220+
212221
examples = load_bird_mini_dev(Path(args.bird_root))
213222
if args.db:
214223
examples = [e for e in examples if e.registry_db_id == args.db]
215224
if not examples:
216225
print(f"[error] no examples for db {args.db!r}", file=sys.stderr)
217226
return 3
218227

219-
try:
220-
only_qids = [int(x) for x in args.only_qids.split(",") if x.strip()]
221-
except ValueError:
222-
print("[error] invalid --only-qids: expected comma-separated integers", file=sys.stderr)
223-
return 3
224228
if only_qids:
225229
examples_by_qid = {e.question_id: e for e in examples}
226230
sample = [examples_by_qid[qid] for qid in only_qids if qid in examples_by_qid]

0 commit comments

Comments
 (0)