Repository navigation
Commit 795f33b
authored
fix(eval): await asynchronous data sources (#1076)
`EvalData` accepts awaitable iterators and async data factories, but
both `Eval` and `EvalAsync` currently pass the resulting coroutine into
a synchronous `for` loop. Evaluation fails with `TypeError: 'coroutine'
object is not iterable` before processing a sample.
Await the data after resolving its factory and before adapting the
iterator. This matches the existing `workflow_eval.py` handling and
preserves synchronous iterators and async generators.
Validation on Python 3.13.18 with locked dependencies:
- The new public-API regression has 4 failures and 4 passing synchronous
controls on base `e26deb0`; all 8 cases pass with the fix.
- `pytest src/braintrust/test_framework.py
src/braintrust/test_framework2.py --vcr-record=none -q`: 180 passed, 5
skipped.
- Ruff 0.15.21 lint and format checks pass for both changed files.
- Full `pre-commit run --all-files` passes (pre-commit 4.6.2, including
Ruff and Codespell).
- Targeted Pylint (`--errors-only`) passes for both changed files.
- Full `nox -R -s test_core -- --vcr-record=none -q`: 1,127 passed, 114
skipped, 12 xfailed, 23 subtests passed. Git signing was disabled only
in this test process for deterministic generated Git fixtures.1 parent 2651d7a commit 795f33b
2 files changed
Lines changed: 43 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2014 | 2014 | | |
2015 | 2015 | | |
2016 | 2016 | | |
| 2017 | + | |
| 2018 | + | |
| 2019 | + | |
2017 | 2020 | | |
2018 | 2021 | | |
2019 | 2022 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
287 | 287 | | |
288 | 288 | | |
289 | 289 | | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
290 | 330 | | |
291 | 331 | | |
292 | 332 | | |
| |||
0 commit comments