fix(evaluation): preserve trace boundaries in eval sets - #907
fix(evaluation): preserve trace boundaries in eval sets#907linhongyu510 wants to merge 1 commit into
Conversation
293239a to
030ac07
Compare
030ac07 to
4e887bd
Compare
4e887bd to
7a83eb1
Compare
|
Full-suite follow-up for current head 7a83eb1: I created a clean isolated Python 3.12.13 environment outside the repository, ran uv sync --all-extras (288 locked packages), then executed python -m pytest -n 16 -q. Final result: 3175 passed, 5 skipped, 66 warnings, 6 subtests passed in 396.41s. The OTLP exporter logged localhost:8000 connection failures during teardown, but pytest completed normally with a successful final summary. This supersedes my earlier incomplete full-suite attempt. |
1 similar comment
|
Full-suite follow-up for current head 7a83eb1: I created a clean isolated Python 3.12.13 environment outside the repository, ran uv sync --all-extras (288 locked packages), then executed python -m pytest -n 16 -q. Final result: 3175 passed, 5 skipped, 66 warnings, 6 subtests passed in 396.41s. The OTLP exporter logged localhost:8000 connection failures during teardown, but pytest completed normally with a successful final summary. This supersedes my earlier incomplete full-suite attempt. |
Summary
Fixes #1021
Fix trace-to-eval-set conversion when a tracing file contains more than one trace.
Previously, conversion state such as
conversation,app_name, anduser_idwas shared across trace groups, whileeval_cases.append()ran after the grouping loop. This could collapse multiple traces into one eval case and mix tool calls or metadata across trace boundaries.This change:
start_timebefore deriving input, output, and tool orderThe regression test uses two interleaved, out-of-order traces and verifies that conversations, tool calls, metadata, and timestamps remain isolated.
Validation
mainwith the regression test: fails with 1 eval case instead of 2uv run pytest tests/test_evaluator.py -q: 3 passeduv sync --all-extras && uv run pytest -n 16 -q: 2929 passed, 6 skipped, 6 subtests passedruff check veadk/evaluation/base_evaluator.py tests/test_evaluator.pyruff format --check veadk/evaluation/base_evaluator.py tests/test_evaluator.pygit diff --checkAI assistance
This change was developed with AI assistance. I reviewed the trace grouping logic, rebased it onto the latest
main, and verified the regression against both the unfixed and fixed implementations.