Skip to content

fix(evaluation): preserve trace boundaries in eval sets - #907

Open
linhongyu510 wants to merge 1 commit into
volcengine:mainfrom
linhongyu510:fix/trace-eval-case-boundaries
Open

fix(evaluation): preserve trace boundaries in eval sets#907
linhongyu510 wants to merge 1 commit into
volcengine:mainfrom
linhongyu510:fix/trace-eval-case-boundaries

Conversation

@linhongyu510

@linhongyu510 linhongyu510 commented Aug 16, 2026

Copy link
Copy Markdown

Summary

Fixes #1021

Fix trace-to-eval-set conversion when a tracing file contains more than one trace.

Previously, conversion state such as conversation, app_name, and user_id was shared across trace groups, while eval_cases.append() ran after the grouping loop. This could collapse multiple traces into one eval case and mix tool calls or metadata across trace boundaries.

This change:

  • creates one isolated eval case per trace
  • sorts spans by start_time before deriving input, output, and tool order
  • resets conversation and metadata state for every trace
  • includes the trace ID in generated eval IDs
  • derives the eval-set timestamp from the earliest generated case

The regression test uses two interleaved, out-of-order traces and verifies that conversations, tool calls, metadata, and timestamps remain isolated.

Validation

  • latest main with the regression test: fails with 1 eval case instead of 2
  • uv run pytest tests/test_evaluator.py -q: 3 passed
  • uv sync --all-extras && uv run pytest -n 16 -q: 2929 passed, 6 skipped, 6 subtests passed
  • ruff check veadk/evaluation/base_evaluator.py tests/test_evaluator.py
  • ruff format --check veadk/evaluation/base_evaluator.py tests/test_evaluator.py
  • git diff --check

AI assistance

This change was developed with AI assistance. I reviewed the trace grouping logic, rebased it onto the latest main, and verified the regression against both the unfixed and fixed implementations.

@linhongyu510
linhongyu510 force-pushed the fix/trace-eval-case-boundaries branch from 293239a to 030ac07 Compare August 18, 2026 02:23
@linhongyu510
linhongyu510 force-pushed the fix/trace-eval-case-boundaries branch from 030ac07 to 4e887bd Compare August 29, 2026 22:47
@linhongyu510
linhongyu510 force-pushed the fix/trace-eval-case-boundaries branch from 4e887bd to 7a83eb1 Compare August 31, 2026 22:45
@linhongyu510

linhongyu510 commented Aug 31, 2026

Copy link
Copy Markdown
Author

A previously completed 2929-pass run is documented in Issue #1021, but it was on the pre-rebase head and is not evidence for 7a83eb1., but it was on the pre-rebase head and is not evidence for 7a83eb1.

@linhongyu510

Copy link
Copy Markdown
Author

Full-suite follow-up for current head 7a83eb1: I created a clean isolated Python 3.12.13 environment outside the repository, ran uv sync --all-extras (288 locked packages), then executed python -m pytest -n 16 -q. Final result: 3175 passed, 5 skipped, 66 warnings, 6 subtests passed in 396.41s. The OTLP exporter logged localhost:8000 connection failures during teardown, but pytest completed normally with a successful final summary. This supersedes my earlier incomplete full-suite attempt.

1 similar comment
@linhongyu510

Copy link
Copy Markdown
Author

Full-suite follow-up for current head 7a83eb1: I created a clean isolated Python 3.12.13 environment outside the repository, ran uv sync --all-extras (288 locked packages), then executed python -m pytest -n 16 -q. Final result: 3175 passed, 5 skipped, 66 warnings, 6 subtests passed in 396.41s. The OTLP exporter logged localhost:8000 connection failures during teardown, but pytest completed normally with a successful final summary. This supersedes my earlier incomplete full-suite attempt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Multi-trace files collapse into one eval case

1 participant