Repository navigation
Conversation
chrikrah
left a comment
There was a problem hiding this comment.
@Nafay-0 Nothing here blocks a merge from my side: at 4bbe63b each evaluator keeps its own stats entry, which is what evaluator_stats as a list already implied.
# git archive 4bbe63b (base 8e6c60d), Python 3.12.14, langfuse.__file__ under the tree
$ python -m pytest tests/unit/test_batch_evaluation.py -q
21 passed in 0.15s
# langfuse/batch_evaluation.py from 8e6c60d, tests kept
2 failed, 19 passed in 0.14s
FAILED test_evaluators_sharing_a_name_get_separate_stats
FAILED test_autoevals_evaluators_get_separate_stats
# not run: tests/e2e, no Langfuse server here
non-blocking: code that builds {s.name: s for s in result.evaluator_stats}, as tests/e2e/test_batch_evaluation.py:238 does, now keeps only the last of two same-named evaluators and loses the first one's stats. I'd keep the names, as the description promises, and add a docstring line that names can repeat. Can that go in this PR?
chrikrah
left a comment
There was a problem hiding this comment.
@Nafay-0 thanks, the docstring line in 9a40b27 covers what I asked for. 36fb23c only merges main and touches neither batch_evaluation.py nor its tests, so the result matches my review at 4bbe63b.
$ uv run --frozen pytest tests/unit/test_batch_evaluation.py -q # head 36fb23c, Python 3.12.14
21 passed in 0.13s
# langfuse/batch_evaluation.py from main 077448c, tests kept
2 failed, 19 passed in 0.14s
My approval carries over to this head, and I have nothing further on this PR.
What does this PR do?
Keep a separate
EvaluatorStatsentry for each evaluator passed torun_batched_evaluation, even when several evaluators have the same function name.The runner keyed statistics by
__name__. Every evaluator returned bycreate_evaluator_from_autoevals()is namedlangfuse_evaluator, so a batch with two scorers returned one combined stats entry. Factory-created evaluators and lambdas have the same problem. If one evaluator always failed and another always succeeded, the combined entry misleadingly showed a 50% success rate instead of identifying the failing evaluator separately.Key the internal dictionary by evaluator position instead. Display names, ordering, and the public result types are preserved.
Reproduced through the public
Langfuse.run_batched_evaluation()API with two real local autoevals scorers (LevenshteinandExactMatch) and in-process observation/score fakes. For three observations, the original code returns one entry with six runs; the fix returns two entries with three runs each. No live Langfuse or model-provider request was made.No existing issue to close. This is separate from #2103, which corrects score-copy accounting and pagination limits rather than same-named evaluator collisions.
Type of change
Verification
Both new regression tests fail on the original code and pass with the fix. They cover local autoevals wrappers and same-named factory evaluators with separate success/failure/score counts.
Without explicit test credentials, 18 existing
test_prompt.pyfixture setup errors also occur on unmodified main; #2072 and #1979 address that independently. E2E and live-provider suites were not run.Checklist
code_review.md..env.templateif needed. (No changes needed.)Claude Code assisted with investigation, implementation, and regression tests. I reviewed the diff and reran the checks before submitting.


Confidence Score: 5/5
This PR appears safe to merge.Summary
This PR keeps separate statistics for evaluators that share a function name.
Reviews (1) · Last reviewed commit: "fix(batch-evaluation): separate stats fo..." · Reviewed by Greptile