[6londe]

What does this PR do?

BatchEvaluationRunner.run_async() calls the synchronous client.flush() directly after evaluation. Waiting for the score queue or network uploads therefore blocks the caller's event loop, including unrelated tasks and callbacks.

Await asyncio.to_thread(self.client.flush), matching the existing experiment runner, so flushing finishes before a normal return while the event loop stays available. Update the method docstring and add a regression using the real SDK client, generated observations/score API clients, score queue and consumer thread with httpx.MockTransport.

The test schedules an event-loop callback from the score upload handler and checks that it runs during delivery. On unchanged main 1fc854dd2b241cf8e3dac38bfd5907ff2fe365e3, it fails because the callback cannot run until the upload wait times out. With the fix, it runs during the upload; both scores are delivered and the queue is empty before the result returns.

Found during a source audit; no matching issue or PR was found. This is separate from #2064, which adds a missing flush on the fetch-failure path. This patch changes only the existing normal-completion flush.

Type of change

• Bug fix

Verification

uv sync --locked
uv run --frozen pytest tests/unit/test_batch_evaluation.py -k flush_keeps_event_loop -q
uv run --frozen pytest tests/unit/test_batch_evaluation.py tests/unit/test_score_consumer.py tests/unit/test_resource_manager.py -q
uv run --frozen ruff check .
uv run --frozen mypy langfuse --no-error-summary
uv run --frozen pre-commit run --files langfuse/batch_evaluation.py tests/unit/test_batch_evaluation.py

• Regression on unchanged production code: 1 failed, 17 deselected.
• After the fix: 71 passed across the three affected modules.
• Repository-wide Ruff and mypy, changed-file formatting/pre-commit and normal commit hooks passed.
• No live Langfuse server or model provider calls. Live-backend e2e tests, the remaining unit suite and the Python-version matrix were not run locally.

Checklist

• Self-reviewed the diff using code_review.md.
• Added a regression test for the behavior change.
• Updated the affected method's docstring; existing examples remain valid.
• Did not hand-edit generated files.
• Did not commit secrets or credentials.

AI assistance: Codex investigated, generated and reviewed the patch and ran the checks under the account owner's authorization. No independent human review is claimed.

Confidence Score: 5/5

The PR appears safe to merge.

Summary

Moves the normal-completion flush in BatchEvaluationRunner.run_async() to an awaited worker thread.

• Batch evaluation flushes scores without blocking the event loop.

Acknowledged scope: 6londe explicitly leaves the existing fetch-failure flush issue to #2064. This patch changes only the normal-completion flush.

Diagram

sequenceDiagram
    participant R as Batch runner
    participant L as Event loop
    participant W as Worker thread
    participant C as Score consumer
    R->>W: Await client.flush()
    W->>C: Wake consumer
    W->>W: Wait for queued work
    C->>C: Upload scores
    L->>L: Run unrelated callbacks
    C-->>W: Mark scores finished
    W-->>R: Flush finished
    R->>R: Return result

Loading

_{Reviews (1) · Last reviewed commit: "fix(evaluations): flush scores without b..." · Reviewed by Greptile}

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[chrikrah]

From my perspective this can be merged; it's the same await asyncio.to_thread(self.flush) the experiment runner already uses at client.py:2791.

$ sed -i 's/await asyncio.to_thread(self.client.flush)/self.client.flush()/' langfuse/batch_evaluation.py && .venv/bin/python -m pytest tests/unit/test_batch_evaluation.py tests/unit/test_score_consumer.py tests/unit/test_resource_manager.py -q -p no:cacheprovider   # head ba1eb3b with the fix reverted
1 failed, 70 passed, 267 warnings in 6.36s

non-blocking: #2064 adds a bare self.client.flush() on the fetch-failure return at line 968, so that path would still block. If it lands first, would you switch that call to the same form when you rebase?