Skip to content

πŸ› Fix: avoid duplicate stream keyword in evaluator LLM calls - #3829

Merged
WMC001 merged 1 commit into
developfrom
codex/fix-evaluation-stream-kwargs
Sep 1, 2026
Merged

πŸ› Fix: avoid duplicate stream keyword in evaluator LLM calls#3829
WMC001 merged 1 commit into
developfrom
codex/fix-evaluation-stream-kwargs

Conversation

@cj2026-bit

@cj2026-bit cj2026-bit commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Problem

Evaluation requests could fail when the evaluator generated system prompts or processed evaluation cases.

The failure was caused by a duplicated stream keyword in the evaluator LLM request.

Root Cause

The in-process LLM client caching feature was introduced by
PR #3668.

After that change, cached model clients could retain a construction-time
stream parameter in the result of _prepare_completion_kwargs().

The normal conversation path was later adapted in
PR #3695 by removing
the stale stream value before forcing streaming.

However, the evaluation path was not updated accordingly. It still called:

When completion_kwargs already contained stream=False, Python received
the same keyword twice and raised:

TypeError: got multiple values for keyword argument 'stream'

The evaluator then forces streaming by calling:

client.chat.completions.create(
    stream=True,
    **completion_kwargs,
)

This passes stream twice and can raise a Python duplicate-keyword error before the request reaches the model.
The normal conversation path already removes the old stream parameter, but the evaluator path did not perform the same adaptation.
Solution
Remove any existing stream value from the prepared completion parameters before forcing evaluator calls to use streaming:

completion_kwargs.pop("stream", None)

current_request = llm.client.chat.completions.create(
    stream=True,
    **completion_kwargs,
)

Other completion parameters remain unchanged.

Validation Screenshots

  1. Restart the runtime service and clear the cache
  2. On the login screen, start a new conversation using Model A
image
  1. Use Model A to run an agent evaluation
image

@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

βœ… All modified and coverable lines are covered by tests.

πŸ“’ Thoughts on this report? Let us know!

@cj2026-bit cj2026-bit changed the title fix(evaluation): avoid duplicate stream keyword Fix: avoid duplicate stream keyword in evaluator LLM calls Aug 31, 2026
@cj2026-bit cj2026-bit changed the title Fix: avoid duplicate stream keyword in evaluator LLM calls πŸ› Fix: avoid duplicate stream keyword in evaluator LLM calls Aug 31, 2026
@WMC001
WMC001 merged commit a026607 into develop Sep 1, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants