[OPIK-8102] [BE] [FE] feat: expose Gemini thinking config for Vertex AI and google_ai - #8024
Conversation
…king config Gemini thinking cannot be configured on the Vertex provider because Opik builds a protobuf GenerationConfig, and the 1.27.0 that langchain4j-vertex-ai-gemini pins transitively has no thinking_config field. 1.52.0 adds GenerationConfig.ThinkingConfig (include_thoughts, thinking_budget). langchain4j's VertexAiGeminiChatModel builder exposes no thinking option either, and Opik bypasses that builder anyway, so the protobuf is the only lever. The org.jetbrains:annotations exclusion is required, not incidental: the bump otherwise downgrades it to 13.0, whose @NotNull lacks TYPE_USE, which fails the test sources' type-context usages and cascades into ~200 misleading "cannot find symbol" errors on Lombok-generated members. Verified: dependency tree resolves vertexai 1.52.0 with annotations held at 17.0.0, and the infrastructure.llm.** suites pass (1077 tests, 0 failures). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gemini calls ran at Google's per-model default thinking behaviour with no way to enable, tune, or disable it. Most visibly, Gemini 2.5 Flash Lite ships with thinking off, so there was no way to turn it on at all. Thinking is carried as custom_parameters.thinking.level, following the Anthropic precedent. This matters on the playground path: the request body is a flat spread of the config and deserializes into langchain4j's ChatCompletionRequest, which discards unknown top-level fields — the pre-existing flat thinking_level was being dropped silently. custom_parameters is the only free-form slot it captures. The two providers do not expose the same knobs. Google AI Studio takes a thinking level directly; the Vertex GenerationConfig.ThinkingConfig protobuf has only thinking_budget and include_thoughts, so a level is translated into a budget there. GeminiThinkingParams holds that translation and decodes both custom parameter shapes (JsonNode for judge rules, Map for the playground). Frontend: the level control was gated to Gemini 3 and now covers the 2.5 family on both providers, with an "off" option and an off default for Flash Lite so its disabled-by-default behaviour is preserved. Rule forms fold the selection into custom_parameters on save and read it back on load, so it survives a round trip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-actions
Bot
added
java
⏱️ pre-commit per-hook timing
⏭️ 40 skipped (no matching files changed)
|
|
This change looks worth a test. The thinking-level control is a pure lookup off the selected model ( Would target What it would check
Deploying a test environment for this PR and exploring it — results will follow in a comment. Not testable yet. The half the PR is named for — a level reaching Google as also touches Backend (Java API / internal) Advisory, from the QA test radar. Nothing here blocks this PR, and anything it proposes is a draft for review. Re-checked after a push on 03 Sep 07:05 UTC — nothing the verdict depends on changed. |
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
|
🌙 Nightly cleanup: The test environment for this PR ( |
…gence Follow-up on the review of this PR. Backend: - VertexAIClientGenerator: filter customParameters to object nodes before converting to a Map. custom_parameters is unvalidated free-form JSON and Jackson throws IllegalArgumentException on an array or scalar, which failed the whole evaluation run on a path that previously ignored the field. - GeminiThinkingConfigMapper: resolve the off-level budget through budgetForLevel() so an explicit budget_tokens wins on Google AI Studio exactly as it does on Vertex, instead of the two providers disagreeing. - ExperimentMessageRenderer: forward custom_parameters, which applyConfigs dropped, so a thinking level selected for an experiment reaches the request rather than falling back to the provider default. Frontend: - Split the 2.5 option list: only Flash and Flash Lite can disable thinking, so 2.5 Pro no longer offers "off" (a zero budget it rejects). - updateProviderConfig: reconcile a stale thinkingLevel on model change, mirroring the OpenAI reasoningEffort and Anthropic thinkingEffort handling. - getThinkingLevelOptions: drop the unreachable Vertex 3 Flash arm, which supportsVertexAIThinkingLevel does not enable. - Rule form: strip the persisted thinking block before re-adding the validated selection, so a rejected level no longer survives the custom_parameters spread; merge into the existing block so budget_tokens and include_thoughts are kept across an unchanged round trip. - sanitizeConfigForRequest: fold the level under custom_parameters only for callers that opt in, since the optimizer gateway consumes the same output as flat llm_model.parameters; merge rather than replace the thinking block. Backend 1150 tests pass (1133 before, 17 added), spotless clean. Frontend 2280 pass (2269 before, 11 added), typecheck and eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-actions
Bot
added
the
dependencies
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
The thinking dropdown fell back to getDefaultThinkingLevel() for display only. In the playground that matched what was sent, because getDefaultConfigByProvider seeds thinkingLevel into the config; the rule form seeds no provider defaults, so an untouched control read "Off" on 2.5 Flash Lite while persisting nothing and letting the provider default apply. updateProviderConfig already runs on model change in both forms, so the fix is to have its Gemini/Vertex branch set the default when no level is held, not only coerce a stale one. The displayed level is now the persisted and sent level. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
|
🌙 Nightly cleanup: The test environment for this PR ( |
aadereiko
left a comment
aadereiko
left a comment
There was a problem hiding this comment.
-
Data loss: saving an LLM-judge rule deletes the persisted thinking block
AddEditRuleDialog strips custom_parameters.thinking for any model in the level table, then re-adds it only if a valid, non-auto level exists. When the level is auto (or absent), nothing is re-added -
Optimizer: the saved level is silently reset to the model default on re-run
Rules parse the level back out ofcustom_parameters.thinking.levelon load. The optimizer form never does. convertOptimizationToFormData just merges the saved parameters blob into modelConfig, so thinkingLevel comes back undefined, and sanitizeConfigForRequest then substitutes the model default -
The optimizer doesn't go through ChatCompletionRequest
…evel From @aadereiko's review. 1. Rule save deleted the persisted thinking block whenever the form produced no level of its own. The Anthropic case was already gated, but a Gemini model whose default level is "auto" (2.5 Pro/Flash) hits the same path: the block is stripped and nothing re-adds it, losing budget_tokens and include_thoughts on an unedited save. The strip is now limited to the two cases that need it — the form is putting a level back, or the form held a level this model rejects (a stale "off" carried onto a model that cannot disable thinking). 2. sanitizeConfigForRequest overwrote a level already nested under custom_parameters with the model default, because only the flat thinkingLevel counted as "stored". A caller that persists the sanitized output and reloads it has no flat field: the optimizer form merges a saved run's parameters blob wholesale, so a saved level was reset to the default on every re-run. A nested level now counts as stored, with the flat field still taking precedence. Only reproduced on models whose default is a real level — on 2.5 the "auto" default skips the fold, which is why the earlier optimizer test did not catch it. Frontend 2301 tests pass (2298 before, 3 added), typecheck and eslint clean. Backend untouched; spotless clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
aadereiko
left a comment
aadereiko
left a comment
There was a problem hiding this comment.
The FE code looks good, thank you for addressing the comments :)
thiagohora
left a comment
thiagohora
left a comment
There was a problem hiding this comment.
Opik reviewer (mined from your team's review history)
25 findings — 1 high · 24 medium · 0 low. Suppressed by team conventions: see suppressed.md.
React 👍/👎 on each comment — your feedback helps tune what it flags.
…discarded From @thiagohora's review. The high finding: GeminiThinkingConfigMapper honoured custom_parameters.thinking.include_thoughts, but returnThinking is pinned to Boolean.FALSE on the model (Gemma 4 returns thought parts unconditionally and they would otherwise be concatenated into the answer). Verified in langchain4j 1.19's PartsAndContentsMapper: at FALSE a thought part is neither collected into thinking() nor appended to the text — it is dropped. So the flag billed for thinking tokens and returned nothing. Vertex has the mirror problem: nothing there filters thought parts, because langchain4j builds the answer from ResponseHandler.getText(), which concatenates every part with no thought check and has no returnThinking equivalent. Forwarding include_thoughts would prepend the reasoning trace to the answer, and on the judge path that breaks the JSON parse in OnlineScoringEngine, producing no scores. Neither provider can currently surface thoughts, and the frontend never authors the flag — it only preserves one already present. So it is no longer forwarded on either path, with the reasoning recorded at both sites. Wire it up alongside a way to actually return the thoughts. Also corrected a comment that claimed Gemini 3+ "does not accept a budget", which contradicted the Vertex path relying on exactly that. Gemini 3 does accept thinking_budget — verified live on both providers earlier in this PR. The "off" early-return is about "off" being meaningless on a model that cannot disable thinking, not about budgets in general. Test coverage for two smaller findings: budgetForLevel() with an explicit zero budget alongside a level (0 must win, not read as absent), and the renderer's non-object custom_parameters branch parameterised over array/string/number/boolean/ null instead of only an array. Backend 1177 tests pass, spotless clean. Frontend 2314 pass, untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ertex @thiagohora is right that the Anthropic half of that claim is false today, and worse than merely inaccurate: LlmProviderAnthropicMapper declares no mapping for thinking/customParameters, so the block never reaches AnthropicCreateMessageRequest, while its thinkingEnabled(request) does read custom_parameters. Forwarding a thinking block on that path therefore gates temperature/top_p off without enabling thinking. Wiring Anthropic thinking through the experiment path is its own change. Comment now says what is actually true and why an Anthropic experiment should not carry the block until then. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thiagohora's second look at the OFF early-return was right on one half. `{level: off, budget_tokens: 4096}` on a Gemini 3+ model hit the early return and discarded the explicit budget, while the same input on 2.5 resolves through budgetForLevel() and sends 4096 — contradicting what explicitBudgetWinsOverLevelOff asserts for 2.5. The early return is now gated on there being no explicit budget, so only the unusable level is dropped and the budget wins on both families. The include_thoughts half of that finding is moot: cc47af6 stopped forwarding it on every path, so the early return has nothing to discard there. Also closes the second half of the renderer-test finding: applyConfigParameters has configs present with no custom_parameters key and never asserted the outcome, so it now asserts customParameters() stays null. Backend 1178 tests pass, spotless clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
🌙 Nightly cleanup: The test environment for this PR ( |
…version-gate tests Three more from @thiagohora's review, all of which turned out to be real. sentGenerationConfig() returned path("generationConfig"), which degrades to a MissingNode. Every negative assertion built on it passed for the wrong reason: renaming the field to a typo left 20 of 23 tests green, including all four that assert a thinking block is absent. It now uses get() and asserts the node exists — the same sabotage fails 10. levelOffDisablesThinking had the same defect one level down: MissingNode .path("thinkingBudget").asInt() is 0, so isZero() succeeded whether budget 0 was sent or nothing was. It now asserts the block and the field are present first. Verified by removing the setThinkingConfig call: the test fails, where before it would have passed. modelAcceptsLevel coverage moved to GeminiThinkingParamsTest, where the method lives, replacing two near-duplicates in the mapper suite. Extended with the unversioned ids the heuristic cannot parse — gemini-omni-flash-preview, gemini-flash-latest — which read as pre-3 and get a budget. That is the safe direction: a budget is accepted on every generation, a level on a pre-3 model is rejected outright. Recorded in the javadoc, since it looked accidental. Also covers both accepted budget boundaries on the map path (0 and MAX_VALUE), which only had rejection cases. Backend 1172 tests pass, spotless clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
4 participants
Details
Gemini thinking could not be configured. The Gemini 3 dropdown that existed did nothing, and 2.5 Flash Lite ships with thinking off so it could not be turned on at all. That was the customer report.
Adds a thinking level dropdown for every Gemini model that supports thinking, on both Vertex AI and
google_ai. Sent ascustom_parameters.thinking.level, same as Anthropic thinking.Two things needed care:
thinking_levelwas being thrown away. The backend deserializes the playground body into langchain4j'sChatCompletionRequest, which drops unknown fields.custom_parametersis the only field taking arbitrary keys, so the level goes there.thinking_levelonly works ongoogle_aiwith Gemini 3+. Older models error, and Vertex has no level field at all. So the UI always shows a level and the backend converts it tothinking_budgetwhere needed. Sending both is a 400, so only one is ever set.Reviewing this
GeminiThinkingParams— decodes both shapes (JsonNodefor rules,Mapfor the playground) and converts level to budget. One place, so the paths cannot disagree.THINKING_LEVELS_BY_MODELinmodelUtils.ts— which levels each model allows, from Google's table. They vary per model: 3.7 Flash has nominimal, 3.1 Flash Lite onlyminimalandhigh. Adding a model is one line. Note new models arrive via an automated sync job that won't know to update this table.high: 2.5 getsauto, 2.5 Flash Liteoff, 3.7/3.6/3.5 Flashmedium. Opening the dropdown doesn't change model behaviour.autoandoffare ours, not Google's, and only on pre-Gemini-3 models.autosends nothing, which is how you ask for a dynamic budget.offsends budget 0, only on 2.5 Flash Lite, since 2.5 Pro can't turn thinking off.Behaviour change: prompts that sent no thinking config now send the model default. For 2.5 that default is
auto, which still sends nothing.Change checklist
Issues
AI-WATERMARK
AI-WATERMARK: yes
Testing
mvn test -Dtest='com.comet.opik.infrastructure.llm.**'→ 1156 pass, was 1077.spotless:checkclean.VertexAIClientGeneratorTestchecks the real outbound request body with WireMock, not internal state.npx vitest run→ 2294 pass, was 2269. Typecheck andeslint --max-warnings=0clean.Not run: full
mvn verify, which fails locally on unrelated ClickHouse/Testcontainers problems, so relying on CI. Budget numbers are confirmed as sent but not as accepted for every model, so worth trying 2.5 Flash and a 3.x Flash before release.Documentation
No docs change. This is a UI control, not a documented API surface, and the dropdown already has a tooltip.