[NA] [BE] feat: always persist online-evaluation monitoring traces - #7317
Open
alexkuzmik wants to merge 4 commits into
Open
[NA] [BE] feat: always persist online-evaluation monitoring traces#7317alexkuzmik wants to merge 4 commits into
alexkuzmik wants to merge 4 commits into
Conversation
Remove the onlineScoringTracingEnabled service toggle (env TOGGLE_ONLINE_SCORING_TRACING_ENABLED) so the LLM-as-judge online evaluation loop is always recorded as hidden source=evaluator monitoring traces. The three scorers now always start a real EvaluationRecorder; the trace-thread best-effort onErrorResume still degrades to NOOP on a project lookup failure. Drops the field from ServiceTogglesConfig, config.yml, config-test.yml, the OpenAPI schemas, and the generated Python/TS SDK types (the toggle was exposed via GET /v1/private/toggles). Origin feature: OPIK_6994. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
documentation
Contributor
Backend Tests - Unit Tests2 590 tests 2 588 ✅ 1m 16s ⏱️ Results for commit 62d0920. ♻️ This comment has been updated with latest results. |
Contributor
Backend Tests - Integration Group 12264 tests 263 ✅ 4m 50s ⏱️ Results for commit 62d0920. ♻️ This comment has been updated with latest results. |
Contributor
|
🌿 Preview your docs: https://opik-preview-c8295b3c-9d7d-44a2-a68c-3b12cc65935f.docs.buildwithfern.com/docs/opik 📌 Results for commit b4c504f |
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
…conditional The scorer unit tests relied on the (unstubbed) onlineScoringTracingEnabled toggle defaulting to false, which took the NOOP branch and never invoked onlineEvaluationRecorder.begin(). With the toggle removed the recorder is always started, so begin() must be stubbed to return EvaluationRecorder.NOOP in setup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
andrescrz
previously approved these changes
Jul 2, 2026
Member
There was a problem hiding this comment.
I like the idea of this feature being always enabled, I believe there's no per customer need etc. to disable. I tested it in prod and looks nice.
However, I wouldn't rush to merge and deploy this. I'd recommend leave it baking for max 1 week in production, so you have this flag as a emergency stop measure, in case some issue happens.
Left couple of non-blocking comments, but I do recommend addressing them.
Per review: don't hand-edit Fern-generated files. Revert the manual removals from the OpenAPI specs and the Python/TS rest_api ServiceTogglesConfig types; the daily Fern codegen job will drop onlineScoringTracingEnabled once the backend stops returning it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
removed
documentation
Contributor
Python SDK E2E Tests Results (Python 3.14)286 tests 284 ✅ 4m 20s ⏱️ Results for commit 708bc95. ♻️ This comment has been updated with latest results. |
andrescrz
previously approved these changes
Jul 3, 2026
Collaborator
Author
Don't merge before 13.07.2025 |
andrescrz
marked this pull request as draft
July 3, 2026 13:42
alexkuzmik
marked this pull request as ready for review
July 7, 2026 09:04
…racing
Resolve conflicts in online-scoring scorers: keep the {{trace}}/{{span}}
agentic triggers from main while dropping the isOnlineScoringTracingEnabled
toggle gate so the monitoring recorder is created unconditionally, and stub
the recorder in the span/thread scorer tests now that tracing always runs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
⏱️ pre-commit per-hook timing
⏭️ 40 skipped (no matching files changed)
|
andrescrz
approved these changes
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details
Online-evaluation monitoring traces (OPIK_6994) were gated behind the
onlineScoringTracingEnabledservice toggle (envTOGGLE_ONLINE_SCORING_TRACING_ENABLED, defaulttrue). This PR removes that configuration so the behavior is always on — every LLM-as-judge online evaluation (trace / span / thread) is always recorded as a hiddensource=evaluatormonitoring trace.EvaluationRecorderinstead of branching on the toggle.onErrorResume→EvaluationRecorder.NOOPfallback so a project lookup failure still can't abort scoring.ServiceTogglesConfig,config.yml, andconfig-test.yml. The toggle is exposed viaGET /v1/private/toggles, but per review the Fern-autogenerated OpenAPI specs and Python/TSrest_apitypes are left untouched — the daily Fern codegen job will droponlineScoringTracingEnabledonce the backend stops returning it.Change checklist
Issues
AI-WATERMARK
AI-WATERMARK: yes
Testing
mvn -o compileinapps/opik-backend— compiles cleanly.onlineScoringTracingEnabled/TOGGLE_ONLINE_SCORING_TRACING_ENABLEDin the repo.true), so they exercise the now-unconditional path.Documentation
The removed toggle was infra config only; no user-facing docs reference it.