[OPIK-8185] opik-instrument: verify span coverage + guard common ingestion traps - #173
Open
aswynz wants to merge 1 commit into
Open
[OPIK-8185] opik-instrument: verify span coverage + guard common ingestion traps#173aswynz wants to merge 1 commit into
aswynz wants to merge 1 commit into
Conversation
…stion traps The verify step confirmed a trace arrived but not that it was complete, and did not warn about the silent ingestion traps that make a trace land partial. Add a coverage check (expected spans vs search_spans; well-formed name/type) and a short "common ingestion traps" guard (batching race, post-hoc feedback, missing flush), so "verified" means a complete trace, not just any trace. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
alexkuzmik
requested changes
Sep 1, 2026
|
|
||
| Traces are asynchronous — allow a few seconds after the run and make sure the flush ran. | ||
|
|
||
| **Verify coverage, not just arrival.** A trace arriving is necessary but not sufficient — batching can silently drop or truncate spans, so a trace can land *incomplete* and still look fine. Before reporting `verified`: |
Contributor
There was a problem hiding this comment.
@aswynz let's add a tip that if opik mcp is available - the agent can use it for verification.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Enhances the
opik-instrumentskill's Step 6 — Verify ingestion so that "verified" means a complete trace, not just that a trace arrived.search_spansagainst the call sites instrumented on the exercised path; every span must have a non-emptyname/type, and LLM spans must carry input/output. Fewer or empty spans ⇒ notverified.Why
The skill already runs a safe path and checks that a trace arrives. But a trace can arrive silently incomplete — batching can drop or unname fast spans, and post-hoc scores can be lost if posted before the span's create reaches the backend. These were hit first-hand while instrumenting a multi-agent app from scratch: a tracer adapter that passed all unit tests and a static audit still lost trace data across three runs; only running the app and inspecting the platform caught it. A "code edits alone are not success" skill should catch that class of failure.
Docs-only change to the authored skill (
src/opik_mcp/skills/opik-instrument/SKILL.md); the publicopik-skillspack is built verbatim from it.Closes OPIK-8185.
🤖 Generated with Claude Code