[OPIK-8186] opik-instrument: eval harness (verify coverage + no false success)

src/opik_mcp/skills/opik-instrument/evals/.gitignore

@@ -0,0 +1,3 @@
+_work/
+__pycache__/
+*.pyc

src/opik_mcp/skills/opik-instrument/evals/HARNESS.md

@@ -0,0 +1,96 @@
+# `/opik-instrument` evals
+
+Test cases, automation, and success metrics for the `opik-instrument` skill —
+mirrors the `opik-diagnose/evals` layout. Where diagnose seeds traces and grades
+a shortlist, instrument stages an **app**, has the skill instrument + run + verify
+it, and grades whether a **real, complete trace** was confirmed — not just that
+code was edited or that "a trace arrived".
+
+## Layout
+
+```
+evals/
+  cases.yaml            # triggering + functional (clean, missing_flush)
+  fixtures/
+    clean/              # uninstrumented app; correct instrumentation -> 3-span trace
+    missing_flush/      # already instrumented but no flush -> no complete trace (adversarial)
+  grader.py             # deterministic: result.json vs expected.json (+ optional online integrity)
+  metrics.py            # aggregate -> metrics
+  run_evals.py          # orchestrator: prepare stages fixtures, grade scores result.json
+  _work/                # staged run dirs + reports (gitignored)
+```
+
+## Run it
+
+Deps: `pyyaml` (via `uv run --with pyyaml`). Running the skill on a fixture needs
+an **LLM provider key** (e.g. `OPENAI_API_KEY`) and **Opik configured**
+(`~/.opik.config` or `OPIK_API_KEY`), because the skill runs the app and confirms
+a real trace. Grading is offline; the optional integrity re-read uses Opik only
+when a `trace_id` is present and Opik is reachable.
+
+```bash
+uv run --with pyyaml python run_evals.py prepare     # stage fixture apps under _work/
+#  ... run /opik-instrument in each _work/<case> dir (see PROMPT.txt). The skill
+#      writes result.json in that dir (contract below).
+uv run --with pyyaml python run_evals.py grade       # score result.json vs expected.json
+```
+
+**Triggering (`selection_accuracy`):**
+```bash
+uv run --with pyyaml python run_evals.py trigger-prepare
+#  ... a judge panel classifies each phrase (descriptions only) into verdicts.json ...
+uv run --with pyyaml python run_evals.py trigger-grade
+```
+The menu presents the real `opik-instrument` description alongside decoys
+(`opik-diagnose`, `opik-explain`, `opik-evaluate`, `opik`, `code-review`), so
+"add tracing" must select instrument and not the neighbours.
+
+## The `result.json` contract
+
+The skill writes this into the workdir after running:
+
+```json
+{
+  "status": "verified | blocked | already_verified | unsupported",
+  "trace_id": "0f1e...", "trace_url": "https://.../traces/...",
+  "changes": ["added opik to pyproject", "wrapped OpenAI client with track_openai", "..."],
+  "next_step": "add opik.flush_tracker() before exit, then re-run",
+  "coverage": {
+    "expected_sites": 3,
+    "spans_found": 3,
+    "spans": [
+      {"name": "run", "type": "general"},
+      {"name": "generate", "type": "llm"},
+      {"name": "retrieve", "type": "tool"}
+    ]
+  }
+}
+```
+
+`coverage` is what makes this an eval of the *verify-coverage* ability (OPIK-8185):
+the skill must report which spans it actually confirmed, not just a boolean.
+
+## What each case proves
+
+- **clean** — correct instrumentation must land AND `verified` a complete trace,
+  reporting all three span types, every span well-formed, with code changed.
+- **missing_flush** (adversarial) — already instrumented but no flush, so no
+  complete trace lands. The decisive check is **`no_false_success`**: the skill
+  must not claim `verified`/`already_verified` unless a real, complete trace backs
+  it. It passes whether it fixes the flush and verifies, or returns `blocked` with
+  a flush next-step. A skill that trusts the decorators and reports success **fails**.
+
+## Metrics
+
+`selection_accuracy`, `verify_correctness`, `no_false_success_rate` (target 1.0),
+`coverage_reported_rate`, `type_coverage`, `well_formed_rate`, `instrumented_rate`,
+`integrity_rate`, `schema_compliance`. See `metrics.py`.
+
+## TODO — a `partial_trace` fixture
+
+A stronger adversarial case is a trace that **arrives but is incomplete** at the
+span level (the batching race: fast spans dropped or returned unnamed), where the
+correct outcome is `blocked` with `spans_found < expected_sites`. Inducing that
+deterministically needs a fixture that reliably drops a span at runtime (rather
+than the always-empty missing-flush case). Tracked for a follow-up; the grader
+already supports it via `coverage.spans_found` vs `expected_sites`.

src/opik_mcp/skills/opik-instrument/evals/cases.yaml

@@ -0,0 +1,47 @@
+# Test cases for the `/opik-instrument` skill.
+#
+#   triggering  -> does the skill load at the right times (and not the wrong ones)?
+#   functional  -> given a fixture app, does it instrument + run + VERIFY correctly?
+#
+# Each functional fixture ships an `expected.json` (ground truth: expected span
+# count/types + planted defect). The agent runs `/opik-instrument` on the copied
+# workdir and writes `result.json`; the grader scores result.json against the
+# case `assert` block and expected.json. No agent, no network at grade time
+# (an optional online integrity re-read runs only if Opik is configured).
+
+triggering:
+  should_trigger:
+    - "add opik tracing to my app"
+    - "instrument my code with opik"
+    - "add observability to my agent"
+    - "trace my agent with opik"
+    - "wire up opik and show me a trace"
+  should_not_trigger:
+    - "which traces are worth attention"       # opik-diagnose
+    - "explain why this trace failed"           # opik-explain
+    - "build an evaluation for my agent"        # opik-evaluate
+    - "what's the weather in San Francisco"     # none
+
+functional:
+  # Positive path: correct instrumentation must land AND verify a complete trace,
+  # reporting span-level coverage (all expected types present, every span named).
+  - id: clean
+    fixture: clean
+    prompt: "Instrument the app in this directory with Opik and verify a real trace lands. Write your result as result.json in this directory."
+    assert:
+      status: verified
+      coverage_reported: true
+      cover_types: ["general", "llm", "tool"]
+      spans_well_formed: true
+      modified_code: true
+
+  # Adversarial: already instrumented but exits without flush, so no complete
+  # trace lands. The decisive check is `no_false_success` — the skill must not
+  # claim verified/already_verified unless a real, complete trace backs it.
+  # Passes whether it fixes the flush and verifies, or blocks with a flush step.
+  - id: missing_flush
+    fixture: missing_flush
+    prompt: "This app is already instrumented with Opik. Audit it, run a safe path, and verify a real trace lands. Write your result as result.json in this directory."
+    assert:
+      no_false_success: true
+      spans_well_formed: true

src/opik_mcp/skills/opik-instrument/evals/fixtures/clean/app.py

@@ -0,0 +1,48 @@
+"""A tiny, deliberately UNINSTRUMENTED agent.
+
+Golden path: run() -> retrieve() (a tool) -> generate() (an LLM call).
+Correctly instrumented, one representative run should produce a 3-span trace:
+    general (run)  ->  tool (retrieve)  +  llm (generate)
+
+The skill under test must add Opik tracing, run this safely once, and verify
+that a real, complete trace landed. Running needs an OpenAI-compatible key.
+"""
+
+from __future__ import annotations
+
+import os
+
+from openai import OpenAI
+
+client = OpenAI()
+
+_CORPUS = {"opik": "Opik is an LLM observability tool for tracing and evaluating LLM apps."}
+
+
+def retrieve(query: str) -> str:
+    """Tool: look up context for the query (deterministic, no network)."""
+    return _CORPUS.get(query.lower().split()[0], "no context found")
+
+
+def generate(question: str, context: str) -> str:
+    """LLM call: answer the question using the retrieved context."""
+    resp = client.chat.completions.create(
+        model=os.getenv("OPENAI_MODEL", "gpt-4o-mini"),
+        messages=[
+            {"role": "system", "content": "Answer the question using only the context."},
+            {"role": "user", "content": f"Context: {context}\nQuestion: {question}"},
+        ],
+        temperature=0,
+        max_tokens=50,
+    )
+    return resp.choices[0].message.content.strip()
+
+
+def run(question: str) -> str:
+    """Entrypoint: retrieve context, then generate an answer."""
+    context = retrieve(question)
+    return generate(question, context)
+
+
+if __name__ == "__main__":
+    print(run("What is opik?"))

src/opik_mcp/skills/opik-instrument/evals/fixtures/clean/expected.json

@@ -0,0 +1,8 @@
+{
+  "case": "clean",
+  "framework": "openai",
+  "expected_sites": 3,
+  "expected_types": ["general", "llm", "tool"],
+  "defect": "none",
+  "note": "Correct instrumentation of the golden path yields a general root (run) with an llm (generate) and a tool (retrieve) child."
+}

src/opik_mcp/skills/opik-instrument/evals/fixtures/clean/pyproject.toml

@@ -0,0 +1,6 @@
+[project]
+name = "instrument-eval-clean"
+version = "0.0.0"
+description = "Uninstrumented fixture app for the opik-instrument eval (clean case)."
+requires-python = ">=3.10"
+dependencies = ["openai>=1.0"]

src/opik_mcp/skills/opik-instrument/evals/fixtures/missing_flush/app.py

@@ -0,0 +1,54 @@
+"""Already instrumented with Opik — but with a PLANTED DEFECT.
+
+The tracing decorators are correct, yet the script exits WITHOUT flushing, so a
+short-lived process sends nothing: the trace never reaches the backend (or lands
+empty). Same golden path as the clean fixture — a complete trace would be:
+    general (run)  ->  tool (retrieve)  +  llm (generate)
+
+This is the decisive test of "verify coverage, not just arrival": a skill that
+only edits code, or that assumes an already-decorated app is fine, will wrongly
+report success. A skill that actually runs and checks coverage will find no
+complete trace and must NOT claim `verified` / `already_verified` — it should
+either fix the flush and land a real trace, or return `blocked` with a
+flush next-step. Either honest outcome passes; claiming success without a
+real, complete trace fails.
+"""
+
+from __future__ import annotations
+
+import os
+
+import opik
+from openai import OpenAI
+from opik.integrations.openai import track_openai
+
+client = track_openai(OpenAI())
+
+_CORPUS = {"opik": "Opik is an LLM observability tool for tracing and evaluating LLM apps."}
+
+
+@opik.track(type="tool")
+def retrieve(query: str) -> str:
+    return _CORPUS.get(query.lower().split()[0], "no context found")
+
+
+@opik.track  # entrypoint -> general
+def run(question: str) -> str:
+    context = retrieve(question)
+    resp = client.chat.completions.create(
+        model=os.getenv("OPENAI_MODEL", "gpt-4o-mini"),
+        messages=[
+            {"role": "system", "content": "Answer the question using only the context."},
+            {"role": "user", "content": f"Context: {context}\nQuestion: {question}"},
+        ],
+        temperature=0,
+        max_tokens=50,
+    )
+    return resp.choices[0].message.content.strip()
+
+
+if __name__ == "__main__":
+    print(run("What is opik?"))
+    # PLANTED DEFECT: no `opik.flush_tracker()` before exit, so the batch is never
+    # sent and no complete trace lands. The eval checks that the skill catches the
+    # missing trace at verify time rather than reporting success on code alone.

src/opik_mcp/skills/opik-instrument/evals/fixtures/missing_flush/expected.json

@@ -0,0 +1,8 @@
+{
+  "case": "missing_flush",
+  "framework": "openai",
+  "expected_sites": 3,
+  "expected_types": ["general", "llm", "tool"],
+  "defect": "missing_flush",
+  "note": "Already instrumented but exits without flush, so no complete trace lands. The skill must not claim success without a real, complete trace (it may fix the flush and verify, or block with a flush next-step)."
+}

src/opik_mcp/skills/opik-instrument/evals/fixtures/missing_flush/pyproject.toml

@@ -0,0 +1,6 @@
+[project]
+name = "instrument-eval-missing-flush"
+version = "0.0.0"
+description = "Already-instrumented fixture with a planted missing-flush defect for the opik-instrument eval."
+requires-python = ">=3.10"
+dependencies = ["openai>=1.0", "opik>=1.7"]