Merge pull request #46 from evalshift/fix/native-message-shapes

fix(capture): promote every provider's native message shape; add capture sync --repair

Author: babaliauskas

CHANGELOG.md

@@ -9,6 +9,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `capture sync --repair` rebuilds the `inputs` and `history` of cases an
+  earlier sync built from stringified message blocks, keeping every other
+  field and the case's input variable name, and re-downloads exactly the
+  captures it needs from `captures.store` even though they are promoted. It
+  also rebuilds cases an earlier sync promoted with empty `inputs` (OpenAI
+  `exclude_none`, Responses, LangChain), which get no warning.
+  `capture sync` and `validate` warn when a suite still holds stringified
+  cases; without `--repair`, sync leaves both kinds untouched. A repaired case that now
+  duplicates another promoted case is named
+  (`repaired case <a> duplicates <b>`) and nothing is deleted.
 - `evalshift init` writes the `semantic` evaluator active for every provider.
   Anthropic and DeepSeek have no embeddings endpoint, so the scaffold borrows
   OpenAI's or Gemini's — whichever key you already have — and `init --ci`
@@ -46,6 +56,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- `capture sync` and `capture promote` now read every wrapped provider's own
+  message shape. Claude follow-up turns no longer get Python-repr strings
+  (`"[{'id': 'toolu_1', … 'type': 'tool_use'}]"`) in their history: `tool_use`
+  blocks become the assistant's `tool_calls`, `tool_result` blocks become `tool`
+  messages, thinking is dropped, and a prompt-cached `system` prompt or a
+  text-block user turn becomes plain text. OpenAI Chat histories with
+  `content: null` no longer abort the whole sync with a traceback, and OpenAI
+  Responses and LangChain captures recover their `inputs` instead of promoting
+  them empty.
+- A turn whose recorded messages carry an image, a document, audio, a
+  server-tool result or an unknown block type is skipped (`capture promote`
+  refuses it) instead of being promoted with that content stringified or lost;
+  `--allow-errored` does not apply.
+- An OpenAI `developer` message is now kept as the system prompt instead of
+  being dropped: it merges with a leading system prompt (a Responses
+  `instructions=` included), and a system or `developer` message after the
+  first turn is dropped with a warning instead of aborting the sync; run
+  `capture sync --force` to refresh cases promoted before.
+- A capture no valid case can be built from is skipped with the reason
+  (`capture promote` refuses it) instead of aborting the whole `capture sync`
+  with a validation traceback; `--allow-errored` does not apply.
 - A judge or embedding model with no API key no longer fails silently on every
   pair: it is skipped (advisory) or refused (blocking) before any call.
 - The bare `text-embedding-3-small` default now resolves to OpenAI, so its key

DOCS.md

@@ -173,7 +173,7 @@ captures:
   store: s3://acme-evals/support-agent
 ```
 
-and `capture sync` and `capture list` fetch what is new before they read the local directory — captures are immutable and toolsets content-addressed, so only objects missing locally are downloaded, captures already promoted are skipped (so `capture clean` never causes a re-download), and each file is written atomically. Bucket keys are untrusted: one not shaped like the SDK's layout — a suite directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with `.` or a space, included — is counted as malformed and never written (a key with a trailing newline included), and no key can place a file outside `.evalshift/`; folder-marker keys ending in `/` (the S3 console's "Create folder", ADLS directories) are skipped without being counted. One summary line reports it (`fetched 12 capture(s) and 1 toolset(s) from s3://…`; with `capture list --json` it goes to stderr, so stdout stays pure JSON). `--since 24h` limits a fetch to recently written captures, `--offline` skips it, and `capture fetch` does only that step. `promote`, `diff` and `clean` stay local. Install the matching client library — `pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity` — and authenticate with the provider's normal chain (an OIDC role in CI, `aws sso login` / `gcloud auth application-default login` / `az login` locally); credentials never go in the URI, and a value containing `@` or `?` fails to load; no rejected value is echoed back, at most its scheme. A missing client library exits 1 naming the `pip install <package>` to run; a list or download failure exits 1, so `sync` promotes nothing from a partial fetch, with a hint naming the provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC role, `gcloud auth application-default login`, `az login`); a local write failure names the file and asks you to check free disk space and permissions instead. If `evalshift.yaml` cannot be read or does not validate, `sync` and `list` skip the fetch and work on the local directory as before, warning only when the file has a top-level `captures:` key. Full field reference: [`captures`](docs/configuration.md#captures).
+and `capture sync` and `capture list` fetch what is new before they read the local directory — captures are immutable and toolsets content-addressed, so only objects missing locally are downloaded, captures already promoted are skipped (so `capture clean` never causes a re-download; `capture sync --repair` alone re-fetches the promoted captures its rebuilds need, past `--since` too), and each file is written atomically. Bucket keys are untrusted: one not shaped like the SDK's layout — a suite directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with `.` or a space, included — is counted as malformed and never written (a key with a trailing newline included), and no key can place a file outside `.evalshift/`; folder-marker keys ending in `/` (the S3 console's "Create folder", ADLS directories) are skipped without being counted. One summary line reports it (`fetched 12 capture(s) and 1 toolset(s) from s3://…`; with `capture list --json` it goes to stderr, so stdout stays pure JSON). `--since 24h` limits a fetch to recently written captures, `--offline` skips it, and `capture fetch` does only that step. `promote`, `diff` and `clean` stay local. Install the matching client library — `pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity` — and authenticate with the provider's normal chain (an OIDC role in CI, `aws sso login` / `gcloud auth application-default login` / `az login` locally); credentials never go in the URI, and a value containing `@` or `?` fails to load; no rejected value is echoed back, at most its scheme. A missing client library exits 1 naming the `pip install <package>` to run; a list or download failure exits 1, so `sync` promotes nothing from a partial fetch, with a hint naming the provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC role, `gcloud auth application-default login`, `az login`); a local write failure names the file and asks you to check free disk space and permissions instead. If `evalshift.yaml` cannot be read or does not validate, `sync` and `list` skip the fetch and work on the local directory as before, warning only when the file has a top-level `captures:` key. Full field reference: [`captures`](docs/configuration.md#captures).
 
 **`capture sync`** is the workhorse. Per suite it:
 
@@ -183,7 +183,7 @@ and `capture sync` and `capture list` fetch what is new before they read the loc
 4. Carries the recorded generation settings onto the promoted example: the first `model_call`'s `metadata["generation_config"]` is copied verbatim to the example's `generation_config`, so the replay dispatches under the temperature, structured-output, and tool-choice settings the production call actually used. `capture sync` appends `wired generation config for N case(s)` to its summary line when any promoted capture carried one.
 5. Records what the captured run cost on the case file as `cost_usd` (its `model_call` events summed) plus `cost_source`. The SDK never prices anything — a `model_call`'s `cost_usd` is `0.0` unless the app's own instrumentation set it, and the provider client wrappers record tokens but no cost by design — so each event with tokens and no cost is priced at promotion from litellm's price table for its own `model_id` (aliases and provider prefixes resolve through the registry first). `cost_source` is `"recorded"` when every non-zero part came from the capture (a recorded cost is never re-estimated), `"estimated"` when any part was priced here, and absent with `cost_usd: 0.0` when nothing was priced — no tokens, or a model litellm does not price. An unpriced local / self-hosted model is the normal case, not a failure: it stays at 0 silently, with no warning. The example in `golden.jsonl` never carries the figure; it is provenance of the capture, not something a replay reproduces.
 6. Skips captures whose replayed content duplicates an already-promoted case or an earlier capture in the same run — duplicate examples inflate *n* and corrupt the paired statistics (`--keep-duplicates` opts out). Dedup is seeded from the cases already in the suite dir, so it holds across repeated syncs.
-7. Skips captures whose turn recorded an `error` event — a turn that died before the agent acted is not ground truth, and promoting it would assert `expected_no_tools: true` on a question that needed a tool. `--allow-errored` promotes it anyway (still never asserting `expected_no_tools`). `capture promote` exits non-zero on the same condition. Separately and unconditionally — `--allow-errored` does not help — a capture whose first `model_call` has no `toolset_ref` is refused: the SDK did not record what tools were offered, so there is nothing to carry, and re-capturing with a current `evalshift-sdk` is the only fix.
+7. Skips captures whose turn recorded an `error` event — a turn that died before the agent acted is not ground truth, and promoting it would assert `expected_no_tools: true` on a question that needed a tool. `--allow-errored` promotes it anyway (still never asserting `expected_no_tools`). `capture promote` exits non-zero on the same condition. Separately and unconditionally — `--allow-errored` does not help — a capture whose first `model_call` has no `toolset_ref` is refused: the SDK did not record what tools were offered, so there is nothing to carry, and re-capturing with a current `evalshift-sdk` is the only fix. Also unconditionally, a capture whose recorded messages carry content a text replay can't carry — an image, a document, audio, a server-tool result, an unknown block type — is skipped and counted on its own (`capture promote` refuses it): a replay without it is not ground truth. Last, as a safety net, a capture whose case fails the suite schema's own validation in a way none of these rules anticipates is skipped with pydantic's first error and counted on its own (`skipped N capture(s) that could not form a valid case`; `capture promote` refuses it) instead of aborting the sync; hand-write that case or report the capture.
 8. Warns when two captures claim the same `(conversation_id, turn_index)` (a retried turn), and when a promoted turn contains a failed tool result (`error`, or `{"success": false}`). Both stay warnings — see [Agent evals → What does not belong in a golden suite](docs/agents.md).
 9. Writes `.evalshift/suites/<suite>/golden.jsonl` and rewrites the managed `suites:` block in `evalshift.yaml` (between the `>>> evalshift suites` markers).
 10. After the write (or after printing the block for you to paste), checks the CI pin: if a workflow under `.github/workflows/` uses `evalshift/evalshift-action` with an `evalshift-version` older than this CLI, with no pin at all, or with pins that are all newer than this CLI, it prints a warning naming the workflow and job plus the fix — the exact `evalshift-version: "<this version>"` line to set, or, for a newer pin, `pip install -U evalshift` locally. Advisory only — sync never edits a workflow and the exit code is unchanged. See [Pin drift](#pin-drift).
@@ -658,7 +658,7 @@ Rules and limitations:
 
 - `history` may contain at most one `system` message, and it must be first. `history: null` means single-turn; `history: []` is a conversational example with no prefix.
 - Re-driving a whole conversation (letting the candidate generate turn 1 and feeding *its own* reply into turn 2) is deliberately unsupported — it breaks the paired-comparison contract.
-- Captures recorded with the SDK's messages-list convention recover `history` **verbatim**; captures that only recorded a bare string get history **reconstructed** from sibling turn captures (an approximation: assistant replies come from final outputs, intermediate tool exchanges are absent). Prefer the messages-list convention in your instrumentation.
+- Captures recorded with the SDK's messages-list convention recover `history` from that list, translated from the provider's own message shape (Claude blocks, OpenAI Chat and Responses, LangChain roles) into the replay shape with text and tool calls kept; thinking is dropped, an OpenAI `developer` message merges into the leading system prompt (a system or `developer` message after the first turn is dropped with a warning — a replay carries one leading system prompt), and a turn carrying an image or other non-text content is skipped. Captures that only recorded a bare string get history **reconstructed** from sibling turn captures (an approximation: assistant replies come from final outputs, intermediate tool exchanges are absent). Prefer the messages-list convention in your instrumentation. Cases an earlier sync promoted with stringified message blocks are flagged by `capture sync` and `validate`; `capture sync --repair` rebuilds them, along with cases an earlier sync left with empty `inputs` (those get no warning). It takes their `inputs` and `history` from the capture, so it replaces hand edits to those two fields (a redaction, say) — review the diff before committing; it leaves a turn as it is, naming the missing turns, when its history was reconstructed from sibling turns and one of those turns' capture is gone. A plain sync rebuilds a stringified case offline to recognise a capture that duplicates it, but a case left with empty `inputs` can't be rebuilt without its capture: on such suites run `--repair` before a plain sync, or a duplicate capture may be promoted (the duplicate warning after `--repair` names it) — see [Provider message shapes](docs/conversations.md#provider-message-shapes).
 - History carries the **agent loop**: an `assistant` turn may have `tool_calls` (`{id, name, arguments}`), and a `tool` message carries that call's result keyed by `tool_call_id` (required — an unpairable result is a load-time error). They are dispatched in the OpenAI wire shape and translated per provider by LiteLLM, so the candidate model sees the tool results production saw. A promoted `tool` message with no recorded id is keyed, in order, to the preceding assistant turn's still-unanswered call (how id-less providers such as Gemini pair them), with a warning; only a result with no call to answer gets a positional `_pos<N>` id.
 - The report shows a `turn N` badge on conversational examples.
 
@@ -910,7 +910,7 @@ All `run` flags, plus `--gate` · `--policy-gate` · `--open` · `--push` · `--
 **`evalshift capture list [suite]`** — `--json` · `-c/--config` · `--offline`
 **`evalshift capture fetch`** — `--suite` · `--since <30m|24h|7d|ISO>` · `-c/--config` (needs `captures.store`)
 **`evalshift capture promote <capture-id>`** — `--as <case-id>` · `--suite` · `--input-var` (default `input`) · `--tag` (repeatable) · `--strict-args` · `--names-only` · `--tool-count` · `--rounds {first,all}` (default `first`) · `--allow-errored` · `-f/--force`
-**`evalshift capture sync`** — `--suite` · `--input-var` · `--tag` · `--strict-args` · `--names-only` · `--tool-count` · `--rounds {first,all}` · `--allow-errored` · `-c/--config` · `-f/--force` · `--write/--print` (default write) · `--keep-duplicates` · `--since` · `--offline`
+**`evalshift capture sync`** — `--suite` · `--input-var` · `--tag` · `--strict-args` · `--names-only` · `--tool-count` · `--rounds {first,all}` · `--allow-errored` · `-c/--config` · `-f/--force` · `--write/--print` (default write) · `--keep-duplicates` · `--since` · `--offline` · `--repair`
 **`evalshift capture clean [suite]`** — `--promoted` (default) · `--all` · `-y/--yes`
 **`evalshift capture diff <cap-a> <cap-b>`**
 
@@ -928,7 +928,7 @@ All `run` flags, plus `--gate` · `--policy-gate` · `--open` · `--push` · `--
 
 ### Hidden debug commands
 
-**`evalshift validate`** — load config + suite + prompts, cross-check compatibility. `-s/--suite` · `-c/--config`. After the success line, prints the [Pin drift](#pin-drift) warning if a workflow pins an older or newer CLI, or none at all (advisory; exit code unchanged, and a no-op in CI where the running CLI is the pin).
+**`evalshift validate`** — load config + suite + prompts, cross-check compatibility. `-s/--suite` · `-c/--config`. After the success line, prints the [Pin drift](#pin-drift) warning if a workflow pins an older or newer CLI, or none at all (advisory; exit code unchanged, and a no-op in CI where the running CLI is the pin). It also warns when the suite's examples carry message blocks an earlier `capture sync` stringified, pointing at `capture sync --repair` (advisory; see [Provider message shapes](docs/conversations.md#provider-message-shapes)).
 **`evalshift test-call`** — one live smoke-test call. `-m/--model` (required) · `-p/--prompt` · `-t/--temperature` (0–2, default 0) · `--max-tokens` (1–8192, default 256) · `--tools <file>` (prints a ToolTrace)
 
 ---

docs/configuration.md

@@ -1097,8 +1097,9 @@ key or connection string); the error names at most its scheme. Under the prefix
 
 What the fetch does: it lists the bucket, downloads only objects missing locally (captures
 are immutable and toolsets content-addressed, so "exists" means "current"), skips captures
-already promoted into a suite (so `capture clean` never causes a re-download), and writes
-each file atomically. `--since 24h` (or `7d`, `30m`, an ISO date) limits a fetch to recently
+already promoted into a suite (so `capture clean` never causes a re-download;
+`capture sync --repair` alone re-fetches the promoted captures its rebuilds need, past `--since`
+too), and writes each file atomically. `--since 24h` (or `7d`, `30m`, an ISO date) limits a fetch to recently
 written captures; `--offline` skips it and works with what is already local. Retention in the bucket
 is yours to set with a lifecycle rule; the SDK's `max_captures` does not apply there.
 
@@ -1154,7 +1155,12 @@ content duplicates an already-promoted case (or an earlier capture in the
 same run) are skipped too — duplicate examples inflate *n* and corrupt the
 paired statistics (`--keep-duplicates` opts out). The dedup set is seeded
 from the cases already in the suite dir, so re-syncing after recording more
-captures can't slip a duplicate past it. Useful flags: `--input-var`
+captures can't slip a duplicate past it. Captures whose messages carry an
+image or other non-text content are skipped too (see
+[Provider message shapes](conversations.md#provider-message-shapes)), as is
+a capture no valid case can be built from, and
+`--repair` rebuilds cases an earlier sync built from stringified message
+blocks. Useful flags: `--input-var`
 (default `input`), `--suite <name>` to filter to one suite, `--tag`,
 `--names-only`, `--tool-count`, `--strict-args`, `--rounds first|all`
 (`first`, the default, scores single-shot replay against round 1; `all`