Skip to content

Commit 6ddfa8a

Browse files
authored
feat: replay a recorded response, starting with the Responses API (#12)
A scenario can now come from a recorded provider stream in kartsan03/wiretruth's wiretruth.fixture/v1 format, not only from code. The first is openai/responses-stream-missing-finish-abort, vendored unchanged with its derived_from block and the MIT notice: eight Responses API events, then the connection drops. It is also the first Responses API scenario, so the app calls client.responses.create and the reference instrumentation covers it. The fake provider never dropped a connection when drop_after equalled the event count: the loop ended first and sent a clean end of stream. A recorded abort cuts after its last frame, so the drop now happens after the loop. Fixes #11
1 parent db7c544 commit 6ddfa8a

15 files changed

Lines changed: 354 additions & 19 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,18 @@ All notable changes to this project are documented here. The format follows
66

77
## [Unreleased]
88

9+
### Added
10+
11+
- A recorded response can be replayed as a scenario, from a fixture in kartsan03/wiretruth's
12+
`wiretruth.fixture/v1` format. The first is `openai/responses-stream-missing-finish-abort`, the
13+
first OpenAI Responses API scenario: eight recorded events, then the connection drops.
14+
- The reference instrumentation covers the Responses API, so both controls grade it.
15+
16+
### Fixed
17+
18+
- The fake provider now drops the connection when `drop_after` equals the number of events, where
19+
it used to end the stream cleanly.
20+
921
## [0.2.0] - 2026-10-12
1022

1123
### Changed

‎README.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -127,17 +127,20 @@ endpoint per library, at `https://plexito-de.github.io/spantruth/badges/<subject
127127

128128
## Coverage
129129

130-
- **Providers:** OpenAI Chat Completions and Anthropic Messages, sync and async.
131-
- **Scenarios (31):** plain call, stream, stream without usage, prompt-cache hit (plain, streamed,
130+
- **Providers:** OpenAI Chat Completions and Anthropic Messages, sync and async. The OpenAI
131+
Responses API has its first scenario, a stream replayed from a recording.
132+
- **Scenarios (32):** plain call, stream, stream without usage, prompt-cache hit (plain, streamed,
132133
and with cumulative counts in `message_delta`), parallel tool calls, reasoning tokens, HTTP 429,
133134
a stream the app leaves early (by `close()` and by a `with` block), a dropped connection, and
134-
Anthropic's mid-stream `error` event. Streams and failures run sync and async.
135+
Anthropic's mid-stream `error` event. Streams and failures run sync and async. A Responses API
136+
stream cut after eight recorded events replays kartsan03/wiretruth's
137+
`stream.missing-finish-abort` fixture, vendored unchanged with its provenance.
135138
- **Subjects:** the [OpenTelemetry GenAI instrumentation](https://github.com/open-telemetry/opentelemetry-python-genai),
136139
[OpenLLMetry](https://github.com/traceloop/openllmetry) and
137140
[OpenInference](https://github.com/Arize-ai/openinference), plus the two controls.
138141

139-
Next: Logfire, OpenLIT, the JavaScript libraries, AWS Bedrock, Gemini and the OpenAI Responses
140-
API. [Help is welcome](CONTRIBUTING.md#ways-to-help).
142+
Next: Logfire, OpenLIT, the JavaScript libraries, AWS Bedrock, Gemini and the rest of the OpenAI
143+
Responses API. [Help is welcome](CONTRIBUTING.md#ways-to-help).
141144

142145
## Related work
143146

‎src/spantruth/calls.py‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,24 @@ def _openai_chunk(c: Any, seen: Seen) -> None:
8181
seen.usage = c.usage.model_dump()
8282

8383

84+
def _responses_event(e: Any, seen: Seen) -> None:
85+
seen.events += 1
86+
if e.type == "response.created":
87+
seen.response_id = e.response.id
88+
elif e.type == "response.output_text.delta":
89+
seen.text += e.delta
90+
elif e.type == "response.completed" and e.response.usage:
91+
seen.usage = e.response.usage.model_dump()
92+
93+
94+
def _openai_create(client: Any, sc: Scenario):
95+
"""The call for the scenario's API, and how the app reads a stream from it."""
96+
if sc.api == "responses":
97+
kw: dict = {"model": sc.request_model, "input": PROMPT, "stream": sc.stream}
98+
return client.responses.create(**kw), None, _responses_event
99+
return client.chat.completions.create(**_openai_kwargs(sc)), _openai_result, _openai_chunk
100+
101+
84102
def _anthropic_result(r: Any, seen: Seen) -> None:
85103
seen.text = "".join(b.text if b.type == "text" else b.name for b in r.content)
86104
seen.response_id, seen.usage = r.id, r.usage.model_dump()
@@ -103,8 +121,7 @@ def run(sc: Scenario, base_url: str, seen: Seen) -> None:
103121
import openai
104122

105123
client = openai.OpenAI(base_url=base_url, api_key="wt-test", max_retries=0, timeout=10)
106-
result = client.chat.completions.create(**_openai_kwargs(sc))
107-
on_result, on_event = _openai_result, _openai_chunk
124+
result, on_result, on_event = _openai_create(client, sc)
108125
else:
109126
import anthropic
110127

@@ -136,8 +153,8 @@ async def _run_async(sc: Scenario, base_url: str, seen: Seen) -> None:
136153
import openai
137154

138155
client = openai.AsyncOpenAI(base_url=base_url, api_key="wt-test", max_retries=0, timeout=10)
139-
result = await client.chat.completions.create(**_openai_kwargs(sc))
140-
on_result, on_event = _openai_result, _openai_chunk
156+
pending, on_result, on_event = _openai_create(client, sc)
157+
result = await pending
141158
else:
142159
import anthropic
143160

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Third-party fixtures
2+
3+
`stream.missing-finish-abort.json` is copied unchanged from
4+
[kartsan03/wiretruth](https://github.com/kartsan03/wiretruth) at commit `4b2a835e76d77304ebb4450239acd609b850aebc`,
5+
`suite/fixtures/openai-responses/stream.missing-finish-abort.json`. Its `derived_from` block names
6+
the recording it comes from and the transform that cut it. It is used under the MIT License below.
7+
8+
```text
9+
MIT License
10+
11+
Copyright (c) 2026 kartsan03
12+
13+
Permission is hereby granted, free of charge, to any person obtaining a copy
14+
of this software and associated documentation files (the "Software"), to deal
15+
in the Software without restriction, including without limitation the rights
16+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
17+
copies of the Software, and to permit persons to whom the Software is
18+
furnished to do so, subject to the following conditions:
19+
20+
The above copyright notice and this permission notice shall be included in all
21+
copies or substantial portions of the Software.
22+
23+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
24+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
25+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
26+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
27+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
28+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
29+
SOFTWARE.
30+
```
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
{
2+
"format": "wiretruth.fixture/v1",
3+
"id": "openai-responses/stream.missing-finish-abort",
4+
"scenario": "stream.missing-finish-abort",
5+
"provider": "openai-responses",
6+
"tier": "derived",
7+
"provenance": {
8+
"requested_model": "gpt-6-luna",
9+
"returned_model": "gpt-6-luna",
10+
"request": {
11+
"model": "gpt-6-luna",
12+
"input": [
13+
{
14+
"role": "user",
15+
"content": "Write a 200-word paragraph about how octopuses change colour to hide from predators."
16+
}
17+
],
18+
"stream": true,
19+
"max_output_tokens": 16,
20+
"reasoning": {
21+
"effort": "none"
22+
}
23+
}
24+
},
25+
"derived_from": {
26+
"fixture": "openai-responses/truncation.text-stream",
27+
"transform": "cut_stream_after",
28+
"args": {
29+
"n": 8,
30+
"end": "abort"
31+
},
32+
"rationale": "The connection closes before the stream has delivered every event.",
33+
"evidence_url": "https://github.com/openai/openai-python/issues/2065"
34+
},
35+
"exchanges": [
36+
{
37+
"match": {
38+
"method": "POST",
39+
"path": "/v1/responses"
40+
},
41+
"response": {
42+
"status": 200,
43+
"headers": {
44+
"content-type": "text/event-stream; charset=utf-8"
45+
},
46+
"body": {
47+
"sse": [
48+
"event: response.created\ndata: {\"type\":\"response.created\",\"response\":{\"id\":\"resp_025c23fff6653590006acbfc0086cc87d18124e70e8d3ff838\",\"object\":\"response\",\"created_at\":1791753216,\"status\":\"in_progress\",\"access_programs\":null,\"background\":false,\"completed_at\":null,\"error\":null,\"frequency_penalty\":0.0,\"incomplete_details\":null,\"instructions\":null,\"max_output_tokens\":16,\"max_tool_calls\":null,\"model\":\"gpt-6-luna\",\"moderation\":null,\"output\":[],\"parallel_tool_calls\":true,\"presence_penalty\":0.0,\"previous_response_id\":null,\"prompt_cache_key\":null,\"prompt_cache_retention\":\"24h\",\"reasoning\":{\"context\":\"all_turns\",\"effort\":\"none\",\"mode\":\"standard\",\"summary\":null},\"safety_identifier\":null,\"service_tier\":\"auto\",\"store\":true,\"temperature\":1.0,\"text\":{\"format\":{\"type\":\"text\"},\"verbosity\":\"medium\"},\"tool_choice\":\"auto\",\"tool_usage\":{\"image_gen\":{\"input_tokens\":0,\"input_tokens_details\":{\"image_tokens\":0,\"text_tokens\":0},\"output_tokens\":0,\"output_tokens_details\":{\"image_tokens\":0,\"text_tokens\":0},\"total_tokens\":0},\"web_search\":{\"num_requests\":0}},\"tools\":[],\"top_logprobs\":0,\"top_p\":0.98,\"truncation\":\"disabled\",\"usage\":null,\"user\":null,\"metadata\":{}},\"sequence_number\":0}\n\n",
49+
"event: response.in_progress\ndata: {\"type\":\"response.in_progress\",\"response\":{\"id\":\"resp_025c23fff6653590006acbfc0086cc87d18124e70e8d3ff838\",\"object\":\"response\",\"created_at\":1791753216,\"status\":\"in_progress\",\"access_programs\":null,\"background\":false,\"completed_at\":null,\"error\":null,\"frequency_penalty\":0.0,\"incomplete_details\":null,\"instructions\":null,\"max_output_tokens\":16,\"max_tool_calls\":null,\"model\":\"gpt-6-luna\",\"moderation\":null,\"output\":[],\"parallel_tool_calls\":true,\"presence_penalty\":0.0,\"previous_response_id\":null,\"prompt_cache_key\":null,\"prompt_cache_retention\":\"24h\",\"reasoning\":{\"context\":\"all_turns\",\"effort\":\"none\",\"mode\":\"standard\",\"summary\":null},\"safety_identifier\":null,\"service_tier\":\"auto\",\"store\":true,\"temperature\":1.0,\"text\":{\"format\":{\"type\":\"text\"},\"verbosity\":\"medium\"},\"tool_choice\":\"auto\",\"tool_usage\":{\"image_gen\":{\"input_tokens\":0,\"input_tokens_details\":{\"image_tokens\":0,\"text_tokens\":0},\"output_tokens\":0,\"output_tokens_details\":{\"image_tokens\":0,\"text_tokens\":0},\"total_tokens\":0},\"web_search\":{\"num_requests\":0}},\"tools\":[],\"top_logprobs\":0,\"top_p\":0.98,\"truncation\":\"disabled\",\"usage\":null,\"user\":null,\"metadata\":{}},\"sequence_number\":1}\n\n",
50+
"event: response.output_item.added\ndata: {\"type\":\"response.output_item.added\",\"item\":{\"id\":\"msg_025c23fff6653590006acbfc016d8087d18e46405588162435\",\"type\":\"message\",\"status\":\"in_progress\",\"content\":[],\"phase\":\"final_answer\",\"role\":\"assistant\"},\"output_index\":0,\"sequence_number\":2}\n\n",
51+
"event: response.content_part.added\ndata: {\"type\":\"response.content_part.added\",\"content_index\":0,\"item_id\":\"msg_025c23fff6653590006acbfc016d8087d18e46405588162435\",\"output_index\":0,\"part\":{\"type\":\"output_text\",\"annotations\":[],\"logprobs\":[],\"text\":\"\"},\"sequence_number\":3}\n\n",
52+
"event: response.output_text.delta\ndata: {\"type\":\"response.output_text.delta\",\"content_index\":0,\"delta\":\"Oct\",\"item_id\":\"msg_025c23fff6653590006acbfc016d8087d18e46405588162435\",\"logprobs\":[],\"obfuscation\":\"JJEbJBKhHOigL\",\"output_index\":0,\"sequence_number\":4}\n\n",
53+
"event: response.output_text.delta\ndata: {\"type\":\"response.output_text.delta\",\"content_index\":0,\"delta\":\"op\",\"item_id\":\"msg_025c23fff6653590006acbfc016d8087d18e46405588162435\",\"logprobs\":[],\"obfuscation\":\"rCPO5tGFW8ghsz\",\"output_index\":0,\"sequence_number\":5}\n\n",
54+
"event: response.output_text.delta\ndata: {\"type\":\"response.output_text.delta\",\"content_index\":0,\"delta\":\"uses\",\"item_id\":\"msg_025c23fff6653590006acbfc016d8087d18e46405588162435\",\"logprobs\":[],\"obfuscation\":\"byausMiYWM8M\",\"output_index\":0,\"sequence_number\":6}\n\n",
55+
"event: response.output_text.delta\ndata: {\"type\":\"response.output_text.delta\",\"content_index\":0,\"delta\":\" change\",\"item_id\":\"msg_025c23fff6653590006acbfc016d8087d18e46405588162435\",\"logprobs\":[],\"obfuscation\":\"lE22JEr0R\",\"output_index\":0,\"sequence_number\":7}\n\n"
56+
]
57+
},
58+
"end": "abort"
59+
}
60+
}
61+
],
62+
"truth": {
63+
"finish": "none",
64+
"raw_finish": null,
65+
"text": "Octopuses change",
66+
"tool_calls": [],
67+
"structured": null,
68+
"error": null,
69+
"stream_completed": false,
70+
"usage": {
71+
"input": null,
72+
"cached_input": null,
73+
"output": null,
74+
"reasoning": null,
75+
"total": null,
76+
"alternatives": {},
77+
"raw": null
78+
}
79+
}
80+
}

‎src/spantruth/mockserver.py‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -65,15 +65,15 @@ def _stream(self, sc: Scenario) -> None:
6565
self.end_headers()
6666
events = sc.response.events or ()
6767
drop = sc.response.drop_after
68-
for index, event in enumerate(events):
69-
if drop is not None and index == drop:
70-
self.wfile.flush()
71-
self.connection.shutdown(socket.SHUT_RDWR)
72-
self.close_connection = True
73-
return
68+
for event in events[:drop]:
7469
chunk = _sse(event)
7570
self.wfile.write(f"{len(chunk):x}\r\n".encode() + chunk + b"\r\n")
7671
self.wfile.flush()
72+
if drop is not None:
73+
# A drop after the last event too: a recorded abort ends where the recording stops.
74+
self.connection.shutdown(socket.SHUT_RDWR)
75+
self.close_connection = True
76+
return
7777
self.wfile.write(b"0\r\n\r\n")
7878
self.wfile.flush()
7979

‎src/spantruth/model.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ class Scenario:
8080
`with_block` makes the app read the stream inside a `with` block and leave it, instead of
8181
calling `close()`.
8282
`stream_usage` False leaves out OpenAI's `include_usage`, so the stream carries no usage.
83+
`api` is the OpenAI API the app calls: `chat` (Chat Completions) or `responses`.
8384
"""
8485

8586
id: str
@@ -93,6 +94,7 @@ class Scenario:
9394
tools: bool = False
9495
stream_usage: bool = True
9596
asynchronous: bool = False
97+
api: str = "chat"
9698
catches: str = ""
9799
tags: tuple[str, ...] = field(default_factory=tuple)
98100

‎src/spantruth/recorded.py‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
"""Replay a recorded provider response, in kartsan03/wiretruth's fixture format.
2+
3+
The scenarios in `scenarios.py` build their responses in code. A recorded fixture is the other
4+
source: a real provider stream, kept byte for byte, with the provenance of where it came from.
5+
Only what replay needs is read: one exchange, its status, its SSE frames, and how it ended.
6+
Format: https://github.com/kartsan03/wiretruth/blob/main/docs/formats.md
7+
"""
8+
9+
from __future__ import annotations
10+
11+
import json
12+
from pathlib import Path
13+
14+
from .model import Event, Response
15+
16+
FIXTURES = Path(__file__).resolve().parent / "fixtures"
17+
FORMAT = "wiretruth.fixture/v1"
18+
19+
20+
def frame(raw: str) -> Event:
21+
"""One recorded SSE frame, `event: <name>\\ndata: <json>`, as an Event."""
22+
name, data = None, []
23+
for line in raw.splitlines():
24+
if line.startswith("event:"):
25+
name = line[len("event:") :].strip()
26+
elif line.startswith("data:"):
27+
data.append(line[len("data:") :].lstrip())
28+
return Event(data="\n".join(data), event=name)
29+
30+
31+
def load(path: Path) -> Response:
32+
"""The response of a recorded fixture. An `abort` end drops the connection after the end."""
33+
doc = json.loads(Path(path).read_text())
34+
if doc.get("format") != FORMAT:
35+
raise ValueError(f"{path}: format {doc.get('format')!r}, expected {FORMAT!r}")
36+
if len(doc["exchanges"]) != 1:
37+
raise ValueError(f"{path}: replay needs exactly one exchange")
38+
response = doc["exchanges"][0]["response"]
39+
events = tuple(frame(raw) for raw in response["body"]["sse"])
40+
end = response.get("end", "complete")
41+
if end not in ("complete", "abort"):
42+
raise ValueError(f"{path}: unknown end {end!r}")
43+
return Response(
44+
status=response["status"],
45+
events=events,
46+
drop_after=len(events) if end == "abort" else None,
47+
)
48+
49+
50+
def requested_model(path: Path) -> str:
51+
return json.loads(Path(path).read_text())["provenance"]["requested_model"]

‎src/spantruth/reference.py‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,22 @@ def _openai_chunk(call: _Call, c: Any) -> None:
9696
_openai_usage(call, c.usage)
9797

9898

99+
def _responses_event(call: _Call, e: Any) -> None:
100+
if e.type in ("response.created", "response.completed"):
101+
call.set("gen_ai.response.model", e.response.model)
102+
call.set("gen_ai.response.id", e.response.id)
103+
if e.type == "response.completed" and e.response.usage:
104+
u = e.response.usage
105+
call.usage(
106+
u.input_tokens,
107+
u.output_tokens,
108+
read=u.input_tokens_details.cached_tokens or 0,
109+
reasoning=u.output_tokens_details.reasoning_tokens or 0,
110+
)
111+
elif e.type == "response.output_text.delta":
112+
call.text += e.delta
113+
114+
99115
def _anthropic_response(call: _Call, r: Any) -> None:
100116
call.set("gen_ai.response.model", r.model)
101117
call.set("gen_ai.response.id", r.id)
@@ -231,11 +247,23 @@ def instrument(tracer_provider, faults: set[str] = frozenset()) -> None:
231247
tracer = tracer_provider.get_tracer("spantruth.reference")
232248
import anthropic.resources.messages as am
233249
import openai.resources.chat.completions as oc
250+
import openai.resources.responses as orr
234251

235252
for cls, is_async in ((oc.Completions, False), (oc.AsyncCompletions, True)):
236253
cls.create = _wrap(
237254
tracer, "openai", set(faults), _openai_response, _openai_chunk, cls.create, is_async
238255
)
256+
for cls, is_async in ((orr.Responses, False), (orr.AsyncResponses, True)):
257+
# Only streams are exercised so far, so a plain response is not read.
258+
cls.create = _wrap(
259+
tracer,
260+
"openai",
261+
set(faults),
262+
lambda call, r: None,
263+
_responses_event,
264+
cls.create,
265+
is_async,
266+
)
239267
for cls, is_async in ((am.Messages, False), (am.AsyncMessages, True)):
240268
cls.create = _wrap(
241269
tracer,

‎src/spantruth/scenarios.py‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88

99
from dataclasses import replace
1010

11+
from . import recorded
1112
from . import wire_anthropic as an
1213
from . import wire_openai as oa
1314
from .model import Scenario, Truth
@@ -135,6 +136,24 @@ def sc(name: str, title: str, catches: str, response, **kw) -> Scenario:
135136
]
136137

137138

139+
def _recorded() -> list[Scenario]:
140+
"""Scenarios replayed from recorded provider streams, rather than built in code."""
141+
abort = recorded.FIXTURES / "openai-responses" / "stream.missing-finish-abort.json"
142+
return [
143+
Scenario(
144+
id="openai/responses-stream-missing-finish-abort",
145+
provider="openai",
146+
api="responses",
147+
title="Responses API stream cut after eight recorded events",
148+
catches="an aborted stream recorded as a success",
149+
request_model=recorded.requested_model(abort),
150+
response=recorded.load(abort),
151+
stream=True,
152+
tags=("recorded",),
153+
)
154+
]
155+
156+
138157
def _anthropic() -> list[Scenario]:
139158
def sc(name: str, title: str, catches: str, response, **kw) -> Scenario:
140159
return Scenario(
@@ -290,7 +309,7 @@ def _with_async(scenarios: list[Scenario]) -> list[Scenario]:
290309

291310

292311
SCENARIOS: dict[str, Scenario] = {
293-
s.id: s for s in _with_async(_with_block(_openai() + _anthropic()))
312+
s.id: s for s in _with_async(_with_block(_openai() + _anthropic())) + _recorded()
294313
}
295314

296315

@@ -306,5 +325,8 @@ def by_key(key: str) -> Scenario:
306325

307326

308327
def truth(scenario: Scenario) -> Truth:
309-
derive = oa.truth if scenario.provider == "openai" else an.truth
328+
if scenario.api == "responses":
329+
derive = oa.responses_truth
330+
else:
331+
derive = oa.truth if scenario.provider == "openai" else an.truth
310332
return derive(scenario.request_model, scenario.response, scenario.read_events)

0 commit comments

Comments
 (0)