Skip to content

Commit f886c03

Browse files
krisztian-gajdarKrisztian Gajdar
andauthored
feat(examples): run the OpenCode example on one GPU with Qwen3.8-27B-FP8 (#756)
* feat(examples): run the OpenCode example on one GPU with Qwen3.8-27B-FP8 * docs(examples): record the one-GPU OpenCode runs on Qwen3.8-27B-FP8 * docs(examples): record the one-GPU and cluster-route runs, and ship presence_penalty 0 * fix(examples): pass the long call only for the asked-for file, and time decode over what was streamed * docs(examples): say how hidden reasoning affects the decode rate * docs(examples): rewrap the session load docstring * docs(examples): record conformance and session load at the final commit * docs(examples): cite the server commits on main --------- Co-authored-by: Krisztian Gajdar <krisztian@superlinked.com>
1 parent d183404 commit f886c03

11 files changed

Lines changed: 1250 additions & 38 deletions

File tree

‎examples/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ to a separately maintained repository (clone and run there).
2020
| [Extract custom entities from primary sources](./named-entity-extraction) | Changing zero-shot labels across financial, healthcare, rail-safety and legal text, with the two labels that came back empty left visible | `extract` | SIE endpoint with GLiNER multi v2.1; `fetch.py` pulls the recorded run, `score.py` reproduces the counts with no key | Runnable recorded example |
2121
| [Extract the entity types you name, with no model to train](./custom-entity-types) | Asking for the 85 entity types of seven zero-shot NER sets and scoring exact spans against GPT-6 Luna, GPT-6 Sol, Claude Haiku 4.5, SIE Qwen3.8 27B, AWS Comprehend and spaCy, with a paired bootstrap and the price of each | `extract` | SIE server with GLiNER BioMed Large; `fetch.py` pulls the recorded run, `score.py` reproduces the figures with no key | Runnable recorded example |
2222
| [Answer a question from a passage, and quote the sentence you used](./chat) | Checking a generated answer for its citation, its length and whether it declines a question the passage cannot answer | `chat/completions` | SIE Cloud with Qwen3.8 27B; `fetch.py` pulls the recorded run, `score.py` reproduces the numbers with no key | Runnable recorded example |
23-
| [Run a coding agent on SIE with OpenCode](./coding-agent-opencode) | Pointing OpenCode at a laptop server, a self-hosted cluster or SIE Cloud, and checking the agent loop the way OpenCode drives it: streaming tool-call deltas, parallel calls, tool-result round trips, a 64K-plus prompt, finish reasons, usage and the reasoning field | `chat/completions` | An SIE Cloud endpoint that lists GLM-5.3-Flash via SIE Relay, or a local server; the laptop recipe was run on Apple Silicon with Qwen3.5-4B; `conformance.py` reports every case, `run_gate.sh` makes OpenCode fix a seeded failing test | Runnable check, laptop and SIE Cloud results recorded |
23+
| [Run a coding agent on SIE with OpenCode](./coding-agent-opencode) | Pointing OpenCode at a laptop server, one 80 GB GPU, a self-hosted cluster or SIE Cloud, and checking the agent loop the way OpenCode drives it: streaming tool-call deltas, parallel calls, tool-result round trips, a 64K-plus prompt, a 64 KiB tool call, finish reasons, usage and the reasoning field; `session_load.py` measures how many agent sessions one endpoint holds | `chat/completions` | An SIE Cloud endpoint that lists GLM-5.3-Flash via SIE Relay, or a local server; the laptop recipe was run on Apple Silicon with Qwen3.5-4B, the GPU and cluster recipes on one H100 with Qwen3.8-27B-FP8; `conformance.py` reports every case, `run_gate.sh` makes OpenCode fix a seeded failing test | Runnable check, laptop, one-GPU and SIE Cloud results recorded |
2424
| [Route requests while skipping most generation calls](./request-routing) | Assigning support, banking and assistant requests to routes with an embedding classifier and conditional generation | `encode`, `chat/completions` | SIE server with Qwen3 Embedding 4B and Qwen3.8 27B; recorded 3,000-case verification needs no API key | Runnable recorded example |
2525
| [Send each request to a cheap or a strong LLM](./request-routing/cost) | Cutting LLM spend with a bge-m3 nearest-neighbour router that picks GPT-6 Luna or GPT-6.1 Sol per request: 38.6% less than always using GPT-6.1 Sol at 95.8% of its quality on 728 held-out requests of a pre-registered study | `encode` | SIE Serverless with bge-m3; `fetch.py` pulls the recorded study, `score.py` reproduces the held-out table with no key | Runnable recorded example |
2626
| [Turn PDF pages into Markdown, scored on olmOCR-Bench](./document-to-markdown-olmocr) | Converting all 1,403 pages of Ai2's olmOCR-Bench with LightOnOCR-2-1B and scoring them with Ai2's own scorer, beside published Azure Document Intelligence and AWS Textract runs | `extract` | SIE Cloud with LightOnOCR-2-1B; `fetch.py` pulls the recorded run, `score.py` reproduces the scores with no key | Runnable recorded example |

‎examples/coding-agent-opencode/README.md‎

Lines changed: 395 additions & 30 deletions
Large diffs are not rendered by default.

‎examples/coding-agent-opencode/conformance.py‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,18 @@
6262
NEEDLE = "The access code is 7741-ZEBRA."
6363
RECORDED_HEADERS = ("x-sie-served-by", "x-sie-model-revision")
6464
REASONING_QUESTION = "Which is larger, 9.11 or 9.9? Answer with just the larger number."
65+
LONG_CALL_MIN_BYTES = 65_536
66+
LONG_CALL_BUDGET = 32_000
67+
LONG_CALL_LINES = 1_320
68+
LONG_CALL_PATH = "notes/pangrams.txt"
69+
LONG_CALL_SENTENCE = "The quick brown fox jumps over the lazy dog."
70+
LONG_CALL_PROMPT = (
71+
f"Use the write_file tool once to create the file {LONG_CALL_PATH}. Its content must be {LONG_CALL_LINES:,} "
72+
f"lines numbered 0001 to {LONG_CALL_LINES:04d}: each line is its four-digit number, one space and the sentence "
73+
f'"{LONG_CALL_SENTENCE}", about {LONG_CALL_LINES * (len(LONG_CALL_SENTENCE) + 6):,} characters in all. Write '
74+
"every line out in full in that one call: no ellipsis, no placeholder, no summary and no code that generates "
75+
"the lines."
76+
)
6577
MESSAGES_HEADERS = {
6678
"anthropic-version": "2023-06-01",
6779
"anthropic-beta": "claude-code-20250219,interleaved-thinking-2025-05-14",
@@ -86,6 +98,22 @@
8698
},
8799
}
88100

101+
WRITE_TOOL = {
102+
"type": "function",
103+
"function": {
104+
"name": "write_file",
105+
"description": "Create or replace one text file.",
106+
"parameters": {
107+
"type": "object",
108+
"properties": {
109+
"path": {"type": "string", "description": "File path relative to the project root"},
110+
"content": {"type": "string", "description": "The whole file content"},
111+
},
112+
"required": ["path", "content"],
113+
},
114+
},
115+
}
116+
89117
FILLER_WORDS = [
90118
"the",
91119
"ledger",
@@ -156,6 +184,9 @@ class StreamResult:
156184
error: dict[str, Any] | None = None
157185
retries: list[dict[str, Any]] = field(default_factory=list)
158186
seconds: float = 0.0
187+
first_chunk_s: float | None = None
188+
stream_s: float = 0.0
189+
longest_gap_s: float = 0.0
159190

160191
def calls(self) -> list[ToolCall]:
161192
return [self.tool_calls[index] for index in sorted(self.tool_calls)]
@@ -305,6 +336,7 @@ def complete_chat(client: openai.OpenAI, model: str, messages: list[dict[str, An
305336

306337
def _stream_once(client: openai.OpenAI, model: str, messages: list[dict[str, Any]], **params: Any) -> StreamResult:
307338
result = StreamResult()
339+
started = previous = time.monotonic()
308340
try:
309341
raw = client.chat.completions.with_raw_response.create(
310342
model=model,
@@ -315,6 +347,13 @@ def _stream_once(client: openai.OpenAI, model: str, messages: list[dict[str, Any
315347
)
316348
_keep_headers(result, raw)
317349
for chunk in raw.parse():
350+
now = time.monotonic()
351+
if result.first_chunk_s is None:
352+
result.first_chunk_s = now - started
353+
else:
354+
result.longest_gap_s = max(result.longest_gap_s, now - previous)
355+
result.stream_s = now - started - result.first_chunk_s
356+
previous = now
318357
result.chunks += 1
319358
result.usage_on_last_chunk = False
320359
usage = getattr(chunk, "usage", None)
@@ -372,6 +411,10 @@ def _complete_once(client: openai.OpenAI, model: str, messages: list[dict[str, A
372411
return result
373412

374413

414+
def _describe_names(calls: list[ToolCall]) -> str:
415+
return ", ".join(str(call.name) for call in calls)
416+
417+
375418
def _describe_calls(result: StreamResult) -> str:
376419
return ", ".join(f"{call.name}({call.arguments})" for call in result.calls()) or "no tool call"
377420

@@ -741,6 +784,76 @@ def case_tool_choice_required(client: openai.OpenAI, model: str) -> Case:
741784
)
742785

743786

787+
def long_call_line(number: int) -> str:
788+
return f"{number:04d} {LONG_CALL_SENTENCE}"
789+
790+
791+
def case_long_tool_call(client: openai.OpenAI, model: str, tool_params: dict[str, Any]) -> Case:
792+
name = "long_tool_call"
793+
messages = [{"role": "user", "content": LONG_CALL_PROMPT}]
794+
result = stream_chat(
795+
client, model, messages, tools=[WRITE_TOOL], max_tokens=LONG_CALL_BUDGET, temperature=0, **tool_params
796+
)
797+
calls = result.calls()
798+
size = sum(len(call.arguments.encode()) for call in calls)
799+
observed = result.observed()
800+
observed.update(
801+
max_tokens=LONG_CALL_BUDGET,
802+
argument_bytes=size,
803+
first_chunk_s=None if result.first_chunk_s is None else round(result.first_chunk_s, 2),
804+
stream_s=round(result.stream_s, 2),
805+
longest_gap_s=round(result.longest_gap_s, 2),
806+
)
807+
timing = f"streamed over {result.stream_s:.0f} s, longest gap between chunks {result.longest_gap_s:.1f} s"
808+
if result.error and _refuses_field(result.error, "max_tokens"):
809+
error = result.error
810+
detail = (
811+
f"not run: max_tokens={LONG_CALL_BUDGET} refused: {error['status']} {error['code']}: {error['message']}"
812+
)
813+
return Case(name, "recorded", detail, observed)
814+
failed = _fail_on_error(name, result)
815+
if failed:
816+
failed.observed = observed
817+
failed.detail = f"{failed.detail} ({timing})"
818+
return failed
819+
if not calls:
820+
detail = f"the model answered without a tool call: {result.content.strip()[:80]!r}"
821+
return Case(name, "recorded", detail, observed)
822+
arguments = [call.parsed_arguments() for call in calls]
823+
if result.finish_reason == "length":
824+
detail = f"max_tokens={LONG_CALL_BUDGET} ended the turn after {size:,} bytes of arguments ({timing})"
825+
return Case(name, "recorded", detail, observed)
826+
if None in arguments:
827+
detail = f"{size:,} bytes of arguments, but not a JSON object: truncated or corrupted ({timing})"
828+
return Case(name, "fail", detail, observed)
829+
if result.finish_reason != "tool_calls":
830+
detail = f"finish_reason is {result.finish_reason!r} after a complete call, expected 'tool_calls'"
831+
return Case(name, "fail", detail, observed)
832+
content = next((args.get("content") for args in arguments if args and isinstance(args.get("content"), str)), "")
833+
observed["content_lines"] = len(content.splitlines())
834+
observed["lines_as_asked"] = sum(
835+
line.strip() == long_call_line(number) for number, line in enumerate(content.splitlines(), 1)
836+
)
837+
kib = size / 1024
838+
if size < LONG_CALL_MIN_BYTES:
839+
detail = f"the model wrote {kib:.1f} KiB of arguments, under the {LONG_CALL_MIN_BYTES // 1024} KiB asked for"
840+
return Case(name, "recorded", detail, observed)
841+
paths = [args.get("path") for args in arguments if args]
842+
as_asked = (
843+
[call.name for call in calls] == ["write_file"]
844+
and paths == [LONG_CALL_PATH]
845+
and observed["lines_as_asked"] == LONG_CALL_LINES == observed["content_lines"]
846+
)
847+
if not as_asked:
848+
detail = (
849+
f"{kib:.1f} KiB arrived as JSON, but not the file asked for: {_describe_names(calls)} to {paths}, "
850+
f"{observed['lines_as_asked']} of {observed['content_lines']} lines as asked, {LONG_CALL_LINES} wanted"
851+
)
852+
return Case(name, "recorded", detail, observed)
853+
detail = f"{kib:.1f} KiB of arguments in {len(calls)} call ({observed['content_lines']} lines), {timing}"
854+
return Case(name, "pass", detail, observed)
855+
856+
744857
def model_entry(base_url: str, api_key: str, model: str) -> dict[str, Any] | None:
745858
headers = {"authorization": f"Bearer {api_key}"} if api_key else {}
746859
try:
@@ -816,6 +929,7 @@ def run(base_url: str, api_key: str, model: str) -> dict[str, Any]:
816929
cases.append(case_usage_on_final_chunk(stop_result))
817930
cases.append(reasoning_case)
818931
cases.append(case_tool_choice_required(client, model))
932+
cases.append(case_long_tool_call(client, model, tool_params))
819933

820934
statuses = ("pass", "fail", "recorded", "capacity")
821935
counts = {status: sum(1 for case in cases if case.status == status) for status in statuses}

‎examples/coding-agent-opencode/recipes/cluster/opencode.json‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"$schema": "https://opencode.ai/config.json",
3-
"model": "sie-cluster/zai-org/GLM-5.3-Flash",
3+
"model": "sie-cluster/Qwen/Qwen3.8-27B-FP8:h100-256k-batch",
44
"provider": {
55
"sie-cluster": {
66
"npm": "@ai-sdk/openai-compatible",
@@ -28,6 +28,17 @@
2828
"output": 4096
2929
},
3030
"tool_call": true
31+
},
32+
"Qwen/Qwen3.8-27B-FP8:h100-256k-batch": {
33+
"name": "Qwen3.8-27B on the cluster, 256K window",
34+
"limit": {
35+
"context": 262144,
36+
"output": 32768
37+
},
38+
"tool_call": true,
39+
"options": {
40+
"presence_penalty": 0
41+
}
3142
}
3243
}
3344
}
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{
2+
"$schema": "https://opencode.ai/config.json",
3+
"model": "sie-gpu/Qwen/Qwen3.8-27B-FP8:h100-256k-batch",
4+
"provider": {
5+
"sie-gpu": {
6+
"npm": "@ai-sdk/openai-compatible",
7+
"name": "SIE on one GPU",
8+
"options": {
9+
"baseURL": "http://localhost:8080/v1",
10+
"apiKey": "local",
11+
"timeout": 1800000,
12+
"headerTimeout": 600000,
13+
"chunkTimeout": 600000
14+
},
15+
"models": {
16+
"Qwen/Qwen3.8-27B-FP8:h100-256k-batch": {
17+
"name": "Qwen3.8-27B, 256K window, one 80 GB GPU",
18+
"limit": {
19+
"context": 262144,
20+
"output": 32768
21+
},
22+
"tool_call": true,
23+
"options": {
24+
"presence_penalty": 0
25+
}
26+
}
27+
}
28+
}
29+
}
30+
}

‎examples/coding-agent-opencode/run_gate.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
1717
RECIPE="${1:-cloud}"
1818
CONFIG="$HERE/recipes/$RECIPE/opencode.json"
1919
if [[ ! -f "$CONFIG" ]]; then
20-
echo "unknown recipe '$RECIPE'; pick laptop, cluster or cloud" >&2
20+
echo "unknown recipe '$RECIPE'; pick laptop, gpu, cluster or cloud" >&2
2121
exit 2
2222
fi
2323
for tool in opencode uv; do
@@ -34,7 +34,7 @@ for var in GATE_STEPS GATE_DEADLINE_SECONDS; do
3434
exit 2
3535
fi
3636
done
37-
if [[ "$RECIPE" != "laptop" ]]; then
37+
if grep -q '{env:SIE_' "$CONFIG"; then
3838
for var in SIE_BASE_URL SIE_API_KEY; do
3939
if [[ -z "${!var:-}" ]]; then
4040
echo "$var is not set" >&2

0 commit comments

Comments
 (0)