Skip to content

Commit 71e17fa

Browse files
author
andrey.korchemkin
committed
perf(bench): update performance metrics and regenerate charts
- Regenerate benchmark results for all scenarios with latest timings, memory allocations, and task latencies - Enable model evaluation for synthetic-50 scenario and update its description - Adjust chart generation to horizontal bar layout and include tool counts in labels - Refine OpenAPI importer to streamline output schema building by removing unused document parameter - Update SVG assets for scale behavior and memory usage visualizations with new data points Closes #benchmark-refresh-2026-04-12
1 parent b0df717 commit 71e17fa

28 files changed

Lines changed: 865 additions & 536 deletions

‎BENCHMARK_REPORT.md‎

Lines changed: 39 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Benchmark Report
22

33
- Suite: `release`
4-
- Started: `2026-04-12 18:12:25 MSK`
5-
- Finished: `2026-04-12 18:21:17 MSK`
4+
- Started: `2026-04-12 19:02:27 MSK`
5+
- Finished: `2026-04-12 19:13:27 MSK`
66
- Environment: `windows/amd64`
77
- Model evaluation: enabled (`z-ai/glm-5.1` via `https://openrouter.ai/api/v1`)
88

@@ -12,50 +12,47 @@ Task success rate in this report means: correct tool selection + schema-valid ar
1212

1313
| Scenario | Layer | Load | Load alloc | Compile | Compile alloc | Tools | Namespaces | Index | Namespace | Schema | Invoke |
1414
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
15-
| synthetic-10 | synthetic | 0 | 57.27 KiB | 523.000 us | 106.31 KiB | 10 | 10 | 0 | 0 | 0 | 0 |
16-
| synthetic-50 | synthetic | 0 | 237.23 KiB | 516.100 us | 469.85 KiB | 50 | 10 | 0 | 0 | 0 | 0 |
17-
| synthetic-200 | synthetic | 522.500 us | 929.90 KiB | 2.993 ms | 1.84 MiB | 200 | 10 | 0 | 0 | 0 | 5.127 us |
18-
| synthetic-500 | synthetic | 1.551 ms | 2.09 MiB | 7.453 ms | 4.59 MiB | 500 | 10 | 0 | 0 | 0 | 0 |
19-
| bfcl-live-simple-subset | real-world | 3.552 ms | 1.33 MiB | 0 | 52.67 KiB | 4 | 1 | 0 | 0 | 0 | 0 |
20-
| github-collaboration-subset | real-world | 135.079 ms | 75.06 MiB | 0 | 348.46 KiB | 14 | 2 | 0 | 0 | 0 | 0 |
21-
| github-rest-full-metrics | real-world | 113.405 ms | 71.08 MiB | 29.781 ms | 21.09 MiB | 1112 | 44 | 0 | 0 | 0 | 0 |
22-
| stripe-rest-full-metrics | real-world | 78.706 ms | 48.84 MiB | 16.677 ms | 11.21 MiB | 587 | 294 | 0 | 0 | 0 | 0 |
23-
| twilio-rest-full-metrics | real-world | 18.657 ms | 11.92 MiB | 5.681 ms | 4.02 MiB | 197 | 75 | 0 | 0 | 0 | 0 |
24-
| openmeteo-runtime-public | execution | 0 | 137.67 KiB | 0 | 22.89 KiB | 1 | 1 | 0 | 0 | 0 | 0 |
15+
| synthetic-10 | synthetic | 519.400 us | 56.84 KiB | 0 | 106.30 KiB | 10 | 10 | 0 | 0 | 0 | 0 |
16+
| synthetic-50 | synthetic | 526.800 us | 232.17 KiB | 967.700 us | 469.85 KiB | 50 | 10 | 0 | 0 | 0 | 0 |
17+
| synthetic-200 | synthetic | 511.300 us | 919.65 KiB | 3.131 ms | 1.84 MiB | 200 | 10 | 0 | 0 | 0 | 0 |
18+
| synthetic-500 | synthetic | 2.066 ms | 2.07 MiB | 7.278 ms | 4.58 MiB | 500 | 10 | 0 | 0 | 0 | 0 |
19+
| bfcl-live-simple-subset | real-world | 3.090 ms | 1.32 MiB | 512.200 us | 52.67 KiB | 4 | 1 | 0 | 0 | 0 | 0 |
20+
| github-collaboration-subset | real-world | 104.329 ms | 75.17 MiB | 0 | 348.46 KiB | 14 | 2 | 0 | 0 | 0 | 0 |
21+
| github-rest-full-metrics | real-world | 119.118 ms | 71.12 MiB | 31.269 ms | 21.09 MiB | 1112 | 44 | 0 | 0 | 0 | 0 |
22+
| stripe-rest-full-metrics | real-world | 81.073 ms | 48.87 MiB | 15.341 ms | 11.21 MiB | 587 | 294 | 0 | 0 | 0 | 0 |
23+
| twilio-rest-full-metrics | real-world | 17.382 ms | 11.88 MiB | 5.448 ms | 4.02 MiB | 197 | 75 | 0 | 0 | 0 | 0 |
24+
| openmeteo-runtime-public | execution | 506.700 us | 137.67 KiB | 0 | 22.89 KiB | 1 | 1 | 0 | 0 | 0 | 0 |
2525

2626
## synthetic-10
2727

2828
Small synthetic catalog with model evaluation enabled.
2929

3030
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
3131
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
32-
| direct | 10.0 | 910.0 | 0.00 | 0.00 | 1.00 | 7.924 s | 7.872 s | 10.189 s | 7.924 s |
33-
| flat | 10.0 | 474.0 | 0.00 | 0.00 | 1.00 | 7.206 s | 784.858 ms | 20.522 s | 7.206 s |
34-
| staged | 10.0 | 604.0 | 0.00 | 0.00 | 1.00 | 9.647 s | 12.430 s | 13.705 s | 9.647 s |
32+
| direct | 10.0 | 910.0 | 0.00 | 0.00 | 1.00 | 5.356 s | 4.209 s | 10.411 s | 5.356 s |
33+
| flat | 10.0 | 474.0 | 0.00 | 0.00 | 1.00 | 6.116 s | 7.896 s | 8.566 s | 6.116 s |
34+
| staged | 10.0 | 604.0 | 0.00 | 0.00 | 1.00 | 3.878 s | 1.318 s | 9.293 s | 3.878 s |
3535

36-
Structural memory: load alloc `57.27 KiB`, compile alloc `106.31 KiB`, heap after compile `547.78 KiB`
36+
Structural memory: load alloc `56.84 KiB`, compile alloc `106.30 KiB`, heap after compile `542.21 KiB`
3737

3838

3939
## synthetic-50
4040

41-
Medium synthetic catalog for structural metrics only in the release run.
41+
Medium synthetic catalog with model evaluation enabled.
4242

4343
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
4444
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
45-
| direct | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
46-
47-
Skipped reason for `direct`: model evaluation disabled for scenario
48-
49-
| flat | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
45+
| direct | 50.0 | 4699.0 | 0.33 | 0.33 | 0.67 | 5.773 s | 2.664 s | 13.108 s | 2.106 s |
5046

51-
Skipped reason for `flat`: model evaluation disabled for scenario
47+
Error counts for `direct`: model_error=1
5248

53-
| staged | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
49+
| flat | 50.0 | 2492.0 | 0.33 | 0.33 | 0.67 | 2.811 s | 2.618 s | 4.113 s | 2.908 s |
5450

55-
Skipped reason for `staged`: model evaluation disabled for scenario
51+
Error counts for `flat`: model_error=1
5652

53+
| staged | 50.0 | 2413.0 | 0.00 | 0.00 | 0.67 | 11.925 s | 11.826 s | 13.327 s | 11.975 s |
5754

58-
Structural memory: load alloc `237.23 KiB`, compile alloc `469.85 KiB`, heap after compile `1.52 MiB`
55+
Structural memory: load alloc `232.17 KiB`, compile alloc `469.85 KiB`, heap after compile `1.52 MiB`
5956

6057

6158
## synthetic-200
@@ -77,7 +74,7 @@ Skipped reason for `flat`: model evaluation disabled for scenario
7774
Skipped reason for `staged`: model evaluation disabled for scenario
7875

7976

80-
Structural memory: load alloc `929.90 KiB`, compile alloc `1.84 MiB`, heap after compile `3.58 MiB`
77+
Structural memory: load alloc `919.65 KiB`, compile alloc `1.84 MiB`, heap after compile `2.23 MiB`
8178

8279

8380
## synthetic-500
@@ -99,7 +96,7 @@ Skipped reason for `flat`: model evaluation disabled for scenario
9996
Skipped reason for `staged`: model evaluation disabled for scenario
10097

10198

102-
Structural memory: load alloc `2.09 MiB`, compile alloc `4.59 MiB`, heap after compile `4.35 MiB`
99+
Structural memory: load alloc `2.07 MiB`, compile alloc `4.58 MiB`, heap after compile `5.29 MiB`
103100

104101

105102
## bfcl-live-simple-subset
@@ -108,11 +105,11 @@ BFCL subset with model evaluation enabled for a bounded set of tasks.
108105

109106
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
110107
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
111-
| direct | 4.0 | 743.0 | 0.00 | 0.00 | 1.00 | 885.463 ms | 394.971 ms | 1.704 s | 885.463 ms |
112-
| flat | 4.0 | 282.0 | 0.00 | 0.00 | 1.00 | 1.581 s | 878.059 ms | 3.026 s | 1.581 s |
113-
| staged | 4.0 | 247.0 | 0.00 | 0.00 | 1.00 | 12.590 s | 7.798 s | 20.122 s | 12.590 s |
108+
| direct | 4.0 | 743.0 | 0.00 | 0.00 | 1.00 | 2.453 s | 1.471 s | 6.018 s | 2.453 s |
109+
| flat | 4.0 | 282.0 | 0.00 | 0.00 | 1.00 | 2.550 s | 528.988 ms | 6.814 s | 2.550 s |
110+
| staged | 4.0 | 247.0 | 0.00 | 0.00 | 1.00 | 4.411 s | 1.384 s | 13.369 s | 4.411 s |
114111

115-
Structural memory: load alloc `1.33 MiB`, compile alloc `52.67 KiB`, heap after compile `2.23 MiB`
112+
Structural memory: load alloc `1.32 MiB`, compile alloc `52.67 KiB`, heap after compile `2.25 MiB`
116113

117114

118115
## github-collaboration-subset
@@ -121,11 +118,11 @@ Official GitHub REST API subset with model evaluation enabled for a bounded set
121118

122119
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
123120
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
124-
| direct | 14.0 | 6323.0 | 0.00 | 0.00 | 1.00 | 2.562 s | 988.665 ms | 6.999 s | 2.562 s |
125-
| flat | 14.0 | 993.0 | 0.00 | 0.00 | 1.00 | 2.022 s | 1.145 s | 3.607 s | 2.022 s |
126-
| staged | 14.0 | 956.0 | 0.00 | 0.00 | 1.00 | 4.815 s | 4.446 s | 7.131 s | 4.815 s |
121+
| direct | 14.0 | 6323.0 | 0.00 | 0.00 | 1.00 | 3.079 s | 3.613 s | 4.021 s | 3.079 s |
122+
| flat | 14.0 | 993.0 | 0.00 | 0.00 | 1.00 | 5.289 s | 2.973 s | 9.708 s | 5.289 s |
123+
| staged | 14.0 | 956.0 | 0.00 | 0.00 | 1.00 | 2.341 s | 1.281 s | 4.135 s | 2.341 s |
127124

128-
Structural memory: load alloc `75.06 MiB`, compile alloc `348.46 KiB`, heap after compile `36.95 MiB`
125+
Structural memory: load alloc `75.17 MiB`, compile alloc `348.46 KiB`, heap after compile `35.52 MiB`
129126

130127

131128
## github-rest-full-metrics
@@ -150,7 +147,7 @@ Skipped reason for `staged`: scenario has no tasks
150147
Notes:
151148
- scenario contains no task oracle; only structural metrics, compile time, and gateway overhead were measured
152149

153-
Structural memory: load alloc `71.08 MiB`, compile alloc `21.09 MiB`, heap after compile `18.52 MiB`
150+
Structural memory: load alloc `71.12 MiB`, compile alloc `21.09 MiB`, heap after compile `19.02 MiB`
154151

155152

156153
## stripe-rest-full-metrics
@@ -175,7 +172,7 @@ Skipped reason for `staged`: scenario has no tasks
175172
Notes:
176173
- scenario contains no task oracle; only structural metrics, compile time, and gateway overhead were measured
177174

178-
Structural memory: load alloc `48.84 MiB`, compile alloc `11.21 MiB`, heap after compile `10.03 MiB`
175+
Structural memory: load alloc `48.87 MiB`, compile alloc `11.21 MiB`, heap after compile `13.38 MiB`
179176

180177

181178
## twilio-rest-full-metrics
@@ -200,7 +197,7 @@ Skipped reason for `staged`: scenario has no tasks
200197
Notes:
201198
- scenario contains no task oracle; only structural metrics, compile time, and gateway overhead were measured
202199

203-
Structural memory: load alloc `11.92 MiB`, compile alloc `4.02 MiB`, heap after compile `6.52 MiB`
200+
Structural memory: load alloc `11.88 MiB`, compile alloc `4.02 MiB`, heap after compile `5.06 MiB`
204201

205202

206203
## openmeteo-runtime-public
@@ -225,12 +222,12 @@ Skipped reason for `staged`: scenario has no tasks
225222
Notes:
226223
- scenario contains no task oracle; only structural metrics, compile time, and gateway overhead were measured
227224

228-
Structural memory: load alloc `137.67 KiB`, compile alloc `22.89 KiB`, heap after compile `1.12 MiB`
225+
Structural memory: load alloc `137.67 KiB`, compile alloc `22.89 KiB`, heap after compile `1.14 MiB`
229226

230227
Runtime execution:
231228
- tasks: 1
232229
- success_rate: 1.00
233-
- avg_latency: 305.447 ms
234-
- p95_latency: 305.447 ms
230+
- avg_latency: 302.711 ms
231+
- p95_latency: 302.711 ms
235232

236233

0 commit comments

Comments
 (0)