Skip to content

Commit 7d8f1ee

Browse files
author
andrey.korchemkin
committed
feat: add mode recommendation and benchmark enhancements
Introduce mode recommendation for catalog compilation strategies, generate static benchmark charts, and enhance benchmarking with new variants and resumable runs. - Add `toolc recommend` command for suggesting optimal modes based on catalog signals - Add `toolc charts` command to render benchmark results as SVG visualizations - Introduce compiled_flat variant for single-turn compiled flows - Support resumable benchmark suites and improved model client with retries - Add MCP stdio backend for live tool execution in auto backend - Precompute runtime views in gateway for reduced latency - Update documentation and README with new architecture and examples - Add comprehensive test coverage for new features and backends
1 parent cb4ce10 commit 7d8f1ee

67 files changed

Lines changed: 6214 additions & 1080 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎BENCHMARK_REPORT.md‎

Lines changed: 77 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,51 +1,56 @@
11
# Benchmark Report
22

33
- Suite: `release`
4-
- Started: `2026-04-11 21:45:27 MSK`
5-
- Finished: `2026-04-11 21:48:36 MSK`
4+
- Started: `2026-04-11 22:23:09 MSK`
5+
- Finished: `2026-04-11 22:31:01 MSK`
66
- Environment: `windows/amd64`
77
- Model evaluation: enabled (`z-ai/glm-5.1` via `https://openrouter.ai/api/v1`)
88

99
Task success rate in this report means: correct tool selection + schema-valid arguments + local task oracle assertions. It is not a claim of end-to-end external API execution success.
1010

1111
## Scenario Summary
1212

13-
| Scenario | Layer | Tools | Namespaces | Compile | Index | Namespace | Schema | Invoke |
14-
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
15-
| synthetic-10 | synthetic | 10 | 10 | 510.600 us | 5.071 us | 0 | 0 | 5.065 us |
16-
| synthetic-50 | synthetic | 50 | 10 | 2.063 ms | 29.439 us | 0 | 0 | 0 |
17-
| synthetic-200 | synthetic | 200 | 10 | 7.174 ms | 85.571 us | 5.205 us | 0 | 0 |
18-
| synthetic-500 | synthetic | 500 | 10 | 15.453 ms | 156.847 us | 0 | 0 | 0 |
19-
| bfcl-live-simple-subset | real-world | 4 | 1 | 0 | 0 | 0 | 0 | 0 |
20-
| github-collaboration-subset | real-world | 14 | 2 | 8.247 ms | 0 | 0 | 125.106 us | 0 |
21-
| github-rest-full-metrics | real-world | 1112 | 44 | 131.878 ms | 287.213 us | 30.259 us | 0 | 0 |
13+
| Scenario | Layer | Load | Compile | Tools | Namespaces | Index | Namespace | Schema | Invoke |
14+
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
15+
| synthetic-10 | synthetic | 514.600 us | 0 | 10 | 10 | 0 | 0 | 0 | 0 |
16+
| synthetic-50 | synthetic | 0 | 519.500 us | 50 | 10 | 0 | 0 | 0 | 5.143 us |
17+
| synthetic-200 | synthetic | 514.200 us | 3.171 ms | 200 | 10 | 0 | 0 | 0 | 0 |
18+
| synthetic-500 | synthetic | 1.554 ms | 7.414 ms | 500 | 10 | 0 | 0 | 0 | 0 |
19+
| bfcl-live-simple-subset | real-world | 2.556 ms | 0 | 4 | 1 | 0 | 0 | 0 | 0 |
20+
| github-collaboration-subset | real-world | 164.571 ms | 4.804 ms | 14 | 2 | 0 | 0 | 0 | 0 |
21+
| github-rest-full-metrics | real-world | 147.333 ms | 82.702 ms | 1112 | 44 | 0 | 0 | 0 | 0 |
2222

2323
## synthetic-10
2424

2525
Small synthetic catalog with model evaluation enabled.
2626

27-
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | Useful selection |
28-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
29-
| raw | 10.0 | 910.0 | 0.00 | 0.00 | 1.00 | 1.241 s | 1.241 s |
30-
| baseline | 2.3 | 246.3 | 0.00 | 0.00 | 1.00 | 1.338 s | 1.338 s |
31-
| compiled | 10.0 | 604.0 | 0.33 | 0.33 | 0.67 | 2.575 s | 3.126 s |
27+
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
28+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
29+
| raw | 10.0 | 910.0 | 0.00 | 0.00 | 1.00 | 4.599 s | 6.356 s | 6.417 s | 4.599 s |
30+
| baseline | 2.3 | 246.3 | 0.00 | 0.00 | 1.00 | 5.719 s | 6.582 s | 10.060 s | 5.719 s |
31+
| compiled_flat | 10.0 | 474.0 | 0.00 | 0.00 | 1.00 | 2.880 s | 1.323 s | 6.428 s | 2.880 s |
32+
| compiled | 10.0 | 604.0 | 0.00 | 0.00 | 1.00 | 2.518 s | 2.590 s | 2.829 s | 2.518 s |
3233

3334

3435
## synthetic-50
3536

3637
Medium synthetic catalog for structural metrics only in the release run.
3738

38-
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | Useful selection |
39-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
40-
| raw | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
39+
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
40+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
41+
| raw | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
4142

4243
Skipped reason for `raw`: model evaluation disabled for scenario
4344

44-
| baseline | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
45+
| baseline | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
4546

4647
Skipped reason for `baseline`: model evaluation disabled for scenario
4748

48-
| compiled | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
49+
| compiled_flat | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
50+
51+
Skipped reason for `compiled_flat`: model evaluation disabled for scenario
52+
53+
| compiled | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
4954

5055
Skipped reason for `compiled`: model evaluation disabled for scenario
5156

@@ -55,17 +60,21 @@ Skipped reason for `compiled`: model evaluation disabled for scenario
5560

5661
Large synthetic catalog for structural metrics only in the release run.
5762

58-
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | Useful selection |
59-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
60-
| raw | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
63+
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
64+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
65+
| raw | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
6166

6267
Skipped reason for `raw`: model evaluation disabled for scenario
6368

64-
| baseline | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
69+
| baseline | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
6570

6671
Skipped reason for `baseline`: model evaluation disabled for scenario
6772

68-
| compiled | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
73+
| compiled_flat | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
74+
75+
Skipped reason for `compiled_flat`: model evaluation disabled for scenario
76+
77+
| compiled | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
6978

7079
Skipped reason for `compiled`: model evaluation disabled for scenario
7180

@@ -75,17 +84,21 @@ Skipped reason for `compiled`: model evaluation disabled for scenario
7584

7685
Very large synthetic catalog for structural metrics only in the release run.
7786

78-
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | Useful selection |
79-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
80-
| raw | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
87+
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
88+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
89+
| raw | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
8190

8291
Skipped reason for `raw`: model evaluation disabled for scenario
8392

84-
| baseline | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
93+
| baseline | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
8594

8695
Skipped reason for `baseline`: model evaluation disabled for scenario
8796

88-
| compiled | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
97+
| compiled_flat | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
98+
99+
Skipped reason for `compiled_flat`: model evaluation disabled for scenario
100+
101+
| compiled | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
89102

90103
Skipped reason for `compiled`: model evaluation disabled for scenario
91104

@@ -95,39 +108,57 @@ Skipped reason for `compiled`: model evaluation disabled for scenario
95108

96109
BFCL subset with model evaluation enabled for a bounded set of tasks.
97110

98-
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | Useful selection |
99-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
100-
| raw | 4.0 | 742.0 | 0.00 | 0.00 | 1.00 | 1.171 s | 1.171 s |
101-
| baseline | 1.8 | 362.8 | 0.00 | 0.00 | 1.00 | 864.549 ms | 864.549 ms |
102-
| compiled | 4.0 | 247.0 | 0.00 | 0.00 | 1.00 | 5.271 s | 5.271 s |
111+
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
112+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
113+
| raw | 4.0 | 742.0 | 0.00 | 0.00 | 1.00 | 1.331 s | 791.524 ms | 3.192 s | 1.331 s |
114+
| baseline | 1.8 | 362.8 | 0.00 | 0.00 | 1.00 | 818.479 ms | 762.101 ms | 998.179 ms | 818.479 ms |
115+
| compiled_flat | 4.0 | 282.0 | 0.00 | 0.00 | 1.00 | 1.012 s | 588.684 ms | 1.700 s | 1.012 s |
116+
| compiled | 4.0 | 247.0 | 0.00 | 0.00 | 1.00 | 3.163 s | 1.318 s | 7.142 s | 3.163 s |
103117

104118

105119
## github-collaboration-subset
106120

107121
Official GitHub REST API subset with model evaluation enabled for a bounded set of tasks.
108122

109-
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | Useful selection |
110-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
111-
| raw | 14.0 | 6322.0 | 0.00 | 0.25 | 0.75 | 1.064 s | 780.402 ms |
112-
| baseline | 5.0 | 2576.8 | 0.00 | 0.25 | 0.75 | 1.217 s | 1.515 s |
113-
| compiled | 14.0 | 956.0 | 0.00 | 0.25 | 0.75 | 1.791 s | 1.797 s |
123+
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
124+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
125+
| raw | 14.0 | 6322.0 | 0.00 | 0.25 | 0.75 | 1.996 s | 1.891 s | 2.286 s | 1.966 s |
126+
127+
Error counts for `raw`: invalid_arguments=1
128+
129+
| baseline | 5.0 | 2576.8 | 0.00 | 0.25 | 0.75 | 1.358 s | 1.007 s | 1.781 s | 1.475 s |
130+
131+
Error counts for `baseline`: invalid_arguments=1
132+
133+
| compiled_flat | 14.0 | 1084.0 | 0.00 | 0.25 | 0.75 | 2.260 s | 1.086 s | 4.140 s | 2.840 s |
134+
135+
Error counts for `compiled_flat`: invalid_arguments=1
136+
137+
| compiled | 14.0 | 956.0 | 0.00 | 0.25 | 0.75 | 2.063 s | 1.738 s | 2.781 s | 2.004 s |
138+
139+
Error counts for `compiled`: invalid_arguments=1
140+
114141

115142

116143
## github-rest-full-metrics
117144

118145
Full official GitHub REST API description for import/compile/gateway metrics only.
119146

120-
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | Useful selection |
121-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
122-
| raw | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
147+
| Variant | Avg visible tools | Avg token proxy | Wrong-tool rate | Invalid-args rate | Task success rate | Avg latency | P50 | P95 | Useful selection |
148+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
149+
| raw | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
123150

124151
Skipped reason for `raw`: scenario has no tasks
125152

126-
| baseline | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
153+
| baseline | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
127154

128155
Skipped reason for `baseline`: scenario has no tasks
129156

130-
| compiled | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 |
157+
| compiled_flat | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
158+
159+
Skipped reason for `compiled_flat`: scenario has no tasks
160+
161+
| compiled | 0.0 | 0.0 | 0.00 | 0.00 | 0.00 | 0 | 0 | 0 | 0 |
131162

132163
Skipped reason for `compiled`: scenario has no tasks
133164

0 commit comments

Comments
 (0)