Skip to content

Commit 6af893a

Browse files
authored
feat(bench): measure code execution's response-side saving (#1160)
* feat(bench): measure code execution's response-side saving, and fix the three gaps that blocked it Every savings figure in bench so far is about the MENU — how many tokens of tool definitions an agent carries. Code execution's menu is already the smallest of the three surfaces, but the menu is not where it pays: a script issues several upstream calls inside the sandbox and only the value it returns enters the model's context. The intermediate responses, which an agent driving those tools itself would have paid for in full, never appear. Nothing measured that. It could not be measured, for three separate reasons, each fixed here: 1. Sandbox sub-calls emitted hardcoded 0,0 byte lengths. In the activity log 0 means UNKNOWN, not free, so every sub-call was unaccountable — the gap bench already names ReasonSubCallZeroBytes. 2. Internal built-in calls never populated the byte fields at all, so the code_execution PARENT was unaccountable too. Fixed centrally in the emitter, which also closes ReasonInternalNoByteCounts for retrieve_tools and describe_tool. The value emitted is the full pre-truncation size, which is what response_bytes is defined to mean; the truncation flag travels beside it so a consumer can still exclude the row. 3. A PARAMETERLESS tool call records no arguments, so requestCost fell through to the byte estimate while its siblings were measured. Colliding bases meant the whole script was withheld — and most fleets have several parameterless tools. Found on real traffic, not in unit tests. Measured over 7 scripts and 63 sub-calls on a live 4-server fleet, the saving ranges from -14.9% (a script that returns its one response verbatim) to +97.0% (thirty file reads summarised to a count). The negative case is real and is reported as one: code execution's saving is a function of how much the sandbox discards, and a script that discards nothing loses tokens. Accounting rules, each pinned by a test that fails when the guard is removed: - never-sum: a measured cost is TOKENS and an estimated cost is BYTES; a call mixing them is withheld rather than added into a number in no unit. - no silent zeros: a withheld saving carries no figure at all, because a 0 reads as "saved nothing" when the truth is "nothing was measurable". - no double-counting: sub-calls live under their parent, never in Calls. - truncation withholds, and does not merely annotate. A truncated component's byte length is the full pre-truncation size, but baseline means what an agent would have paid, and that agent receives a cut response — so counting it overstates the baseline and INFLATES the saving. Caught by cross-model review; the test that previously asserted the annotate behaviour is inverted rather than deleted. Also adds breakevencurve, which computes fleet-size break-even by bootstrapping random orderings of the 527-tool corpus: definition sizes span 145..6485 bytes, so a single ordering would be one arbitrary curve rather than an answer. Corrects specs/103 findings/webui-savings-vs-bench.md: BOTH figures in the +68.4% / -32.5% contradiction were wrong. The Web-UI number is non-deterministic (map-order sampling, 52.35%-78.80% over an unchanging fleet), and bench's own proxy menu was overstated because ProxyModeToolDefs pins code execution on. The sign of the finding survives; its magnitudes did not. * docs(103): the "40-128 tools" constraint is stale — research the 2026 state Spec 103's break-even framing inherited a 2025 claim: that above roughly 40-128 tools a user physically cannot connect everything, so mcpproxy's value is CAPABILITY rather than token saving. Checked it. It mostly does not hold. - Cursor's "40-tool cap" is 2025 forum lore, absent from current docs, and recharacterized by staff as an upstream per-model limit. - Gemini went 128 -> 512. Anthropic, Bedrock, Claude Code, Kiro, Cline and Zed document no cap, and the MCP spec caps nothing. - The 2026 answer to large tool surfaces was not a raised ceiling but DEFERRED LOADING, which Claude Code, Cursor, VS Code and Kiro now ship natively — the same mechanism this proxy sells. The binding constraint moved, and it is far LOWER than any cap: Anthropic publishes tool-selection degradation past 30-50 available tools, where nothing errors and quality just falls. Every crossing point on our break-even curve is above that. Fleet size may be the wrong x-axis — median MCP server is 10 tools and only 1.2% expose more than 100, so WHICH servers dominates HOW MANY. Two consequences for what we may claim: 1. Do not say "you couldn't connect these otherwise" without naming the client. On Claude Code in 2026 you could. 2. Against a deferring client the token baseline is already reduced, so the defensible claim is retrieval QUALITY above 30-50, not raw token saving. The all-loaded baseline is honest only on non-deferring surfaces. The document keeps its own completeness critic, which judges the research NOT thorough and names three holes — tools/list pagination non-compliance, the gateway/aggregator tier (AWS AgentCore and Docker MCP Gateway have shipped this design already), and prompt caching. The caching one matters most and is unresolved: tool definitions are the canonical cacheable prefix, so a proxy that varies the exposed tool list per turn invalidates that prefix every turn and can cost MORE than the static baseline it is measured against. Nothing in bench models this. * docs(103): prefix-stability audit — caching shrinks the saving 10x but does not move break-even Step 1 of the caching question: can mcpproxy invalidate the provider's prompt cache, and does that flip the sign of the savings claim? Static audit of every tool-list mutation path plus the refresh log of one ordinary 4-server startup. No API spend. Pinned to primary source (platform.claude.com, verified today): prefix order is tools -> system -> messages; writes 1.25x (5m) / 2x (1h); reads 0.1x; and "modifying tool definitions ... invalidates the entire cache". Findings: - The DEFAULT retrieve_tools surface is cache-stable. It is rebuilt only on config.reloaded, never on server churn. - The code_execution surface fires notifications/tools/list_changed on every servers.changed for a tool list that structurally cannot change — buildCodeExecModeTools never iterates upstreams. Measured 9 of 9 broadcasts gratuitous on one startup. - The direct surface DOES re-write real tool definitions on ordinary churn (1 -> 3 -> 18 -> 27 tools across one startup), and that is a genuine full invalidation. RefreshDirectModeToolsOnSerializationChange is drift-guarded; the servers.changed path calls the UNGUARDED RefreshDirectModeTools. - 17 sites emit servers.changed, including server_connected/_disconnected/ _state_changed. Coalescing is a 50ms window, so a reconnect loop is not collapsed. Two corrections to my own earlier reasoning, both in the write-up: 1. A gratuitous list_changed is NOT a cache hazard. mcp-go sorts tools/list (sort.Strings), so an unchanged set returns byte-identical content and the cache still hits. It is wasted client round-trips, not lost cache. 2. Caching does NOT push break-even further right. Both arms cache the same way, so the (B - M) factor divides out and the crossover fleet size is roughly invariant. What caching changes is the ABSOLUTE saving: in a cached steady state a token bills at 0.1x, so every raw token figure this spec publishes overstates money value by about 10x. The mechanisms that do move the line, both against the proxy: discovery responses are per-turn cache WRITES at 1.25x while the baseline pays nothing extra, and direct-mode invalidation re-writes the whole prefix including the conversation. The latter is the only mechanism found that can plausibly flip the sign, and it is confined to direct mode with a churning fleet. Recommends guarding both refresh paths before any paid step-2 measurement, so the experiment does not price avoidable churn as if it were inherent. * fix(server): stop the tool surfaces re-registering when nothing changed mcp-go's SetTools notifies EVERY initialized client unconditionally — it swaps the registry and sends notifications/tools/list_changed without comparing old and new content, despite its own comment saying "when the list of available tools changes". Both routing-mode surfaces rebuilt speculatively on servers.changed, which has 17 emitters including server_connected, server_disconnected and server_state_changed. So every reconnect attempt told every connected client its tool set had moved, and the 50ms coalescer collapses nothing on that timescale. Measured on one ordinary 4-server startup, before and after: code_execution surface : 9 publishes -> 1 direct surface : 10 publishes -> 4 The direct tool_count sequence is the clearest evidence: before: 1 1 1 3 3 18 27 27 27 27 after: 1 3 18 27 Every duplicate gone, every genuine transition preserved. The code-execution surface could never change at all — buildCodeExecModeTools carries built-ins only and never iterates upstreams — so all 9 of its publishes were spurious. Guarded on a content FINGERPRINT rather than by dropping the servers.changed call, so both stay correct if either surface ever gains fleet-dependent content. The direct guard compares TWO fingerprints, and the second is the load-bearing one. Registry and catalog publish as a pair (a handler closes over its catalog entry), and routing can move while the listing stays byte-identical: a display-name collision resolving to a different upstream, or a changed RequiredPermission that gates scoped tokens without appearing in any tool definition. Guarding on the listing alone breaks three existing tests, including spec-102's direct_tool_response_mode hot reload — with no upstream tools a serialization flip changes only the catalog's mode, leaving the listing identical. A listing-only guard would have silently disabled a shipped feature. RenderedDescription is deliberately excluded from the routing fingerprint: it is captured at render time and the signature cache mutates independently of rebuilds, so including it would report a cache warm or evict as a catalog change — the distinction D13 rule 5 exists to preserve. directSurfaceUnchanged is a named predicate rather than an inline && because the reload harness has no upstreams and so cannot stage a routing-only change; the requirement is pinned by a test instead of by a comment. What this does NOT do: it removes wasted client round-trips, not lost provider cache. mcp-go sorts tools/list, so a re-fetch of an unchanged set returns byte-identical content and a cached prefix still hits (see findings/cache-prefix-stability-audit.md). The four remaining direct-mode publishes are genuine tool-definition changes and are genuine invalidations. * fix(bench): split code-exec savings by billing direction — tokens saved can be money lost The saving I shipped two hours ago sums a tool call's ARGUMENTS with its RESPONSE. They are not the same kind of token. Arguments are text the MODEL GENERATED (output, billed ~5x input); the response is text it reads (input, billed 1x, or 0.1x against a cached prefix). Summing them prices a 5x asymmetry at 1x — a milder form of the never-sum error this file already polices. It is not academic. Code execution trades the cheap direction for the expensive one: it replaces N argument-writings with ONE script, whose cost is fixed however few calls it replaces. Measured on real traffic, filesystem reads of ~700-byte files: sub-calls | output base->ce | input base->ce | tokens | USD (uncached / cached in) 1 | 5 -> 44 | 62 -> 26 | -3 | -0.000477 / -0.000574 2 | 24 -> 78 | 210 -> 11 | +145 | -0.000213 / -0.000750 3 | 36 -> 78 | 315 -> 11 | +262 | +0.000282 / -0.000539 5 | 60 -> 78 | 525 -> 12 | +495 | +0.001269 / -0.000116 7 | 84 -> 78 | 735 -> 12 | +729 | +0.002259 / +0.000307 30 | 360 -> 78 | 3150 -> 12 | +3420 | +0.013644 / +0.005171 Break-even is ~2 sub-calls in raw tokens but ~6 in money once input is cached. Between 3 and 5 the token figure reads "+262 to +495 saved" while the wallet loses. Publishing the token number alone would be exactly the kind of flattering arithmetic spec 103 exists to stop. CodeExecSaving now carries BaselineOutput/BaselineInput and ProxyOutput/ ProxyInput beside the existing totals, and PricedSavingUSD converts them. Pricing REFUSES two cases rather than returning a plausible number: a withheld saving (0.0 would read as "free" rather than "not measurable") and an estimated one (those are BYTE lengths, and pricing bytes per token is wrong by the bytes-per-token ratio while looking entirely correct). Adds bench/cmd/iosplit, which reports the split straight from an activity export — that is how the table above was measured. Response-side totals are unchanged; this only stops them being read as cost. * fix(bench,runtime): don't bill server-resolved script source, and stop truncated built-ins inflating usage Three defects from cross-model review of the branch. HIGH (bench): a stored-script code_execution was charged for a script the model never wrote. mcpproxy records the RESOLVED source under "code" alongside the name so the audit trail shows what ran, but the model sent only {"script": "<name>", "input": {...}}. Billing the recorded arguments as model-generated output charges ~5,200 output tokens per run for a ~17-token request, at roughly 5x the input rate — and it does so in exactly the configuration stored scripts exist to make cheap, so it understated code execution by the whole size of the script on every invocation. billableArguments now drops "code" when a non-empty "script" name is present, and only then; an inline call genuinely did send its source and keeps it. MEDIUM (bench): PricedSavingUSD accepted NaN, infinity and negative prices and returned ok=true, so nonsense propagated silently into a confident dollar figure. Rejected now, the same way a withheld or byte-estimated saving is. MEDIUM (runtime): countInTimeBucket added ResponseBytes without regard to ResponseTruncated. That was harmless while internal calls carried no byte counts at all; this branch gave them some, so a truncated retrieve_tools began contributing its FULL pre-truncation size to delivered traffic. The direction differs by record type and that is the point — an upstream tool_call is cut on the way into the LOG while the agent got the whole response, so its pre-truncation length is honest; a built-in is the reverse, the log keeps the full text and the agent consumed the cut one. Counting the second inflates the usage timeline in the flattering direction. Each fix is pinned by a test that fails when the guard is removed, verified by mutation. Also adds a runbook for measuring the "Twenty-Eight Round Trips" workload on the machine it was recorded on. * fix(api): registry success payloads must not deref a nil entry redactedRegistrySummary (#1164) dereferences its argument, and all three registry handlers — add, edit and remove — called it on the SUCCESS path with no nil check. A controller reporting success without an entry therefore panicked inside the handler, and chi's recoverer turned that into a bare 500 with an EMPTY body: the caller saw an unexplained server error on a request that had actually succeeded, and the cause survived only as a stack in the log. Not hypothetical. MockServerController already returns (nil, nil, nil) from these methods, so every test driving those routes through it hit the panic. It surfaced as an intermittent windows-amd64 CI failure where the recovered panic escaped the test process — intermittent because the recoverer usually swallows it, which is exactly why it went unnoticed: the package stayed green while three routes were answering 500 to successful requests. The guard lives in the renderer rather than at the three call sites so a fourth caller cannot reintroduce it. A nil entry now renders the zero summary, which reports no identity and — deliberately — Trusted=false: nothing unknown may be described as trusted. Tests assert both that the routes do not panic and that the body is NON-EMPTY, since an empty 500 is the signature of a recovered panic and a test that only checked the status code would have passed throughout.
1 parent a8e9af7 commit 6af893a

24 files changed

Lines changed: 2893 additions & 6 deletions

‎bench/cmd/breakevencurve/main.go‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
// Command breakevencurve computes the fleet-size break-even curve: at how many
2+
// upstream tools (and servers) does the proxy stop costing more than it saves?
3+
//
4+
// Net(n) = B(n) - M - sum(r), where B(n) is the cumulative cost of n upstream
5+
// tool definitions, M is mcpproxy's own advertised menu (fixed), and sum(r) is
6+
// the measured cost of the session's discovery responses.
7+
//
8+
// B(n) depends on WHICH n tools: definition sizes span 145..6485 bytes on the
9+
// reference snapshot, so a single ordering yields one arbitrary curve. This
10+
// bootstraps over many random orderings and reports the median and a p10..p90
11+
// band, so the answer is a distribution and reads as one.
12+
package main
13+
14+
import (
15+
"encoding/json"
16+
"flag"
17+
"fmt"
18+
"math/rand"
19+
"os"
20+
"sort"
21+
22+
"github.com/smart-mcp-proxy/mcpproxy-go/bench"
23+
"github.com/smart-mcp-proxy/mcpproxy-go/bench/corpusio"
24+
)
25+
26+
func main() {
27+
corpusPath := flag.String("corpus", "specs/083-discovery-profiler/datasets/livemcptool_snapshot/tools.json", "livemcptool snapshot")
28+
trials := flag.Int("trials", 400, "bootstrap orderings")
29+
meanResp := flag.Float64("mean-response", 1704.3, "mean discovery response tokens")
30+
calls := flag.Int("calls", 3, "discovery calls per session")
31+
jsonOut := flag.String("json", "", "write curve data to this path")
32+
seed := flag.Int64("seed", 20260901, "rng seed (pinned for reproducibility)")
33+
flag.Parse()
34+
35+
corpus, _, _, err := corpusio.LoadLiveMCPTool(*corpusPath)
36+
if err != nil {
37+
fmt.Fprintln(os.Stderr, "load:", err)
38+
os.Exit(1)
39+
}
40+
tk, err := bench.NewTokenizer("cl100k_base")
41+
if err != nil {
42+
fmt.Fprintln(os.Stderr, "tokenizer:", err)
43+
os.Exit(1)
44+
}
45+
46+
// Per-tool cost, computed once. BPE is not additive across concatenation,
47+
// so this is an approximation of B(n) by summation -- stated, not hidden.
48+
cost := make([]int, len(corpus.Tools))
49+
servers := make([]string, len(corpus.Tools))
50+
for i, tl := range corpus.Tools {
51+
cost[i] = tk.CountToolWithSchema(tl)
52+
servers[i] = tl.Server
53+
}
54+
total := 0
55+
for _, c := range cost {
56+
total += c
57+
}
58+
59+
spend := *meanResp * float64(*calls)
60+
fmt.Printf("corpus : %s (%d tools)\n", corpus.Version, len(corpus.Tools))
61+
fmt.Printf("B(all) : %d tokens mean %.1f/tool\n", total, float64(total)/float64(len(cost)))
62+
fmt.Printf("session spend : %d calls x %.1f = %.0f tokens\n\n", *calls, *meanResp, spend)
63+
64+
for _, mode := range []string{bench.ModeRetrieveTools, bench.ModeCodeExecution} {
65+
m := 0
66+
for _, tl := range bench.ProxyToolsForMode(mode) {
67+
m += tk.CountToolWithSchema(tl)
68+
}
69+
threshold := float64(m) + spend
70+
rng := rand.New(rand.NewSource(*seed))
71+
crossings := make([]int, 0, *trials)
72+
never := 0
73+
for t := 0; t < *trials; t++ {
74+
perm := rng.Perm(len(cost))
75+
run, hit := 0.0, -1
76+
for k, idx := range perm {
77+
run += float64(cost[idx])
78+
if run >= threshold {
79+
hit = k + 1
80+
break
81+
}
82+
}
83+
if hit < 0 {
84+
never++
85+
continue
86+
}
87+
crossings = append(crossings, hit)
88+
}
89+
sort.Ints(crossings)
90+
fmt.Printf("mode %-15s menu M = %d tokens\n", mode, m)
91+
fmt.Printf(" break-even threshold : %.0f tokens (M + session spend)\n", threshold)
92+
if never > 0 {
93+
fmt.Printf(" NEVER breaks even in %d/%d orderings (whole fleet is cheaper than the proxy)\n", never, *trials)
94+
}
95+
if len(crossings) > 0 {
96+
fmt.Printf(" break-even tools : p10=%d median=%d p90=%d\n",
97+
crossings[len(crossings)/10], crossings[len(crossings)/2], crossings[len(crossings)*9/10])
98+
}
99+
fmt.Println()
100+
}
101+
102+
if *jsonOut != "" {
103+
emitCurve(*jsonOut, tk, cost, servers, spend, *trials, *seed, corpus.Version)
104+
}
105+
106+
// Servers are what a user actually adds or removes.
107+
byServer := map[string]int{}
108+
for i := range cost {
109+
byServer[servers[i]] += cost[i]
110+
}
111+
sizes := make([]int, 0, len(byServer))
112+
for _, v := range byServer {
113+
sizes = append(sizes, v)
114+
}
115+
sort.Ints(sizes)
116+
fmt.Printf("servers : %d, median %d tokens each (p10=%d p90=%d)\n",
117+
len(sizes), sizes[len(sizes)/2], sizes[len(sizes)/10], sizes[len(sizes)*9/10])
118+
}
119+
120+
type curvePoint struct {
121+
N int `json:"n"`
122+
P10 float64 `json:"p10"`
123+
Med float64 `json:"median"`
124+
P90 float64 `json:"p90"`
125+
}
126+
127+
// emitCurve writes net-token curves (B(n) - M - spend) per mode. Menus are
128+
// listed with the value bench computes AND the deployed default, which differ
129+
// because ProxyModeToolDefs pins EnableCodeExecution:true while the shipped
130+
// default serves a disabled stub.
131+
func emitCurve(path string, tk *bench.Tokenizer, cost []int, servers []string, spend float64, trials int, seed int64, version string) {
132+
rng := rand.New(rand.NewSource(seed))
133+
perms := make([][]int, trials)
134+
for t := range perms {
135+
perms[t] = rng.Perm(len(cost))
136+
}
137+
// Cumulative B(n) per ordering.
138+
cum := make([][]float64, trials)
139+
for t, perm := range perms {
140+
row := make([]float64, len(cost)+1)
141+
for k, idx := range perm {
142+
row[k+1] = row[k] + float64(cost[idx])
143+
}
144+
cum[t] = row
145+
}
146+
menus := map[string]int{}
147+
for _, mode := range []string{bench.ModeRetrieveTools, bench.ModeCodeExecution} {
148+
m := 0
149+
for _, tl := range bench.ProxyToolsForMode(mode) {
150+
m += tk.CountToolWithSchema(tl)
151+
}
152+
menus[mode] = m
153+
}
154+
out := map[string]interface{}{
155+
"corpus": version, "tools": len(cost), "spend": spend,
156+
"menus": menus, "trials": trials,
157+
}
158+
catalog := []curvePoint{}
159+
for n := 0; n <= len(cost); n++ {
160+
vals := make([]float64, trials)
161+
for t := range cum {
162+
vals[t] = cum[t][n]
163+
}
164+
sort.Float64s(vals)
165+
catalog = append(catalog, curvePoint{N: n, P10: vals[trials/10], Med: vals[trials/2], P90: vals[trials*9/10]})
166+
}
167+
out["tool_catalog_curve"] = catalog
168+
curves := map[string][]curvePoint{}
169+
step := 1
170+
for mode, m := range menus {
171+
pts := []curvePoint{}
172+
thr := float64(m) + spend
173+
for n := 0; n <= len(cost); n += step {
174+
vals := make([]float64, trials)
175+
for t := range cum {
176+
vals[t] = cum[t][n] - thr
177+
}
178+
sort.Float64s(vals)
179+
pts = append(pts, curvePoint{N: n,
180+
P10: vals[trials/10], Med: vals[trials/2], P90: vals[trials*9/10]})
181+
}
182+
curves[mode] = pts
183+
}
184+
out["curves"] = curves
185+
byServer := map[string]int{}
186+
for i := range cost {
187+
byServer[servers[i]] += cost[i]
188+
}
189+
// Bootstrap over SERVER orderings too: a user adds and removes servers,
190+
// not individual tools, and server sizes are far more skewed than tool
191+
// sizes (p10=152, p90=3184 on this snapshot).
192+
svcCost := make([]int, 0, len(byServer))
193+
for _, v := range byServer {
194+
svcCost = append(svcCost, v)
195+
}
196+
srng := rand.New(rand.NewSource(seed))
197+
scum := make([][]float64, trials)
198+
for t := range scum {
199+
perm := srng.Perm(len(svcCost))
200+
row := make([]float64, len(svcCost)+1)
201+
for k, idx := range perm {
202+
row[k+1] = row[k] + float64(svcCost[idx])
203+
}
204+
scum[t] = row
205+
}
206+
scurve := []curvePoint{}
207+
for n := 0; n <= len(svcCost); n++ {
208+
vals := make([]float64, trials)
209+
for t := range scum {
210+
vals[t] = scum[t][n]
211+
}
212+
sort.Float64s(vals)
213+
scurve = append(scurve, curvePoint{N: n, P10: vals[trials/10], Med: vals[trials/2], P90: vals[trials*9/10]})
214+
}
215+
out["server_catalog_curve"] = scurve
216+
out["servers"] = len(byServer)
217+
b, _ := json.MarshalIndent(out, "", " ")
218+
_ = os.WriteFile(path, b, 0o644)
219+
fmt.Println("curve written:", path)
220+
}

‎bench/cmd/codeexecsaving/main.go‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
// Command codeexecsaving reports the RESPONSE-SIDE saving of code execution
2+
// from a recorded activity export: what the sandbox's sub-call responses would
3+
// have cost an agent that made those calls itself, against the script it sent
4+
// and the single result it got back.
5+
package main
6+
7+
import (
8+
"encoding/json"
9+
"flag"
10+
"fmt"
11+
"os"
12+
"sort"
13+
14+
"github.com/smart-mcp-proxy/mcpproxy-go/bench/replaycorpus"
15+
)
16+
17+
func main() {
18+
in := flag.String("in", "", "activity export JSONL (must live outside the repository)")
19+
inPrice := flag.Float64("in-price", 3.0, "USD per Mtok, input")
20+
outPrice := flag.Float64("out-price", 15.0, "USD per Mtok, output")
21+
cacheMult := flag.Float64("cache-mult", 0.1, "input multiplier for a cached prefix (1.0 = uncached)")
22+
// Default OFF, matching replaycorpus's zero value: bodies-on reads recorded
23+
// request and response CONTENT unmasked. With the sub-call byte counts now
24+
// emitted, bodies-off still yields a byte-basis figure.
25+
bodies := flag.Bool("bodies-unmasked", false, "read recorded bodies and count real TOKENS (reads unmasked content)")
26+
flag.Parse()
27+
if *in == "" {
28+
fmt.Fprintln(os.Stderr, "-in is required")
29+
os.Exit(2)
30+
}
31+
32+
policy := replaycorpus.BodiesOff
33+
if *bodies {
34+
policy = replaycorpus.BodiesOnUnmasked
35+
}
36+
corpus, err := replaycorpus.LoadFile(*in, replaycorpus.Options{Bodies: policy})
37+
if err != nil {
38+
fmt.Fprintln(os.Stderr, "load:", err)
39+
os.Exit(1)
40+
}
41+
rep := replaycorpus.CodeExecSavingsFor(corpus.Sessions)
42+
43+
fmt.Printf("sessions %d bodies=%v\n", len(corpus.Sessions), policy)
44+
for _, w := range corpus.Warnings {
45+
fmt.Println(" warning:", w)
46+
}
47+
bases := make([]string, 0, len(rep.Buckets))
48+
for b := range rep.Buckets {
49+
bases = append(bases, string(b))
50+
}
51+
sort.Strings(bases)
52+
if len(bases) == 0 {
53+
fmt.Println("\nno code_execution call produced a figure")
54+
}
55+
for _, b := range bases {
56+
k := rep.Buckets[replaycorpus.CostBasis(b)]
57+
unit := "bytes"
58+
if b == string(replaycorpus.CostMeasured) {
59+
unit = "tokens"
60+
}
61+
fmt.Printf("\n[%s] %d call(s), %d sub-call(s) — unit: %s\n", b, k.Calls, k.SubCalls, unit)
62+
fmt.Printf(" baseline (agent makes the sub-calls itself) : %d\n", k.Baseline)
63+
fmt.Printf(" proxy (script sent + result returned) : %d\n", k.Proxy)
64+
fmt.Printf(" saving : %+d", k.Saving)
65+
if k.Baseline > 0 {
66+
fmt.Printf(" (%.1f%% of baseline)", 100*float64(k.Saving)/float64(k.Baseline))
67+
}
68+
fmt.Println()
69+
}
70+
// Money, not just tokens: a call's ARGUMENTS are output (billed ~5x) and its
71+
// RESPONSE is input, so the token total is not proportional to cost. Priced
72+
// per-call and summed, since only measured calls can be priced at all.
73+
pricedTotal, priced, unpriceable := 0.0, 0, 0
74+
for _, sv := range rep.Savings {
75+
if usd, ok := sv.PricedSavingUSD(*inPrice, *outPrice, *cacheMult); ok {
76+
pricedTotal += usd
77+
priced++
78+
} else {
79+
unpriceable++
80+
}
81+
}
82+
if priced > 0 {
83+
fmt.Printf("\npriced over %d measured call(s) at $%.2f/$%.2f per Mtok, input x%.2f:\n",
84+
priced, *inPrice, *outPrice, *cacheMult)
85+
fmt.Printf(" net saving: %+.6f USD\n", pricedTotal)
86+
if pricedTotal < 0 {
87+
fmt.Println(" NOTE: negative in money. Code execution trades cheap input for expensive")
88+
fmt.Println(" output — the script is a fixed cost however few calls it replaces.")
89+
}
90+
}
91+
if unpriceable > 0 {
92+
fmt.Printf(" %d call(s) could not be priced (withheld, or byte estimates rather than tokens)\n", unpriceable)
93+
}
94+
95+
for reason, n := range rep.Withheld {
96+
fmt.Printf("\nwithheld: %d call(s) — %s\n", n, reason)
97+
}
98+
if b, err := json.MarshalIndent(rep.Savings, "", " "); err == nil && len(rep.Savings) > 0 {
99+
fmt.Println("\nper-call detail:")
100+
fmt.Println(string(b))
101+
}
102+
}

0 commit comments

Comments
 (0)