Repository navigation
SDK benchmark: Ponytail ON → more tool calls & cost, fewer output lines; skillCount ≠ skills read #121
Description
Cursor SDK benchmark findings (Ponytail A/B)
Measured with ponytail-benchmark — isolated git worktrees, settingSources: ["project"], same prompt per model, Ponytail ON vs OFF toggling only .cursor/rules/ponytail.mdc.
Date: June 2026
Models: Composer 2.5 Fast, Gemini 3.5 Flash, Claude Opus 4.8, Claude Sonnet 4.6, GPT-5.5 High (10 finished runs; GPT-5.5 Medium failed)
Task: Structured test-design prompt with frozen benchmark/context/ copied into each worktree.
Cost figures are estimated from SDK turn-ended usage × public list pricing, not Admin API billing.
1. Skills indexed vs skills actually read
The SDK startup index (skillCount / AgentSkillsCursorRulesService) reported ~36–44 skills available in the workspace, but stream analysis showed only 1–4 unique SKILL.md files actually read per run via tool calls.
| Metric | Typical Ponytail ON | Typical Ponytail OFF |
|---|---|---|
SDK skillCount (startup) |
~40 | ~40 |
Unique SKILL.md read (tool read) |
2–4 | 2–4 |
Total SKILL.md read events |
2–6 (re-reads) | 2–6 |
Implication: Reporting “40 skills loaded” does not mean the model used 40 skills. Glob/list events do not count as usage — only read of …/SKILL.md does.
2. Ponytail ON: higher cost and more tool calls, often fewer lines written
Across several models, Ponytail ON correlated with more SDK tool calls and higher estimated token cost, while git lines added to the draft file were often lower than Ponytail OFF for the same model and prompt.
Representative per-model results
| Model | Est. cost ON | Est. cost OFF | Δ cost | Tool calls ON | Tool calls OFF | Lines added ON | Lines added OFF |
|---|---|---|---|---|---|---|---|
| Composer 2.5 Fast | ~$1.07 | ~$0.69 | +55% | higher | lower | lower | higher |
| Gemini 3.5 Flash | ~$1.32 | ~$1.25 | +6% | 22 | lower | 693 | higher |
| Claude Sonnet 4.6 | ~$1.55 | ~$1.48 | +5% | higher | lower | ~626 | ~617 |
| Claude Opus 4.8 | ~$2.35 | ~$2.39 | −2% | similar | similar | ~523 | ~560 |
| GPT-5.5 High | ~$1.80 | ~$2.06 | −13% | lower | higher | ~698 | ~647 |
Pattern (most models): Ponytail ON → more exploration (reads/skills/context) → more tool calls and tokens → higher estimated cost, but leaner draft output (fewer lines in .cursor/drafts/benchmark-*.md).
Exceptions: Claude Opus and GPT-5.5 High did not follow the cost/tool pattern in this matrix; Opus ON was slightly cheaper and faster.
Composer 2.5 Fast (strongest cost gap)
| Ponytail ON | Ponytail OFF | |
|---|---|---|
| Total tokens (input + output) | ~308k | ~187k |
| Estimated cost | ~$1.07 | ~$0.69 |
| Tool calls | more | fewer |
| Lines added (git) | fewer | more |
3. Interpretation (not a product bug report)
These runs used a large, completable task (full test-design draft), not a “blocked MCP / stop early” scenario. In blocked scenarios, Ponytail’s “lean / don’t snowball” behavior can reduce cost dramatically; in completion-forced benchmarks, ON often means more disciplined reading before writing, which shows up as higher tool and token use with shorter written output.
4. Reproduce
git clone https://github.com/RicardoCostaGit/ponytail-benchmark-from-cursor.git
cd ponytail-benchmark-from-cursor
npm install && cp .env.example .env
# set CURSOR_API_KEY, point benchmark.config.json at your repo
bash scripts/prepare-example-target.sh
npm run run && npm run aggregate
Activity
DietrichGebert commentedon Jun 16, 2026
This is a great writeup, thank you. This is the multi-turn agentic side my own benchmark does not cover (mine is single-shot: one prompt, count the code and the generation cost), so it is genuinely useful to see the tool-call and token numbers.
Taking it honestly:
The cost-up, lines-down pattern is real and I will not argue it. Ponytail re-injects its full ruleset every turn, so it adds a fixed input-token cost per turn no matter what. On a big forced-completion task, if ON also makes the model read and check more before it writes, you get exactly what you measured: more tool calls and tokens, leaner output. That is the same overhead the README now flags (a caveat just landed in #115). Your runs put real numbers on it on an agentic harness, which mine never did.
One scoping note, not a defense: these are two different axes. My repo number is single-shot generation cost and lines. Yours is multi-turn process cost on a completable task. Both can be true at once: leaner artifact, heavier process. So whether ON saves money depends entirely on the workload, which your "blocked vs completion-forced" split captures well.
skillCount vs skills read: agreed, and worth saying loudly for anyone benchmarking. Ponytail never claims a skill count though, so that one is a general Cursor SDK caveat (index size is not usage), not a ponytail discrepancy. Good to document either way.
On the mechanism: my hypothesis for "ON reads more" is the ladder itself. Checking stdlib, native, already-installed before writing pushes the model to look around first. That is a feature when it stops the model reinventing something, and a cost when the answer was always net-new. Worth measuring directly. I would also flag the sample is small and model-dependent: Opus and GPT-5.5 High went the other way in your own table, so I would not generalize the cost direction across models yet.
Honestly the most valuable thing here is the harness. Would you be open to me linking ponytail-benchmark-from-cursor from the benchmarks README, or folding a short summary of this in with credit to you? People keep asking for a multi-turn benchmark and you built one.
Lakshya77089 commentedon Jun 16, 2026
This is excellent, thanks for the rigor — isolated worktrees and toggling only the rule file is exactly the right setup.
On skillCount vs reads: fully agree, and it's worth nailing down.
skillCountis the SDK's startup index of what's available in the workspace, not what the model pulled into context — only areadof aSKILL.mdis usage; glob/list events aren't. "40 skills loaded" was never 40 skills used.On ON → more tool calls/cost, fewer lines: your interpretation matches how I'd read it. The headline benchmark numbers are single-shot completions on five tiny tasks via the Claude API — one call, no agentic loop. Your matrix is multi-turn, completion-forced agentic runs, and there ponytail's "understand before you write" discipline shows up as more reading/exploration up front: more tool calls and tokens, leaner draft out. So ON trades input cost for output restraint, and on a large completable task that trade can go net-negative on cost — while the savings really land on the snowball-prone / blocked tasks you flagged.
Your finding sits in slight tension with the current
benchmarks/README.mdnote that "in real sessions … the cost gap widens further in ponytail's favor" — true for the single-shot regime, but it over-generalizes to agentic completion-forced ones. So I opened #122: it scopes that claim, adds a "Where ponytail saves vs. costs" section splitting the two regimes, and folds in your skillCount-is-availability-not-usage point, with a credit + link to your benchmark repo. Happy to reword if you'd frame any of it differently.Genuinely useful data — thanks for measuring it.
RicardoCostaGit commentedon Jun 16, 2026
Thank you both for the detailed feedback and for taking the findings seriously.
I have now made the benchmark repository public.
Please feel free to link it from the benchmarks README and include a short summary with credit.
I appreciate the distinction you both made between single-shot generation and multi-turn agentic execution.
The benchmark was intended to complement the existing results rather than contradict them.
I agree that the current sample is still small and that the results are highly dependent on the model, task type, and whether the task is completion-forced or likely to trigger unnecessary scaffolding and snowballing.
There is definitely room to improve the harness, particularly around:
Contributions are very welcome.
Feel free to open issues or pull requests with improvements, additional scenarios, or changes to the methodology.
I would be happy to review and collaborate on them.
Thanks again for the quick response and for already incorporating these findings into the documentation.
docs: state the single-shot limitation honestly
docs: correct cost claim to 42-75% from 30-rep re-verification (#129)
Fix/examples issue 127 (#131)
docs(benchmarks): note ponytail's cost is task-shaped in agentic runs
docs(benchmarks): add Independent benchmarks section (DietrichGebert#460
docs(benchmarks): add Independent benchmarks section (DietrichGebert#460
docs: correct cost claim to 42-75% from 30-rep re-verification (Dietr…
Fix/examples issue 127 (DietrichGebert#131)
docs: correct cost claim to 42-75% from 30-rep re-verification (Dietr…
Fix/examples issue 127 (DietrichGebert#131)
docs(benchmarks): add Independent benchmarks section (DietrichGebert#460