Skip to content

docs(benchmarks): note ponytail's cost is task-shaped in agentic runs (#121) - #122

Closed
Lakshya77089 wants to merge 1 commit into
DietrichGebert:mainfrom
Lakshya77089:bench-regime-note
Closed

Lakshya77089 wants to merge 1 commit into
DietrichGebert:mainfrom
Lakshya77089:bench-regime-note

Conversation

@Lakshya77089

@Lakshya77089 Lakshya77089 commented Jun 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #121

Adds a "Where ponytail saves vs. costs" section to the benchmarks README, splitting the two regimes seen in multi-turn agentic runs:

  • Snowball-prone / blocked tasks: ponytail's restraint cuts the runaway work (largest cost win).
  • Large completion-forced tasks: it can add up-front reading and raise tool calls/tokens while shrinking the written output, so net cost can go either way.

Credits @RicardoCostaGit's independent Cursor-SDK A/B, and adds the clarification that an SDK's startup skillCount is skills available in the workspace, not skills the model read.

Rebased on main per review: dropped the edit to the single-shot cost bullet (main already scopes that claim and links #121), so this is now purely the new section. Kept it em-dash-free to match the recent docs sweep. Docs only, so check-rule-copies.js and the tests are unaffected.

@DietrichGebert

Copy link
Copy Markdown
Owner

This is great and I want it merged with your name on it, the credit to Ricardo's benchmark repo is exactly right.

One snag: it conflicts now. Main already rewrote that "Cost reflects single-shot calls..." bullet under ## Notes (the later agentic-README/#115 work) to scope the claim and link #121, so your edit to that one line collides with it.

Easy fix on your end:

  1. Rebase on main.
  2. Drop your change to that single-shot cost bullet, main's version already covers it.
  3. Keep your "Where ponytail saves vs. costs" section as-is, the RicardoCostaGit repo link and the skillCount note are the parts I most want landed.
  4. Add Closes #121 to the PR description so the issue closes itself when this merges.

Ping me when it's pushed and I'll merge it. Thanks for the rigor on this one.

Add a "Where ponytail saves vs. costs" section splitting the two regimes
(snowball-prone/blocked saves; large completion-forced can cost more),
credit RicardoCostaGit's Cursor-SDK A/B, and clarify that an SDK's startup
skillCount is skills available, not skills read. The single-shot cost bullet
under Notes already scopes the claim on main, so this leaves it untouched.

Closes DietrichGebert#121
@Lakshya77089

Copy link
Copy Markdown
Contributor Author

@DietrichGebert rebased and pushed. Dropped the single-shot cost bullet edit (main's version covers it), kept the "Where ponytail saves vs. costs" section with the RicardoCostaGit link and the skillCount note, and made it em-dash-free to match the docs sweep. Added Closes #121 to the description. Conflict is gone, ready to merge whenever.

@DietrichGebert

Copy link
Copy Markdown
Owner

Heads-up: I opened #460, which adds an 'Independent benchmarks' section to benchmarks/README.md covering this (Ricardo's Cursor benchmark) plus Kuldeep's plugin-installed run, with method/date framing and credit links to both authors. It overlaps this PR's intent. Not closing this one, that is the maintainer's call, but flagging so the work is not duplicated. If #460 is the direction, this can be closed as superseded; if you would rather land your version, say so and I will drop the Ricardo row from #460.

@DietrichGebert

Copy link
Copy Markdown
Owner

Closing as superseded by #460, now merged, which adds the Independent benchmarks section covering both Ricardo's and Kuldeep's runs. Thanks @Lakshya77089 for scoping the cost claim and pushing the credit/link in the first place, that framing carried straight into the merged version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SDK benchmark: Ponytail ON → more tool calls & cost, fewer output lines; skillCount ≠ skills read

2 participants