Repository navigation
docs(benchmarks): note ponytail's cost is task-shaped in agentic runs (#121) - #122
Lakshya77089 wants to merge 1 commit into
Conversation
|
This is great and I want it merged with your name on it, the credit to Ricardo's benchmark repo is exactly right. One snag: it conflicts now. Main already rewrote that "Cost reflects single-shot calls..." bullet under Easy fix on your end:
Ping me when it's pushed and I'll merge it. Thanks for the rigor on this one. |
Add a "Where ponytail saves vs. costs" section splitting the two regimes (snowball-prone/blocked saves; large completion-forced can cost more), credit RicardoCostaGit's Cursor-SDK A/B, and clarify that an SDK's startup skillCount is skills available, not skills read. The single-shot cost bullet under Notes already scopes the claim on main, so this leaves it untouched. Closes DietrichGebert#121
b781acf to
c5205c7
Compare
|
@DietrichGebert rebased and pushed. Dropped the single-shot cost bullet edit (main's version covers it), kept the "Where ponytail saves vs. costs" section with the RicardoCostaGit link and the skillCount note, and made it em-dash-free to match the docs sweep. Added |
|
Heads-up: I opened #460, which adds an 'Independent benchmarks' section to benchmarks/README.md covering this (Ricardo's Cursor benchmark) plus Kuldeep's plugin-installed run, with method/date framing and credit links to both authors. It overlaps this PR's intent. Not closing this one, that is the maintainer's call, but flagging so the work is not duplicated. If #460 is the direction, this can be closed as superseded; if you would rather land your version, say so and I will drop the Ricardo row from #460. |
|
Closing as superseded by #460, now merged, which adds the Independent benchmarks section covering both Ricardo's and Kuldeep's runs. Thanks @Lakshya77089 for scoping the cost claim and pushing the credit/link in the first place, that framing carried straight into the merged version. |
Closes #121
Adds a "Where ponytail saves vs. costs" section to the benchmarks README, splitting the two regimes seen in multi-turn agentic runs:
Credits @RicardoCostaGit's independent Cursor-SDK A/B, and adds the clarification that an SDK's startup
skillCountis skills available in the workspace, not skills the model read.Rebased on main per review: dropped the edit to the single-shot cost bullet (main already scopes that claim and links #121), so this is now purely the new section. Kept it em-dash-free to match the recent docs sweep. Docs only, so
check-rule-copies.jsand the tests are unaffected.