Skip to content

Commit b781acf

Browse files
committed
docs(benchmarks): note ponytail's cost is task-shaped in agentic runs
The headline benchmark is single-shot completions; a Notes bullet generalized 'the cost gap widens further in ponytail's favor' to real sessions. Independent multi-turn Cursor-SDK A/B (#121) shows that on large completion-forced tasks ponytail ON can raise tool calls and tokens (more reading before writing) while shrinking output — net cost either way; the win is clearest on snowball-prone/blocked tasks. Scope the single-shot claim and add a 'saves vs. costs' section, plus the skillCount-is-availability-not-usage clarification.
1 parent 99139a2 commit b781acf

1 file changed

Lines changed: 10 additions & 1 deletion

File tree

‎benchmarks/README.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,5 +79,14 @@ Running the benchmark requires **Python 3**, **pandas**, and **Node.js** (18+).
7979
## Notes
8080

8181
- Caveman is a prose-compression skill (it leaves code "normal"), so it lands between baseline and ponytail on code size and wins mainly on prose tokens.
82-
- Cost reflects single-shot calls that re-send the skill every time. In real sessions the skill is injected once and prompt-cached, so the cost gap widens further in ponytail's favor.
82+
- Cost reflects single-shot calls that re-send the skill every time. In real sessions the skill is injected once and prompt-cached, so on tasks shaped like these the cost gap widens further in ponytail's favor.
8383
- These are everyday tasks. For production-grade specs, where an unconstrained agent bloats much harder, see the writeups in `results/`.
84+
85+
## Where ponytail saves vs. costs
86+
87+
These numbers are single-shot completions. In multi-turn agentic runs the result is task-shaped, not universal:
88+
89+
- **Snowball-prone or blocked tasks** (agent keeps adding, installs a dep, scaffolds "for later"): ponytail's restraint cuts the runaway work — this is where the cost win is largest.
90+
- **Large completion-forced tasks** (a full draft the agent must finish): ponytail's "understand before you write" discipline can add reading/exploration up front, so it may raise tool calls and tokens while shrinking the written output. Net cost can go either way.
91+
92+
An independent Cursor-SDK A/B measuring this (isolated worktrees, toggling only the rule file) saw ponytail ON correlate with more tool calls and higher estimated cost but leaner drafts on completion-forced tasks, with per-model exceptions: [RicardoCostaGit/ponytail-benchmark-from-cursor](https://github.com/RicardoCostaGit/ponytail-benchmark-from-cursor) (#121). Note also that an SDK's startup `skillCount` is the count of skills *available* in the workspace, not skills the model read — only a `read` of a `SKILL.md` is usage.

0 commit comments

Comments
 (0)