Skip to content

docs: align README benchmark claims with the agentic results - #1055

Closed
ggeandersonreis wants to merge 1 commit into
DietrichGebert:mainfrom
ggeandersonreis:docs/readme-benchmark-claims
Closed

ggeandersonreis wants to merge 1 commit into
DietrichGebert:mainfrom
ggeandersonreis:docs/readme-benchmark-claims

Conversation

@ggeandersonreis

Copy link
Copy Markdown

Docs only: no ruleset, skill or hook changes. Every number here comes from benchmarks/results/2026-06-18-agentic.md; nothing was re-measured.

What changed

  • "ponytail is the only arm that cuts every metric" (README and results file). The README's own table shows the "YAGNI + one-liners" prompt also cuts all four (−33% LOC, −14% tokens, −21% cost, −30% time), and it is slightly cheaper and faster than ponytail. The sentence now states what does set ponytail apart: by far the biggest code cut, and the only arm that cuts every metric while staying fully safe (the YAGNI prompt dropped a path-traversal guard in 1 of 20 runs).
  • Chart alt text: "lowest on every metric" → lowest on LOC and tokens; the YAGNI prompt is slightly lower on cost and time (79% / 70% vs 80% / 73%).
  • Headline: "100% safe" → "safe in 20/20 adversarial runs", and "up to 94%" → "up to 94% on one task" (the date picker). Without the qualifier, the 94% reads like the single-shot 80–94% figure the README itself retracts further down.
  • Codex install: "trust its two lifecycle hooks" → "trust its lifecycle hooks". .codex-plugin/plugin.json points at hooks/claude-codex-hooks.json, which defines three (SessionStart, SubagentStart, UserPromptSubmit). If Codex really shows two, putting the number back with a note on why would be clearer than my edit.
  • Hosts with commands: added Kimi Code and Goose, which INSTALL.md documents as exposing the six commands.

Not changed, for you to decide

  • Translations: i18n/README.{es,ko,zh-CN,ja}.md carry the same headline, alt text and host list. I left them alone rather than hand-edit Korean, Chinese and Japanese prose.
  • Results file: the safety section's header and summary say "6 safety tasks", but the safe-rate table says "5 security tasks × 4 runs = 20". It doesn't say what the sixth task is.

🤖 Generated with Claude Code

- 'only arm that cuts every metric' contradicts the table: the YAGNI
  one-liner prompt also cuts all four and is slightly cheaper and faster;
  state what does set ponytail apart (biggest code cut, fully safe)
- qualify the headline: 94% is one task, safety is 20/20 adversarial runs
- fix the chart alt text the same way
- drop the hook count for Codex (the shared hook file defines three)
- list Kimi Code and Goose among hosts with commands, per INSTALL.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ggeandersonreis
ggeandersonreis force-pushed the docs/readme-benchmark-claims branch from 57458ea to 66e5707 Compare October 6, 2026 23:51
@DietrichGebert

Copy link
Copy Markdown
Owner

Thanks! Ponytail 5 replaced the headline, the chart text and the "only arm" sentence with the new benchmark, so these edits don't apply anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants