Skip to content

Commit 45f7d2f

Browse files
Fix/examples issue 127 (#131)
* docs: correct cost claim to 42-75% from 30-rep re-verification Re-ran the cost benchmark at 30 reps per cell on Claude (Haiku/Sonnet/Opus): ponytail is 42-75% cheaper than no-skill, not the previously published 47-77%. The direction holds, both ends came in a few points lower. Updates the README headline and body, the benchmark chart subtitle, and the benchmarks/README cost table, and adds a dated results doc with full method. Also adds the OpenAI (gpt-4.1-mini/gpt-5.4-mini/gpt-5.5) and Gemini configs. On OpenAI reasoning models ponytail costs more, not less, so the claim stays Claude-scoped. Gemini run pending a fresh-quota day. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: scope the body claim to Claude models "on every model" read as cross-provider, but the 30-rep verification shows the cost win reverses on OpenAI reasoning models. Match the caption and benchmarks/README, which already say Claude. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reframe the pitch as the discipline, not token savings The cost/code/latency numbers vary by model and on some (terse reasoning models like GPT-5.5) ponytail costs more, so leading with them as a universal win was misleading. Adds model-variance to the headline caption and a paragraph making the stated point the mental model: write only what the task needs, safety kept, maintainable code. Savings are a model-dependent side effect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: name the ladder's reasoning cost The ladder is a deliberation step: on reasoning models the agent spends thinking tokens working through the rungs before it saves any output, which together with the always-on ruleset can outweigh the shorter code. Makes the GPT-5.5 cost increase legible rather than just stating it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: state the single-shot limitation honestly The benchmark is single-shot (one prompt, one completion); it does not measure a real multi-turn agent session, where the ruleset re-injects and the ladder deliberates every turn. Adds that caveat to the README, and corrects the benchmarks/README note that claimed caching widens the gap "in ponytail's favor" (unverified, and a measured agentic A/B in #121 found the opposite can happen). Per-session cost can land either way. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix run count in caption (cost is 30 runs, not 10) Cost was re-verified at 30 reps; code and latency are still the original 10. The headline caption said "10 runs" across the board, which undersold the cost verification. Now states the split, matching benchmarks/README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(examples): replace hand-written examples with real benchmark output The examples/ before/after blocks were authored by hand, not produced by a model. Issue #127 correctly noted that nobody hand-rolls quicksort for "sort this array" - every model just calls .sort(). Regenerate all examples verbatim from a real benchmark run (Claude Haiku 4.5, no-skill arm vs ponytail arm, benchmarks/output.json) so the before/after is reproducible, not authored: email 75->3, debounce 116->10, csv 20->3, countdown 267->9, rate-limit 128->10 LOC - Delete sorting.md (pure strawman) plus the other hand-written caricatures (api-endpoint, caching, date-picker) - Add benchmarks/generate-examples.mjs to regenerate examples from any run - examples/README.md indexes the set and documents how to reproduce Closes #127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 1b49141 commit 45f7d2f

11 files changed

Lines changed: 1158 additions & 261 deletions

‎benchmarks/generate-examples.mjs‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
// Generate examples/*.md verbatim from a real benchmark run (output.json):
2+
// each file shows the same task answered with no skill vs with ponytail, same model.
3+
// node benchmarks/generate-examples.mjs
4+
import { readFileSync, writeFileSync } from 'node:fs';
5+
import loc from './loc.js';
6+
7+
const j = JSON.parse(readFileSync(new URL('./output.json', import.meta.url), 'utf8'));
8+
const isHaiku = (id) => id.includes('haiku');
9+
10+
const meta = [
11+
[/validates email/, 'email-validation', 'Email Validation'],
12+
[/debounce/, 'debounce', 'Debounce'],
13+
[/sales\.csv/, 'csv-sum', 'CSV Sum'],
14+
[/countdown timer/, 'react-countdown', 'Countdown Timer'],
15+
[/rate limiting/, 'rate-limit', 'Rate Limiting'],
16+
];
17+
18+
const pick = (re, armIdx) =>
19+
j.results.results.find((r) => isHaiku(r.provider.id) && r.promptIdx === armIdx && re.test(r.vars.task));
20+
21+
const rows = [];
22+
for (const [re, slug, title] of meta) {
23+
const b = pick(re, 0), p = pick(re, 2);
24+
if (!b || !p) { console.log('MISS', slug, !!b, !!p); continue; }
25+
const bL = loc(b.response.output).score, pL = loc(p.response.output).score;
26+
const md = `# ${title}
27+
28+
**Task:** "${b.vars.task}"
29+
30+
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source \`benchmarks/output.json\`. Reproduce: \`npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml\`.
31+
32+
## Without Ponytail — ${bL} lines of code
33+
34+
${b.response.output.trim()}
35+
36+
## With Ponytail — ${pL} lines of code
37+
38+
${p.response.output.trim()}
39+
40+
**${bL} → ${pL} lines of code** — same model, same prompt.
41+
`;
42+
writeFileSync(new URL(`../examples/${slug}.md`, import.meta.url), md);
43+
rows.push([title, slug, bL, pL]);
44+
console.log('wrote examples/' + slug + '.md', bL, '->', pL);
45+
}
46+
47+
const tbl = rows.map(([t, s, b, p]) => `| [${t}](${s}.md) | ${b} | ${p} |`).join('\n');
48+
const readme = `# Examples
49+
50+
Real model output, verbatim from benchmark runs — the same task answered by the same model
51+
with no skill (\`## Without Ponytail\`) and with ponytail (\`## With Ponytail\`), so you can
52+
compare side by side. Model: Claude Haiku 4.5, temperature 1, source \`benchmarks/output.json\`.
53+
54+
These are not hand-written. Reproduce them yourself:
55+
\`npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml\`. Method, all three models, and
56+
median-of-10 numbers: [../benchmarks/](../benchmarks/).
57+
58+
| Example | Without (LOC) | With (LOC) |
59+
|---|--:|--:|
60+
${tbl}
61+
`;
62+
writeFileSync(new URL('../examples/README.md', import.meta.url), readme);
63+
console.log('wrote examples/README.md');

‎examples/README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Examples
2+
3+
Real model output, verbatim from benchmark runs — the same task answered by the same model
4+
with no skill (`## Without Ponytail`) and with ponytail (`## With Ponytail`), so you can
5+
compare side by side. Model: Claude Haiku 4.5, temperature 1, source `benchmarks/output.json`.
6+
7+
These are not hand-written. Reproduce them yourself:
8+
`npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`. Method, all three models, and
9+
median-of-10 numbers: [../benchmarks/](../benchmarks/).
10+
11+
| Example | Without (LOC) | With (LOC) |
12+
|---|--:|--:|
13+
| [Email Validation](email-validation.md) | 75 | 3 |
14+
| [Debounce](debounce.md) | 116 | 10 |
15+
| [CSV Sum](csv-sum.md) | 20 | 3 |
16+
| [Countdown Timer](react-countdown.md) | 267 | 9 |
17+
| [Rate Limiting](rate-limit.md) | 128 | 10 |

‎examples/api-endpoint.md‎

Lines changed: 0 additions & 68 deletions
This file was deleted.

‎examples/caching.md‎

Lines changed: 0 additions & 74 deletions
This file was deleted.

‎examples/csv-sum.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# CSV Sum
2+
3+
**Task:** "Write Python code that reads sales.csv and sums the 'amount' column."
4+
5+
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source `benchmarks/output.json`. Reproduce: `npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`.
6+
7+
## Without Ponytail — 20 lines of code
8+
9+
# Python code to read sales.csv and sum the 'amount' column
10+
11+
```python
12+
import pandas as pd
13+
14+
# Read the CSV file
15+
df = pd.read_csv('sales.csv')
16+
17+
# Sum the 'amount' column
18+
total_amount = df['amount'].sum()
19+
20+
# Print the result
21+
print(f"Total amount: ${total_amount:,.2f}")
22+
```
23+
24+
## Alternative methods:
25+
26+
**Using CSV module (without pandas):**
27+
```python
28+
import csv
29+
30+
total_amount = 0
31+
32+
with open('sales.csv', 'r') as file:
33+
reader = csv.DictReader(file)
34+
for row in reader:
35+
total_amount += float(row['amount'])
36+
37+
print(f"Total amount: ${total_amount:,.2f}")
38+
```
39+
40+
**Using pandas with error handling:**
41+
```python
42+
import pandas as pd
43+
44+
try:
45+
df = pd.read_csv('sales.csv')
46+
total_amount = df['amount'].sum()
47+
print(f"Total amount: ${total_amount:,.2f}")
48+
except FileNotFoundError:
49+
print("Error: sales.csv not found")
50+
except KeyError:
51+
print("Error: 'amount' column not found")
52+
```
53+
54+
**The pandas method is recommended** because it's:
55+
- More concise and readable
56+
- Handles data types automatically
57+
- Provides built-in error handling
58+
- Efficient for large files
59+
60+
## With Ponytail — 3 lines of code
61+
62+
```python
63+
import csv
64+
65+
total = sum(float(row['amount']) for row in csv.DictReader(open('sales.csv')))
66+
print(total)
67+
```
68+
69+
Skipped: pandas, error handling, file closing — add when the CSV is large, malformed, or you need more analysis.
70+
71+
**20 → 3 lines of code** — same model, same prompt.

‎examples/date-picker.md‎

Lines changed: 0 additions & 48 deletions
This file was deleted.

0 commit comments

Comments
 (0)