Repository navigation
Commit e25ea48
docs(eval): correct two overstated claims in the ceiling anatomy
Self-review of the published numbers caught two that did not survive checking.
The wall was described as evenly spread across difficulty tiers ("so these are
not just the hard questions"). That compared raw counts across tiers of very
different sizes. Normalised, it takes 11.9% of simple (8/67), 16.3% of moderate
(16/98) and 23.5% of challenging (8/34) — it grows with difficulty. The point
the sentence was making still stands, but for the real reason: a quarter of the
wall sits on the simple tier, where the generator otherwise scores 76-88%.
The fine-tune section called the training prompt "exactly the production one".
The template is, but the schema block is raw DDL at training time and retrieval
cards with value samples at eval time — a known gap of this dataset version,
now stated instead of glossed.
Every other number in the file was re-derived from the raw reports: all 25
table entries, the 105/61/33 split, per-tier accuracy, repair counts, the wall
breakdown, and the paired fine-tune verdict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent c8ef8b2 commit e25ea48
1 file changed
Lines changed: 9 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
177 | | - | |
178 | | - | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
179 | 183 | | |
180 | 184 | | |
181 | 185 | | |
| |||
195 | 199 | | |
196 | 200 | | |
197 | 201 | | |
198 | | - | |
199 | | - | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
200 | 205 | | |
201 | 206 | | |
202 | 207 | | |
| |||
0 commit comments