Skip to content

Commit 34ad03c

Browse files
wormeymanclaude
andcommitted
docs: measure the island finder end to end, and correct four stale numbers (#27)
The end-to-end search time at the default 5,000-tile radius was recorded as UNMEASURED, with the design spec's ~15s standing in as a projection. Measured now, in Chrome 151 on a 12-core Mac against the deployed build 0a160df, seed 2967702466, Fulgora at default settings: three consecutive runs took 28,041 / 28,261 / 27,976 ms, a spread under 1%. So ~28s, which is 1.9x the estimate. That run did 2,335 units of work and returned 1,922 islands. The same run surfaced something the radius-600 test fixtures could not. The window-growth cap leaves 43 rows clipped, and they are not spread evenly: 43 of 1,922 rows is 2.2% overall, but 17 of the 50 refined rows (34%) and 5 of the top 10 (50%) carry the marker. The cap bites hardest on the biggest islands, which are the ones the tool exists to find, so several top rows report a lower bound rather than the whole island. The `!` marker is load-bearing, not decoration. Also corrects four numbers in CLAUDE.md that measurement contradicted: 1. The CI shard timings, and this one refuted its own first draft. #207's run came in at 391 / 378 / 469 / 400, which read as +80s (+21%) against the recorded 389s. Then #208 - this PR, docs-only, so the SAME 218 spec files and the same shard assignment - came in at 248 / 269 / 259 / 294. Binding shard 469s against 294s is a 59% swing on identical test code, and 294s is below the 389s being compared to. So the finder's cost is not resolvable from these runs at all, and the section says that rather than picking the number that was measured first. 2. `verify`'s own runtime, recorded as ~65-90s and measured at 3m28s. This line has now been wrong twice in the same direction, so it says so and says to treat the figure as perishable. 3. The timeout-annotation counts, which were stale on the day they were last corrected: 86/29 recorded against a real 89/30, and now 94/31. Adds the one-line command to re-count instead of quoting. 4. "300s is one file's exception, as of #203". Wrong on both halves - 17 files use 300s and the practice arrived with the cliff work in #122. The claim it supported, that an 85.2s case in vulcanusCliffRejectionStage is nearest the 120s edge, is void too: that file carries zero 120s annotations and three 300s ones. Which test now sits nearest its budget is left explicitly open rather than replaced with a guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0114mWC72TAgZzNjMejBpdTt
1 parent 0a160df commit 34ad03c

2 files changed

Lines changed: 106 additions & 37 deletions

File tree

CLAUDE.md

Lines changed: 76 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -248,14 +248,16 @@ pnpm vp dev --port 5199 --strictPort # expect a Local: URL, not a picker or ex
248248
Docker at all, which is what makes the CI workflow possible. Auto-start is
249249
opt-in behind `FMW_AUTO_START_DOCKER=1`.
250250
- `pnpm run verify` - `verify:lint` + `vp test` + `preview:test` in
251-
one gate. **~65-90s on a dev machine.** On a runner it is no longer one job -
252-
see the CI section, which shards it. The `~9.5s` this line
253-
claimed for a long time was simply wrong - already wrong by a factor of six
254-
before `check:vue` existed, because the suite grew through the Vulcanus and
255-
cliff work (143 files then; **152 as of 2026-08-01**, which is what the
256-
`test/**/*.spec.ts` include actually matches) - and the gap mattered: ~63s is exactly the duration
257-
at which people start skipping a manual gate, which is half the argument for
258-
the CI workflow below. Don't budget 10 seconds for this.
251+
one gate. **~3m30s cold on a dev machine** (measured 2026-08-15 at #207:
252+
3m28s wall, 218 test files, 1,922 tests). On a runner it is no longer one job -
253+
see the CI section, which shards it. This line has been wrong twice and in the
254+
same direction, so treat the number as perishable. It claimed `~9.5s` for a
255+
long time - wrong by a factor of six even before `check:vue` existed, because
256+
the suite grew through the Vulcanus and cliff work. It was then corrected to
257+
`~65-90s`, which the island finder (#207) invalidated within two weeks by
258+
adding one 134.6s spec file. The gap matters both times: a gate people believe
259+
is instant and is not is a gate they stop running, which is half the argument
260+
for the CI workflow below. Don't budget seconds for this; budget minutes.
259261

260262
The test phase runs through **`vp run --cache test`**, not a bare `vp test`.
261263
Measured 2026-08-02: the four phases are `vp check` 2.0s, `check:vue` 3.0s,
@@ -456,6 +458,42 @@ against **366 / 273 / 327 / 137** - shard 1 up 41%, shard 3 down 16%, and the
456458
binding shard changed identity from 3 to 1. Any rebalancing worth doing has to
457459
beat that, and a single run cannot show that it did.
458460

461+
**Re-measured on CI 2026-08-15 after the Fulgora island finder (#207) - three
462+
runs over the SAME 218 spec files, and they disagree by more than the effect
463+
anyone would want to read out of them.** #207 and both runs of #208 (docs-only,
464+
so identical spec files and identical shard assignment):
465+
466+
| run | shards (s) | binding |
467+
| ----------------- | --------------------- | ------- |
468+
| #207 | 391 / 378 / 469 / 400 | 469 |
469+
| #208 first | 248 / 269 / 259 / 294 | **294** |
470+
| #208 after a redo | 366 / 281 / 416 / 368 | 416 |
471+
472+
Range **294-469 on identical test code**, a 59% spread, against a recorded 389s
473+
for #202. Two of the three sit above 389 and one sits well below it, so the
474+
finder probably did add gate wall - but the noise is the same size as the thing
475+
being measured, and no honest point estimate comes out of this.
476+
477+
Note how that table was built, because it is the cheapest way to get one: three
478+
runs of the same tests arrived for free from one PR's normal life (open, amend,
479+
push). If a number here matters, collect it that way rather than from whichever
480+
run you happened to look at. The first draft of this very paragraph read
481+
"+80s, +21%" off #207 alone, and the next run refuted it.
482+
483+
That is the #202/#203 lesson arriving a second time, and it should be the
484+
default assumption now: a single CI run here measures the runner as much as the
485+
suite. Do not tune on one.
486+
487+
What IS solid, because it was measured locally where the spread is small:
488+
**`test/findIslands.spec.ts` is the new heaviest file at 134.6s**, taking the
489+
crown from `previewAgreement.spec.ts`. Read that with its history - it was
490+
**240.4s** when the branch's last fix landed, and four of its tests were then
491+
cut to a small `refineCount` for identical coverage. Why it is expensive at all:
492+
the finder re-renders a candidate at a doubled pad whenever its island mask
493+
touches the window border, and refinement pays 16x the pixels of the coarse
494+
pass. One test in that file **cannot** be cheapened the same way and its own
495+
comment explains why, so do not "finish the job" by lowering its refine count.
496+
459497
So the gate wall stays where it is, and the thing that actually broke was a
460498
**timeout, not the wall**. On #203 - a docs-only change - the unchanged
461499
"Vulcanus rock and cliff coverage" test hit its 120s budget at 150.5s. Across
@@ -621,25 +659,36 @@ wrong belief that a green suite endorsed.
621659

622660
#### `testTimeout` is 30s, deliberately, and retries are not used
623661

624-
Vitest's 5s default was too tight for this suite long before CI existed - **86
625-
individual tests across 29 files** carry an explicit `}, 120000)`, and five more
626-
in `previewAgreement.spec.ts` now carry `}, 300000)`. That is the same complaint
627-
made 91 times by hand. (The count read "24 across 10" for a year and went stale
628-
as the suite grew; re-counted 2026-08-15.) The first CI run
629-
proved the default was the real problem rather than any one test: on a 4-core
630-
runner (~3x slower, 230s vs 71s for the same suite)
631-
`elevationRenderRequest.spec.ts`'s `view 'all'` case needs **9.8s**, and that
632-
file has 27 tests and zero annotations. `vite.config.ts` now sets
633-
`testTimeout: 30_000`; the existing annotations still win over it, so raising
634-
the global does nothing for any of those 91 tests.
635-
636-
**120000 is not a safe ceiling any more, and only one file has been moved off
637-
it.** `previewAgreement.spec.ts` uses 300s as of #203, after its heaviest case
638-
timed out at 150.5s (see the CI section above for the four-run table). The rest
639-
still sit at 120s, and the nearest to the edge is an 85.2s case in
640-
`vulcanusCliffRejectionStage.spec.ts`. At the ~40% run-to-run spread measured on
641-
these runners, 85.2s is about one bad draw from red. Nothing is broken there
642-
today - this is where to look first if a shard goes red on a timeout.
662+
Vitest's 5s default was too tight for this suite long before CI existed. Counted
663+
on `test/*.spec.ts` at #207 (2026-08-15): **94 tests across 31 files** carry an
664+
explicit `}, 120000)`, and **74 tests across 17 files** carry `}, 300000)`. That
665+
is the same complaint made 168 times by hand. The first CI run proved the
666+
default was the real problem rather than any one test: on a 4-core runner (~3x
667+
slower, 230s vs 71s for the same suite) `elevationRenderRequest.spec.ts`'s
668+
`view 'all'` case needs **9.8s**, and that file has 27 tests and zero
669+
annotations. `vite.config.ts` now sets `testTimeout: 30_000`; the existing
670+
annotations still win over it, so raising the global does nothing for any of
671+
those 168 tests.
672+
673+
**Do not trust a hand-maintained count here - this one has now gone stale
674+
twice.** It read "24 across 10" for a year, was corrected to "86 across 29" on
675+
2026-08-15, and was still wrong the same day: the real figures were 89/30 and
676+
66/16 before #207 even landed. Re-count before quoting:
677+
678+
```bash
679+
git grep -c '}, 120000)' -- 'test/*.spec.ts' | awk -F: '{s+=$3} END {print s}'
680+
```
681+
682+
**120000 is not a safe ceiling, and 300s is not one file's exception.** This
683+
paragraph used to say `previewAgreement.spec.ts` took 300s "as of #203" and that
684+
it was the only file moved off 120s. Both halves are wrong. 17 files use 300s,
685+
and the practice long predates #203 - the earliest arrived with the cliff work
686+
in #122. It also named an 85.2s case in `vulcanusCliffRejectionStage.spec.ts` as
687+
the nearest to the edge at 120s; that file carries **zero** 120s annotations and
688+
three 300s ones, so the claim's premise is void. Which test now sits nearest its
689+
own budget has not been re-derived - it needs a fresh per-test read off a CI run,
690+
not a grep. Treat that as an open question, not a settled one, if a shard goes
691+
red on a timeout.
643692

644693
Do **not** reach for `retry` when a heavy render test fails in CI. Nothing here is
645694
nondeterministic - these tests compare pixels against captured game output - so a

docs/noise/client-preview-ROADMAP.md

Lines changed: 30 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -601,13 +601,23 @@ Done = ore patches overlaid on land, responding to the frequency/size/richness s
601601
`"all"` - see `findIslands.ts`'s header for why) costs **~24 ms** in
602602
this Node test harness.
603603

604-
**The full end-to-end search time for the default 5,000-tile radius is
605-
unmeasured.** This perf block times the two per-unit costs above, not
606-
the whole survey -> coarse -> refine -> chain pipeline run across the
607-
app's worker pool. The design spec's own ~15s figure for that case
608-
(section 4) is a design-time estimate from a throwaway benchmark, not a
609-
number this task measured - treat it as unconfirmed until a real
610-
end-to-end run is timed and recorded here.
604+
**The full end-to-end search time is now MEASURED: ~28 seconds** at the
605+
default 5,000-tile radius. Taken 2026-08-15 in Chrome 151 on a 12-core
606+
Mac, against the deployed build 0a160df, with seed 2967702466 and
607+
Fulgora at default settings. Three consecutive runs took 28,041 /
608+
28,261 / 27,976 ms - a spread under 1%, so this is a stable figure and
609+
not one lucky draw. That search did 2,335 units of work (2,285 coarse
610+
candidates plus 50 refines) and returned 1,922 islands.
611+
612+
That is **1.9x the design spec's ~15s estimate** (section 4), which was
613+
always a design-time projection off a throwaway benchmark. The estimate
614+
was not wrong in kind - the search still finishes while you wait, and
615+
the progress readout counts the whole way - but quote 28s, not 15s.
616+
617+
Two limits on that number. It came from a 12-core machine, and the
618+
search's concurrency follows the worker pool, so a 4-core laptop should
619+
be expected to take considerably longer. And it is one machine and one
620+
browser, so it bounds nothing; it is a reference point.
611621

612622
**The chain stage is NOT negligible, and now has a measured figure.**
613623
The design spec's own estimate (section 4) called stage 4 (dedup +
@@ -637,9 +647,19 @@ Done = ore patches overlaid on land, responding to the frequency/size/richness s
637647
reported rectangle is a real measurement of a truncated slice, not the
638648
whole island, and `IslandResult.clipped` records that so the panel can
639649
say so (the `!` marker beside a clipped row's rectangle, next to the
640-
existing `~` marker for an unrefined one). No case in the current
641-
seed/radius test fixtures needs more than pad 256, but nothing rules
642-
one out at a larger radius.
650+
existing `~` marker for an unrefined one).
651+
652+
**At the default radius, clipped rows are common, and they cluster at
653+
the top.** The radius-600 test fixtures made this look like a corner
654+
case. The radius-5,000 run timed above returned 43 clipped rows, spread
655+
very unevenly: 43 of 1,922 rows is 2.2% overall, but **17 of the 50
656+
refined rows (34%) and 5 of the top 10 (50%)** carry the marker. The cap
657+
bites hardest on the biggest islands, which are the ones the tool exists
658+
to find, so the headline rectangle on several top rows is a lower bound
659+
rather than a measurement of the whole island. The `!` marker is
660+
therefore load-bearing, not decoration. Raising `MAX_WINDOW_GROWTHS` is
661+
the obvious lever and it is not free - each growth quadruples that
662+
candidate's window area, on a search that already takes ~28s.
643663

644664
**Accuracy**: Fulgora's land/ocean split agrees with the real game on
645665
99.86% of positions (Fulgora V1, above), and the residual mismatches

0 commit comments

Comments
 (0)