Skip to content

docs: measure the island finder end to end, and correct four stale numbers (#27) #208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wormeyman merged 1 commit into from
Aug 16, 2026
Merged

docs: measure the island finder end to end, and correct four stale numbers (#27) #208

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 76 additions & 27 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,14 +248,16 @@ pnpm vp dev --port 5199 --strictPort # expect a Local: URL, not a picker or ex
Docker at all, which is what makes the CI workflow possible. Auto-start is
opt-in behind `FMW_AUTO_START_DOCKER=1`.
- `pnpm run verify` - `verify:lint` + `vp test` + `preview:test` in
one gate. **~65-90s on a dev machine.** On a runner it is no longer one job -
see the CI section, which shards it. The `~9.5s` this line
claimed for a long time was simply wrong - already wrong by a factor of six
before `check:vue` existed, because the suite grew through the Vulcanus and
cliff work (143 files then; **152 as of 2026-08-01**, which is what the
`test/**/*.spec.ts` include actually matches) - and the gap mattered: ~63s is exactly the duration
at which people start skipping a manual gate, which is half the argument for
the CI workflow below. Don't budget 10 seconds for this.
one gate. **~3m30s cold on a dev machine** (measured 2026-08-15 at #207:
3m28s wall, 218 test files, 1,922 tests). On a runner it is no longer one job -
see the CI section, which shards it. This line has been wrong twice and in the
same direction, so treat the number as perishable. It claimed `~9.5s` for a
long time - wrong by a factor of six even before `check:vue` existed, because
the suite grew through the Vulcanus and cliff work. It was then corrected to
`~65-90s`, which the island finder (#207) invalidated within two weeks by
adding one 134.6s spec file. The gap matters both times: a gate people believe
is instant and is not is a gate they stop running, which is half the argument
for the CI workflow below. Don't budget seconds for this; budget minutes.

The test phase runs through **`vp run --cache test`**, not a bare `vp test`.
Measured 2026-08-02: the four phases are `vp check` 2.0s, `check:vue` 3.0s,
Expand Down Expand Up @@ -456,6 +458,42 @@ against **366 / 273 / 327 / 137** - shard 1 up 41%, shard 3 down 16%, and the
binding shard changed identity from 3 to 1. Any rebalancing worth doing has to
beat that, and a single run cannot show that it did.

**Re-measured on CI 2026-08-15 after the Fulgora island finder (#207) - three
runs over the SAME 218 spec files, and they disagree by more than the effect
anyone would want to read out of them.** #207 and both runs of #208 (docs-only,
so identical spec files and identical shard assignment):

| run | shards (s) | binding |
| ----------------- | --------------------- | ------- |
| #207 | 391 / 378 / 469 / 400 | 469 |
| #208 first | 248 / 269 / 259 / 294 | **294** |
| #208 after a redo | 366 / 281 / 416 / 368 | 416 |

Range **294-469 on identical test code**, a 59% spread, against a recorded 389s
for #202. Two of the three sit above 389 and one sits well below it, so the
finder probably did add gate wall - but the noise is the same size as the thing
being measured, and no honest point estimate comes out of this.

Note how that table was built, because it is the cheapest way to get one: three
runs of the same tests arrived for free from one PR's normal life (open, amend,
push). If a number here matters, collect it that way rather than from whichever
run you happened to look at. The first draft of this very paragraph read
"+80s, +21%" off #207 alone, and the next run refuted it.

That is the #202/#203 lesson arriving a second time, and it should be the
default assumption now: a single CI run here measures the runner as much as the
suite. Do not tune on one.

What IS solid, because it was measured locally where the spread is small:
**`test/findIslands.spec.ts` is the new heaviest file at 134.6s**, taking the
crown from `previewAgreement.spec.ts`. Read that with its history - it was
**240.4s** when the branch's last fix landed, and four of its tests were then
cut to a small `refineCount` for identical coverage. Why it is expensive at all:
the finder re-renders a candidate at a doubled pad whenever its island mask
touches the window border, and refinement pays 16x the pixels of the coarse
pass. One test in that file **cannot** be cheapened the same way and its own
comment explains why, so do not "finish the job" by lowering its refine count.

So the gate wall stays where it is, and the thing that actually broke was a
**timeout, not the wall**. On #203 - a docs-only change - the unchanged
"Vulcanus rock and cliff coverage" test hit its 120s budget at 150.5s. Across
Expand Down Expand Up @@ -621,25 +659,36 @@ wrong belief that a green suite endorsed.

#### `testTimeout` is 30s, deliberately, and retries are not used

Vitest's 5s default was too tight for this suite long before CI existed - **86
individual tests across 29 files** carry an explicit `}, 120000)`, and five more
in `previewAgreement.spec.ts` now carry `}, 300000)`. That is the same complaint
made 91 times by hand. (The count read "24 across 10" for a year and went stale
as the suite grew; re-counted 2026-08-15.) The first CI run
proved the default was the real problem rather than any one test: on a 4-core
runner (~3x slower, 230s vs 71s for the same suite)
`elevationRenderRequest.spec.ts`'s `view 'all'` case needs **9.8s**, and that
file has 27 tests and zero annotations. `vite.config.ts` now sets
`testTimeout: 30_000`; the existing annotations still win over it, so raising
the global does nothing for any of those 91 tests.

**120000 is not a safe ceiling any more, and only one file has been moved off
it.** `previewAgreement.spec.ts` uses 300s as of #203, after its heaviest case
timed out at 150.5s (see the CI section above for the four-run table). The rest
still sit at 120s, and the nearest to the edge is an 85.2s case in
`vulcanusCliffRejectionStage.spec.ts`. At the ~40% run-to-run spread measured on
these runners, 85.2s is about one bad draw from red. Nothing is broken there
today - this is where to look first if a shard goes red on a timeout.
Vitest's 5s default was too tight for this suite long before CI existed. Counted
on `test/*.spec.ts` at #207 (2026-08-15): **94 tests across 31 files** carry an
explicit `}, 120000)`, and **74 tests across 17 files** carry `}, 300000)`. That
is the same complaint made 168 times by hand. The first CI run proved the
default was the real problem rather than any one test: on a 4-core runner (~3x
slower, 230s vs 71s for the same suite) `elevationRenderRequest.spec.ts`'s
`view 'all'` case needs **9.8s**, and that file has 27 tests and zero
annotations. `vite.config.ts` now sets `testTimeout: 30_000`; the existing
annotations still win over it, so raising the global does nothing for any of
those 168 tests.

**Do not trust a hand-maintained count here - this one has now gone stale
twice.** It read "24 across 10" for a year, was corrected to "86 across 29" on
2026-08-15, and was still wrong the same day: the real figures were 89/30 and
66/16 before #207 even landed. Re-count before quoting:

```bash
git grep -c '}, 120000)' -- 'test/*.spec.ts' | awk -F: '{s+=$3} END {print s}'
```

**120000 is not a safe ceiling, and 300s is not one file's exception.** This
paragraph used to say `previewAgreement.spec.ts` took 300s "as of #203" and that
it was the only file moved off 120s. Both halves are wrong. 17 files use 300s,
and the practice long predates #203 - the earliest arrived with the cliff work
in #122. It also named an 85.2s case in `vulcanusCliffRejectionStage.spec.ts` as
the nearest to the edge at 120s; that file carries **zero** 120s annotations and
three 300s ones, so the claim's premise is void. Which test now sits nearest its
own budget has not been re-derived - it needs a fresh per-test read off a CI run,
not a grep. Treat that as an open question, not a settled one, if a shard goes
red on a timeout.

Do **not** reach for `retry` when a heavy render test fails in CI. Nothing here is
nondeterministic - these tests compare pixels against captured game output - so a
Expand Down
40 changes: 30 additions & 10 deletions docs/noise/client-preview-ROADMAP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -601,13 +601,23 @@ Done = ore patches overlaid on land, responding to the frequency/size/richness s
`"all"` - see `findIslands.ts`'s header for why) costs **~24 ms** in
this Node test harness.

**The full end-to-end search time for the default 5,000-tile radius is
unmeasured.** This perf block times the two per-unit costs above, not
the whole survey -> coarse -> refine -> chain pipeline run across the
app's worker pool. The design spec's own ~15s figure for that case
(section 4) is a design-time estimate from a throwaway benchmark, not a
number this task measured - treat it as unconfirmed until a real
end-to-end run is timed and recorded here.
**The full end-to-end search time is now MEASURED: ~28 seconds** at the
default 5,000-tile radius. Taken 2026-08-15 in Chrome 151 on a 12-core
Mac, against the deployed build 0a160df, with seed 2967702466 and
Fulgora at default settings. Three consecutive runs took 28,041 /
28,261 / 27,976 ms - a spread under 1%, so this is a stable figure and
not one lucky draw. That search did 2,335 units of work (2,285 coarse
candidates plus 50 refines) and returned 1,922 islands.

That is **1.9x the design spec's ~15s estimate** (section 4), which was
always a design-time projection off a throwaway benchmark. The estimate
was not wrong in kind - the search still finishes while you wait, and
the progress readout counts the whole way - but quote 28s, not 15s.

Two limits on that number. It came from a 12-core machine, and the
search's concurrency follows the worker pool, so a 4-core laptop should
be expected to take considerably longer. And it is one machine and one
browser, so it bounds nothing; it is a reference point.

**The chain stage is NOT negligible, and now has a measured figure.**
The design spec's own estimate (section 4) called stage 4 (dedup +
Expand Down Expand Up @@ -637,9 +647,19 @@ Done = ore patches overlaid on land, responding to the frequency/size/richness s
reported rectangle is a real measurement of a truncated slice, not the
whole island, and `IslandResult.clipped` records that so the panel can
say so (the `!` marker beside a clipped row's rectangle, next to the
existing `~` marker for an unrefined one). No case in the current
seed/radius test fixtures needs more than pad 256, but nothing rules
one out at a larger radius.
existing `~` marker for an unrefined one).

**At the default radius, clipped rows are common, and they cluster at
the top.** The radius-600 test fixtures made this look like a corner
case. The radius-5,000 run timed above returned 43 clipped rows, spread
very unevenly: 43 of 1,922 rows is 2.2% overall, but **17 of the 50
refined rows (34%) and 5 of the top 10 (50%)** carry the marker. The cap
bites hardest on the biggest islands, which are the ones the tool exists
to find, so the headline rectangle on several top rows is a lower bound
rather than a measurement of the whole island. The `!` marker is
therefore load-bearing, not decoration. Raising `MAX_WINDOW_GROWTHS` is
the obvious lever and it is not free - each growth quadruples that
candidate's window area, on a search that already takes ~28s.

**Accuracy**: Fulgora's land/ocean split agrees with the real game on
99.86% of positions (Fulgora V1, above), and the residual mismatches
Expand Down