docs: measure the island finder end to end, and correct four stale numbers (#27) - #208
Merged
Merged
Conversation
wormeyman
added a commit
that referenced
this pull request
Aug 16, 2026
…mbers (#27) The end-to-end search time at the default 5,000-tile radius was recorded as UNMEASURED, with the design spec's ~15s standing in as a projection. Measured now, in Chrome 151 on a 12-core Mac against the deployed build 0a160df, seed 2967702466, Fulgora at default settings: three consecutive runs took 28,041 / 28,261 / 27,976 ms, a spread under 1%. So ~28s, which is 1.9x the estimate. That run did 2,335 units of work and returned 1,922 islands. The same run surfaced something the radius-600 test fixtures could not. The window-growth cap leaves 43 rows clipped, and they are not spread evenly: 43 of 1,922 rows is 2.2% overall, but 17 of the 50 refined rows (34%) and 5 of the top 10 (50%) carry the marker. The cap bites hardest on the biggest islands, which are the ones the tool exists to find, so several top rows report a lower bound rather than the whole island. The `!` marker is load-bearing, not decoration. Also corrects four numbers in CLAUDE.md that measurement contradicted: 1. The CI shard timings, and this one refuted its own first draft. #207's run came in at 391 / 378 / 469 / 400, which read as +80s (+21%) against the recorded 389s. Then #208 - this PR, docs-only, so the SAME 218 spec files and the same shard assignment - came in at 248 / 269 / 259 / 294. Binding shard 469s against 294s is a 59% swing on identical test code, and 294s is below the 389s being compared to. So the finder's cost is not resolvable from these runs at all, and the section says that rather than picking the number that was measured first. 2. `verify`'s own runtime, recorded as ~65-90s and measured at 3m28s. This line has now been wrong twice in the same direction, so it says so and says to treat the figure as perishable. 3. The timeout-annotation counts, which were stale on the day they were last corrected: 86/29 recorded against a real 89/30, and now 94/31. Adds the one-line command to re-count instead of quoting. 4. "300s is one file's exception, as of #203". Wrong on both halves - 17 files use 300s and the practice arrived with the cliff work in #122. The claim it supported, that an 85.2s case in vulcanusCliffRejectionStage is nearest the 120s edge, is void too: that file carries zero 120s annotations and three 300s ones. Which test now sits nearest its budget is left explicitly open rather than replaced with a guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0114mWC72TAgZzNjMejBpdTt
wormeyman
force-pushed
the
docs/island-finder-measurements
branch
from
August 16, 2026 04:30
63bbf9e to
db8cf9e
Compare
…mbers (#27) The end-to-end search time at the default 5,000-tile radius was recorded as UNMEASURED, with the design spec's ~15s standing in as a projection. Measured now, in Chrome 151 on a 12-core Mac against the deployed build 0a160df, seed 2967702466, Fulgora at default settings: three consecutive runs took 28,041 / 28,261 / 27,976 ms, a spread under 1%. So ~28s, which is 1.9x the estimate. That run did 2,335 units of work and returned 1,922 islands. The same run surfaced something the radius-600 test fixtures could not. The window-growth cap leaves 43 rows clipped, and they are not spread evenly: 43 of 1,922 rows is 2.2% overall, but 17 of the 50 refined rows (34%) and 5 of the top 10 (50%) carry the marker. The cap bites hardest on the biggest islands, which are the ones the tool exists to find, so several top rows report a lower bound rather than the whole island. The `!` marker is load-bearing, not decoration. Also corrects four numbers in CLAUDE.md that measurement contradicted: 1. The CI shard timings, and this one refuted its own first draft. #207's run came in at 391 / 378 / 469 / 400, which read as +80s (+21%) against the recorded 389s. Then #208 - this PR, docs-only, so the SAME 218 spec files and the same shard assignment - came in at 248 / 269 / 259 / 294. Binding shard 469s against 294s is a 59% swing on identical test code, and 294s is below the 389s being compared to. So the finder's cost is not resolvable from these runs at all, and the section says that rather than picking the number that was measured first. 2. `verify`'s own runtime, recorded as ~65-90s and measured at 3m28s. This line has now been wrong twice in the same direction, so it says so and says to treat the figure as perishable. 3. The timeout-annotation counts, which were stale on the day they were last corrected: 86/29 recorded against a real 89/30, and now 94/31. Adds the one-line command to re-count instead of quoting. 4. "300s is one file's exception, as of #203". Wrong on both halves - 17 files use 300s and the practice arrived with the cliff work in #122. The claim it supported, that an 85.2s case in vulcanusCliffRejectionStage is nearest the 120s edge, is void too: that file carries zero 120s annotations and three 300s ones. Which test now sits nearest its budget is left explicitly open rather than replaced with a guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0114mWC72TAgZzNjMejBpdTt
wormeyman
force-pushed
the
docs/island-finder-measurements
branch
from
August 16, 2026 04:38
db8cf9e to
34ad03c
Compare
wormeyman
added a commit
that referenced
this pull request
Aug 25, 2026
* Trim CLAUDE.md back under the 150k limit Claude Code was refusing to load the whole file: 179,094 chars against a 150,000 limit. It is 131,561 now, a 27% cut, with ~18k of headroom. Nothing was dropped. The two longest sections were snapshotted verbatim into docs/ and rewritten in place as live rules plus a pointer: - docs/rust-wasm-port-history.md - the Rust/WASM port's measurement record. That section was 77,578 chars, 43% of the file on its own, and most of its length was the story of how a count moved rather than a rule anyone needs at the keyboard: the superseded before-and-after tables from #269/#290/#293, the twelve rejected sweep candidates from #273, and the per-phase landing lists. - docs/ci-sharding-measurements.md - every CI timing behind the four-shard layout: the N=3 through N=6 comparison, the per-run tables from #202, #203, #207 and #208, and the three reasons splitting a heavy spec file was rejected. Every trap stays in CLAUDE.md: the capture-grid snap, the poison-hook doctrine, the three engine.wasm diff fingerprints, aux.rs on Windows, f64::max against Math.max, the exchange-format drift, pnpm's minimumReleaseAge reasoning, and the ruleset hazard around the verify job's name. Three things were corrected while rewriting rather than carried over: - The Rust heading said "phases 1-4 landed, phase 5 in progress". Phase 5 is complete and phase 6 is half done. - The CI job table costed `rust` at 19s. It has been 1m45s-2m50s since #225's cliff half landed, and it is a range because a single run measures the runner. - The `maxUnknown: 0` note had drifted into the Rust section. It is fixture provenance, so it moved to that section. Flesch-Kincaid on the rewritten prose is 8.3, down from 9.0. `pnpm vp check` passes on 436 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hVzDgUq6g7LATfczGxUrE * Point the oracle sections at the oracle repo instead of restating it The two factorio-oracle sections were 13,061 chars and most of that restated things ~/GitHub/factorio-oracle now documents itself: every `refs` subcommand (its README), the `oracle-dump.json` name contract and `error("DUMPED-OK")` exiting non-zero as success (docs/gotchas.md), the binary shipping unstripped (docs/order-of-attack.md), and how to write a probe (README again). They are 6,893 chars now and name those four documents with full `~/GitHub/factorio-oracle/` paths, so a bare `docs/gotchas.md` cannot be mistaken for a file in this repo. docs/factorio-reference-and-oracle.md carries the verbatim long form, including the full WSL capture recipe. What stayed is what the oracle repo does not know about this one: - Which API-docs entry points this app's codec depends on, and that the JSON dumps are NOT a superset of the HTML - `control:temperature:frequency` is in noise-expressions.html and nowhere in runtime-api.json. - The map-gen Lua files the preview ports, the `name = "<expression>"` grep, and the starting_patches 2.0.77 -> 2.1.9 skew that produces a port which passes its own tests and disagrees with the game. - `pnpm refs:sync` pinning nothing, and why it is not in `verify`. - The "new probes only" rule for test/oracle/, and the two worked probes in scripts/probes/. - That the installed binary is the authority on which version is meant, and Steam moves it. CLAUDE.md is 126,713 chars now, down from 179,094 - a 29% cut with 23k of headroom under the 150k limit. Flesch-Kincaid on the rewritten prose is 8.2. `pnpm vp check` passes on 436 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hVzDgUq6g7LATfczGxUrE * Ask the binary for CLI options, not the wiki `factorio --help` prints every option and ships with the game, so it describes the version you actually have. It is also ahead of the wiki: it documents `--map-preview-planet`, `--map-gen-seed-max` and `--exchange-string`, and it states outright that `--map-gen-seed` "will override seed specified in map gen settings" - the trap #232 hit, answered by the tool itself. The wiki page stays named as a fallback for prose the help text does not carry. The docs snapshot is verbatim, so it keeps the old wording and its header now flags that one line as superseded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hVzDgUq6g7LATfczGxUrE --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs only. Closes the one deferred item from #207 and fixes four numbers that measurement contradicted - including one this PR refuted on its own CI run.
The end-to-end search time is no longer a projection
The roadmap recorded it as unmeasured, with the design spec's ~15s standing in. Measured in Chrome 151 on a 12-core Mac against the deployed build
0a160df, seed 2967702466, Fulgora at default settings, radius 5,000:Spread under 1%, so this is a stable figure rather than one lucky draw. That search did 2,335 units of work (2,285 coarse candidates plus 50 refines) and returned 1,922 islands. ~28s, which is 1.9x the estimate. The estimate was not wrong in kind - the search finishes while you wait - but the roadmap now says 28s and names the two limits on that number (one machine, and concurrency follows the worker pool, so fewer cores will be slower).
The growth cap bites where it matters most
The same run surfaced something the radius-600 fixtures could not show. 43 rows come back clipped, and they are not spread evenly:
So the headline rectangle on several top rows is a lower bound, not a measurement of the whole island. The
!marker is load-bearing rather than decoration. RaisingMAX_WINDOW_GROWTHSis the obvious lever and it is not free - each growth quadruples that candidate's window area, on a search already taking ~28s.Four corrections to CLAUDE.md
1. The CI shard timings - and this correction refuted its own first draft. #207's run came in at 391 / 378 / 469 / 400, which read as +80s (+21%) against the recorded 389s, and that is what this PR originally said. Then this PR's own run - docs-only, so the same 218 spec files and the same shard assignment - came in at 248 / 269 / 259 / 294.
A 59% swing on identical test code, and 294s is below the 389s being compared against. So the island finder's effect on the gate is not resolvable from these runs, and the section now says so instead of keeping the number that happened to be measured first. This is the #202/#203 lesson arriving a second time.
What survives is the local measurement, where the spread is small:
test/findIslands.spec.tsis the new heaviest file at 134.6s, down from 240.4s before four of its tests were cut to a smallrefineCount.2.
verify's own runtime, recorded as ~65-90s and measured at 3m28s. That line has now been wrong twice in the same direction (it said ~9.5s before), so it says so and marks the number perishable.3. The timeout-annotation counts, which were stale on the same day they were last corrected - 86/29 recorded against a real 89/30, now 94/31 and 74/17. Replaced with the one-line command to re-count.
4. "300s is one file's exception, as of #203." Wrong on both halves: 17 files use 300s, and the practice arrived with the cliff work in #122. The claim it supported - that an 85.2s case in
vulcanusCliffRejectionStage.spec.tssits nearest the 120s edge - is void too, because that file carries zero 120s annotations and three 300s ones. Which test now sits nearest its budget is left explicitly open rather than replaced with a guess; answering it needs a per-test read off a CI run, not a grep.Prose measures Flesch-Kincaid grade 8.25.
🤖 Generated with Claude Code
https://claude.ai/code/session_0114mWC72TAgZzNjMejBpdTt