Repository navigation
Commit f358e59
test(router): repeat the requests of each match bench so a compile can't land in CodSpeed's measured call (#345)
<!-- claude-projects-attribution: {"github_login":"dinwwwh"} -->
_Requested by **dinwwwh** · [project
thread](https://claude.ai/code/project/chan_01UcKMz91tMarPV1kWLVgLYE?thread=cmsg_01UcKMz91tMarPV1kWLVgLYEKguiPzeDgT5b6rRXC3tGer)_
## What
The two match benches (250 RPC-style and 703 REST-style procedures) now
run their request list 16 times per call, for both this router and rou3.
The add bench and the prefix bench are unchanged. This is a bench-only
change with no source changes.
## Why
In simulation mode, CodSpeed runs each bench function a few times as
warmup and then measures one call. The RPC match bench makes only 4
requests per call, about 30k instructions. A one-time V8 event, such as
a Sparkplug batch compile, can land in that call and swing the result by
up to 10x. This caused the false alarms on #233 (rou3 prefix -12.9%) and
#336 (RPC match x2.1).
I replayed the bench the way the plugin runs it. The replay used
`optimizeFunction`-style warmup calls, then `gc()`, then one call
between `InstrumentHooks.start/stopBenchmark`, under callgrind with
CodSpeed's analysis V8 flags (`--no-opt --predictable ...`).
Instructions in the measured call:
| warmup calls | 1 pass, before #336 | 1 pass, main | 16 passes, before
#336 | 16 passes, main |
| --- | --- | --- | --- | --- |
| 5 | 53,596 | 75,341 | 261,803 | 321,344 |
| 7 | 324,590 | 32,281 | 261,803 | 321,344 |
| 9 | 28,098 | 41,465 | 261,803 | 321,344 |
With one pass, the result depends on where the compile lands: in one
case 7 warmups read 10x higher than 9. With 16 passes, it is the same
for every warmup count.
The stable numbers also show something the noise hid. Since #336, this
request mix costs about 23% more in the tiers CodSpeed runs, because the
`POST /resource49/unknown` miss now splits the path and walks the tree.
In TurboFan that is about +450 instructions per miss on a method with
only static routes, which #336 accepted. So expect CodSpeed on this PR
to show every match bench about 16x slower, because each call does 16x
the work. Later PRs then compare against a stable baseline.
I tried the same check on the prefix bench, and it is already stable
with one pass (142,366 instructions for 5, 7 and 9 warmups). It stays as
it is.
## Checks
- `vitest bench --run packages/router` runs, and the hooks (oxlint,
oxfmt, tsc) pass.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01VVaw7PzysACDLzaReKDUKE)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 8a5abff commit f358e59
1 file changed
Lines changed: 14 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
18 | 24 | | |
19 | 25 | | |
20 | 26 | | |
| |||
37 | 43 | | |
38 | 44 | | |
39 | 45 | | |
40 | | - | |
41 | | - | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
42 | 50 | | |
43 | 51 | | |
44 | 52 | | |
45 | 53 | | |
46 | | - | |
47 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
48 | 58 | | |
49 | 59 | | |
50 | 60 | | |
| |||
0 commit comments