Skip to content

Commit f358e59

Browse files
dinwwwhclaude
andauthored
test(router): repeat the requests of each match bench so a compile can't land in CodSpeed's measured call (#345)
<!-- claude-projects-attribution: {"github_login":"dinwwwh"} --> _Requested by **dinwwwh** · [project thread](https://claude.ai/code/project/chan_01UcKMz91tMarPV1kWLVgLYE?thread=cmsg_01UcKMz91tMarPV1kWLVgLYEKguiPzeDgT5b6rRXC3tGer)_ ## What The two match benches (250 RPC-style and 703 REST-style procedures) now run their request list 16 times per call, for both this router and rou3. The add bench and the prefix bench are unchanged. This is a bench-only change with no source changes. ## Why In simulation mode, CodSpeed runs each bench function a few times as warmup and then measures one call. The RPC match bench makes only 4 requests per call, about 30k instructions. A one-time V8 event, such as a Sparkplug batch compile, can land in that call and swing the result by up to 10x. This caused the false alarms on #233 (rou3 prefix -12.9%) and #336 (RPC match x2.1). I replayed the bench the way the plugin runs it. The replay used `optimizeFunction`-style warmup calls, then `gc()`, then one call between `InstrumentHooks.start/stopBenchmark`, under callgrind with CodSpeed's analysis V8 flags (`--no-opt --predictable ...`). Instructions in the measured call: | warmup calls | 1 pass, before #336 | 1 pass, main | 16 passes, before #336 | 16 passes, main | | --- | --- | --- | --- | --- | | 5 | 53,596 | 75,341 | 261,803 | 321,344 | | 7 | 324,590 | 32,281 | 261,803 | 321,344 | | 9 | 28,098 | 41,465 | 261,803 | 321,344 | With one pass, the result depends on where the compile lands: in one case 7 warmups read 10x higher than 9. With 16 passes, it is the same for every warmup count. The stable numbers also show something the noise hid. Since #336, this request mix costs about 23% more in the tiers CodSpeed runs, because the `POST /resource49/unknown` miss now splits the path and walks the tree. In TurboFan that is about +450 instructions per miss on a method with only static routes, which #336 accepted. So expect CodSpeed on this PR to show every match bench about 16x slower, because each call does 16x the work. Later PRs then compare against a stable baseline. I tried the same check on the prefix bench, and it is already stable with one pass (142,366 instructions for 5, 7 and 9 warmups). It stays as it is. ## Checks - `vitest bench --run packages/router` runs, and the hooks (oxlint, oxfmt, tsc) pass. --- _Generated by [Claude Code](https://claude.ai/code/session_01VVaw7PzysACDLzaReKDUKE)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 8a5abff commit f358e59

1 file changed

Lines changed: 14 additions & 4 deletions

File tree

‎packages/router/benches/router.bench.ts‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,12 @@ import {
1515
// rou3 runs as is, without the percent-decoding oRPC does around it today,
1616
// so each comparison is against less work than this router does.
1717

18+
// CodSpeed measures one call of each bench after a few warmup calls. Repeating
19+
// the few requests of a match bench in each call lets V8 finish compiling them
20+
// during the warmup, so a compile can't land in the measured call and swing its
21+
// result many times.
22+
const REPEAT = 16
23+
1824
function createOurs(routes: Routes): Router<string> {
1925
const router = new Router<string>()
2026
for (const [method, path] of routes) {
@@ -37,14 +43,18 @@ function compare(name: string, routes: Routes, requests: Requests): void {
3743
const rou3 = createRou3(routes)
3844

3945
bench('@openapi-spec/router', () => {
40-
for (const [method, pathname] of requests) {
41-
ours.match(method, fresh(pathname))
46+
for (let i = 0; i < REPEAT; i++) {
47+
for (const [method, pathname] of requests) {
48+
ours.match(method, fresh(pathname))
49+
}
4250
}
4351
})
4452

4553
bench('rou3', () => {
46-
for (const [method, pathname] of requests) {
47-
findRoute(rou3, method, fresh(pathname))
54+
for (let i = 0; i < REPEAT; i++) {
55+
for (const [method, pathname] of requests) {
56+
findRoute(rou3, method, fresh(pathname))
57+
}
4858
}
4959
})
5060
})

0 commit comments

Comments
 (0)