Skip to content

Commit e628d9d

Browse files
committed
test(bench): make the py-lsp time budget a liveness backstop, not a perf gate
test_py_lsp_bench.c:262 failed the v0.10.0 release run TWICE on genuine attempts (ubuntu-24.04-arm 3/3), while the SAME sha passed that shard in the morning's dry run. A wall-clock bound on a shared 4-vCPU runner is not a property of the code, and this repo's testing doctrine says exactly that: perf tests assert a ratio; a time bound may exist only as a liveness backstop. This test's perf content already IS ratio-form (resolution ratio with a sanitizer-aware floor). The 150/1500 ms budget was the anti-pattern half. It becomes a 30 s liveness backstop so a catastrophic hang still fails the suite rather than the 900 s shard wall clock, and absolute timings stay where they belong: scripts/benchmark-*.sh, which record rather than gate. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent aa6d740 commit e628d9d

1 file changed

Lines changed: 9 additions & 8 deletions

File tree

‎tests/test_py_lsp_bench.c‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -255,14 +255,15 @@ TEST(pylsp_bench_resolution_ratio) {
255255
ASSERT_GTE(resolved * 2, calls);
256256
}
257257

258-
/* Time budget. ASan+UBSan instrumentation slows the parse ~5-10×, so
259-
* scale the budget when a sanitizer is active. Native: 150 ms for a
260-
* ~200-line fixture; sanitized: 1500 ms. */
261-
#if defined(CBM_SANITIZED_BUILD) || defined(__SANITIZE_ADDRESS__)
262-
ASSERT(ms < 1500.0);
263-
#else
264-
ASSERT(ms < 150.0);
265-
#endif
258+
/* Liveness backstop ONLY — not a perf gate. The perf content of this test
259+
* is the resolution-ratio assertion above; wall-clock on a shared CI
260+
* runner is not a property of the code (the 1500 ms sanitized budget was
261+
* exceeded twice on loaded ubuntu-24.04-arm runners by the SAME sha that
262+
* passed hours earlier). Absolute timings belong to
263+
* scripts/benchmark-*.sh, which record rather than gate. This bound
264+
* exists solely so a catastrophic hang fails the suite instead of the
265+
* 900 s shard wall clock. */
266+
ASSERT(ms < 30000.0);
266267
PASS();
267268
}
268269

0 commit comments

Comments
 (0)