Repository navigation
test(bench): make the py-lsp time budget a liveness backstop, not a perf gate - #1517
Merged
Merged
Conversation
…erf gate test_py_lsp_bench.c:262 failed the v0.10.0 release run TWICE on genuine attempts (ubuntu-24.04-arm 3/3), while the SAME sha passed that shard in the morning's dry run. A wall-clock bound on a shared 4-vCPU runner is not a property of the code, and this repo's testing doctrine says exactly that: perf tests assert a ratio; a time bound may exist only as a liveness backstop. This test's perf content already IS ratio-form (resolution ratio with a sanitizer-aware floor). The 150/1500 ms budget was the anti-pattern half. It becomes a 30 s liveness backstop so a catastrophic hang still fails the suite rather than the 900 s shard wall clock, and absolute timings stay where they belong: scripts/benchmark-*.sh, which record rather than gate. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v0.10.0 release run failed twice on genuine attempts at
test_py_lsp_bench.c:262 ASSERT(ms < 1500.0)(ubuntu-24.04-arm 3/3), while the same SHA passed that shard in the morning's dry run. Two same-SHA verdicts that differ = the wall-clock bound is measuring runner load, not code.Per the repo's testing doctrine, perf tests assert a ratio; a time bound may exist only as a liveness backstop. This test's perf content already is ratio-form (resolution ratio with the sanitizer-aware floor) — the 150/1500 ms absolute budget was the anti-pattern half. It becomes a 30 s liveness backstop (catastrophic hangs still fail the suite instead of eating the 900 s shard wall clock); absolute timings stay in
scripts/benchmark-*.sh, which record rather than gate.Test-only change; unblocks re-dispatching the v0.10.0 release pipeline on a fresh SHA rather than re-rolling the same one.