Repository navigation
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 (optional) The new gate still passes the 15 GiB residual this PR cites, so TP8 vLLM jobs at 0.90 can still hit the free-memory startup failure.
wait_for_amd_gpu_cleancompares rocm-smi's whole-numberVRAM%with-le 5. On a 287.98 GiB GPU, 15 GiB is 5.2% and prints as 5, so it passes. Anything up to about 15.8 GiB (rounded) or 17.2 GiB (truncated) passes too. Fix: gate on used or free bytes instead (for examplerocm-smi --showmeminfo vram), or use 4%, so every GPU holds at most about 14.4 GiB. That leaves the 14.4 GiB engine-context budget this comment relies on.Why this was flagged
The trigger is a prior job leaving about 15-17 GiB per GPU with no process attached, which is the low end of the 15-26 GiB range in setup.sh:11. setup.sh:14 calls
wait_for_amd_gpu_clean 5. common.sh:11-14 takes the integer fromGPU Memory Allocated \(VRAM%\): [0-9]+and checks-le "$threshold". 15 GiB / 287.98 GiB = 5.2%, which rocm-smi prints as 5 whether it rounds or truncates, so the wait ends at once. vLLM at 0.90 needs used memory <= 28.8 GiB, including its own context, when it starts. The PR's failing runs reported free memory of 251-259 GiB, so used was 29-37 GiB. That means a 15 GiB residual plus roughly 14 GiB of engine context still lands above 28.8 GiB. Those jobs fail the same way as on the base branch, even though the comment at setup.sh:13 says 5% leaves room for the engine's context. Nothing else in the hook re-checks memory, and the rocm-smi percent is too coarse for a gate this tight (1% is about 2.9 GiB).Verification: nit: this fix is incomplete, not a regression. The base 10% gate already let these residuals through by the same route. The trigger is a prior job leaving about 15-15.8 GiB per GPU with no process attached. setup.sh:14 calls
wait_for_amd_gpu_clean 5. 15 GiB / 287.98 GiB = 5.21%, which prints as 5 whether rocm-smi rounds or truncates, so5 -le 5passes on the first poll.