add GPU metrics examples for the run details metrics tab - #1478
Merged
Conversation
gpu_metrics.py walks a GPU through distinct load regimes (tensor, bandwidth, PCIe, NVLink, idle), each held long enough to be visible at the console's polling resolution, so every panel on the Metrics tab gets a recognizable signature. gpu_metrics_multinode.py is the ClusteredTaskEnvironment variant, one pod per node with an all-reduce phase so the cross-node interconnect shows up too. Both can end with a deliberate Xid 31 from an out-of-bounds Triton store, which is how the GPU fault path gets exercised end to end: after_xid picks whether the task then returns, raises, or exits outright, matching the three shapes a fault takes by the time it reaches the console. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
The Triton import is optional, so `tl` is `Unknown | None` and every use of it read as an attribute on None. Guarding on both names narrows the kernel's signature; the two lines inside the kernel body still need a suppression, because that body runs at launch time and the checker resolves `tl` as a global rather than the guarded value. The accelerator request is a `str | int` by construction, since a bare number asks for that many GPUs with no device pin, and that does not fit the literal union `Resources.gpu` declares. Both examples carry the paired mypy and ty suppression the rest of the codebase uses for this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
cosmicBboy
enabled auto-merge (squash)
September 3, 2026 01:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
gpu_metrics.py walks a GPU through distinct load regimes (tensor, bandwidth, PCIe, NVLink, idle), each held long enough to be visible at the console's polling resolution, so every panel on the Metrics tab gets a recognizable signature. gpu_metrics_multinode.py is the ClusteredTaskEnvironment variant, one pod per node with an all-reduce phase so the cross-node interconnect shows up too.
Both can end with a deliberate Xid 31 from an out-of-bounds Triton store, which is how the GPU fault path gets exercised end to end: after_xid picks whether the task then returns, raises, or exits outright, matching the three shapes a fault takes by the time it reaches the console.