refactor(gpufault): the fault relevance window lives with the contract - #7957
Draft
samhita-alla wants to merge 2 commits into
Draft
refactor(gpufault): the fault relevance window lives with the contract#7957samhita-alla wants to merge 2 commits into
samhita-alla wants to merge 2 commits into
Conversation
Whether a fault was firing close enough to a failure to explain it is part of what a GPU fault means, not a detail of how the executor reads events. Keeping it in the executor meant the next consumer had to copy it, and one already has: the leaseworker in unionai/cloud carries its own transcription of the same interval arithmetic and the same two durations, which is two places to get the aggregated event case wrong. RelevantToFailure and the two durations it reads move into gpufault, which is already the one definition of the fault contract. It takes plain times rather than the executor's cached event type, so the package stays free of executor imports and any consumer with a first-recorded and a last-observed time can ask the question. The semantics are unchanged: a fault is active from when it was first recorded until it was last seen, either time standing in for the other when one is missing, and it explains a failure when that stretch overlaps the window before the failure and the small slack after it. The executor keeps its tests. They go through classifyGpuFailure rather than calling the check directly, so they still earn their place: they are what catches the executor handing the two times over in the wrong order, which the new unit tests in gpufault cannot see. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
pingsutw
previously approved these changes
Sep 2, 2026
Working out when a pod's trouble actually happened is the other half of deciding whether a GPU fault explains a failure, and it was still private to the executor. The leaseworker in unionai/cloud is now adopting the relevance check, so without this it would carry a second copy that has to move in lockstep with the first. PodFailureTime moves to flytek8s, which is where pod inspection already lives, and lands next to GetLastTransitionOccurredAt on purpose: that function is the reason this one exists. It reports the latest transition across every container including the init containers, so for a pod that failed while its containers were still running it hands back the time the container started. Anchoring a six-hour task on its own start puts every real fault outside the window. PodFailureTime takes the latest terminated main container instead, then the deletion timestamp an eviction leaves behind, and only then whatever time the caller already had. It stays out of gpufault, which has no Kubernetes imports and should keep none. phaseInfoOccurredAt stays in the executor: it reads a PhaseInfo, not a pod, and nothing outside the plugin machinery has one. The table tests move with the function and gained the cases that were awkward to reach through the executor, including one asserting the divergence from GetLastTransitionOccurredAt that motivates the whole helper. The executor keeps its end-to-end test for the evicted pod, which is what catches the anchor being wired up wrong rather than computed wrong. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
samhita-alla
marked this pull request as draft
September 3, 2026 12:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The interval-overlap fault relevance check introduced in #7879 answers a question every consumer of GPU fault events has to ask: does this fault, active over some stretch of time, explain this failure? The logic lived as an unexported helper inside the executor's plugin manager, and a second consumer outside this repository has already had to copy it verbatim to stay correct. The gpufault package exists so that every consumer of the fault contract reads one implementation; the relevance window belongs there with the rest of the contract.
What
Pure refactor, no behavior change.
Tests
Follow-up outside this repository: the downstream consumer that carries the copy switches to this function and deletes it once this lands.
🤖 Generated with Claude Code