Skip to content

refactor(gpufault): the fault relevance window lives with the contract - #7957

Draft
samhita-alla wants to merge 2 commits into
mainfrom
gpu-forward/shared-fault-relevance
Draft

refactor(gpufault): the fault relevance window lives with the contract#7957
samhita-alla wants to merge 2 commits into
mainfrom
gpu-forward/shared-fault-relevance

Conversation

@samhita-alla

@samhita-alla samhita-alla commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Why

The interval-overlap fault relevance check introduced in #7879 answers a question every consumer of GPU fault events has to ask: does this fault, active over some stretch of time, explain this failure? The logic lived as an unexported helper inside the executor's plugin manager, and a second consumer outside this repository has already had to copy it verbatim to stay correct. The gpufault package exists so that every consumer of the fault contract reads one implementation; the relevance window belongs there with the rest of the contract.

What

Pure refactor, no behavior change.

  • gpufault gains relevance.go: RelevantToFailure(activeFrom, activeUntil, failureAt) plus the RelevanceWindow and AfterFailureSlack constants, moved from the executor with their doc comments. The function takes plain times, so the package depends on nothing but time. An event is active over its creation-to-last-observation interval, with either time standing in when the other is zero; it is relevant when that interval overlaps the window reaching RelevanceWindow before the failure and AfterFailureSlack past it, the slack bounding how late a fault may have started, not how long it may keep firing.
  • The executor deletes its local faultOverlapsFailure and constants and calls the exported function.
  • A second commit exports the failure anchor the same way: flytek8s.PodFailureTime(pod, occurredAt) moves next to GetLastTransitionOccurredAt, whose start-time semantics are exactly why the anchor exists, and the executor deletes its local copy. The same downstream consumer has been asked to adopt this helper too, so exporting both lets one follow-up delete both of its lockstep copies.

Tests

  • New 14-case table in gpufault/relevance_test.go: both boundary directions, both single-time stand-in forms, the two degenerate inputs, and window anchoring on the failure rather than on now.
  • The executor keeps TestClassifyGpuFailureRelevanceIsAnInterval deliberately: swapping the time arguments at the call site fails it, a wiring bug the package's own tests cannot see.
  • 38 packages pass across flyteplugins and the executor; lint diff against main is clean.

Follow-up outside this repository: the downstream consumer that carries the copy switches to this function and deletes it once this lands.

🤖 Generated with Claude Code

Whether a fault was firing close enough to a failure to explain it is part of what
a GPU fault means, not a detail of how the executor reads events. Keeping it in
the executor meant the next consumer had to copy it, and one already has: the
leaseworker in unionai/cloud carries its own transcription of the same interval
arithmetic and the same two durations, which is two places to get the aggregated
event case wrong.

RelevantToFailure and the two durations it reads move into gpufault, which is
already the one definition of the fault contract. It takes plain times rather than
the executor's cached event type, so the package stays free of executor imports
and any consumer with a first-recorded and a last-observed time can ask the
question. The semantics are unchanged: a fault is active from when it was first
recorded until it was last seen, either time standing in for the other when one is
missing, and it explains a failure when that stretch overlaps the window before
the failure and the small slack after it.

The executor keeps its tests. They go through classifyGpuFailure rather than
calling the check directly, so they still earn their place: they are what catches
the executor handing the two times over in the wrong order, which the new
unit tests in gpufault cannot see.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
@github-actions github-actions Bot added the flyte2 label Sep 2, 2026
pingsutw
pingsutw previously approved these changes Sep 2, 2026
Working out when a pod's trouble actually happened is the other half of deciding
whether a GPU fault explains a failure, and it was still private to the executor.
The leaseworker in unionai/cloud is now adopting the relevance check, so without
this it would carry a second copy that has to move in lockstep with the first.

PodFailureTime moves to flytek8s, which is where pod inspection already lives, and
lands next to GetLastTransitionOccurredAt on purpose: that function is the reason
this one exists. It reports the latest transition across every container including
the init containers, so for a pod that failed while its containers were still
running it hands back the time the container started. Anchoring a six-hour task on
its own start puts every real fault outside the window. PodFailureTime takes the
latest terminated main container instead, then the deletion timestamp an eviction
leaves behind, and only then whatever time the caller already had.

It stays out of gpufault, which has no Kubernetes imports and should keep none.
phaseInfoOccurredAt stays in the executor: it reads a PhaseInfo, not a pod, and
nothing outside the plugin machinery has one.

The table tests move with the function and gained the cases that were awkward to
reach through the executor, including one asserting the divergence from
GetLastTransitionOccurredAt that motivates the whole helper. The executor keeps
its end-to-end test for the evicted pod, which is what catches the anchor being
wired up wrong rather than computed wrong.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants