GPU faults are catchable typed exceptions

[samhita-alla]

Why

The backend now classifies GPU faults into the failure a task reports: the error code names the fault (GpuXidError, GpuFallenOffBus, GpuEccUncorrectable, GpuRowRemapPending, GpuNvlinkError, GpuGspError), the message leads with the driver's own sentence, and ExecutionError carries a typed gpu_fault with the Xid, severity, GPU UUID and node. Until now the SDK converted all of that into a generic runtime error, so a parent task could only string-match on the code. This makes GPU faults catchable as typed exceptions.

What

• flyte.errors gains GPUFaultError, carrying fault_kind, fault_code, fault_name, severity, gpu_uuid, gpu_index, node, pci_bus_id and process, plus xid and sxid properties that answer only for the matching kind, since Xid and SXid share a numbering space but not a meaning. Two leaves, GPUFaultUserError and GPUFaultSystemError, keep the error kind honest: the backend files a user-severity Xid (an out-of-bounds access in the workload's own kernel) as USER and a critical hardware fault as SYSTEM, and a single class would have to lie about kind for one of them. except flyte.errors.GPUFaultError catches both.
• The error conversion selects the GPU classes on an exact match of the six backend codes, ahead of the existing substring branches, and tolerates server-prefixed codes. Fields fill from the typed gpu_fault when present; a fault arriving from an older backend falls back to parsing the message sentence, defensively, so conversion can never fail over it.
• The docstring states the semantics plainly: user-severity Xids arrive on the task's own retry budget, while critical hardware faults are retried by the platform without charging that budget, so user code sees those only after platform policy gives up.

try:
    result = await train(shard)
except flyte.errors.GPUFaultError as e:
    if e.severity == "USER":
        raise  # the workload's own kernel is at fault
    result = await train(shard)

Tests

36 new tests: selection for every code under both kinds, catchability of both leaves through the base class, typed-field population including typed values winning over a conflicting sentence, every sentence device shape plus the aggregated-event prefix, unreadable messages, and unchanged fallthrough for OOMKilled, Interrupted, unknown codes and UNKNOWN kind. ruff, mypy and ty clean; the touched test modules pass 187 tests total.

🤖 Generated with Claude Code

[kumare3]

this looks very brittle, i dont think we should create an expectation that we will have this. We can provide this as a helper. Otherwise users will expect we will always come up with this error. but, this cannot be guaranteed