fix: clear stale error.pb at the start of every clustered attempt - #1519
Merged
Conversation
Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
pingsutw
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
For clustered (JobSet) tasks the SDK writes
error.pbonly on what it believes is the terminalrestart attempt, using
JOBSET_RESTART_ATTEMPT >= JOBSET_MAX_RESTARTS. That guess mirrors theJobSet's
Status.Restarts, which also counts free host-maintenance restarts(
RestartJobSetAndIgnoreMaxRestarts, flyteorg/flyte#7947), while the budget is charged fromRestartsCountTowardsMax, which pods never see. So after a free restart the guess fires early, agenuine failure writes
error.pbon a non-terminal attempt, and because the executor readserror.pbbeforeoutputs.pb, a later successful attempt is still reported as a failure.Summary
io.clear_stale_clustered_error(), which removes a leftovererror.pbfrom the attempt'soutput prefix. It runs on every clustered rank-0 start, not just restarted ones, since the
restart counter is exactly what proved untrustworthy. Only rank-0 deletes, restart attempts are
strictly sequential, and a failed delete only restores today's behavior.
load_and_run_task, ahead of the code-bundle download and of all threeupload_errorcall sites.docstring: the guess can be early but never late, and the cleanup is what guarantees correctness.
examples/clustered/ddp_train_restart.pyinto a regression test that reproduces thestale file with no backend dependency, using
max_restarts=0plus one in-pod torchrun restart.Test Plan
uv run pytest tests/flyte/clustered tests/flyte/internal/runtime tests/flyte/internal/bin -qcovers the async and sync delete branches, absent-file and non-rank-0 and non-clustered no-ops,
soft failure on both the existence check and the delete, and that the cleanup is awaited before
the task load, the load-failure error upload, and the task body. 364 passed.
make lintandmake mypyare clean.make dist && uv run python examples/clustered/ddp_train_restart.py.Expect SUCCEEDED.
flyte get logs <run>should show attempt 0 uploading.../a0/0/error.pb,then the restarted rank-0 logging
Removed stale ... error.pb (JOBSET_RESTART_ATTEMPT=0, TORCHELASTIC_RESTART_COUNT=1), then training. Verified as runudvvrchxvfqpnwjvbfwp.Reverting the
entrypoints.pyhook makes the same run end FAILED with the attempt-0 error.