[Docs] Add resilient Ray training example - #10602
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| deadline = time.monotonic() + duration_seconds | ||
| batches = 0 | ||
| while time.monotonic() < deadline: | ||
| activations = self.torch.tanh(activations @ self.weight) | ||
| batches += 1 |
There was a problem hiding this comment.
🟡 Recovered rollouts change training results
After worker loss, rollout repeats a wall-clock-dependent number of updates despite reusing its seed. Recovery can change the reward and policy version.
Prompt for agents
Make GPUWorker.rollout deterministic across Ray task retries. The current loop in examples/ray_resilient_training/train.py stops according to elapsed wall time, so a reconstructed actor can perform a different number of matrix updates for the same step and seed. Use a deterministic work count derived from configuration, or checkpoint enough rollout state to reproduce the same result. Preserve the example's ability to keep a rollout active long enough to exercise node recovery.
Was this helpful? React with 👍 or 👎 to provide feedback.
| deadline = time.monotonic() + duration_seconds | ||
| batches = 0 | ||
| while time.monotonic() < deadline: | ||
| activations = self.torch.tanh(activations @ self.weight) | ||
| batches += 1 | ||
| self.torch.cuda.synchronize() |
There was a problem hiding this comment.
🟡 GPU steps exceed configured duration
Because CUDA operations are asynchronous, rollout enqueues work until the deadline and synchronizes afterward. Healthy steps can greatly exceed STEP_SECONDS and hit the recovery timeout.
Prompt for agents
Make STEP_SECONDS bound actual GPU execution time in GPUWorker.rollout. PyTorch CUDA operations are asynchronous, so the current loop measures CPU enqueue time and cuda.synchronize() drains a potentially large backlog only after the deadline. Synchronize within the timing loop, poll completed CUDA events, or otherwise limit outstanding work while retaining continuous GPU utilization.
Was this helpful? React with 👍 or 👎 to provide feedback.
| export RAY_gcs_storage=rocksdb | ||
| export RAY_gcs_storage_path="${RAY_GCS_ROOT}/rocksdb" | ||
|
|
||
| ray start \ | ||
| --head \ | ||
| --port=6379 \ | ||
| --dashboard-host=0.0.0.0 \ | ||
| --num-cpus=0 \ | ||
| --num-gpus=0 \ | ||
| --disable-usage-stats |
There was a problem hiding this comment.
🔍 Head recovery remains unverified
The central recovery path lacks an end-to-end test. It combines volume reattachment, embedded RocksDB restoration, worker reconnection, and detached-actor discovery.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
I think Devin has a point here — as written, the head can come back and silently create new actors instead of reattaching.
Worth confirming that's not what happens in the head-recovery example, or gating the create branch on last_completed_step == -1 so a failed reattach can't pass as success.
|
/build-docs |
|
✅ ReadTheDocs build triggered for branch The documentation will be available at: https://docs.skypilot.co/en/docs-ray-resilient-training/ |
| # Resilient Ray training on SkyPilot | ||
|
|
||
| This example keeps a multi-node Ray training job alive through head and worker | ||
| failures. Dynamic Node Sets provide fast failover to warm standby capacity: |
There was a problem hiding this comment.
🔍 Unconfigured warm standby claim
warm standby capacity implies preprovisioned spare resources, but the manifest defines only active nodes. Clarify the wording or document external standby requirements.
Was this helpful? React with 👍 or 👎 to provide feedback.
| export RAY_gcs_storage=rocksdb | ||
| export RAY_gcs_storage_path="${RAY_GCS_ROOT}/rocksdb" | ||
|
|
||
| ray start \ | ||
| --head \ | ||
| --port=6379 \ | ||
| --dashboard-host=0.0.0.0 \ | ||
| --num-cpus=0 \ | ||
| --num-gpus=0 \ | ||
| --disable-usage-stats |
There was a problem hiding this comment.
I think Devin has a point here — as written, the head can come back and silently create new actors instead of reattaching.
Worth confirming that's not what happens in the head-recovery example, or gating the create branch on last_completed_step == -1 so a failed reattach can't pass as success.
2 participants
Summary
Test plan
bash format.sh --files examples/ray_resilient_training/train.py examples/ray_resilient_training/wait_for_head.pyReadWriteOncePodaccess mode.The exact checked-in L4 manifest was not launched. Head replacement and restoration from the persisted GCS state were not exercised in the live run.