[jdinalt]

Summary

Fixes issue #218 (the launch-time backend-derivation race that failed all 4 workers of a smoke run), plus a small DiLoCo-server-modal usability add.

#218 — requeue instead of fail on transient unreachability (scheduler.py)

The scheduler derives a worker's sync backend by GETting the param server's /info at launch — a single attempt, 15 s timeout, no retry. A worker dispatched in the brief window where the server is still finishing startup (or momentarily busy as 4 workers make first contact at once) was permanently failed by that one timed-out probe, even though the server was healthy seconds later. (Observed live; re-submitting against the warm server worked.)

The probe runs inside the serial dispatch tick, so a blocking sleep-retry would stall every other dispatch. Instead:

• per-attempt timeout shortened to 5 s (_DILOCO_DERIVE_TIMEOUT_S) so one probe can't hold up the tick;
• _diloco_query_info raises a typed _DilocoServerUnreachable; _launch catches it and requeues the worker — drops the record (freeing the reserved GPU) and re-adds the item with its original submitted_at (FIFO position kept) — for the next tick. It fails for real only after _DILOCO_DERIVE_RETRY_BUDGET_S (180 s), covering a slow server startup without hanging forever on a server that never comes up.

A permanent config error (backend/topology mismatch) stays a plain RuntimeError and still fails immediately — only the network-unreachable case is retried.

webui — "Reset to defaults" on the DiLoCo server modal

The other service modals have it; the DiLoCo one didn't, making it hard to definitively clear persisted ad-hoc overrides. Create-mode only.

Tests

test_scheduler_diloco_retry.py: typed-exception raise, requeue-within-budget, fail-after-budget, success-clears-bookkeeping. Scheduler + diloco-env suites green; black/isort clean; webui tsc --noEmit clean.

Docs

docs/trainers/diloco-backends.md — the "fails loud if unreachable" line now describes the requeue/retry.

🤖 Generated with Claude Code