fix(training): W&B actually works — submit-time key preflight, real resume semantics, dead install gate removed (MT40)

[Mokuroh54]

STACKED PR — base branch is fix/mt47-cloud-log-tail (#71), not main. Merge #71 first, then retarget this to main (auto if the branch is deleted on merge). If #71 is squash-merged, this branch needs a manual rebase onto main before merging. Intended order: #71 → this.

fix(training): W&B actually works — submit-time key preflight, real resume semantics, dead install gate removed (MT40)

Stacked on #71 — merge that first.

What

W&B support existed but had three holes: a resume could run W&B in a billed GPU container with no API key, the "is wandb installed?" gate asked a question that could never fail while the one that mattered went unasked, and the form's defaults quietly worked against the user. This closes all three.

Why

The MT40 defect. build_training_command's resume branch is a whitelist and emitted no --wandb.* at all, so a resumed run inherited the checkpoint's wandb settings from train_config.json. A W&B-enabled parent therefore resumed with wandb on inside the container — while the WANDB_API_KEY secret was injected only when the request said wandb_enable, which a resume's form did not. Net effect: wandb active in-container with no API key, failing inside a paid GPU container rather than at submit time.

The install gate was dead code. /system/wandb-extra* probed whether the wandb package was importable. wandb is a hard transitive dependency of the pinned lerobot's training extra, so the answer was always yes. Worse, the toggle's probe had a fall-through bug: on any fetch error it enabled W&B anyway, so an unreachable backend silently turned logging on.

The defaults fought the user. W&B logging was off even with a key sitting in ~/.netrc, and per-checkpoint artifact uploads were on by default — pushing every checkpoint to W&B on top of the Hub.

Mechanism

Resume — the core fix. The resume branch now emits an explicit --wandb.enable <bool>, so the checkpoint's config never gets a silent vote. The value is not the form's: JobRegistry.start copies wandb_enable / wandb_project / wandb_entity off the parent record and ignores what was sent. That is forced by lerobot, not a preference — WandBLogger resumes with wandb.init(resume="must") using the run id in the checkpoint's train_config.json, so a continuation always re-opens the parent's W&B run. Enabling W&B on a resume of a non-W&B parent is structurally impossible; disabling is the only other lever. Nothing else in the group is emitted, because lerobot rebuilds mode/notes/disable_artifact from the checkpoint itself — verified against the installed lerobot v0.6.0 by reloading a real checkpoint config through the resume path.

Preflight — a hard 400 before any side effect. Any W&B-enabled run needs a key resolvable on this machine, on either runner: a cloud job gets it forwarded as an HF Jobs secret, and a local trainer is a non-tty subprocess in which wandb.init cannot prompt for a login and simply dies after the record already says running. Three layers:

1. POST /jobs/training refuses a fresh run with no resolvable key — the fast half, skipped for resumes because a resume's W&B state is not the request's to state.
2. JobRegistry.start re-asks once the resume block has copied the parent's value, and before it creates a record, spawns a local subprocess, pushes a dataset, or spawns the deferred _upload_resume_then_start thread. That thread is why the check can't live deeper: it returns 201 first and uploads on a thread, so a later refusal would surface as a failed job instead of a message on the button. This check is the authority.
3. HfCloudJobRunner.start repeats it belt-and-braces, moved above _ensure_dataset_on_hub so a missing key can no longer leave a freshly published dataset behind for a job that never ran.

Install gate → credential probe. /system/wandb-extra*, the wandb InstallManager, and WandbInstallDialog.tsx are deleted. GET /system/wandb-credentials replaces them, reporting whether a key is resolvable from WANDB_API_KEY or a ~/.netrc entry for api.wandb.ai — a boolean and a login hint, never the key. resolve_wandb_api_key already existed and is reused unchanged.

Defaults and copy. A fresh run defaults W&B on once the probe confirms a key — applied at most once per mounted form and never overwriting a toggle the user already touched, since the probe answers asynchronously and a late "yes" must not reverse a decision made while it was in flight. wandb_disable_artifact now defaults true (artifacts off; per-checkpoint uploads are opt-in). Field copy adopts W&B's own vocabulary: Project with placeholder lerobot (default) (lerobot's real config default, so an empty field isn't a surprise about where runs land), and Entity with helper "Your W&B username or a team you belong to. Blank = your personal account." — the last clause defuses a real 403 a user hit by typing a placeholder word into that field, which W&B rejects only at run start.

URL scrape. The comment was wrong: the URL comes from lerobot's own logging.info("Track this run --> <coloured url>"), not from wandb's wandb: 🚀 View run at … banner — lerobot sets WANDB_SILENT=True before importing wandb, suppressing that banner entirely. Corrected, and the run-id character class widened. A missing URL is normal (W&B off, offline/disabled mode, self-hosted W&B) and never an error.

Tests

• Resume emits an explicit --wandb.enable and nothing else; a non-W&B parent forces it off.
• wandb_disable_artifact defaults true and serializes both ways.
• Serialization is runner-blind (identical argv for the cloud and local call sites).
• Registry: a cloud resume inherits the parent's enable/project/entity; a resume of a non-W&B parent cannot turn it on and never demands a key; a W&B resume with no key is refused before any record exists; a local run with no key is refused before the subprocess; a W&B-off run never probes at all.
• Endpoint: 400 before job_registry.start on both runners; not blocked with a key; /system/wandb-credentials returns a boolean and never leaks the key.
• Scrape pinned to lerobot's real ANSI-wrapped line shape.
• Mock hygiene: bare MagicMock runners returned a Mock from wandb_run_url(), which the watchdog wrote onto the record and persisted as JSON — silent garbage plus a pydantic serializer warning per tick. A _mock_runner helper plus explicit None on the direct mocks; zero W&B-related warnings remain.

pytest green; ruff clean; pre-commit run --all-files green.

Deliberate non-ports

This is the wandb surface as main knows it, not a port of the rig branch's diff. Not included, and arriving with the future carve train: the compressed tabbed training panel (RunPane/OptimizerPane/ComputePane), sticks-only resume and one-row-per-leaf lineage, and chain-rewind's owner-vs-leaf inheritance. On a chain-rewound lineage the W&B run belongs to the record that wrote the chosen checkpoint, not the leaf — here, with main's single-parent resume model, those are the same record, so the simple form is correct on this branch and will need revisiting when chain-rewind lands.

wandb_run_id is left in place on main. rig removed it (it is checkpoint-inherited state, never a per-launch choice), but removing a request field is a contract change this PR wasn't asked to make, and it is inert here. Flagging it as a known divergence for whoever reconciles the branches.

Note for the merger

PR #42 has since merged and this branch is reconciled with it (rebased over 921cfc4; the resume-inheritance and preflight anchors were re-verified against #42's restructuring, and the rebase-shadowing traps — duplicated regex/test definitions that would have silently reverted #42's versions — were audited out). One natural follow-up deliberately not taken here: #42's new resume-seed prefill is the right hook for showing the parent's W&B values read-only on the resume form.

Also pre-existing on main and untouched here: npx tsc -b is red for five unrelated files (CI runs the no-op npx tsc --noEmit), and tests/test_models.py has 7 environment-dependent failures that reproduce on a clean checkout.

🤖 Generated with Claude Code