Report why fetching a pre-trained checkpoint failed - #7535
Conversation
``get_published_pretrained_checkpoint`` wrapped the download in a bare ``except Exception`` and reported every failure as "A pre-trained checkpoint is currently unavailable for this task.", then returned None so the play scripts exited without further explanation. A checkpoint that downloads fine on one machine and fails on another -- an unwritable ``.pretrained_checkpoints`` directory left behind by a container run, a full disk, a copy the asset server refused -- was indistinguishable from a task that has no published checkpoint, and the underlying error was discarded. Separate the two cases. A checkpoint the asset server does not provide keeps the same first line, so callers matching on it are unaffected, and adds the location that was tried, the task and backends the location was derived from, and what to do instead. Any other failure now raises RuntimeError naming the checkpoint, the cache directory and the originating error, which is chained as the cause.
Greptile SummaryThis PR improves pre-trained checkpoint error reporting by preserving
Confidence Score: 5/5The PR appears safe to merge, with the new failure classification matching the underlying retrieval behavior and documented caller contract. Missing or unreachable remote assets are translated to Important Files Changed
Reviews (1): Last reviewed commit: "Report why fetching a pre-trained checkp..." | Re-trigger Greptile |
There was a problem hiding this comment.
Isaac Lab Review Bot
The change usefully distinguishes unavailable checkpoints from local download failures, preserves the existing leading unavailable message, and adds a chained RuntimeError for download problems. One documentation inconsistency should be corrected before merge.
- Design and architecture: Handling FileNotFoundError separately from other download exceptions is a focused change that preserves the existing unavailable-checkpoint path while exposing actionable local failures.
- API: The public docstring and changelog state that None means the checkpoint is not published, but the implementation also returns None when the asset server is unreachable because both cases surface as FileNotFoundError. These descriptions should explicitly cover both outcomes so callers do not interpret a transient outage as confirmed absence.
- Implementation: The implementation reports the checkpoint and absolute cache path, chains the originating exception, and adds focused tests for unavailable checkpoints and unwritable caches. The remaining issue is the mismatch between the documented None semantics and the FileNotFoundError branch's acknowledged behavior.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
|
|
||
| Returns: | ||
| The path. | ||
| The path, or None when the asset server does not publish a checkpoint for this |
There was a problem hiding this comment.
🔵 Suggestion · Api — Docstring understates when None is returned
The Returns clause states None means the asset server does not publish a checkpoint for this task and backend combination, but the added handler and its own printed message acknowledge that an unreachable server surfaces as the same FileNotFoundError and also yields None. Callers relying on this docstring may treat a transient outage as a permanently unpublished checkpoint. Reword to say None means the checkpoint was absent or the server could not be reached, and align the changelog wording.
There was a problem hiding this comment.
Fixed in b910097. The docstring now says None covers both a checkpoint that was never published and a server that could not be reached, since omni.client does not distinguish them, and the changelog fragment uses the same wording.
|
run-ci |
``omni.client`` reports a checkpoint that was never published and a server it cannot reach identically, so both leave the download raising FileNotFoundError and both return None. The printed message says so, but the docstring and the changelog described None as an unpublished checkpoint only, inviting callers to read a transient outage as a confirmed absence.
|
Review bot finding addressed in b910097 (the only actionable one; Greptile reported none).
No behaviour change in that commit — docstring and changelog wording only. On the previous run, |
|
run-ci |
Description
Running a play script with
--checkpoint pretrainedon a machine whose.pretrained_checkpointsdirectory was left behind by a container run (owned byroot) prints:and exits. The checkpoint is published and reachable; the download failed with
PermissionError: [Errno 13] Permission denied: '.../.pretrained_checkpoints/rsl_rl/https'. That errorwas discarded by a bare
except Exceptioninisaaclab_rl.utils.pretrained_checkpoint.get_published_pretrained_checkpoint, which reported everyfailure — a missing checkpoint, an unreachable server, an unwritable cache, a full disk — with the same
sentence and returned
None, after which the play scripts return without further explanation.This separates the two cases:
The asset server does not provide the checkpoint.
Noneis still returned and the message stillstarts with
A pre-trained checkpoint is currently unavailable for this task., so callers and testsmatching on that line are unaffected. It now also names the location that was tried, the task and
backends that location was derived from, and what to do instead:
The checkpoint exists but could not be downloaded. This is a local problem the user has to fix, so
it raises
RuntimeErrornaming the checkpoint, the cache directory and the originating error, which ischained as the cause:
Note that
omni.clientreports a checkpoint that was never published and a server it cannot reachidentically, so both remain covered by the "unavailable" message.
Both paths were verified end to end against the live asset server, using a published checkpoint with a
read-only cache directory and an unpublished checkpoint name.
Type of change
Behaviour note: callers that relied on
Noneto mask a local download failure now see aRuntimeError.Callers that treat
Noneas "no checkpoint published for this task" — every caller in the repository —are unchanged. This is called out in the changelog fragment.
Release backport
developChecklist
pre-commitchecks with./isaaclab.sh --formatsource/<pkg>/changelog.d/for every touched packageCONTRIBUTORS.mdor my name already exists there