Skip to content

Report why fetching a pre-trained checkpoint failed - #7535

Merged
kellyguo11 merged 2 commits into
developfrom
antoiner/fix/pretrained-checkpoint-errors
Sep 4, 2026
Merged

Report why fetching a pre-trained checkpoint failed#7535
kellyguo11 merged 2 commits into
developfrom
antoiner/fix/pretrained-checkpoint-errors

Conversation

@AntoineRichard

Copy link
Copy Markdown
Collaborator

Description

Running a play script with --checkpoint pretrained on a machine whose
.pretrained_checkpoints directory was left behind by a container run (owned by root) prints:

Fetching pre-trained checkpoint : https://.../rsl_rl/Isaac-Humanoid-Direct_newtonmjwarp_none_rsl_rl.pt
A pre-trained checkpoint is currently unavailable for this task.

and exits. The checkpoint is published and reachable; the download failed with
PermissionError: [Errno 13] Permission denied: '.../.pretrained_checkpoints/rsl_rl/https'. That error
was discarded by a bare except Exception in
isaaclab_rl.utils.pretrained_checkpoint.get_published_pretrained_checkpoint, which reported every
failure — a missing checkpoint, an unreachable server, an unwritable cache, a full disk — with the same
sentence and returned None, after which the play scripts return without further explanation.

This separates the two cases:

  • The asset server does not provide the checkpoint. None is still returned and the message still
    starts with A pre-trained checkpoint is currently unavailable for this task., so callers and tests
    matching on that line are unaffected. It now also names the location that was tried, the task and
    backends that location was derived from, and what to do instead:

    A pre-trained checkpoint is currently unavailable for this task.
      The asset server does not provide 'https://.../Isaac-Not-A-Real-Task_newtonmjwarp_none_rsl_rl.pt'.
      Either no checkpoint is published for task 'Isaac-Not-A-Real-Task' with the 'newtonmjwarp' physics
      and 'none' render backends, or the asset server could not be reached.
      Train the task, or pass --checkpoint <path> to use a checkpoint of your own.
    
  • The checkpoint exists but could not be downloaded. This is a local problem the user has to fix, so
    it raises RuntimeError naming the checkpoint, the cache directory and the originating error, which is
    chained as the cause:

    RuntimeError: Failed to download the pre-trained checkpoint 'https://.../Isaac-Humanoid-Direct_newtonmjwarp_none_rsl_rl.pt'
    into '/home/user/IsaacLab/.pretrained_checkpoints/rsl_rl': PermissionError: [Errno 13] Permission denied:
    '/home/user/IsaacLab/.pretrained_checkpoints/rsl_rl/https'. Check that the cache directory is writable and
    that the disk is not full; a directory left behind by a container run is owned by root.
    

Note that omni.client reports a checkpoint that was never published and a server it cannot reach
identically, so both remain covered by the "unavailable" message.

Both paths were verified end to end against the live asset server, using a published checkpoint with a
read-only cache directory and an unpublished checkpoint name.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Behaviour note: callers that relied on None to mask a local download failure now see a RuntimeError.
Callers that treat None as "no checkpoint published for this task" — every caller in the repository —
are unchanged. This is called out in the changelog fragment.

Release backport

  • Backport this pull request to the active release branch after it merges into develop

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

``get_published_pretrained_checkpoint`` wrapped the download in a bare
``except Exception`` and reported every failure as "A pre-trained
checkpoint is currently unavailable for this task.", then returned None
so the play scripts exited without further explanation. A checkpoint
that downloads fine on one machine and fails on another -- an unwritable
``.pretrained_checkpoints`` directory left behind by a container run, a
full disk, a copy the asset server refused -- was indistinguishable from
a task that has no published checkpoint, and the underlying error was
discarded.

Separate the two cases. A checkpoint the asset server does not provide
keeps the same first line, so callers matching on it are unaffected, and
adds the location that was tried, the task and backends the location was
derived from, and what to do instead. Any other failure now raises
RuntimeError naming the checkpoint, the cache directory and the
originating error, which is chained as the cause.
@AntoineRichard
AntoineRichard requested a review from a team September 3, 2026 13:19
@github-actions github-actions Bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Sep 3, 2026
@greptile-apps

greptile-apps Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR improves pre-trained checkpoint error reporting by preserving None for checkpoints unavailable during the remote lookup while raising a chained RuntimeError for failures encountered while downloading a discovered checkpoint.

  • Adds detailed task, backend, asset-path, and remediation information to unavailable-checkpoint messages.
  • Reports local download failures with the checkpoint URL, cache directory, original exception, and filesystem guidance.
  • Adds tests covering missing remote checkpoints and unwritable cache directories, plus a changelog fragment documenting the behavior change.

Confidence Score: 5/5

The PR appears safe to merge, with the new failure classification matching the underlying retrieval behavior and documented caller contract.

Missing or unreachable remote assets are translated to FileNotFoundError and retain the graceful None path, while failures downloading an asset already found are deliberately surfaced with their original cause and actionable cache context.

Important Files Changed

Filename Overview
source/isaaclab_rl/isaaclab_rl/utils/pretrained_checkpoint.py Separates unavailable remote checkpoints from download failures and provides actionable diagnostics while preserving the established None contract for lookup failures.
source/isaaclab_rl/test/test_pretrained_checkpoint.py Adds focused tests verifying unavailable-checkpoint output and chained runtime errors for local cache failures.
source/isaaclab_rl/changelog.d/antoiner-pretrained-checkpoint-errors.minor.rst Accurately documents the improved diagnostics and the intentional exception-contract change for download failures.

Reviews (1): Last reviewed commit: "Report why fetching a pre-trained checkp..." | Re-trigger Greptile

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The change usefully distinguishes unavailable checkpoints from local download failures, preserves the existing leading unavailable message, and adds a chained RuntimeError for download problems. One documentation inconsistency should be corrected before merge.

  • Design and architecture: Handling FileNotFoundError separately from other download exceptions is a focused change that preserves the existing unavailable-checkpoint path while exposing actionable local failures.
  • API: The public docstring and changelog state that None means the checkpoint is not published, but the implementation also returns None when the asset server is unreachable because both cases surface as FileNotFoundError. These descriptions should explicitly cover both outcomes so callers do not interpret a transient outage as confirmed absence.
  • Implementation: The implementation reports the checkpoint and absolute cache path, chains the originating exception, and adds focused tests for unavailable checkpoints and unwritable caches. The remaining issue is the mismatch between the documented None semantics and the FileNotFoundError branch's acknowledged behavior.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.


Returns:
The path.
The path, or None when the asset server does not publish a checkpoint for this

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Suggestion · Api — Docstring understates when None is returned

The Returns clause states None means the asset server does not publish a checkpoint for this task and backend combination, but the added handler and its own printed message acknowledge that an unreachable server surfaces as the same FileNotFoundError and also yields None. Callers relying on this docstring may treat a transient outage as a permanently unpublished checkpoint. Reword to say None means the checkpoint was absent or the server could not be reached, and align the changelog wording.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b910097. The docstring now says None covers both a checkpoint that was never published and a server that could not be reached, since omni.client does not distinguish them, and the changelog fragment uses the same wording.

@kellyguo11

Copy link
Copy Markdown
Contributor

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 4, 2026
``omni.client`` reports a checkpoint that was never published and a
server it cannot reach identically, so both leave the download raising
FileNotFoundError and both return None. The printed message says so, but
the docstring and the changelog described None as an unpublished
checkpoint only, inviting callers to read a transient outage as a
confirmed absence.
@AntoineRichard

Copy link
Copy Markdown
Collaborator Author

Review bot finding addressed in b910097 (the only actionable one; Greptile reported none).

None no longer reads as "this checkpoint does not exist": both the docstring and the changelog fragment
now state that it covers a checkpoint that was never published and a server that could not be reached,
because omni.client reports the two identically (check_file_path returns 0 in both cases, as
isaaclab.utils.assets._remote_fingerprint documents). The printed message already said both; only the
docs were narrower than the behaviour.

No behaviour change in that commit — docstring and changelog wording only. test_pretrained_checkpoint.py
still passes (14 tests), as do test_pretrained_checkpoint_lookup.py and test_entrypoints.py.

On the previous run, isaaclab (core) [3/3] failed on
source/isaaclab/test/utils/test_math.py::test_quat_unique[cuda:0] — 1024 unseeded random_orientation
draws asserting a strictly positive real part. That is unrelated to this PR, which touches isaaclab_rl
only; the isaaclab_rl job passed.

@AntoineRichard

Copy link
Copy Markdown
Collaborator Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 4, 2026
@kellyguo11
kellyguo11 merged commit 768c305 into develop Sep 4, 2026
55 of 57 checks passed
@kellyguo11
kellyguo11 deleted the antoiner/fix/pretrained-checkpoint-errors branch September 4, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants