Report why fetching a pre-trained checkpoint failed (#7535)

# Description

Running a play script with `--checkpoint pretrained` on a machine whose
`.pretrained_checkpoints` directory was left behind by a container run
(owned by `root`) prints:

```
Fetching pre-trained checkpoint : https://.../rsl_rl/Isaac-Humanoid-Direct_newtonmjwarp_none_rsl_rl.pt
A pre-trained checkpoint is currently unavailable for this task.
```

and exits. The checkpoint is published and reachable; the download
failed with
`PermissionError: [Errno 13] Permission denied:
'.../.pretrained_checkpoints/rsl_rl/https'`. That error
was discarded by a bare `except Exception` in

`isaaclab_rl.utils.pretrained_checkpoint.get_published_pretrained_checkpoint`,
which reported every
failure — a missing checkpoint, an unreachable server, an unwritable
cache, a full disk — with the same
sentence and returned `None`, after which the play scripts return
without further explanation.

This separates the two cases:

* **The asset server does not provide the checkpoint.** `None` is still
returned and the message still
starts with `A pre-trained checkpoint is currently unavailable for this
task.`, so callers and tests
matching on that line are unaffected. It now also names the location
that was tried, the task and
  backends that location was derived from, and what to do instead:

  ```
  A pre-trained checkpoint is currently unavailable for this task.
The asset server does not provide
'https://.../Isaac-Not-A-Real-Task_newtonmjwarp_none_rsl_rl.pt'.
Either no checkpoint is published for task 'Isaac-Not-A-Real-Task' with
the 'newtonmjwarp' physics
and 'none' render backends, or the asset server could not be reached.
Train the task, or pass --checkpoint <path> to use a checkpoint of your
own.
  ```

* **The checkpoint exists but could not be downloaded.** This is a local
problem the user has to fix, so
it raises `RuntimeError` naming the checkpoint, the cache directory and
the originating error, which is
  chained as the cause:

  ```
RuntimeError: Failed to download the pre-trained checkpoint
'https://.../Isaac-Humanoid-Direct_newtonmjwarp_none_rsl_rl.pt'
into '/home/user/IsaacLab/.pretrained_checkpoints/rsl_rl':
PermissionError: [Errno 13] Permission denied:
'/home/user/IsaacLab/.pretrained_checkpoints/rsl_rl/https'. Check that
the cache directory is writable and
that the disk is not full; a directory left behind by a container run is
owned by root.
  ```

Note that `omni.client` reports a checkpoint that was never published
and a server it cannot reach
identically, so both remain covered by the "unavailable" message.

Both paths were verified end to end against the live asset server, using
a published checkpoint with a
read-only cache directory and an unpublished checkpoint name.

## Type of change

- Bug fix (non-breaking change which fixes an issue)

Behaviour note: callers that relied on `None` to mask a local download
failure now see a `RuntimeError`.
Callers that treat `None` as "no checkpoint published for this task" —
every caller in the repository —
are unchanged. This is called out in the changelog fragment.

## Release backport

- [ ] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`

## Checklist

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

Author: AntoineRichard

source/isaaclab_rl/changelog.d/antoiner-pretrained-checkpoint-errors.minor.rst

@@ -0,0 +1,17 @@
+Fixed
+^^^^^
+
+* Fixed :func:`~isaaclab_rl.utils.pretrained_checkpoint.get_published_pretrained_checkpoint` reporting
+  every download failure as ``A pre-trained checkpoint is currently unavailable for this task.``. A
+  checkpoint the asset server does not provide is still reported that way, but the message now names the
+  location that was tried, the task and backends it was derived from, and what to do instead.
+
+Changed
+^^^^^^^
+
+* Changed :func:`~isaaclab_rl.utils.pretrained_checkpoint.get_published_pretrained_checkpoint` to raise
+  ``RuntimeError`` when a published checkpoint cannot be downloaded, for instance when the
+  ``.pretrained_checkpoints`` cache directory is not writable, instead of returning ``None``. The
+  originating error is chained as the cause. ``None`` is now returned only when the asset server does not
+  report the checkpoint, which covers both an unpublished checkpoint and a server that could not be
+  reached; callers that relied on ``None`` to mask local download failures must catch ``RuntimeError``.

source/isaaclab_rl/isaaclab_rl/utils/pretrained_checkpoint.py

@@ -239,7 +239,16 @@ def get_published_pretrained_checkpoint(
             to use the legacy checkpoint layout.
 
     Returns:
-        The path.
+        The path, or None when the asset server does not report a checkpoint for this task
+        and backend combination. That covers both a checkpoint that was never published and
+        a server that could not be reached, which ``omni.client`` does not distinguish, so a
+        transient outage is not evidence that a checkpoint does not exist. The reason is
+        printed before returning.
+
+    Raises:
+        RuntimeError: If the checkpoint is published but could not be downloaded, for
+            instance because the local cache directory is not writable. The originating
+            error is chained as the cause.
     """
     filename = get_pretrained_checkpoint_filename(workflow, task_name, physics_backend, render_backend)
     ov_path = get_published_pretrained_checkpoint_path(workflow, task_name, physics_backend, render_backend)
@@ -252,9 +261,35 @@ def get_published_pretrained_checkpoint(
         print(f"Fetching pre-trained checkpoint : {ov_path}")
         try:
             resume_path = retrieve_file_path(ov_path, download_dir)
-        except Exception:
-            print("A pre-trained checkpoint is currently unavailable for this task.")
+        except FileNotFoundError:
+            # the asset server reports a checkpoint that was never published and a server it
+            # cannot reach the same way, so both are covered by the same message
+            backends = (
+                ""
+                if physics_backend is None
+                else f" with the '{physics_backend}' physics and '{render_backend}' render backends"
+            )
+            print(
+                "A pre-trained checkpoint is currently unavailable for this task.\n"
+                f"  The asset server does not provide '{ov_path}'.\n"
+                f"  Either no checkpoint is published for task '{task_name}'{backends}, or the asset"
+                " server could not be reached.\n"
+                "  Train the task, or pass --checkpoint <path> to use a checkpoint of your own."
+            )
             return None
+        except Exception as exc:
+            # the checkpoint exists on the server, so this is a local failure the user has to fix;
+            # reporting it as an unavailable checkpoint would send them looking in the wrong place
+            hint = ""
+            if isinstance(exc, OSError):
+                hint = (
+                    " Check that the cache directory is writable and that the disk is not full;"
+                    " a directory left behind by a container run is owned by root."
+                )
+            raise RuntimeError(
+                f"Failed to download the pre-trained checkpoint '{ov_path}' into"
+                f" '{os.path.abspath(download_dir)}': {type(exc).__name__}: {exc}.{hint}"
+            ) from exc
     else:
         print("Using pre-fetched pre-trained checkpoint")
     return resume_path

source/isaaclab_rl/test/test_pretrained_checkpoint.py

@@ -174,3 +174,55 @@ def _retrieve_file_path(remote_path: str, download_dir: str) -> str:
         "remote_path": "omniverse://IsaacLab/PretrainedCheckpoints/rsl_rl/Isaac-Cartpole_physx_none_rsl_rl.pt",
         "download_dir": expected_download_dir,
     }
+
+
+def test_get_published_pretrained_checkpoint_reports_unpublished_checkpoint(
+    monkeypatch: pytest.MonkeyPatch,
+    tmp_path: Path,
+    capsys: pytest.CaptureFixture[str],
+):
+    """Test that a checkpoint missing from the asset server names the location that was tried."""
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(pretrained_checkpoint, "ISAACLAB_NUCLEUS_DIR", "omniverse://IsaacLab")
+
+    def _retrieve_file_path(remote_path: str, download_dir: str) -> str:
+        raise FileNotFoundError(f"Unable to find the file: {remote_path}")
+
+    monkeypatch.setattr(pretrained_checkpoint, "retrieve_file_path", _retrieve_file_path)
+
+    path = pretrained_checkpoint.get_published_pretrained_checkpoint(
+        "rsl_rl",
+        "Isaac-Cartpole",
+        "physx",
+        "none",
+    )
+
+    assert path is None
+    output = capsys.readouterr().out
+    assert "A pre-trained checkpoint is currently unavailable for this task." in output
+    assert "omniverse://IsaacLab/PretrainedCheckpoints/rsl_rl/Isaac-Cartpole_physx_none_rsl_rl.pt" in output
+    assert "'physx' physics and 'none' render backends" in output
+
+
+def test_get_published_pretrained_checkpoint_raises_on_unwritable_cache(
+    monkeypatch: pytest.MonkeyPatch,
+    tmp_path: Path,
+):
+    """Test that a local download failure is reported instead of being reported as unavailable."""
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(pretrained_checkpoint, "ISAACLAB_NUCLEUS_DIR", "omniverse://IsaacLab")
+    cause = PermissionError(13, "Permission denied", str(tmp_path / ".pretrained_checkpoints"))
+
+    def _retrieve_file_path(remote_path: str, download_dir: str) -> str:
+        raise cause
+
+    monkeypatch.setattr(pretrained_checkpoint, "retrieve_file_path", _retrieve_file_path)
+
+    with pytest.raises(RuntimeError) as error:
+        pretrained_checkpoint.get_published_pretrained_checkpoint("rsl_rl", "Isaac-Cartpole", "physx", "none")
+
+    message = str(error.value)
+    assert "Isaac-Cartpole_physx_none_rsl_rl.pt" in message
+    assert str(tmp_path / ".pretrained_checkpoints" / "rsl_rl") in message
+    assert "Permission denied" in message
+    assert error.value.__cause__ is cause