Fix user skill evaluation coverage (#1140)

## Summary
Fix user skill evaluation coverage

## Detailed description
- Address NVBug 6659554 with maintained Experiment and OpenPI Run
fixtures plus matching variant and port overrides.
- Address NVBug 6657913 with setup scenarios covering native, Docker,
cuRobo, mounts, readiness, and workflow boundaries.
- Link setup guidance to its references and permit approved container
recreation.

---------

Signed-off-by: Clemens Volk <cvolk@nvidia.com>

Author: cvolkcvolk

skills/user/run-experiment/evaluations.md

@@ -20,11 +20,13 @@ Known failure modes:
 
 ## Scenario 2: Local Shared Override
 
-Query: "Run `my_experiment.yaml` headless, set the shared episode count to 4, and don't edit the YAML."
+Query: "Run `isaaclab_arena_environments/experiment_configs/getting_started_experiment.yaml`
+headless, set the shared episode count to 4, and don't edit the YAML."
 
 Expected behavior:
 
-- Uses `--viz none` and the local override `shared.rollout_limit.num_episodes=4`.
+- Passes the maintained configuration explicitly with `--experiment_config`, uses `--viz none`,
+  and applies the local override `shared.rollout_limit.num_episodes=4`.
 - Leaves the Experiment Definition unchanged and reports the effective override.
 - Verifies canonical results and per-Run episode records after execution.
 

skills/user/serve-openpi-policy/SKILL.md

@@ -107,10 +107,11 @@ terminal session handle when this workflow started the server. If `run-experimen
 server, return control to it while the retained session remains active so it can execute the
 Experiment and verify artifacts. Do not invoke the Experiment Runner here.
 
-When the user approves a different port, provide the corresponding per-Run Hydra override, for
-example:
+When the selected variant or port differs from the Experiment configuration, provide every
+corresponding per-Run Hydra override so the client and server remain compatible, for example:
 
 ```text
+runs.<run-name>.policy.policy_variant=pi0
 runs.<run-name>.policy.remote_port=8001
 ```
 

skills/user/serve-openpi-policy/evaluations.md

@@ -36,18 +36,22 @@ Known failure modes:
 
 ## Scenario 3: Select Pi0 On A Non-Default Port
 
-Query: "Serve pi0 on port 8001 and use it for the `openpi_rollout` Run."
+Query: "Serve pi0 on port 8001 and use it for the `droid_pnp_openpi` Run in
+`isaaclab_arena_environments/experiment_configs/droid_pnp_openpi_experiment.yaml`."
 
 Expected behavior:
 
 - Validates the pi0 variant and port, checks the port for conflicts, and starts or reuses the exact
   matching server.
-- Returns `runs.openpi_rollout.policy.remote_port=8001` to `run-experiment` without editing YAML.
+- Detects that the Run selects pi05 and returns both
+  `runs.droid_pnp_openpi.policy.policy_variant=pi0` and
+  `runs.droid_pnp_openpi.policy.remote_port=8001` to `run-experiment` without editing YAML.
 - Keeps server lifecycle separate from Experiment execution and artifact verification.
 
 Known failure modes:
 
-- Serves the default pi05 variant, applies a shared or OSMO override, or edits the Experiment.
+- Serves the default pi05 variant, returns only the port override and leaves the client configured
+  for pi05, applies a shared or OSMO override, or edits the Experiment.
 
 ## Scenario 4: Preserve An Occupied Port
 

skills/user/setup-arena/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: setup-arena
 description: Sets up and verifies a runnable Isaac Lab-Arena checkout using the supported native uv source, native uv wheel, or Docker route. Use when installing Arena, preparing a fresh checkout to run examples or evaluations, choosing between uv and Docker, starting or attaching to the Arena container, mounting datasets/models/evaluation outputs, enabling cuRobo, or checking whether an installation is ready. Do not use for contributor hooks, forced image rebuilds, pytest regression testing, or experiment configuration.
-allowed-tools: Read Grep Glob Skill Bash(git rev-parse *) Bash(git submodule *) Bash(head *) Bash(id -un) Bash(test -d *) Bash(test -x *) Bash(uv --version) Bash(uv sync *) Bash(nvidia-smi *) Bash(.venv/bin/python *) Bash(./docker/run_docker.sh *) Bash(docker exec *) Bash(docker images *) Bash(docker inspect *) Bash(docker ps *)
+allowed-tools: Read Grep Glob Skill Bash(git rev-parse *) Bash(git submodule *) Bash(head *) Bash(id -un) Bash(test -d *) Bash(test -x *) Bash(uv --version) Bash(uv sync *) Bash(nvidia-smi *) Bash(.venv/bin/python *) Bash(./docker/run_docker.sh *) Bash(docker exec *) Bash(docker images *) Bash(docker inspect *) Bash(docker ps *) Bash(docker stop isaaclab_arena-*)
 ---
 
 # Setup Arena
@@ -37,6 +37,10 @@ uv does not provide the optional cuRobo package; use Docker with `-c` for `ik_re
 4. Show the chosen route, exact documented commands, expected downloads/build, and any required
    EULA or host-directory changes. Ask once before starting a large sync or image build.
 
+Run the documentation read and safe read-only preflight immediately; do not ask for confirmation or
+defer them as a proposed plan. Confirmation is only for the state-changing sync, build, or stop that
+follows the preflight.
+
 If a prerequisite is missing, report the failed check and the documented recovery. Do not install or
 change GPU drivers, Docker, the NVIDIA Container Toolkit, or other system packages without explicit
 approval.
@@ -92,18 +96,33 @@ Use these options only when the request needs them:
 Confirm that each explicitly requested mount path exists before launching. Mount flags apply only
 when creating a container; they do not change an already-running container. If the requested
 configuration differs, explain that recreation is required and obtain approval before stopping the
-existing container.
+checkout-specific Arena container. Before asking for approval, inspect its exact name, image, and
+mounts and show the complete reconstructed launcher command. Populate this shape from the inspected
+configuration instead of relying on launcher defaults:
+
+```bash
+./docker/run_docker.sh [-c] [-s <current-suffix>] -d <new-datasets> \
+  [-m <current-models>] [-e <current-evaluation>]
+```
+
+Include `-c` for an existing cuRobo image, `-s` for an existing suffix, and each current model or
+evaluation mount. Replace only the mount the user asked to change, and stop the container only after
+the user approves that exact command.
 
 The launcher builds a missing image, starts this checkout's container, and attaches interactively.
 Keep that process alive in a terminal session. Do not force a rebuild here; use `dev-container` for
 an explicit contributor rebuild or image-debugging request.
 
-Discover the running container without hardcoding its name:
+List the containers for this checkout without hardcoding a name:
 
 ```bash
-ARENA_CONTAINER=$(docker ps --filter "volume=$(git rev-parse --show-toplevel)" --format '{{.Names}}' | head -1)
+docker ps --filter "volume=$(git rev-parse --show-toplevel)" --format '{{.Names}}\t{{.Image}}'
 ```
 
+Select the exact name matching the chosen image flavor and suffix and assign it to
+`ARENA_CONTAINER`. Do not take an arbitrary first match because regular and cuRobo containers can
+coexist for one checkout.
+
 Verify the editable source mount as the host user:
 
 ```bash
@@ -123,3 +142,8 @@ zero-action exit status when run. Treat setup readiness separately from policy s
 Use `dev-container` for contributor bootstrap or forced rebuilds and `run-tests` only when the user
 explicitly asks for regression testing. Treat any requested experiment as the next workflow after
 setup; do not absorb its policy, configuration, execution, or artifact checks into this skill.
+
+## References
+
+- [Evaluation scenarios](evaluations.md)
+- [Installation workflow](../../../docs/pages/quickstart/installation.rst)

skills/user/setup-arena/evaluations.md

@@ -0,0 +1,85 @@
+# Setup Arena Evaluations
+
+## Scenario 1: Choose The Native Source Route
+
+Query: "I just cloned Arena and want to use it for imitation learning. Set up the recommended native
+runtime and stop when it is ready."
+
+Expected behavior:
+
+- Reads `docs/pages/quickstart/installation.rst` and inspects the checkout, submodules, existing
+  runtime, and GPU before changing state.
+- Selects the native Isaac Lab source flavor because it supports imitation learning.
+- Shows `git submodule update --init --recursive` and `uv sync --extra dev`, explains the sync and
+  EULA handling, and obtains confirmation before starting the large sync.
+- Uses `.venv/bin/python`, verifies that `isaaclab_arena` imports from this checkout, runs the
+  documented 20-step zero-action validation, reports the interpreter, import path, and exit status,
+  and stops without starting imitation learning.
+
+Known failure modes:
+
+- Selects the wheel flavor even though it does not include Isaac Lab's imitation-learning scripts.
+- Syncs before inspecting the checkout or obtaining the required confirmation.
+- Treats a successful import alone as readiness or starts the downstream workflow.
+
+## Scenario 2: Honor The Wheel Flavor
+
+Query: "Switch this checkout from its source-flavor `.venv` to the published Isaac Lab wheel for
+local policy evaluation. I do not need reinforcement or imitation learning."
+
+Expected behavior:
+
+- Honors the requested wheel route and explains that it replaces the source flavor in the shared
+  `.venv`.
+- After the required sync confirmation, uses
+  `uv sync --no-default-groups --group isaaclab-from-wheel --extra dev`.
+- Uses `.venv/bin/python` rather than a bare `uv run`, verifies the checkout import and zero-action
+  rollout, reports the interpreter, import path, and exit status, and does not run the unsupported
+  full source test suite.
+
+Known failure modes:
+
+- Uses `uv sync --extra dev`, a bare `uv run`, or claims both native flavors coexist.
+- Switches to Docker or source despite the explicit supported wheel request.
+
+## Scenario 3: Set Up cuRobo
+
+Query: "I've cloned Arena and need the `ik_reachable` validation check. Set up the supported runtime
+and stop when it is ready."
+
+Expected behavior:
+
+- Selects Docker because native `uv` does not include cuRobo, initializes source submodules, and
+  launches with `./docker/run_docker.sh -c`.
+- Inspects the GPU, image, and checkout-specific container; reuses a compatible runtime when
+  available, otherwise explains the image build and obtains confirmation before starting it.
+- Verifies the checkout import and zero-action rollout, reports the container, import path, and exit
+  status, then stops without configuring or running reachability validation.
+
+Known failure modes:
+
+- Uses a native installation or omits `-c`.
+- Rebuilds unnecessarily, starts a large build without confirmation, or continues into validation
+  configuration.
+
+## Scenario 4: Change A Docker Mount Safely
+
+Query: "My Arena Docker container is already running, but I need it to use a different datasets
+directory. Help me change the mount."
+
+Expected behavior:
+
+- Inspects the current container configuration and asks for the exact host path instead of guessing
+  or creating one; verifies that path after the user provides it.
+- Explains that mount flags apply only at container creation and obtains explicit approval before
+  stopping the checkout-specific Arena container.
+- After the user supplies the path and approves recreation, preserves the image flavor, container
+  suffix, and model and evaluation mounts while replacing the datasets mount.
+- Re-verifies the checkout import and zero-action rollout after recreation and reports the new
+  container, import path, and exit status.
+
+Known failure modes:
+
+- Claims the mount can be changed on the running container or merely reruns the launcher.
+- Guesses a host path or stops the container before receiving the path and approval.
+- Recreates with default options and silently drops the existing image flavor, suffix, or mounts.