[Backport release/3.0.0] Consolidate multi-GPU documentation (#7458)

## Description

Backports #7182 to `release/3.0.0` by cherry-picking merged commit
`c1b0cbd25c488d3c30d070636f8b402e9cbcdc17`.

This consolidates the multi-GPU guide, standardizes benchmark workflows
on `startup_multigpu`, `runtime_multigpu`, and `training_multigpu`, and
retains the hyphenated forms as deprecated compatibility aliases.

No new dependencies.

## Type of change

- Documentation update
- Non-breaking benchmark CLI change with deprecated compatibility
aliases

## Release backport

Not applicable; this PR targets the active release branch.

## Validation

- [x] Focused benchmark tests: 22 passed
- [x] `uv run isaaclab -f` equivalent with
`ISAACLAB_CHANGELOG_BASE_REF=release/3.0.0`: passed
- [x] Warning-as-error docs build: passed
- [x] Backport patch ID matches merged #7182

The host was macOS, while the project lock and `ovstage` wheel support
Linux/Windows. Validation therefore used a temporary uv-managed Python
3.12 environment; Sphinx autodoc mocked only the unavailable Linux
`ovstage` module.

## Checklist

- [x] I have read and understood the contribution guidelines
- [x] I have run the pre-commit checks
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove the canonical and compatibility
command dispatch
- [x] I have included the original changelog fragment for the touched
package
- [x] The original author already exists in `CONTRIBUTORS.md`

Signed-off-by: Mustafa H <34825877+StafaH@users.noreply.github.com>
Co-authored-by: Mustafa H <34825877+StafaH@users.noreply.github.com>
Co-authored-by: Antoine RICHARD <antoiner@nvidia.com>

Author: 3 people

docs/source/features/include/cluster_details.inc

@@ -204,7 +204,7 @@ conversion is needed and there is no code copy step -- the image is the unit of
 
 The workflow definition lives in ``docker/cluster/osmo_multi_gpu_workflow.yaml``. It requests a
 single node and scales training across the GPUs on that node through the
-:ref:`train_multigpu <train-multigpu-command>` command. The following parameters can be overridden
+:ref:`train_multigpu <train_multigpu-command>` command. The following parameters can be overridden
 at submission time:
 
 .. list-table::

docs/source/features/multi_gpu.rst

@@ -299,11 +299,12 @@ Benchmark a task
    * - ``play``
      - Trained-policy rollout throughput. Requires ``--rl_library`` and
        ``--checkpoint``.
-   * - ``-multigpu``
-     - Run ``startup``, ``runtime``, or ``training`` across multiple GPUs by adding the suffix ``-multigpu``. For example, ``runtime-multigpu``.
+   * - ``_multigpu``
+     - Run ``startup``, ``runtime``, or ``training`` across multiple GPUs by
+       adding the suffix ``_multigpu``. For example, ``runtime_multigpu``.
 
-See :ref:`testing_benchmarks` for warm-up, formatters, multi-GPU details, and
-how to read results.
+See :ref:`testing_benchmarks` for benchmark fundamentals and
+:ref:`train_multigpu-command` for multi-GPU training and benchmarks.
 
 Next steps
 ----------

docs/source/setup/quickstart.rst

@@ -32,7 +32,7 @@ Choose A Workflow
    * - Launch, import, configuration, scene creation, or first-step latency
      - ``startup``
    * - Any of the above across several GPUs
-     - ``<workflow>-multigpu``
+     - ``<workflow>_multigpu``
    * - One asset or sensor operation
      - :ref:`testing_micro_benchmarks`
 
@@ -343,90 +343,10 @@ they established final policy quality.
 Multi-GPU
 ---------
 
-Use It When
-~~~~~~~~~~~
-
-Append ``-multigpu`` to ``startup``, ``runtime``, or ``training`` to run the same
-workflow with one rank per GPU. Use it to measure synchronized multi-GPU training
-throughput, or to measure how much a workflow slows down when every GPU on the
-node is busy.
-
-Command
-~~~~~~~
-
-.. code-block:: bash
-
-   ./isaaclab.sh benchmark training-multigpu \
-       --rl_library rsl_rl \
-       --num_gpus 2 \
-       --task Isaac-Cartpole-Direct \
-       --num_envs 4096 \
-       --max_iterations 100 \
-       --seed 42 \
-       --visualizer none \
-       --output_path ./benchmark_results/multigpu \
-       physics=isaacsim_physx
-
-The launcher accepts ``--num_gpus``, ``--nnodes``, ``--node_rank``, and the
-``torchrun`` rendezvous options, exactly like :ref:`train-multigpu-command`. Every
-other argument is forwarded to the single-GPU workflow unchanged. Add ``--dry_run``
-to print the ``torchrun`` command without running it, and ``--log_all_ranks`` to
-show console output from every rank instead of local rank 0 only.
-
-For a multi-node run, issue the same command on every node with a distinct
-``--node_rank``:
-
-.. code-block:: bash
-
-   ./isaaclab.sh benchmark training-multigpu \
-       --rl_library rsl_rl --nnodes 2 --node_rank 0 --num_gpus 8 \
-       --rdzv_backend c10d --rdzv_endpoint host0:29400 --rdzv_id bench \
-       --task Isaac-Cartpole-Direct
-
-``training-multigpu`` supports ``rsl_rl``, ``rl_games``, and ``skrl`` with Torch. It
-does not support skrl JAX or SB3, and it rejects ``--video``,
-``--capture_env_sensors``, and ``--check_success``, none of which are meaningful
-across ranks. Use :ref:`train-multigpu-command` for general distributed training.
-
-Read The Result
-~~~~~~~~~~~~~~~
-
-``--num_envs`` is the number of environments **per rank**, and each rank creates its
-own Isaac Lab instance on its own GPU. Only global rank 0 writes a bundle. What that
-bundle covers depends on the workflow, and ``extra`` records it:
-
-.. list-table::
-   :header-rows: 1
-   :widths: 30 70
-
-   * - ``extra`` field
-     - Meaning
-   * - ``world_size``, ``local_world_size``, ``num_nodes``
-     - Rank layout of the job.
-   * - ``num_envs_per_rank``
-     - Environments hosted by each rank.
-   * - ``workload_scope``
-     - ``global`` for ``training-multigpu``: ranks train in lockstep, so ``run.num_envs``,
-       ``runtime.steps_per_iteration``, and every FPS field cover all ranks.
-       ``rank0`` for ``startup-multigpu`` and ``runtime-multigpu``: those ranks run
-       independent workloads, so the reported values are rank 0's own, measured while
-       the other ranks contend for the same host.
-   * - ``measurement_scope``
-     - ``rank0_process`` — timings, learning curves, CPU, and RAM come from rank 0 alone.
-   * - ``gpu_measurement_scope``
-     - ``rank0_node`` — ``resources.devices`` reports every GPU visible to rank 0, so a
-       single-node run shows all ranks. ``resources.gpu_util_pct`` and
-       ``resources.gpu_mem_gb`` remain scoped to rank 0's own device.
-
-Do Not Infer
-~~~~~~~~~~~~
-
-Do not compare a multi-GPU result against a single-GPU result at the same
-``--num_envs``: the multi-GPU run has ``world_size`` times as many environments. To
-measure scaling, compare the global throughput of an ``N``-GPU run against ``N``
-times the throughput of a single-GPU run at the same per-rank environment count. Do
-not read ``startup-multigpu`` or ``runtime-multigpu`` throughput as a global rate;
-their ``workload_scope`` is ``rank0``, and the other ranks were not measured.
+Append ``_multigpu`` to ``startup``, ``runtime``, or ``training`` to run one
+benchmark rank per GPU. The :ref:`train_multigpu-command` guide is the canonical
+reference for the three workflows, launcher options, multi-node setup, supported
+RL libraries, and result interpretation.
 
 Startup Profiling
 -----------------

docs/source/testing/benchmarks.rst

@@ -0,0 +1,7 @@
+Changed
+^^^^^^^
+
+* Standardized multi-GPU benchmark commands on the underscore suffix. Replace
+  ``startup-multigpu``, ``runtime-multigpu``, and ``training-multigpu`` with
+  ``startup_multigpu``, ``runtime_multigpu``, and ``training_multigpu``. The hyphenated forms
+  remain available as deprecated aliases.

source/isaaclab/changelog.d/mh-train-multigpu-docs.rst

@@ -70,7 +70,7 @@ def run_benchmark_request(request: BenchmarkRequest) -> BenchmarkResult:
 def run_benchmark_cli(argv: list[str] | None = None) -> int:
     """Run a runtime, startup, training, or play benchmark from CLI arguments.
 
-    Appending ``-multigpu`` to a workflow name runs it across several GPUs; see
+    Appending ``_multigpu`` to a workflow name runs it across several GPUs; see
     :mod:`isaaclab.benchmark.entrypoints.multigpu`.
     """
     from .entrypoints import multigpu
@@ -79,14 +79,27 @@ def run_benchmark_cli(argv: list[str] | None = None) -> int:
         argv = sys.argv[1:]
     argv = _fuse_kit_args(argv)
     multigpu_workflows = tuple(f"{name}{multigpu.MULTIGPU_SUFFIX}" for name in multigpu.MULTIGPU_WORKFLOWS)
+    legacy_multigpu_workflows = tuple(
+        f"{name}{multigpu.LEGACY_MULTIGPU_SUFFIX}" for name in multigpu.MULTIGPU_WORKFLOWS
+    )
     parser = argparse.ArgumentParser(description="Run an Isaac Lab benchmark.")
-    parser.add_argument("workflow", choices=(*WORKFLOW_MODULES, *_RL_WORKFLOW_MODULES, *multigpu_workflows))
+    parser.add_argument(
+        "workflow",
+        choices=(*WORKFLOW_MODULES, *_RL_WORKFLOW_MODULES, *multigpu_workflows, *legacy_multigpu_workflows),
+    )
     if not argv or argv[0] in ("-h", "--help"):
         parser.parse_args(argv)
     selected = parser.parse_args(argv[:1])
     if selected.workflow in multigpu_workflows:
         workflow = selected.workflow[: -len(multigpu.MULTIGPU_SUFFIX)]
         return multigpu.run_multigpu_benchmark_cli(workflow, argv[1:])
+    if selected.workflow in legacy_multigpu_workflows:
+        workflow = selected.workflow[: -len(multigpu.LEGACY_MULTIGPU_SUFFIX)]
+        print(
+            f"'{selected.workflow}' is deprecated. Use '{workflow}{multigpu.MULTIGPU_SUFFIX}' instead.",
+            file=sys.stderr,
+        )
+        return multigpu.run_multigpu_benchmark_cli(workflow, argv[1:])
     if selected.workflow in _RL_WORKFLOW_MODULES:
         return _run_rl_cli(selected.workflow, argv[1:])
     module = importlib.import_module(WORKFLOW_MODULES[selected.workflow])

source/isaaclab/isaaclab/benchmark/dispatch.py

@@ -59,7 +59,7 @@ def from_env(cls, enabled: bool, *, workflow: str = "training") -> DistributedCo
         if "RANK" not in os.environ or "WORLD_SIZE" not in os.environ:
             raise ValueError(
                 "--distributed was requested but no distributed launcher exported RANK and WORLD_SIZE."
-                f" Launch the benchmark with `isaaclab benchmark {workflow}-multigpu`."
+                f" Launch the benchmark with `isaaclab benchmark {workflow}_multigpu`."
             )
         return cls(
             enabled=True,
@@ -164,7 +164,7 @@ def add_distributed_arg(parser: argparse.ArgumentParser) -> None:
         action="store_true",
         default=False,
         help=(
-            "Run as one rank of a distributed launch. Set automatically by `isaaclab benchmark <workflow>-multigpu`."
+            "Run as one rank of a distributed launch. Set automatically by `isaaclab benchmark <workflow>_multigpu`."
         ),
     )
 

source/isaaclab/isaaclab/benchmark/distributed.py

@@ -15,12 +15,12 @@
 
 Usage example::
 
-    uv run isaaclab benchmark startup-multigpu \\
+    uv run isaaclab benchmark startup_multigpu \\
         --task Isaac-Cartpole-Direct \\
         --num_gpus 2 \\
         presets=newton_mjwarp
 
-    uv run isaaclab benchmark training-multigpu \\
+    uv run isaaclab benchmark training_multigpu \\
         --rl_library rsl_rl \\
         --num_gpus 2 \\
         --task Isaac-Cartpole-Direct \\
@@ -36,9 +36,12 @@
 
 WORKER_SCRIPT = str(Path(__file__).resolve())
 
-MULTIGPU_SUFFIX = "-multigpu"
+MULTIGPU_SUFFIX = "_multigpu"
 """Suffix that turns a benchmark workflow name into its multi-GPU variant."""
 
+LEGACY_MULTIGPU_SUFFIX = "-multigpu"
+"""Deprecated suffix retained for command-line compatibility."""
+
 MULTIGPU_WORKFLOWS = ("startup", "runtime", "training")
 """Benchmark workflows that offer a multi-GPU variant."""
 

source/isaaclab/isaaclab/benchmark/entrypoints/multigpu.py

@@ -18,7 +18,7 @@
         --num_envs 16 --num_steps 1000 --warmup_steps 50 \\
         presets=newton_mjwarp --visualizer none
 
-Use ``isaaclab benchmark runtime-multigpu`` to measure rank 0 while every GPU steps an
+Use ``isaaclab benchmark runtime_multigpu`` to measure rank 0 while every GPU steps an
 independent workload; see :mod:`isaaclab.benchmark.entrypoints.multigpu`.
 """
 

source/isaaclab/isaaclab/benchmark/entrypoints/runtime.py

@@ -25,7 +25,7 @@
         --num_envs 16 \\
         presets=newton_mjwarp
 
-Use ``isaaclab benchmark startup-multigpu`` to profile rank 0 while every GPU launches
+Use ``isaaclab benchmark startup_multigpu`` to profile rank 0 while every GPU launches
 concurrently; see :mod:`isaaclab.benchmark.entrypoints.multigpu`.
 """