Skip to content

[Workflow/Docs] Consolidate multi-GPU documentation - #7182

Merged
StafaH merged 14 commits into
isaac-sim:developfrom
StafaH:mh/train-multigpu-docs
Aug 31, 2026
Merged

[Workflow/Docs] Consolidate multi-GPU documentation#7182
StafaH merged 14 commits into
isaac-sim:developfrom
StafaH:mh/train-multigpu-docs

Conversation

@StafaH

@StafaH StafaH commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Description

Reworks the multi-GPU guide around a task-first flow modeled after the quickstart and installation pages while retaining the established train_multigpu command spelling.

  • starts with the single-GPU-to-multi-GPU workflow using train_multigpu
  • explains GPU selection, per-rank environment counts, supported RL libraries, and logging
  • standardizes benchmark workflows as startup_multigpu, runtime_multigpu, and training_multigpu
  • retains the former hyphenated benchmark names as deprecated, warning-emitting compatibility aliases
  • clarifies benchmark measurement scope and scaling comparisons
  • simplifies multi-node setup and keeps focused NCCL troubleshooting guidance
  • updates OSMO and repository-owned skill references to the underscore convention

No new dependencies.

Type of change

  • Documentation update
  • Non-breaking benchmark CLI change with deprecated compatibility aliases

Release backport

  • Backport this pull request to the active release branch after it merges into develop

Screenshots

Not applicable.

Validation

  • uv run --extra test pytest -q source/isaaclab/test/benchmark/test_multigpu_launcher.py source/isaaclab/test/benchmark/test_distributed.py (22 passed)
  • uv run isaaclab -f
  • uv run --no-project python tools/skills/cli.py check (21 skills validated)
  • uv run --isolated --extra test -- make -C docs current-docs (warning-free)

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove canonical and compatibility command dispatch
  • I have added changelog fragments for every touched source package
  • My name already exists in CONTRIBUTORS.md

@StafaH
StafaH requested a review from a team August 19, 2026 19:49
@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team infrastructure labels Aug 19, 2026

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The CLI rename preserves the deprecated underscore alias and updates documentation and tests, but the OSMO workflow now invokes the new command inside a pinned container image that predates the rename.

  • Design and architecture: The alias-based migration preserves compatibility for updated checkouts. However, the OSMO workflow executes the CLI supplied by its container rather than the current checkout, so changing the template command independently of the pinned image creates a producer/consumer version mismatch.
  • API: The local CLI supports both train-multigpu and the warning-emitting train_multigpu alias. The pinned nvcr.io/nvidia/isaac-lab:3.0.0-beta2 image does not include this newly introduced canonical spelling, making the updated OSMO default incompatible with that image.
  • Implementation: The CLI dispatch and alias forwarding are covered by focused tests. The OSMO multi-GPU path at line 64 should retain train_multigpu until its default container image includes the renamed command, or the image and command must be updated together.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

Comment thread docker/cluster/osmo_multi_gpu_workflow.yaml Outdated
@greptile-apps

greptile-apps Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR consolidates multi-GPU documentation around the canonical underscore command spelling while preserving deprecated hyphenated benchmark aliases.

  • Reorganizes the multi-GPU guide around training, GPU selection, benchmarking, multi-node operation, and NCCL troubleshooting.
  • Standardizes benchmark commands as startup_multigpu, runtime_multigpu, and training_multigpu.
  • Retains the former hyphenated benchmark commands as warning-emitting compatibility aliases.
  • Updates CLI help, diagnostics, tests, and changelog documentation to match the new convention.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
docs/source/features/multi_gpu.rst Replaces the previous guide with a task-first reference covering distributed training, benchmarks, logs, multi-node setup, and troubleshooting.
source/isaaclab/isaaclab/benchmark/dispatch.py Dispatches canonical underscore benchmark names and preserves hyphenated names as deprecated aliases.
source/isaaclab/isaaclab/benchmark/entrypoints/multigpu.py Changes the canonical benchmark suffix to _multigpu and defines the former suffix as the compatibility spelling.
source/isaaclab/test/benchmark/test_multigpu_launcher.py Verifies canonical command dispatch and deprecated alias warning behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  CLI[Benchmark CLI] --> Canonical{Command spelling}
  Canonical -->|workflow_multigpu| Launcher[Multi-GPU launcher]
  Canonical -->|workflow-multigpu| Warning[Deprecation warning]
  Warning --> Launcher
  Launcher --> Workers[One worker per GPU]
Loading

Reviews (2): Last reviewed commit: "Apply suggestion from @AntoineRichard" | Re-trigger Greptile

Comment thread docker/cluster/osmo_multi_gpu_workflow.yaml Outdated
@StafaH StafaH changed the title Rename multi-GPU training command and rewrite guide [Workflow/Docs] Rename multi-GPU training command and rewrite guide Aug 19, 2026
Comment on lines +210 to +215
.. note::

The template uses the deprecated ``train_multigpu`` spelling because its default
``3.0.0-beta2`` container image predates the rename. Custom images built from current source can
use ``train-multigpu``.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to support this? I don't know if for this in particular applying a deprecation cycle on beta 2 makes sense.

Comment thread docs/source/features/multi_gpu.rst Outdated
"train": train,
"train_multigpu": train_multigpu,
"train-multigpu": train_multigpu,
"train_multigpu": _train_multigpu_deprecated,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we remove this?

@StafaH
StafaH marked this pull request as draft August 23, 2026 07:26
@StafaH StafaH changed the title [Workflow/Docs] Rename multi-GPU training command and rewrite guide [Workflow/Docs] Consolidate multi-GPU documentation Aug 29, 2026
Co-authored-by: Antoine RICHARD <antoiner@nvidia.com>
Signed-off-by: Mustafa H <34825877+StafaH@users.noreply.github.com>
@StafaH
StafaH marked this pull request as ready for review August 29, 2026 21:38
@StafaH

StafaH commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Aug 29, 2026
@StafaH

StafaH commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

run-co

@StafaH

StafaH commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Aug 30, 2026
@StafaH

StafaH commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Aug 31, 2026
@StafaH
StafaH merged commit c1b0cbd into isaac-sim:develop Aug 31, 2026
48 checks passed
kellyguo11 added a commit that referenced this pull request Sep 2, 2026
## Description

Backports #7182 to `release/3.0.0` by cherry-picking merged commit
`c1b0cbd25c488d3c30d070636f8b402e9cbcdc17`.

This consolidates the multi-GPU guide, standardizes benchmark workflows
on `startup_multigpu`, `runtime_multigpu`, and `training_multigpu`, and
retains the hyphenated forms as deprecated compatibility aliases.

No new dependencies.

## Type of change

- Documentation update
- Non-breaking benchmark CLI change with deprecated compatibility
aliases

## Release backport

Not applicable; this PR targets the active release branch.

## Validation

- [x] Focused benchmark tests: 22 passed
- [x] `uv run isaaclab -f` equivalent with
`ISAACLAB_CHANGELOG_BASE_REF=release/3.0.0`: passed
- [x] Warning-as-error docs build: passed
- [x] Backport patch ID matches merged #7182

The host was macOS, while the project lock and `ovstage` wheel support
Linux/Windows. Validation therefore used a temporary uv-managed Python
3.12 environment; Sphinx autodoc mocked only the unavailable Linux
`ovstage` module.

## Checklist

- [x] I have read and understood the contribution guidelines
- [x] I have run the pre-commit checks
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove the canonical and compatibility
command dispatch
- [x] I have included the original changelog fragment for the touched
package
- [x] The original author already exists in `CONTRIBUTORS.md`

Signed-off-by: Mustafa H <34825877+StafaH@users.noreply.github.com>
Co-authored-by: Mustafa H <34825877+StafaH@users.noreply.github.com>
Co-authored-by: Antoine RICHARD <antoiner@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants