Skip to content

Document cuDNN fallback for deterministic training - #7559

Merged
kellyguo11 merged 1 commit into
isaac-sim:developfrom
kellyguo11:kellyguo11/document-cudnn-determinism
Sep 4, 2026
Merged

Document cuDNN fallback for deterministic training#7559
kellyguo11 merged 1 commit into
isaac-sim:developfrom
kellyguo11:kellyguo11/document-cudnn-determinism

Conversation

@kellyguo11

@kellyguo11 kellyguo11 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Description

Document TORCH_CUDNN_V8_API_DISABLED=1 as an optional setting that may improve run-to-run determinism for convolutional training workloads. The note explains that the setting selects PyTorch's cuDNN v7 API instead of cuDNN v8 execution plans.

No new dependencies are required.

Type of change

  • Documentation update

Release backport

  • Backport this pull request to the active release branch after it merges into develop

Screenshots

Not applicable.

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with uv run isaaclab -f
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works (not applicable to this documentation-only change)
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (no source package is touched)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

Validation

  • uv run --isolated --extra dev --extra ov -- make -C docs current-docs
  • uv run isaaclab -f

@kellyguo11
kellyguo11 requested a review from a team September 4, 2026 03:12
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 4, 2026

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The change adds a narrowly scoped note describing the optional cuDNN v7 fallback for convolutional determinism. The proposed objection that this environment variable became inert in PyTorch 2.3 is not supported by the supplied repository context or evidence, so it does not warrant maintainer action.

  • Design and architecture: The note is appropriately placed in the existing PyTorch determinism discussion and leaves the separate rendering and physics determinism responsibilities unchanged.
  • API: No Isaac Lab public API, configuration field, CLI argument, dependency, or compatibility surface is changed; the documentation only mentions an external PyTorch environment variable.
  • Implementation: The reStructuredText note is correctly formed and clearly presents the setting as optional and workload-specific. No documentation references, includes, or surrounding instructions are disrupted.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.

@greptile-apps

greptile-apps Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Documents an optional environment setting for convolutional training workloads that may improve run-to-run determinism by selecting PyTorch’s cuDNN v7 API path.

  • Adds a reproducibility note for TORCH_CUDNN_V8_API_DISABLED=1.
  • Clarifies that the setting should be applied before training starts.

Confidence Score: 5/5

The documentation-only change appears safe to merge.

The new note is narrowly scoped, uses appropriately qualified language, and introduces no accepted correctness, security, or maintainability issue.

Important Files Changed

Filename Overview
docs/source/features/reproducibility.rst Adds concise documentation for an optional cuDNN API fallback, with no concrete correctness or documentation issue identified.

Reviews (1): Last reviewed commit: "Document cuDNN fallback for deterministi..." | Re-trigger Greptile

@kellyguo11

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 4, 2026
@kellyguo11
kellyguo11 merged commit 1a986c9 into isaac-sim:develop Sep 4, 2026
54 checks passed
@isaaclab-bot

isaaclab-bot Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Backported to release/3.0.0 as 11c0a0d.

isaaclab-bot Bot pushed a commit that referenced this pull request Sep 4, 2026
# Description

Document `TORCH_CUDNN_V8_API_DISABLED=1` as an optional setting that may
improve run-to-run determinism for convolutional training workloads. The
note explains that the setting selects PyTorch's cuDNN v7 API instead of
cuDNN v8 execution plans.

No new dependencies are required.

## Type of change

- Documentation update

## Release backport

- [x] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`

## Screenshots

Not applicable.

## Checklist

Docker and GPU tests run on demand. Push the commits you want tested,
then
comment `run-ci` on the pull request.

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`uv run isaaclab -f`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works (not applicable to this documentation-only change)
- [x] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package (no source package
is touched)
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

## Validation

- `uv run --isolated --extra dev --extra ov -- make -C docs
current-docs`
- `uv run isaaclab -f`

(cherry picked from commit 1a986c9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant