Keep the CloudXR runtime on the GPU the XR frames are rendered on

[2047767028-lang]

Description

On a multi-GPU workstation an XR teleop session connects, starts, and streams frames whose
contents are garbage — the headset shows noise. Nothing reports an error: the client
connects, IsaacTeleop session started is logged, and the CloudXR encoder reports normal
per-frame timings (GpuEndToEncodeEnd 7.1ms), which makes it look like a client or network
problem rather than a device-selection one.

Two independent device selections drift apart.

1. The CloudXR runtime picks its own GPU. _ensure_cloudxr_runtime constructs
CloudXRLauncher without saying which device to use, so the runtime falls back to automatic
selection (gpuIndexVulkan: -1 in cxr_server.log) and takes the first Vulkan physical
device. Vulkan's enumeration is unrelated to the CUDA ordering Isaac Lab selects the
simulation and renderer devices with. On the host this was found on:

Vulkan index	nvidia-smi index
0	2
1	llvmpipe (software rasterizer)
2	0
3	1

So the compositor imported swapchain memory from a card that holds no rendered frames.

2. The renderer is not pinned either, so there is no single "render GPU" to match.
_resolve_kit_args applies --/renderer/multiGpu/activeCudaGpus only when
launcher_args["multi_gpu"] is False, and that key is assigned in exactly one place: the
distributed branch of _resolve_device_settings. There is no --multi_gpu or
--distributed CLI argument, and _sim_app_config is built by intersecting with the keys
actually present, so a plain run never sets it. Probing carb settings after startup:

Command	/physics/cudaDevice	activeCudaGpus	multiGpu/enabled
--device cuda:1	1	None	True
--device cuda:1 --kit_args "--/renderer/multiGpu/activeCudaGpus=1,"	1	'1,'	True

After the change, the same probe confirms the narrowed scope:

Command	activeCudaGpus	/physics/cudaDevice
--xr (resolves to cpu)	None — unchanged from develop	0
--xr --device cuda:1	'1,'	1

Physics lands on GPU 1 while the renderer spans every visible GPU — visible in the log as
Usdrt Hydra CUDA Peer Memory Copies from device[0] to device[2] is NOT possible as peer access is disabled, and as an OOM abort when one of those cards is busy. The comment above
launcher_args["physics_gpu"] already states that "the renderer device is selected in
_resolve_kit_args", which is what the gate prevents outside distributed runs.

The change

Pin the renderer to the simulation device when XR is enabled and a CUDA device has been
selected, and point the CloudXR runtime at that same device through
NV_CXR_GPU_INDEX_CUDA.

A bare --xr resolves to device = "cpu" in _resolve_device_settings, and is deliberately
left alone: there is no simulation GPU to align to, and forcing CUDA 0 would break hosts whose
display is not on GPU 0 (Kit's auto-selection and CloudXR's Vulkan auto-selection agree there
today). With the renderer unpinned the teleop side also stands down, so that path is unchanged
on both ends.

The scope is deliberately limited to XR, where a single stereo swapchain has to be imported
by the compositor. Non-XR single-process runs keep their current behaviour, so the trade-off
settled in #7057 is untouched — that PR's concern was distributed ranks under a
CUDA_VISIBLE_DEVICES mask, and its gate stays exactly as it is for those runs.

The runtime rejects setting both index variables at once
(Only one of gpu-index-vulkan and gpu-index-cuda may be set at a time), so an index already
present in the process environment or in the --cloudxr_env profile is left alone. The CUDA
index is used rather than the Vulkan one because it is the same index space --device cuda:N
already speaks, so no Vulkan enumeration has to be parsed.

Validation

End to end on a 4× RTX 5090 host with a Meta Quest 3 over CloudXR.js, Isaac Lab 3.0.0,
IsaacContrib-Stack-Cube-Franka-IK-Abs, headless (--visualizer none --xr --device cuda:1):

• Before — headset shows noise. gpuIndexVulkan: -1, and
  compositor_set_cuda_device_for_vk lands on a different physical device than the renderer.
• After, with no manual --kit_args and no GPU index in the profile —
  Pinned the CloudXR runtime to CUDA device 1, gpuIndexCuda: 1, and the compositor logs
  Physical device 1 is being used by Vulkan, selecting that device for use in CUDA, the same
  physical device the renderer is on. The scene renders correctly and the robot is
  teleoperable.

The cxr_server.log settings block is byte-identical between the patched run and a
known-good run that had been pinned by hand, apart from the index being expressed as
gpuIndexCuda: 1 instead of gpuIndexVulkan: 2 — both resolve to the same physical device,
which the compositor confirms with the same Physical device 1 line. Negotiated stream size
is unchanged (packed stream size: 4096x4032).

Type of change

• Bug fix (non-breaking change which fixes an issue)

Release backport

• Backport this pull request to the active release branch after it merges into develop

Screenshots

The failure is a headset-only artefact (a full-frame noise pattern) and the fix restores the
normal scene; I do not have a capture path off the device that would show it faithfully.

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

_{test_xr_pins_the_renderer_only_for_a_cuda_device covers the renderer half of the gate in
CI. The CloudXR half needs a multi-GPU host whose Vulkan and CUDA orderings disagree plus a
connected headset, so that part rests on the manual validation above. CONTRIBUTORS.md is
untouched; let me know if you would like me to add an entry.}

[greptile-apps]

Greptile Summary

The PR aligns CloudXR and Kit rendering on the selected CUDA device during XR sessions.

• Pins the Kit renderer to the selected simulation GPU when XR uses CUDA.
• Derives the CloudXR CUDA index from the renderer setting, while preserving explicit environment or profile selections.
• Documents the renderer and runtime device-selection fix in both affected packages.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/app/app_launcher.py	Extends renderer GPU pinning to CUDA-backed XR launches while leaving CPU and non-XR behavior unchanged.
source/isaaclab_teleop/isaaclab_teleop/session_lifecycle.py	Pins auto-launched CloudXR to the renderer CUDA index unless the process or selected profile already specifies an index.
source/isaaclab/changelog.d/pk-xr-multigpu-device-pinning.rst	Documents the XR renderer pinning correction.
source/isaaclab_teleop/changelog.d/pk-xr-multigpu-device-pinning.rst	Documents the CloudXR compositor device-selection correction.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[XR launch with CUDA device] --> B[AppLauncher]
  B --> C[Pin Kit renderer to CUDA index]
  C --> D[Renderer activeCudaGpus setting]
  D --> E[Teleop session lifecycle]
  E --> F[Set CloudXR CUDA index]
  F --> G[CloudXR compositor]
  C --> H[Rendered XR swapchain]
  H --> G

Loading

_{Reviews (2): Last reviewed commit: "Leave `--xr` without an explicit device ..." | Re-trigger Greptile}

[isaaclab-review-bot]

Isaac Lab Review Bot

The change aligns XR rendering and CloudXR runtime device selection by pinning the renderer under XR and deriving the runtime’s CUDA index from the resulting Kit setting.

• Design and architecture: The responsibility split is coherent: AppLauncher produces the renderer GPU setting, while the teleop lifecycle consumes that setting before launching CloudXR. The behavior is scoped to XR, preserving non-XR multi-GPU behavior.
• API: No public API or CLI surface changes. Existing renderer overrides and explicit CloudXR GPU selections remain authoritative, and both affected packages include changelog fragments.
• Implementation: The producer and consumer use the same CUDA index space and handle the trailing-comma Kit setting correctly. The process-global environment mutation is a tradeoff, but the candidate did not establish a concrete supported path where the renderer device changes between CloudXR sessions in one process or where the value must be restored.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.

[nv-jakob]

Unsure if we need the context, but the reason it picks the first Vulkan device is that most (all) windowed applications will pick the first Vulkan device and then ignore what the runtime asks the app to use, becuse the first Vulkan device is the GPU that the monitor is plugged into (breaks down in "unusual" setups).

[nv-jakob]

Maybe add warnings here if they don't match, and ask the user to check the configs?

[rwiltz]

One thing I realized is that this changes the behavior with --xr and --device cpu (which is the default for --xr). Previously, CXR was bound to whatever the first GPU was matching the vulkan behavior. Now, it is always bound to GPU 0 which may cause a mismatch if for example, the monitor is plugged into GPU 1.

Can we leave the --xr --device cpu behavior unchanged in this PR?

[2047767028-lang]

Good catch — thank you, that is a real regression and I had not covered it.

_resolve_device_settings resolves a bare --xr to device = "cpu" while device_id stays
at its 0 initialiser, so my gate pinned the renderer to CUDA 0 on that path. On a host whose
display is not on GPU 0 that breaks the very agreement this PR is trying to establish: Kit's
auto-selection and CloudXR's Vulkan auto-selection previously landed on the same card, and
forcing 0 would split them. (The host I tested on is one of those — its display is on
nvidia-smi GPU 2, which is Vulkan index 0.)

Pushed c6f7f90 to require a CUDA device before pinning:

if launcher_args.get("multi_gpu") is False or (self._xr and "cuda" in self.device):

The teleop side needs no change to match: with the renderer unpinned,
_renderer_cuda_index() returns None, and the simulation device is CPU, so
_pin_cloudxr_to_render_device returns without touching NV_CXR_GPU_INDEX_CUDA and the
runtime keeps its automatic selection.

Verified both paths by reading the carb settings back after startup:

Command	/renderer/multiGpu/activeCudaGpus	/physics/cudaDevice
--xr (resolves to cpu)	None — unchanged from before this PR	0
--xr --device cuda:1	'1,'	1

So --xr --device cpu behaves exactly as it does on develop, and the fix still applies once
a CUDA device is explicitly selected. The isaaclab changelog fragment now states that scope.

Happy to look at the bare---xr case separately if you would like it addressed — it has the
same underlying mismatch, but the right answer there is to follow Kit's chosen device rather
than impose one, which is a different change.

[rwiltz]

@greptile

[kellyguo11]

run-ci

[kellyguo11]

run-ci

[2047767028-lang]

Thanks for merging develop in, @kellyguo11. Two CI failures on that run — one mine, one not.

isaaclab (core) [2/3] was mine. Fixed in 21f2d44.

test_app_launcher_argv.py::test_spectator_view_follows_visual_output_intent[xr]  FAILED
test_app_launcher_argv.py::test_explicit_spectator_setting_overrides_visualizer_default  FAILED
E   AttributeError: 'AppLauncher' object has no attribute 'device'
E   AttributeError: 'AppLauncher' object has no attribute '_xr'

Both build a partial launcher with AppLauncher.__new__ and set only what
_resolve_kit_args read before this PR; gating the renderer pin on XR added two more reads.
They now stub device (and _xr, which the second one never set).

I also added direct coverage rather than only unbreaking the existing tests:
_resolve_devices_and_kit_args takes an xr flag, and
test_xr_pins_the_renderer_only_for_a_cuda_device asserts both halves of the gate —
--xr --device cuda:1 emits --/renderer/multiGpu/activeCudaGpus=1, with physics_gpu = 1,
and a bare --xr emits nothing, which is the case @rwiltz raised. Reverting the
app_launcher.py change fails the first parameter set, so it pins the behaviour instead of
passing vacuously. 17 passed locally, up from 15 with 2 failing.

test-contrib-environments looks unrelated. It fails on
test_contrib_environments[IsaacContrib-Franka-Pour] with

ValueError: Reset dataset task contract field 'metadata.task_contract.robot_asset'
does not match the runtime.

raised from franka_pour/reset_dataset_io.py:242. That suite launches
AppLauncher(headless=True, enable_cameras=True) with no XR, so self._xr is False and the
gate short-circuits to exactly the previous expression — the change is a provable no-op for
every non-XR run. Nothing in this PR touches reset datasets, task contracts or the pouring
task. I did not want to assume, though: if you would rather I confirm it reproduces without
these commits, say the word and I will check.

[kellyguo11]

run-ci

[isaaclab-bot]

Backported to release/3.0.0 as 0fadcf8.