Move the remaining keyboard scripts to Isaac Capture and deprecate the isaaclab.devices keyboard, gamepad and SpaceMouse by rwiltz · Pull Request #8798 · isaac-sim/IsaacLab

rwiltz · 2026-10-23T15:20:51Z

Description

Stacked on #8797, which is stacked on #8158; review those first. This PR's own changes are the last two commits
(compare).

Finishes moving keyboard, gamepad and SpaceMouse input off the isaaclab.devices devices:

isaaclab_teleop.script_input adds two helpers for scripts. add_key_callbacks calls a script's hotkey callbacks for keys pressed in a visualizer window, with no teleop session. open_teleop_device
opens a keyboard, gamepad or SpaceMouse through Isaac Capture (standalone CloudXR profile, no headset) when isaacteleop is installed, and through isaaclab.devices otherwise.
replay_demos.py and annotate_demos.py read their hotkeys with add_key_callbacks; pick_and_place.py and consolidated_demo.py open their device with open_teleop_device. Key bindings are
unchanged. consolidated_demo.py also reads the device command as the 7-element tensor the devices return; it unpacked it into two values, which raised.
Se2/Se3Keyboard, Se2/Se3Gamepad and Se2/Se3SpaceMouse are deprecated: their constructors emit a DeprecationWarning pointing to the isaaclab_teleop builders. Their configs are not deprecated;
environments keep declaring devices with them.

Fixes # (issue)

Type of change

New feature (non-breaking change which adds functionality)
Documentation update

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

I have read and understood the contribution guidelines
I have run the pre-commit checks with uv run isaaclab --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
I have added my name to the CONTRIBUTORS.md or my name already exists there

greptile-apps · 2026-10-23T15:26:13Z

Confidence Score: 5/5

[Medium impact] This update appears safe to merge; no new actionable issue was found, and the numbered previous finding is fixed.

Summary

The PR moves keyboard, gamepad, and SpaceMouse scripts toward Isaac Capture while keeping legacy fallbacks.

Scripts and teleoperation pipelines can read keys from focused Kit and Newton windows.
Isaac Capture pipelines now support keyboard, gamepad, and SpaceMouse teleoperation.
Demo and teleoperation scripts use the shared keyboard and device input paths.
Legacy keyboard, gamepad, and SpaceMouse device constructors now warn that they are deprecated.

_{Reviews (3) · Last reviewed commit: "Move add_key_callbacks into isaaclab.vis..." · Reviewed by Greptile}

isaaclab-review-bot

Isaac Lab Review Bot

Needs attention: Design and architecture (1), Compatibility and deprecation (1), Implementation (3), Style consistency (2), Test quality (1). Other reviewed categories look good.

PR description: The title undercovers the 71-file visualizer, input, and dependency diff; remove empty “Fixes #”, reconcile intentional deprecation warnings with the checklist, and list focused validation.
Design and architecture: The generic add_key_callbacks helper depends only on visualizers but lives in optional isaaclab_teleop, forcing non-teleop hotkey consumers through the wrong package; move it to isaaclab.visualizers.
Scope and over-engineering: No over-engineering issues.
API: API changes look good.
Compatibility and deprecation: Breaking changes: automatic Se3GamepadCfg routing reverses Y and yaw relative to legacy Se3Gamepad; preserve signs before enabling replacement, while constructor deprecations otherwise retain functional warnings.
Implementation: IsaacTeleopDevice translates R into RESET but script_input promises add_callback("R"), so consolidated_demo loses reset under Capture; register the actual RESET callback or preserve R dispatch.
Style consistency: The three added package deprecation directives omit Sphinx’s required version argument; documentation rendering can fail, so add the release version.
Test quality: The device-selection fixture replaces routing with its own expected map, making builder-name assertions self-fulfilling; preserve production routing and assert observable forwarding.

Significant concerns. Posted 8 inline findings.

Automated review; human maintainers own approval decisions.

rwiltz · 2026-10-23T20:18:57Z

@greptile

rwiltz · 2026-10-23T20:30:24Z

@greptile

rwiltz · 2026-10-23T20:41:37Z

run-ci

# Description Adds a keyboard input API to visualizers, so code can read the keys pressed in a visualizer's window, e.g. for keyboard teleoperation or script hotkeys, independent of Kit/carb. - `BaseVisualizer.key_event_source` returns a `KeyEventSource` for the visualizer's window, or `None` when it has none (the default; also before initialization, after close, and headless). It is created on first access and closed with the visualizer. - `KeyEventSource.add_key_listener(on_key, on_focus_lost)` returns a closable `KeyboardSubscription`. Presses and releases arrive as W3C `KeyboardEvent.code` strings (`"KeyW"`), forwarded as the window reports them; `on_focus_lost` runs on blur, when the visualizer's UI takes the keyboard, and on close. Subscribing, closing and callbacks all happen on the visualizer's thread. - `BaseVisualizer.set_camera_keys_enabled(enabled)` lets the caller turn off keyboard camera movement (Newton WASD/QE and arrow keys) while it uses those keys; a no-op by default. - Implemented for Kit (physical keys; Kit visualizers share the app window, so each reports the same keys) and for Newton GL and Newton RTX (one pyglet viewer window; US-layout key mapping, presses typed into the ImGui UI are withheld). Viser, Rerun and headless runs return `None`. - Each visualizer closes its keyboard source first in `close()`, while the window is still open; a failure there is logged and the rest of the teardown still runs. Opt-in: nothing is hooked until a listener subscribes, and existing visualizer behavior is unchanged. Used by #8797 (keyboard, gamepad and SpaceMouse teleop through Isaac Capture) and #8798 (remaining scripts and deprecation of the `isaaclab.devices` keyboard, gamepad and SpaceMouse devices). Fixes # (issue) ## Type of change  - New feature (non-breaking change which adds functionality) ## Release backport - [ ]  Backport this pull request to the active release branch after it merges into `develop` ## Checklist Docker and GPU tests run on demand. Push the commits you want tested, then comment `run-ci` on the pull request. - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `uv run isaaclab --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have validated the changed behavior with focused coverage and applied the [test-audit criteria](https://github.com/isaac-sim/IsaacLab/blob/develop/skills/developer/test-audit/SKILL.md) to any test changes - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package (do **not** edit `CHANGELOG.rst` or bump package versions — CI handles that) - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: ooctipus <zhengyuz@nvidia.com>

Add builders for keyboard, gamepad and SpaceMouse SE(2)/SE(3) Isaac Capture pipelines. The keyboard is read inside the teleop session from whichever visualizer window has focus (no input group, no keys from other apps); while a pipeline reads it, the session turns off the visualizers' keyboard camera movement. The gamepad and SpaceMouse configs start the Isaac Capture gamepad and spacemouse plugins with the session, which read /dev/input/jsN and /dev/hidrawN. Like every Isaac Capture session they open an OpenXR session; without a headset they run on the standalone CloudXR profile. IsaacTeleopDevice now starts, pauses and resets teleoperation on the B, P and R keys typed into a visualizer window and resets on a SpaceMouse pipeline's right button, the bindings the teleop scripts made themselves through a Carb keyboard. Each key press is handled once, whatever the retargeting mode. device_selection translates an isaaclab.devices keyboard, gamepad or SpaceMouse config into the equivalent pipeline, so a script can serve --teleop_device and an environment's teleop_devices entries through Isaac Capture.

teleop_se3_agent.py and record_demos.py now serve keyboard, gamepad and SpaceMouse devices through Isaac Capture when the isaacteleop package is installed, and through the isaaclab.devices devices otherwise (e.g. on aarch64 or Windows). --teleop_device keeps its meaning: an entry of the environment's teleop_devices, or a built-in device. Environments need no changes. The scripts no longer create a Carb keyboard for the B, P and R control keys, which the device now handles.

…vices The keyboard, gamepad and SpaceMouse sources and the gamepad and spacemouse plugins first ship in Isaac Capture 1.7, published as isaaccapture (isaacteleop renamed; it still provides the isaacteleop import). The teleop extra now installs isaaccapture~=1.7.44a1 instead of isaacteleop~=1.4.145. uv.lock adds isaaccapture and cupy-cuda12x and moves the isaacteleop alias package to the same release. The docs point the release branch, web client and Haptikos plugin links at 1.7.x.

record_demos.py now applies the R callback's guard to an Isaac Capture reset: while a successful demo is still recording its post-success steps, the reset is refused with the same message as for the isaaclab.devices keyboard. Other resets, including the host reset after each episode, are handled as before. The session checks for unattached visualizer windows on every step, not only until the first one attaches, so a window that initializes later also gets keyboard input.

uv.lock is regenerated from main's lock with uv 0.13.5, which keeps lock revision 5 and viser's ipywidgets dependency; only isaaccapture, cupy-cuda12x and the isaacteleop alias package change. The B / P / R key codes are class constants at the top of IsaacTeleopDevice, and the device-selection test records the builders the routing table actually calls rather than names the test supplies.

isaaccapture publishes Linux aarch64 wheels, so the teleop extra now installs it on any Linux system instead of x86_64 only. dex-retargeting stays x86_64 only. uv.lock keeps isaaccapture 1.7.44a1 and adds its aarch64 wheel.

select_teleop_cfg no longer falls back to the isaaclab.devices keyboard, gamepad and SpaceMouse when Isaac Capture is missing: it raises an ImportError naming the teleop extra, as it does for an environment's declared Isaac Capture pipeline. Devices without a Capture pipeline (e.g. OpenXR hand tracking) keep their native config and factory.

Restore the prior PR contents so the proposed cleanup can be reviewed as comments.

… docs Attach the visualizer keyboards at the end of session start, so a start that fails part way does not leave listeners attached or camera keys disabled. Update the teleop docs for the isaaccapture 1.7 pin and its Linux aarch64 install. Co-authored-by: AntoineRichard <antoiner@nvidia.com> Signed-off-by: Rafael Wiltz <rwiltz@nvidia.com>

Signed-off-by: Rafael Wiltz <rwiltz@nvidia.com>

replay_demos.py and annotate_demos.py read their hotkeys from the visualizer windows with add_key_callbacks, without a teleop device. pick_and_place.py and consolidated_demo.py open their keyboard or SpaceMouse with open_teleop_device: an Isaac Capture pipeline on the standalone CloudXR profile when isaacteleop is installed, and the isaaclab.devices device otherwise. Their key bindings are unchanged. consolidated_demo.py also reads the device command as the 7-element tensor the devices return, instead of unpacking it into a pose and a gripper flag, which raised.

Se2/Se3Keyboard, Se2/Se3Gamepad and Se2/Se3SpaceMouse now emit a DeprecationWarning pointing to the isaaclab_teleop builders that replace them. Their configs are not deprecated: environments keep declaring their devices with them, and the teleop scripts run those through the equivalent Isaac Capture pipeline when isaacteleop is installed.

pick_and_place.py and consolidated_demo.py read the device's control events, as teleop_se3_agent.py does: while an Isaac Capture device is paused (P, or after R), pick_and_place.py renders without stepping and consolidated_demo.py holds the robot still. The isaaclab.devices devices report no control events, so they are unaffected. add_key_callbacks tracks held keys across all visualizer windows, so a press several visualizers report (Kit visualizers share the app window) fires its callback once instead of once per visualizer.

isaaclab_teleop comes only with the teleop and mimic extras, so replay_demos.py and pick_and_place.py could no longer run on a base install. add_key_callbacks uses only the simulation's visualizers, so it now lives in isaaclab.visualizers next to KeyEventSource, and those scripts read their hotkeys from there. pick_and_place.py opens its keyboard through isaaclab_teleop when it is installed and through isaaclab.devices otherwise, and reads pause from the device's control events without importing isaaclab_teleop.

open_teleop_device no longer falls back to the isaaclab.devices device: without Isaac Capture it raises an ImportError naming the teleop extra. pick_and_place.py imports it only when it opens its keyboard, so a windowless run still needs no teleop extra. The deprecated devices remain for code that constructs them directly.

The teleop session now reads keyboard input only from the visualizers it is given. open_teleop_device takes them and opens the device through select_teleop_cfg and create_teleop_device, and add_key_callbacks takes them instead of looking up the simulation. The replay, annotate, consolidated Mimic and pick-and-place scripts pass sim.visualizers.

rwiltz requested a review from a team October 23, 2026 15:20

rwiltz requested review from AntoineRichard, StafaH, fatimaanes, hougantc-nvda, kellyguo11, matthewtrepte, ooctipus, pbarejko and peterd-NV as code owners October 23, 2026 15:20

github-actions Bot added documentation Improvements or additions to documentation isaac-mimic Related to Isaac Mimic team labels Oct 23, 2026

rwiltz changed the title ~~Rwiltz/teleop scripts isaac capture~~ Move the remaining keyboard scripts to Isaac Capture and deprecate the isaaclab.devices keyboard, gamepad and SpaceMouse Oct 23, 2026

This was referenced Oct 23, 2026

Run keyboard, gamepad and SpaceMouse teleop through Isaac Capture #8797

Open

Add keyboard input from the focused visualizer window #8158

Merged

greptile-apps Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread examples/demos/pick_and_place.py

Comment thread source/isaaclab_teleop/isaaclab_teleop/script_input.py Outdated

rwiltz force-pushed the rwiltz/teleop-scripts-isaac-capture branch from d3ce34f to 16d7e5a Compare October 23, 2026 15:36

github-actions Bot added the infrastructure label Oct 23, 2026

isaaclab-review-bot Bot reviewed Oct 23, 2026

View reviewed changes

rwiltz force-pushed the rwiltz/teleop-scripts-isaac-capture branch 2 times, most recently from 8b1a343 to 3aa3e49 Compare October 23, 2026 16:03

greptile-apps Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread examples/demos/pick_and_place.py Outdated

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 23, 2026

rwiltz force-pushed the rwiltz/teleop-scripts-isaac-capture branch from 9f87fdf to c3147df Compare October 24, 2026 03:06

rwiltz added 5 commits October 24, 2026 08:46

rwiltz force-pushed the rwiltz/teleop-scripts-isaac-capture branch from c3147df to d886c4b Compare October 24, 2026 12:50

ooctipus added 3 commits October 24, 2026 23:18

Centralize teleop setup and pass window inputs explicitly

bfec221

Merge main and preserve explicit teleop scene dependencies

df69d9e

Consolidate device pipelines and remove redundant action copies

00db2a9

rwiltz force-pushed the rwiltz/teleop-scripts-isaac-capture branch from d886c4b to 2f7f158 Compare October 25, 2026 12:17

rwiltz added 2 commits October 25, 2026 08:18

Install Isaac Capture on Linux aarch64 too

eec644d

isaaccapture publishes Linux aarch64 wheels, so the teleop extra now installs it on any Linux system instead of x86_64 only. dex-retargeting stays x86_64 only. uv.lock keeps isaaccapture 1.7.44a1 and adds its aarch64 wheel.

rwiltz force-pushed the rwiltz/teleop-scripts-isaac-capture branch from 2f7f158 to 8bd1884 Compare October 25, 2026 23:11

AntoineRichard and others added 3 commits October 26, 2026 11:39

Fix teleop startup cleanup and paused demonstration exports

c096cf7

Revert unintended teleop cleanup code push

06eacaa

Restore the prior PR contents so the proposed cleanup can be reviewed as comments.

rwiltz force-pushed the rwiltz/teleop-scripts-isaac-capture branch from 8bd1884 to 5752dba Compare October 26, 2026 12:34

rwiltz added 7 commits October 26, 2026 10:43

Note which visualizers the teleop scripts support

f23a2d6

Signed-off-by: Rafael Wiltz <rwiltz@nvidia.com>

rwiltz force-pushed the rwiltz/teleop-scripts-isaac-capture branch from 5752dba to 38f51f3 Compare October 26, 2026 14:43

Navigation Menu

Repository navigation

Move the remaining keyboard scripts to Isaac Capture and deprecate the isaaclab.devices keyboard, gamepad and SpaceMouse - #8798

Move the remaining keyboard scripts to Isaac Capture and deprecate the isaaclab.devices keyboard, gamepad and SpaceMouse#8798
rwiltz wants to merge 20 commits into
isaac-sim:mainfrom
rwiltz:rwiltz/teleop-scripts-isaac-capture

rwiltz commented Oct 23, 2026

Uh oh!

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

isaaclab-review-bot Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rwiltz commented Oct 23, 2026

Uh oh!

Uh oh!

rwiltz commented Oct 23, 2026

Uh oh!

rwiltz commented Oct 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

rwiltz commented Oct 23, 2026

Description

Type of change

Checklist

Uh oh!

greptile-apps Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

Isaac Lab Review Bot

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rwiltz commented Oct 23, 2026

Uh oh!

Uh oh!

rwiltz commented Oct 23, 2026

Uh oh!

rwiltz commented Oct 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading