[Odin] Fix semantic_segmentation camera observations reaching the policy as integers

[AntoineRichard]

Description

Every semantic_segmentation camera row in benchmark dispatch 20260901-153531 (image built from origin/release/3.0.0 at f88dbc59c82, rsl_rl, all core tasks) failed at the first training step — 60 failed rows total, with:

RuntimeError: Input type (unsigned char) and bias type (float) should be the same

The failure is renderer-independent: isaacsim_rtx 18 rows, ovrtx 18, newton_renderer 24. The segmentation output reaches the feature extractor's first convolution still as an integer tensor.

Root cause

Both Cartpole camera observation paths normalize only RGB-like and depth data types, so semantic_segmentation falls through unconverted:

• source/isaaclab_tasks/isaaclab_tasks/core/cartpole/mdp/observations.py (CameraImageStack.__call__)
• source/isaaclab_tasks/isaaclab_tasks/core/cartpole/cartpole_direct_camera_env.py (CartpoleCameraEnv._get_observations)

Both files are byte-identical between release/3.0.0 and develop, so the bug is present on develop as-is.

Why the fix belongs in the observation term

The renderer is producing exactly what its published contract says it should — NewtonWarpRenderer.supported_output_types deliberately emits RGBA uint8 when colorized and a single int32 id channel when not, "matching the Isaac RTX / OVRTX contract so backend-independent consumers see the same dtype". Segmentation label ids are integers; making a renderer emit floats would break every consumer that reads ids (visualization, semantic id lookup, dataset export).

Nor does it belong in the feature extractor: the extractor's contract is "float32 image in", and the observation pipeline already owns dtype/layout normalization for every other camera data type via normalize_camera_image. Making the CNN defensively cast would paper over the same gap for every future task and duplicate logic that already exists.

So the fix goes where the gap is: the observation term, routed through the shared helper.

The int32-vs-uint8 subtlety

Segmentation has two dtypes depending on colorize_semantic_segmentation:

• colorize=True (the CameraCfg default, and what the failing sweep used): RGBA uint8, 4 channels
• colorize=False: a single int32 label-id channel

normalize_camera_image already handled the colorized uint8 case correctly — it was simply never called for this data type. The non-colorized int32 case was not handled: it fell through every branch and was returned unchanged, so a fix that only wired up the existing call would still break colorize=False.

The int32 half is not demonstrated by the sweep (see the caveat below), but it is not opportunistic scope creep: the caller now dispatches on the data type alone, so the helper is the single place that decides what segmentation means, and leaving it to silently return int32 unchanged would ship a fix that reads as complete while still crashing under colorize=False.

Handled by keying on the tensor dtype rather than on the colorize flag or a hardcoded uint8 assumption:

• In normalize_camera_image, non-uint8 segmentation is cast to float32. Label ids carry no meaningful scale, so they are cast and not rescaled — applying (x / 255) - mean to label ids would be inventing semantics.
• In both Cartpole callers, the uint8 deferred-normalize fast path (which keeps the frame-stack ring buffer in uint8 for cheaper per-step copies) is gated on camera_data.dtype == torch.uint8, so colorized segmentation rides it and int32 label maps are normalized before entering the ring.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Test evidence

Construction-only reproduction, no simulator needed. The bug fires when the extractor first sees an observation, so exercising the observation term directly on stub camera output is sufficient and much faster.

Extended source/isaaclab/test/utils/test_images.py with the int32 case next to the existing colorized case, and dropped semantic_segmentation from the "unknown type passthrough" parametrization — that class asserts out is src under the heading "Unknown data_types return the input unchanged", and segmentation is no longer unknown. It would still pass by the accident that .float() on a float32 tensor returns self, so leaving it would have left the suite documenting the opposite of the new behaviour.

Added source/isaaclab_tasks/test/core/test_cartpole_camera_observations.py for the observation term itself; no sim-free test of it existed (the test_rendering_cartpole*.py neighbours need a renderer and golden images, and the *_camera_presets.py files only resolve configs). Two tests, each parametrized over frame_stack [1, 2] so both the immediate and the deferred-normalize branch are covered, and each asserting exact values rather than just dtype.

Both dtypes are covered: colorized uint8 RGBA and non-colorized int32.

Without the fix (the three source files reverted to origin/develop, tests kept):

$ uv run --frozen --extra dev python -m pytest source/isaaclab/test/utils/test_images.py \
      source/isaaclab_tasks/test/core/test_cartpole_camera_observations.py -q
FAILED test_images.py::TestNormalizeCameraImageSegmentation::test_non_colorized_semantic_segmentation_is_cast_to_float[cpu]
FAILED test_images.py::TestNormalizeCameraImageSegmentation::test_non_colorized_semantic_segmentation_is_cast_to_float[cuda:0]
FAILED test_cartpole_camera_observations.py::test_colorized_segmentation_is_normalized_like_rgb[cpu-1]
FAILED test_cartpole_camera_observations.py::test_colorized_segmentation_is_normalized_like_rgb[cpu-2]
FAILED test_cartpole_camera_observations.py::test_colorized_segmentation_is_normalized_like_rgb[cuda:0-1]
FAILED test_cartpole_camera_observations.py::test_colorized_segmentation_is_normalized_like_rgb[cuda:0-2]
FAILED test_cartpole_camera_observations.py::test_non_colorized_segmentation_is_cast_to_float[cpu-1]
FAILED test_cartpole_camera_observations.py::test_non_colorized_segmentation_is_cast_to_float[cpu-2]
FAILED test_cartpole_camera_observations.py::test_non_colorized_segmentation_is_cast_to_float[cuda:0-1]
FAILED test_cartpole_camera_observations.py::test_non_colorized_segmentation_is_cast_to_float[cuda:0-2]
10 failed, 62 passed in 8.56s

Note that the pre-existing colorized-uint8 helper test passes on develop: normalize_camera_image always handled that case correctly, and the crash came from the Cartpole callers never invoking it.

With the fix:

$ uv run --frozen --extra dev python -m pytest source/isaaclab/test/utils/test_images.py \
      source/isaaclab_tasks/test/core/test_cartpole_camera_observations.py -q
72 passed in 3.53s

uv run --frozen isaaclab -f passes clean.

Caveats a reviewer should know

• The sweep exercised the colorized uint8 path: CameraCfg.colorize_semantic_segmentation defaults to True and the Cartpole config declares observation_space=[4, 96, 96] (4 channels = RGBA). The int32 path is reachable only with colorize=False; it was genuinely broken (the helper returned it unchanged) but the 60 rows do not prove it.
• The direct-environment edit is not covered by a test. CartpoleCameraEnv._get_observations calls super()._get_observations(), which needs a constructed environment, so it cannot be exercised sim-free. The edit is line-for-line identical to the manager-term edit, which is tested.

Relationship to #7440

#7440 touches isaaclab/utils/images.py and the shared isaaclab/envs/mdp/observations.py::image term, but neither Cartpole file, so it does not fix this. Its images.py work is a fused normalize+layout-conversion perf change that adds an output_channel_dim parameter; it leaves the segmentation dispatch condition semantically unchanged and does not add int32 handling. This PR adds an early-return branch above that condition and leaves #7440's line untouched, so the two should merge cleanly in either order.

Checklist

• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation (docstrings for normalize_camera_image)
• My changes generate no new warnings
• I have added tests that prove my fix is effective
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file (changelog fragments; extension.toml is generated)

Release backport

• Backport this pull request to the active release branch after it merges into develop

[greptile-apps]

Greptile Summary

This PR fixes Cartpole semantic-segmentation observations reaching policy convolutions as integer tensors.

• Casts non-colorized int32 label maps to float32 without rescaling label IDs.
• Routes colorized uint8 segmentation through the existing image-normalization path, including deferred normalization after frame stacking.
• Adds utility and sim-free observation tests covering colorized and non-colorized segmentation with and without frame stacking.

Confidence Score: 5/5

The PR appears safe to merge, with the reachable colorized and non-colorized segmentation paths producing correctly shaped float32 policy observations.

The changed dispatch handles the documented renderer outputs—uint8 RGBA and int32 label maps—and the stacking paths preserve dtype, layout, and label values without exposing a concrete regression.

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/utils/images.py	Adds the intended float32 cast for non-colorized semantic label maps while preserving the existing uint8 normalization path.
source/isaaclab_tasks/isaaclab_tasks/core/cartpole/cartpole_direct_camera_env.py	Extends immediate and deferred normalization to semantic-segmentation observations based on their dtype.
source/isaaclab_tasks/isaaclab_tasks/core/cartpole/mdp/observations.py	Applies matching semantic-segmentation normalization behavior to the manager-based frame-stack observation term.
source/isaaclab/test/utils/test_images.py	Covers normalized uint8 segmentation and value-preserving int32-to-float32 conversion.
source/isaaclab_tasks/test/core/test_cartpole_camera_observations.py	Adds sim-free coverage for both segmentation representations across single-frame and stacked observations.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Camera semantic segmentation] --> B{Tensor dtype}
    B -->|uint8 RGBA| C{Frame stacking enabled?}
    C -->|Yes| D[Store uint8 frames in ring buffer]
    D --> E[Stack channels]
    C -->|No| F[Normalize image immediately]
    E --> G[Normalize stacked image]
    B -->|non-uint8 label map| H[Cast label IDs to float32 without rescaling]
    H --> I[Stack float32 frames if configured]
    F --> J[Float32 policy observation]
    G --> J
    I --> J

Loading

_{Reviews (1): Last reviewed commit: "Fix semantic_segmentation camera observa..." | Re-trigger Greptile}

[isaaclab-review-bot]

Isaac Lab Review Bot

The patch routes semantic-segmentation observations through the shared normalization helper and casts non-colorized integer label maps to float32 without rescaling, while preserving colorized uint8 normalization.

• Design and architecture: Normalization remains correctly owned by the observation pipeline. Both Cartpole paths distinguish colorized uint8 segmentation for deferred normalization from int32 label maps normalized before stacking. The repeated semantic-segmentation check is a small, non-blocking tradeoff and does not justify introducing another shared predicate in this focused fix.
• API: The helper’s behavior changes specifically for non-uint8 semantic segmentation, which previously passed through unchanged. Existing RGB-like, depth, normals, colorized-segmentation, out-buffer, and channel-layout behavior remains intact, and both affected packages include changelog fragments.
• Implementation: The manager-based and direct Cartpole paths consistently produce float32 segmentation observations. Colorized stacked frames retain the uint8 ring-buffer optimization, while non-colorized label maps are cast before stacking without altering label values. Focused tests cover both segmentation representations and stacked and unstacked manager-based observations.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.

[kellyguo11]

run-ci

[isaaclab-bot]

Backported to release/3.0.0 as 21b60f9.