Skip to content

Re-enable the Kit visualizer pause checks - #7049

Merged
kellyguo11 merged 8 commits into
isaac-sim:developfrom
fatimaanes:fanes/nvbug-6570125-adjust-threshold
Aug 14, 2026
Merged

Re-enable the Kit visualizer pause checks#7049
kellyguo11 merged 8 commits into
isaac-sim:developfrom
fatimaanes:fanes/nvbug-6570125-adjust-threshold

Conversation

@fatimaanes

@fatimaanes fatimaanes commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Temporary workaround for NVBUG 6570125 while we wait for the upstream renderer fix.

Problem

RTX responsiveDenoising continues refining the image after physics stops, so fresh renders captured during a pause are no longer stable.
The affected Kit visualizer checks are currently green for the wrong reasons:
Tiled camera: #6658 changed the paused captures to reuse the cached sensor frame, so the assertion effectively compares the same frame against itself and can't detect renderer instability.
Viewport: still performs a real comparison, but the known denoiser noise can push it over the existing threshold and the failure is absorbed by retries.

What this changes

This PR restores fresh renders for the tiled pause check and adjusts the per-channel difference threshold only for the affected Kit RTX pause paths.
The existing 100-pixel gate stays unchanged. We're changing how large a per-channel difference must be before a pixel is counted, rather than simply allowing more differing pixels.

site @50 @80 @100 @120 @150 @160 outcome
Kit viewport (Newton) 67 12 8 7 3 80
Kit viewport (PhysX) 14 4 3 1 0 80
Kit tiled (Newton) 999 604 226 120 59 49 160
Kit tiled (PhysX) 103 4 4 2 0 80
Newton ViewerGL 0 0 0 0 0 50

Values are the number of differing pixels at each per-channel threshold.
For the margin-critical case (Kit tiled camera, Newton backend), threshold 160 leaves at most 49 px of paused denoiser noise (worst of n=36) against at least 160 px with real motion (weakest of n=72), both measured with the denoiser enabled. With the existing 100-pixel gate the known renderer noise passes at 2.0× margin while real motion is still detected at 1.6×.

Changes

_assert_frames_remain_stable now accepts a per-call channel_diff_threshold; the default remains 50.
Kit RTX pause checks use 80 for the viewport, 160 for the tiled camera on Newton, and 80 for the tiled camera on PhysX — three separate constants, since the tiled PhysX cell measures 103 px at the default threshold and must not silently track a later change to the viewport value. Newton ViewerGL remains at the strict default since it doesn't exercise the affected DLSS path.
max_differing_pixels remains 100 everywhere.
The tiled pause check is restored to fresh renders instead of cached frames, so the assertion is actually exercising renderer stability again.

Validation

With retries disabled, the workaround passed 40/40 runs across all four affected configurations.

As a control, restoring the original threshold of 50 reproduced the failure 6/6 times at 766–949 differing pixels. Those six failures are all the tiled assertion: the control ran under pytest -x and the viewport test is defined first in the same file, so the run aborted at the tiled test and the viewport site passed at 50 in those reps. The control therefore demonstrates that the tiled workaround filters real denoiser noise rather than simply making the test pass; it is not evidence for the viewport site.

The viewport raise from 50 to 80 rests instead on production CI, where the assertion has really been failing (107 pixels differed, expected at most 100, with a historical band reaching 161 px), plus n=5 bench samples showing 67 px at threshold 50 falling to 12 px at 80.

The same change was validated on three independent L40S CI runners through #7055, with all four visualizer integration tests passing first-pass, no retries. Restoring fresh renders also had no meaningful runtime impact (~27s median).

This is a temporary workaround for NVBUG 6570125 and should be removed or re-measured once the upstream fix is available. That revert is load-bearing rather than cosmetic: with the denoiser disabled — the world after the renderer fix — the weakest real motion in the governing cell measures 80 px at threshold 160, which is below the 100-pixel gate. Leaving 160 in place after the fix ships would make the tiled check blind to the weakest motion it exists to catch, so the removal comment on the constants is the thing guarding that.

RTX ResponsiveDenoising keeps refining a paused frame, so two captures taken while
simulation is frozen differ by a small number of high-amplitude pixels. Amplitude
separates that residue from real motion and the pixel count does not: at the shared
threshold of 50 the paused noise reaches 1.55x the weakest real motion signal, so any
count-based gate loose enough to stay quiet is also blind to a genuine regression.

_assert_frames_remain_stable now takes a channel_diff_threshold, raised to 80 at the
Kit viewport pause and 150 at the Kit tiled camera pause. The tiled override is gated
on KitVisualizer because all four tiled cases share _attempt_pause and the
Newton-visualizer cells sit at zero noise. Newton ViewerGL rasterises without DLSS and
keeps the strict default. The pixel gate stays at 100.

Both paused tiled captures render fresh again, reverting isaac-sim#6658. Comparing the sensor's
cached frame against itself cannot detect a renderer that keeps changing the image
after physics stops, which left that assertion unable to fail.

Measured on L40 / driver 595.58.03 / Isaac Sim 6.1.0-alpha.47. Work-around for
NVBUG 6570125; remove once OVRTX 0.5 ships the fix.
@github-actions github-actions Bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Aug 12, 2026
@fatimaanes fatimaanes changed the title Re-enable the Kit visualizer pause checks with amplitude thresholds (NVBUG 6570125) Re-enable the Kit visualizer pause checks Aug 12, 2026
fatimaanes and others added 3 commits August 12, 2026 15:49
These checks have been dead weight for a while. RTX responsive denoising keeps
refining the image after physics stops, so two frames captured during a pause
differ by ~900 pixels and the test went red constantly. isaac-sim#6658 quietened it by
comparing the camera's cached frame against itself, which can't fail whatever
the renderer does.

The denoiser residue is a few very bright pixels; real motion is spread across
many. So count on amplitude instead: only pixels differing by 160 or more (80 in
the viewport). The shimmer drops out, real movement doesn't. Measured over 24
runs on an L40, and confirmed the check still fails when fed actual motion.

Worth being clear about the scope: no physics can step inside this pause window,
so this is a renderer-stability check, not a pause-correctness one. It catches a
gross instability, not a subtle one.

Work-around/Temp fix for NVBUG 6570125. Need to remove once the real fix is in.
Shortened the comment regarding NVBUG 6570125 for clarity.

Signed-off-by: fanes <74020209+fatimaanes@users.noreply.github.com>
The pause pixel-count log never reaches CI: every pause call site runs inside
caplog.at_level(logging.WARNING), which hard-sets the root logger, so the module
logger's effective level is WARNING and info() short-circuits. Raising it to
WARNING would instead self-trip _assert_no_visualizer_log_issues once
ASSERT_VISUALIZER_WARNINGS is enabled. The assert message already carries the
count, threshold and gate on the run where it matters.

Give the tiled PhysX cell its own constant rather than borrowing the viewport
one. It measures 103 differing pixels at the default threshold against a gate of
100, so lowering the viewport value later would silently break it.
@fatimaanes
fatimaanes marked this pull request as ready for review August 13, 2026 22:53
@fatimaanes
fatimaanes requested a review from a team August 13, 2026 22:53
@greptile-apps

greptile-apps Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR re-enables meaningful Kit tiled-camera pause checks by capturing fresh renders and introduces temporary, per-configuration channel-difference thresholds for known RTX responsive-denoising noise.

  • Adds separate pause thresholds for Kit viewport, Kit tiled/Newton, and Kit tiled/PhysX paths while retaining the existing differing-pixel gate.
  • Leaves Newton ViewerGL comparisons at the strict default threshold.
  • Documents the NVBUG 6570125 workaround and the need to remove or remeasure it after the renderer fix.

Confidence Score: 5/5

The PR appears safe to merge with no actionable defects identified in the changed test paths.

Current callers select the intended threshold for each visualizer/backend configuration, fresh captures restore a non-vacuous renderer-stability check, and no evidence establishes a build, lifecycle, or test-contract failure.

Important Files Changed

Filename Overview
source/isaaclab_visualizers/test/visualizer_integration_utils.py Restores fresh paused tiled renders and applies correctly scoped temporary thresholds without an identified actionable defect.
source/isaaclab_visualizers/changelog.d/fanes-nvbug-6570125-adjust-threshold.rst Accurately documents the test correction, threshold overrides, affected rendering paths, and temporary-workaround lifecycle.

Reviews (1): Last reviewed commit: "Drop dead pause log and split the tiled ..." | Re-trigger Greptile

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The pause-check implementation is confined to private visualizer integration-test utilities, but its changelog treatment needs correction: this test-only workaround should use a .skip fragment rather than publish a user-facing release note.

  • Design and architecture: The separate Kit viewport, tiled/Newton, and tiled/PhysX thresholds keep the temporary NVBUG 6570125 workaround explicit and removable. No architecture concern was identified in the changed test utility.
  • API: The new keyword-only channel_diff_threshold retains the existing default, preserving unchanged callers. The affected helper and constants are private test utilities rather than a public API surface.
  • Implementation: Fresh tiled-camera captures and backend-specific pause thresholds implement the stated test behavior. However, because the change only adjusts private integration tests and has no user-visible package behavior, repository release-metadata rules require a .skip fragment instead of the added .rst release note.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

Comment thread source/isaaclab_visualizers/changelog.d/fanes-nvbug-6570125-adjust-threshold.rst Outdated
The change is confined to the visualizer integration test helper and has no
user-visible package behavior, so it should not publish a release note. The
contributing guide reserves .skip for CI/docs/test-only PRs, and every prior
test-only fragment in this package used it.
_KIT_PAUSED_VIEWPORT_CHANNEL_DIFF_THRESHOLD = 80
"""Per-channel threshold for paused Kit viewport comparisons (0–255 space)."""

_KIT_PAUSED_TILED_CAMERA_NEWTON_CHANNEL_DIFF_THRESHOLD = 160

@matthewtrepte matthewtrepte Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm more of a note than a request: 160 out of 256 is a large part of the range; its not rejecting many cases. not sure if there's a cleaner way like loosening also the pixel count threshold so that this threshold doesn't need to be as high

but given that its temporary and well documented that is so and what the bug is, i think its fine.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I had the same concern. 160 felt high to me too. The measured separation from real motion is what made me comfortable with it as a temporary WAR, and I wouldn't want this to quietly become permanent.

@kellyguo11
kellyguo11 merged commit ca23aca into isaac-sim:develop Aug 14, 2026
45 of 46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants