Re-enable the Kit visualizer pause checks - #7049
Conversation
RTX ResponsiveDenoising keeps refining a paused frame, so two captures taken while simulation is frozen differ by a small number of high-amplitude pixels. Amplitude separates that residue from real motion and the pixel count does not: at the shared threshold of 50 the paused noise reaches 1.55x the weakest real motion signal, so any count-based gate loose enough to stay quiet is also blind to a genuine regression. _assert_frames_remain_stable now takes a channel_diff_threshold, raised to 80 at the Kit viewport pause and 150 at the Kit tiled camera pause. The tiled override is gated on KitVisualizer because all four tiled cases share _attempt_pause and the Newton-visualizer cells sit at zero noise. Newton ViewerGL rasterises without DLSS and keeps the strict default. The pixel gate stays at 100. Both paused tiled captures render fresh again, reverting isaac-sim#6658. Comparing the sensor's cached frame against itself cannot detect a renderer that keeps changing the image after physics stops, which left that assertion unable to fail. Measured on L40 / driver 595.58.03 / Isaac Sim 6.1.0-alpha.47. Work-around for NVBUG 6570125; remove once OVRTX 0.5 ships the fix.
These checks have been dead weight for a while. RTX responsive denoising keeps refining the image after physics stops, so two frames captured during a pause differ by ~900 pixels and the test went red constantly. isaac-sim#6658 quietened it by comparing the camera's cached frame against itself, which can't fail whatever the renderer does. The denoiser residue is a few very bright pixels; real motion is spread across many. So count on amplitude instead: only pixels differing by 160 or more (80 in the viewport). The shimmer drops out, real movement doesn't. Measured over 24 runs on an L40, and confirmed the check still fails when fed actual motion. Worth being clear about the scope: no physics can step inside this pause window, so this is a renderer-stability check, not a pause-correctness one. It catches a gross instability, not a subtle one. Work-around/Temp fix for NVBUG 6570125. Need to remove once the real fix is in.
Shortened the comment regarding NVBUG 6570125 for clarity. Signed-off-by: fanes <74020209+fatimaanes@users.noreply.github.com>
The pause pixel-count log never reaches CI: every pause call site runs inside caplog.at_level(logging.WARNING), which hard-sets the root logger, so the module logger's effective level is WARNING and info() short-circuits. Raising it to WARNING would instead self-trip _assert_no_visualizer_log_issues once ASSERT_VISUALIZER_WARNINGS is enabled. The assert message already carries the count, threshold and gate on the run where it matters. Give the tiled PhysX cell its own constant rather than borrowing the viewport one. It measures 103 differing pixels at the default threshold against a gate of 100, so lowering the viewport value later would silently break it.
Greptile SummaryThe PR re-enables meaningful Kit tiled-camera pause checks by capturing fresh renders and introduces temporary, per-configuration channel-difference thresholds for known RTX responsive-denoising noise.
Confidence Score: 5/5The PR appears safe to merge with no actionable defects identified in the changed test paths. Current callers select the intended threshold for each visualizer/backend configuration, fresh captures restore a non-vacuous renderer-stability check, and no evidence establishes a build, lifecycle, or test-contract failure. Important Files Changed
Reviews (1): Last reviewed commit: "Drop dead pause log and split the tiled ..." | Re-trigger Greptile |
There was a problem hiding this comment.
Isaac Lab Review Bot
The pause-check implementation is confined to private visualizer integration-test utilities, but its changelog treatment needs correction: this test-only workaround should use a .skip fragment rather than publish a user-facing release note.
- Design and architecture: The separate Kit viewport, tiled/Newton, and tiled/PhysX thresholds keep the temporary NVBUG 6570125 workaround explicit and removable. No architecture concern was identified in the changed test utility.
- API: The new keyword-only
channel_diff_thresholdretains the existing default, preserving unchanged callers. The affected helper and constants are private test utilities rather than a public API surface. - Implementation: Fresh tiled-camera captures and backend-specific pause thresholds implement the stated test behavior. However, because the change only adjusts private integration tests and has no user-visible package behavior, repository release-metadata rules require a
.skipfragment instead of the added.rstrelease note.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
The change is confined to the visualizer integration test helper and has no user-visible package behavior, so it should not publish a release note. The contributing guide reserves .skip for CI/docs/test-only PRs, and every prior test-only fragment in this package used it.
| _KIT_PAUSED_VIEWPORT_CHANNEL_DIFF_THRESHOLD = 80 | ||
| """Per-channel threshold for paused Kit viewport comparisons (0–255 space).""" | ||
|
|
||
| _KIT_PAUSED_TILED_CAMERA_NEWTON_CHANNEL_DIFF_THRESHOLD = 160 |
There was a problem hiding this comment.
hm more of a note than a request: 160 out of 256 is a large part of the range; its not rejecting many cases. not sure if there's a cleaner way like loosening also the pixel count threshold so that this threshold doesn't need to be as high
but given that its temporary and well documented that is so and what the bug is, i think its fine.
There was a problem hiding this comment.
Yeah, I had the same concern. 160 felt high to me too. The measured separation from real motion is what made me comfortable with it as a temporary WAR, and I wouldn't want this to quietly become permanent.
Temporary workaround for NVBUG 6570125 while we wait for the upstream renderer fix.
Problem
RTX responsiveDenoising continues refining the image after physics stops, so fresh renders captured during a pause are no longer stable.
The affected Kit visualizer checks are currently green for the wrong reasons:
Tiled camera: #6658 changed the paused captures to reuse the cached sensor frame, so the assertion effectively compares the same frame against itself and can't detect renderer instability.
Viewport: still performs a real comparison, but the known denoiser noise can push it over the existing threshold and the failure is absorbed by retries.
What this changes
This PR restores fresh renders for the tiled pause check and adjusts the per-channel difference threshold only for the affected Kit RTX pause paths.
The existing 100-pixel gate stays unchanged. We're changing how large a per-channel difference must be before a pixel is counted, rather than simply allowing more differing pixels.
Values are the number of differing pixels at each per-channel threshold.
For the margin-critical case (Kit tiled camera, Newton backend), threshold 160 leaves at most 49 px of paused denoiser noise (worst of n=36) against at least 160 px with real motion (weakest of n=72), both measured with the denoiser enabled. With the existing 100-pixel gate the known renderer noise passes at 2.0× margin while real motion is still detected at 1.6×.
Changes
_assert_frames_remain_stablenow accepts a per-callchannel_diff_threshold; the default remains 50.Kit RTX pause checks use 80 for the viewport, 160 for the tiled camera on Newton, and 80 for the tiled camera on PhysX — three separate constants, since the tiled PhysX cell measures 103 px at the default threshold and must not silently track a later change to the viewport value. Newton ViewerGL remains at the strict default since it doesn't exercise the affected DLSS path.
max_differing_pixelsremains 100 everywhere.The tiled pause check is restored to fresh renders instead of cached frames, so the assertion is actually exercising renderer stability again.
Validation
With retries disabled, the workaround passed 40/40 runs across all four affected configurations.
As a control, restoring the original threshold of 50 reproduced the failure 6/6 times at 766–949 differing pixels. Those six failures are all the tiled assertion: the control ran under
pytest -xand the viewport test is defined first in the same file, so the run aborted at the tiled test and the viewport site passed at 50 in those reps. The control therefore demonstrates that the tiled workaround filters real denoiser noise rather than simply making the test pass; it is not evidence for the viewport site.The viewport raise from 50 to 80 rests instead on production CI, where the assertion has really been failing (
107 pixels differed, expected at most 100, with a historical band reaching 161 px), plus n=5 bench samples showing 67 px at threshold 50 falling to 12 px at 80.The same change was validated on three independent L40S CI runners through #7055, with all four visualizer integration tests passing first-pass, no retries. Restoring fresh renders also had no meaningful runtime impact (~27s median).
This is a temporary workaround for NVBUG 6570125 and should be removed or re-measured once the upstream fix is available. That revert is load-bearing rather than cosmetic: with the denoiser disabled — the world after the renderer fix — the weakest real motion in the governing cell measures 80 px at threshold 160, which is below the 100-pixel gate. Leaving 160 in place after the fix ships would make the tiled check blind to the weakest motion it exists to catch, so the removal comment on the constants is the thing guarding that.