Skip to content

[Tests] Trim redundant isaaclab_rl export and wrapper tests - #7604

Open
StafaH wants to merge 1 commit into
isaac-sim:developfrom
StafaH:test/optimize-isaaclab-rl-tests
Open

[Tests] Trim redundant isaaclab_rl export and wrapper tests#7604
StafaH wants to merge 1 commit into
isaac-sim:developfrom
StafaH:test/optimize-isaaclab-rl-tests

Conversation

@StafaH

@StafaH StafaH commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Description

Trims the isaaclab_rl test suite where it repeats an export path or a check that already exists elsewhere. On this workstation the suite took ~15 min, of which test_leapp_export_flow.py alone was 7 to 9 min and test_rsl_rl_export_flow.py another 2.5 min; every other file finishes in under 35 s.

export/test_leapp_export_flow.py

  • Dropped Isaac-Humanoid from the RSL-RL task list. test_rsl_rl_humanoid_export_across_physics_backends[newton_mjwarp] already creates the same initialized checkpoint with the same preset and exports it, so the matrix entry was an exact duplicate (one ~20 s checkpoint subprocess and one ~14 s export subprocess).
  • Dropped Isaac-Reach-UR10. The list is documented as "one example of each distinct observation / command / action pattern"; UR10 is the same pose-command + joint-position-arm pattern as Isaac-Reach-Franka, which stays.
  • Removed test_initialized_checkpoints. It only re-asserted that the module fixture wrote its path files; the fixture already fails the run on a non-zero subprocess and every export test asserts its own checkpoint path.

export/test_rsl_rl_export_flow.py

  • Reduced the pretrained-checkpoint export matrix from 28 tasks (4 Kit batches) to 12 tasks (2 batches) using the same "diversity over breadth" rule: classic, navigation, quadruped flat, quadruped rough (height scan), humanoid rough, a contrib task with its own MDP terms, reach, lift, cabinet, and both dexterous tasks. The 16 dropped entries are robot variants of the quadruped/humanoid velocity pattern plus Reach-UR10. Note that the batches containing those variants finished in ~12 s for 8 tasks in both baseline runs, i.e. the exports short-circuit (checkpoint unavailable or actor_state_dict mismatch are silently skipped), so their nominal coverage was mostly not exercised.
  • Removed test_recurrent_state_helpers_support_modular_rnn_model_lstm, which is line-for-line TestRslRlRecurrentState.test_modular_rnn_model_lstm_round_trip in test_leapp_recurrent_state.py, and test_export_flow_fails_on_sim_traceback, which tested a helper of the test file itself.

test_rsl_rl_wrapper.py

  • test_no_time_outs now builds two environments instead of five. The finite-horizon contract it checks belongs to the wrapper, not the task; test_random_actions still runs all five.

Before and after measurement

Per-file wall time from the CI-style per-file runner on the same workstation (RTX PRO 6000). Baseline numbers are from a second, warm-cache run of the unchanged files; the first cold run was 548 s / 160 s for the two export files.

File Cases before Cases after Before After
export/test_leapp_export_flow.py 18 15 411.9 s 348.1 s
export/test_rsl_rl_export_flow.py 6 2 150.8 s 70.3 s
test_rsl_rl_wrapper.py 2 2 31.8 s 31.3 s
Total (changed files) 594 s 450 s

That is about 2.4 min (16%) off a ~15 min suite. test_leapp_recurrent_state.py still passes (7 cases) and carries the recurrent-state check removed from the export flow file.

Type of change

  • Test and CI performance improvement (non-breaking)

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

Drop the LEAPP export tasks that repeat an already covered export pattern,
the checkpoint-fixture-only test, and a recurrent-state unit test that is
duplicated in test_leapp_recurrent_state.py. Limit the finite-horizon
wrapper check to two environments since the contract is task independent.
@StafaH
StafaH requested a review from a team September 5, 2026 15:39
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Sep 5, 2026
@greptile-apps

greptile-apps Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR shortens the isaaclab_rl suite by removing duplicated export cases and narrowing task matrices to representative configurations.

  • Removes redundant Humanoid, UR10, checkpoint-fixture, and recurrent-state cases from LEAPP-related tests.
  • Reduces the pretrained RSL-RL export matrix from 28 tasks to 12 representative tasks.
  • Runs the finite-horizon wrapper assertion against two environments rather than five.
  • Leaves the silent-subprocess-failure detector without a focused test of its failure branch.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking recommendation to preserve focused coverage of silent subprocess failure detection.

The task-matrix and wrapper reductions retain representative coverage, but deleting the only failure-path test for _fail_on_process_error weakens protection against successful subprocesses that emit simulator tracebacks.

Files Needing Attention: source/isaaclab_rl/test/export/test_rsl_rl_export_flow.py

Important Files Changed

Filename Overview
source/isaaclab_rl/test/export/test_leapp_export_flow.py Removes export cases whose graph or backend coverage is preserved by representative tests.
source/isaaclab_rl/test/export/test_rsl_rl_export_flow.py Narrows the pretrained task matrix and removes duplicate unit coverage, but also removes the only test of silent traceback detection.
source/isaaclab_rl/test/test_rsl_rl_wrapper.py Reduces repetitions of a wrapper-level finite-horizon contract without changing the asserted behavior.
source/isaaclab_rl/changelog.d/test-optimize-isaaclab-rl-tests.skip Documents the test-only optimization and absence of user-visible behavior changes.

Reviews (1): Last reviewed commit: "Trim redundant isaaclab_rl export and wr..." | Re-trigger Greptile

@@ -287,39 +253,6 @@ def _run_export_batch_entrypoint() -> None:
_run_export_batch(tasks)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Traceback Detection Loses Coverage

Removing the only test of _fail_on_process_error leaves its traceback-detection branch untested, even though the export flow relies on this helper to reject simulator failures that exit successfully. A later change to the patterns or failure logic could make CI accept a subprocess containing a traceback. Please retain a small focused test for the zero-exit-code failure path.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

This test-only change reduces LEAPP/RSL-RL export matrices, removes test-local redundant checks, limits the finite-horizon wrapper test to two registered environments, and adds the required .skip changelog fragment. No candidate findings identified a concrete issue requiring action before merge.

  • Design and architecture: The patch consistently adopts representative export-path coverage instead of robot-variant breadth. Retained tasks span the documented classic, navigation, locomotion, manipulation, and dexterous patterns, while production architecture and export paths remain unchanged.
  • API: No production API, public symbol, task registration, or export script is modified. Removed classes and test functions are local to test modules, so the patch does not introduce a public compatibility or migration concern.
  • Implementation: The reduced RSL-RL task list still forms two valid batches under the existing batch size, and slicing registered_tasks[:2] preserves the wrapper test mechanics while reducing task breadth. The principal non-blocking tradeoff is reduced variant-specific regression coverage, with the patch explicitly relying on representative shared-pattern and per-backend tests instead.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.

@StafaH

StafaH commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant