Skip to content

[Workflow] Remove unused/poor tests from isaaclab_tasks - #7088

Merged
StafaH merged 1 commit into
isaac-sim:developfrom
StafaH:mh/remove-tasks-badtests
Aug 14, 2026
Merged

[Workflow] Remove unused/poor tests from isaaclab_tasks#7088
StafaH merged 1 commit into
isaac-sim:developfrom
StafaH:mh/remove-tasks-badtests

Conversation

@StafaH

@StafaH StafaH commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Changes

Removed test_record_video.py.
Reason: It stepped environments but never checked recorded output. Changed the dedicated video CI job to run test_video_recording.py. It verifies video files and frame content for each source and simultaneous recorders. That test already existed before, so we were actually double-testing but doing a bad job at it...

Excluded test_video_recording.py from ordinary task shards.
Reason: It needs the movie dependency and has a dedicated CI job.

Removed the environments-training CI job and skipped test module.
Reason: The entire collected matrix was permanently skipped anyways, and was never actually run. This testing is also now in the hands of the perf/benchmarking team, and there is an expected regression test coming soon!

Removed test_lift_package_structure.py.
Reason: It locked a completed migration's file layout, and just asserted stuff. There was no real behaviour testing.

Removed the GearAssembly default-count and DR Legs tuning snapshot tests.
Reason: They restated hard-coded configuration values without exercising outcomes.

Removed Lift migration and runner configuration snapshots.
Reason: They repeated authored settings; the camera-normalization behavior test remains.

Removed the disabled deterministic-training E2E and backend-path assertions.
Reason: They were never run or only enforced internal file placement.

Consolidated runtime/Kit preset decision tests.
Reason: Two modules covered the same resolution outcomes.

Consolidated Franka and UR10 reach preset tests.
Reason: Shared preset behavior now uses one matrix; robot-specific contracts remain explicit.

Consolidated Shadow Hand camera preset tests and removed skipped OVRTX cases.
Reason: The prior cases duplicated validation and permanently skipped coverage.

Validation

  • uv run python -m pytest source/isaaclab_tasks/test/core/test_preset_kit_decision.py source/isaaclab_tasks/test/core/test_train_scripts_deterministic.py -q -k 'not registered_task_physx_presets_keep_auto_selection_explicit'
    • 15 passed.
  • uv run python -m pytest --collect-only -q source/isaaclab_tasks/test/core/test_reach_franka_presets.py source/isaaclab_tasks/test/core/test_runtime_compatibility.py source/isaaclab_tasks/test/core/test_shadow_hand_camera_presets.py
    • 103 tests collected.
  • uv run isaaclab -f
    • Formatting and lint hooks passed; the changelog hook requires the new untracked .skip fragment to be staged before it can detect it.

@StafaH
StafaH requested a review from a team August 14, 2026 00:23
@github-actions github-actions Bot added isaac-lab Related to Isaac Lab team infrastructure labels Aug 14, 2026
@greptile-apps

greptile-apps Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR streamlines isaaclab_tasks test coverage and CI selection without changing package runtime code.

  • Moves video validation to the existing content-checking test and excludes it from ordinary task shards.
  • Removes permanently skipped, migration-layout, and hard-coded configuration snapshot tests.
  • Consolidates runtime, reach-preset, and Shadow Hand camera-preset coverage.

Confidence Score: 4/5

The PR appears safe to merge, with one non-blocking gap in coverage for automatic PhysX fallback when OvPhysX is unavailable.

CI routing and consolidated tests remain functional, but the removed single-backend PhysX fallback case has no direct equivalent in the remaining suite.

Files Needing Attention: source/isaaclab_tasks/test/core/test_preset_kit_decision.py

Important Files Changed

Filename Overview
.github/workflows/build.yaml Correctly routes video recording coverage to the dedicated dependency-equipped job and removes the permanently skipped training job.
source/isaaclab_tasks/test/core/test_preset_kit_decision.py Removes duplicated runtime-resolution cases but also drops direct coverage of auto-PhysX fallback when OvPhysX is unavailable.
source/isaaclab_tasks/test/core/test_runtime_compatibility.py Retains broad validation and backend-resolution coverage after consolidation.
source/isaaclab_tasks/test/core/test_reach_franka_presets.py Consolidates Franka and UR10 preset contracts using valid parameterization and fresh configurations.
source/isaaclab_tasks/test/core/test_shadow_hand_camera_presets.py Consolidates camera and renderer preset assertions while removing permanently skipped OVRTX cases.
source/isaaclab_tasks/test/core/test_train_scripts_deterministic.py Removes disabled end-to-end determinism execution and internal path-layout assertions while retaining CLI and seed-order checks.

Comments Outside Diff (1)

  1. source/isaaclab_tasks/test/core/test_preset_kit_decision.py, line 68 (link)

    P2 Auto-PhysX fallback coverage removed

    The consolidation removes the only direct regression test for a PhysxAutoCfg that provides Isaac Sim PhysX without OvPhysX. This leaves fallback to the sole available backend and the associated Kit requirement unprotected against regressions.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Reviews (1): Last reviewed commit: "Remove unused tests and bad tests from t..." | Re-trigger Greptile

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Reviewed the test-suite cleanup, preset-test consolidation, and CI retargeting. The dedicated video job now runs test_video_recording.py, ordinary task shards exclude that dependency-specific test, and the removed training job targeted a permanently skipped module. No actionable defect is established by the patch.

  • Design and architecture: The change is confined to tests and workflow organization. UR10 Reach coverage is consolidated into the existing Reach preset matrix, Shadow Hand camera checks retain preset resolution and Warp compatibility cases, and implementation-layout or configuration snapshot assertions are removed in favor of retained behavioral coverage. A non-blocking discoverability tradeoff is that UR10 checks now reside in a Franka-named test module.
  • API: No production API, task registration, configuration field, export, or CLI contract is changed. The retained test imports match their remaining uses, and the package includes an appropriate .skip changelog fragment for this non-user-visible cleanup.
  • Implementation: The workflow path is internally consistent: test_video_recording.py is excluded from all three standard task shards and selected by the dedicated job that installs MoviePy, while the obsolete environment-training workflow and skipped test are removed together. The workflow still relies on the shared action correctly parsing the comma-separated exclude-pattern, but the supplied patch does not demonstrate a failure in that path.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.

@StafaH
StafaH merged commit 7f0d4f3 into isaac-sim:develop Aug 14, 2026
48 of 49 checks passed
@StafaH
StafaH deleted the mh/remove-tasks-badtests branch August 14, 2026 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants