Skip to content

Commit ded9dfd

Browse files
mataylor-nvidiamataylor-nvidia
committed
Add a CI probe measuring what sharing one Kit app saves
Whether migrating the remaining ~125 test files off module-scope AppLauncher is worth doing depends on how much Kit startup actually costs, which is not something the current pipeline reports directly. Add two temporary jobs that run the same 30 files from source/isaaclab/test/sim and differ only in how many Kit apps they boot. kit-reuse-probe-per-file keeps the default test-path of "tools", so tools/conftest.py gives each file its own subprocess and Kit boots 30 times. kit-reuse-probe-batched points pytest at the files directly, so they share one process and launch_kit() boots Kit once. The difference between the two job durations is what reuse is worth per 30 files. Both jobs list their files explicitly instead of selecting with `-m kit`, because pytest's marker filtering deselects tests but still imports every collected module, and importing a kit_cameras module calls launch_kit(cameras=True) regardless of whether its tests will run. The batched job lists the four kit_cameras files first: a camera-enabled app can serve tests that do not need cameras, but cameras cannot be enabled after startup, so the opposite order makes launch_kit() raise. Files in TESTS_TO_SKIP are excluded from both sides so the jobs cover the same tests. To let a job bypass the per-file orchestrator, run-package-tests gains a test-path input. It defaults to "tools", the value that was previously hard-coded, so every existing caller is unaffected. Both jobs are continue-on-error and are meant to be deleted once the measurement is recorded.
1 parent 0201086 commit ded9dfd

2 files changed

Lines changed: 124 additions & 1 deletion

File tree

‎.github/actions/run-package-tests/action.yml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,13 @@ inputs:
7979
description: 'Additional pytest options'
8080
default: ''
8181
required: false
82+
test-path:
83+
description: >-
84+
Path handed to pytest. Defaults to "tools", which loads tools/conftest.py and runs each
85+
test file in its own subprocess. Point it at a test directory instead to run those files
86+
together in a single pytest process, bypassing the per-file orchestrator.
87+
default: 'tools'
88+
required: false
8289
extra-pip-packages:
8390
description: 'Space-separated pip packages to install inside the Docker container before pytest starts'
8491
default: ''
@@ -291,7 +298,7 @@ runs:
291298
- name: Run Tests
292299
uses: ./.github/actions/run-tests
293300
with:
294-
test-path: "tools"
301+
test-path: ${{ inputs.test-path }}
295302
result-file: "${{ inputs.result-file != '' && inputs.result-file || format('{0}-report.xml', github.job) }}"
296303
container-name: "${{ inputs.container-name }}-${{ github.run_id }}-${{ github.run_attempt }}"
297304
image-tag: ${{ inputs.image-tag }}

‎.github/workflows/build.yaml‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -892,6 +892,122 @@ jobs:
892892
omni-github-test-type: warp-cache-warm
893893
#endregion
894894

895+
#region kit-reuse timing probe
896+
# TEMPORARY. Measures what sharing one Kit app across test files is worth, so the decision to
897+
# migrate the remaining ~125 files is based on a number rather than an estimate. Both jobs run
898+
# the same 30 files from source/isaaclab/test/sim; the only difference is how many Kit apps get
899+
# booted. Compare the two job durations in the Actions UI, then delete this region.
900+
#
901+
# The file lists are spelled out rather than selected with `-m kit` because pytest's marker
902+
# filtering deselects tests but still imports every collected module, and importing a
903+
# kit_cameras module calls launch_kit(cameras=True). Files in TESTS_TO_SKIP are left out of
904+
# both sides so the two jobs cover exactly the same tests.
905+
test-kit-reuse-probe-per-file:
906+
name: "kit-reuse-probe-per-file"
907+
runs-on: [self-hosted, gpu]
908+
timeout-minutes: 120
909+
continue-on-error: true
910+
needs: [build, config]
911+
if: needs.build.result == 'success'
912+
steps:
913+
- uses: actions/checkout@v6
914+
with:
915+
fetch-depth: 1
916+
lfs: true
917+
# Baseline: the default test-path of "tools" runs tools/conftest.py, which gives each file
918+
# its own subprocess, so Kit boots 30 times.
919+
- uses: ./.github/actions/run-package-tests
920+
with:
921+
image-tag: ${{ needs.config.outputs.ci_image_tag }}
922+
isaacsim-base-image: ${{ needs.config.outputs.isaacsim_image_name }}
923+
isaacsim-version: ${{ needs.config.outputs.isaacsim_image_tag }}
924+
filter-pattern: "isaaclab/test/sim"
925+
include-files: >-
926+
test_simulation_stage_in_memory.py,
927+
test_utils_prims.py,
928+
test_utils_queries.py,
929+
test_utils_semantics.py,
930+
test_articulation_fragments.py,
931+
test_build_simulation_context_headless.py,
932+
test_cloner.py,
933+
test_collision_fragments.py,
934+
test_joint_drive_fragments.py,
935+
test_mass_fragments.py,
936+
test_material_fragments.py,
937+
test_mesh_collision_fragments.py,
938+
test_mesh_converter.py,
939+
test_schema_fragments.py,
940+
test_schema_writer_nested_targets.py,
941+
test_schemas.py,
942+
test_simulation_context.py,
943+
test_spawn_from_files.py,
944+
test_spawn_lights.py,
945+
test_spawn_materials.py,
946+
test_spawn_meshes.py,
947+
test_spawn_sensors.py,
948+
test_spawn_shapes.py,
949+
test_spawn_wrappers.py,
950+
test_tendon_fragments.py,
951+
test_utils_stage.py,
952+
test_utils_transforms.py,
953+
test_views_xform_prim.py
954+
container-name: isaac-lab-kit-reuse-probe-per-file
955+
omni-github-test-type: kit-reuse-probe-per-file
956+
957+
test-kit-reuse-probe-batched:
958+
name: "kit-reuse-probe-batched"
959+
runs-on: [self-hosted, gpu]
960+
timeout-minutes: 120
961+
continue-on-error: true
962+
needs: [build, config]
963+
if: needs.build.result == 'success'
964+
steps:
965+
- uses: actions/checkout@v6
966+
with:
967+
fetch-depth: 1
968+
lfs: true
969+
# Batched: pointing test-path at the files themselves bypasses tools/conftest.py, so all 30
970+
# run in one pytest process and launch_kit() boots Kit once. The four kit_cameras files are
971+
# listed first on purpose: a camera-enabled app can serve tests that do not need cameras, but
972+
# cameras cannot be turned on after startup, so the reverse order makes launch_kit() raise.
973+
- uses: ./.github/actions/run-package-tests
974+
with:
975+
image-tag: ${{ needs.config.outputs.ci_image_tag }}
976+
isaacsim-base-image: ${{ needs.config.outputs.isaacsim_image_name }}
977+
isaacsim-version: ${{ needs.config.outputs.isaacsim_image_tag }}
978+
test-path: >-
979+
source/isaaclab/test/sim/test_simulation_stage_in_memory.py
980+
source/isaaclab/test/sim/test_utils_prims.py
981+
source/isaaclab/test/sim/test_utils_queries.py
982+
source/isaaclab/test/sim/test_utils_semantics.py
983+
source/isaaclab/test/sim/test_articulation_fragments.py
984+
source/isaaclab/test/sim/test_build_simulation_context_headless.py
985+
source/isaaclab/test/sim/test_cloner.py
986+
source/isaaclab/test/sim/test_collision_fragments.py
987+
source/isaaclab/test/sim/test_joint_drive_fragments.py
988+
source/isaaclab/test/sim/test_mass_fragments.py
989+
source/isaaclab/test/sim/test_material_fragments.py
990+
source/isaaclab/test/sim/test_mesh_collision_fragments.py
991+
source/isaaclab/test/sim/test_mesh_converter.py
992+
source/isaaclab/test/sim/test_schema_fragments.py
993+
source/isaaclab/test/sim/test_schema_writer_nested_targets.py
994+
source/isaaclab/test/sim/test_schemas.py
995+
source/isaaclab/test/sim/test_simulation_context.py
996+
source/isaaclab/test/sim/test_spawn_from_files.py
997+
source/isaaclab/test/sim/test_spawn_lights.py
998+
source/isaaclab/test/sim/test_spawn_materials.py
999+
source/isaaclab/test/sim/test_spawn_meshes.py
1000+
source/isaaclab/test/sim/test_spawn_sensors.py
1001+
source/isaaclab/test/sim/test_spawn_shapes.py
1002+
source/isaaclab/test/sim/test_spawn_wrappers.py
1003+
source/isaaclab/test/sim/test_tendon_fragments.py
1004+
source/isaaclab/test/sim/test_utils_stage.py
1005+
source/isaaclab/test/sim/test_utils_transforms.py
1006+
source/isaaclab/test/sim/test_views_xform_prim.py
1007+
container-name: isaac-lab-kit-reuse-probe-batched
1008+
omni-github-test-type: kit-reuse-probe-batched
1009+
#endregion
1010+
8951011
#region disabled quarantined tests
8961012
# test-quarantined:
8971013
# name: "Quarantined Tests"

0 commit comments

Comments
 (0)