Add a task discovery API for backend combinations - #7046
Draft
AntoineRichard wants to merge 10 commits into
Draft
Conversation
AntoineRichard
force-pushed
the
antoiner/task-discovery-api
branch
4 times, most recently
from
August 20, 2026 12:39
c557cfe to
0d9227d
Compare
AntoineRichard
force-pushed
the
antoiner/task-discovery-api
branch
from
August 21, 2026 14:52
0d9227d to
16dfb9d
Compare
The Gym registry is walked in two places with two different answers. tools/environ_docs.py reads what a task declares, which is what the environment tables publish. Nothing reports what actually resolves, so a combination can be documented while being impossible to run: that is how the AnymalC-Direct rows in environments.rst came to advertise presets the task does not have. Add tools/task_discovery.py. discover_tasks(resolve=False) reports the declared view; resolve=True additionally builds each combination and runs the runtime validator, keeping only combinations that can run. Automatic selectors are reported separately from concrete backends. physics=physx resolves to OvPhysX kitless and to Isaac Sim PhysX under Kit, and renderer=rtx behaves the same way, so a selector and its target are the same run. Which of the two to drop depends on how the caller launches, so discovery reports both and lets callers decide. They are detected by config type rather than by name, so a new selector upstream needs no edit here.
Two defects in the first cut. ModuleNotFoundError subclasses ImportError, which was listed as a structural failure, so one uninstalled extra aborted the whole walk rather than skipping that task. A task whose config needs teleop or mimic simply cannot be inspected from a partial install; that is a property of the environment, not a broken registry. ``declared`` also collapsed two different answers into one. A config that cannot be loaded is unknown; a config that loads and declares nothing means the task runs on a fixed backend with no selectable alternative. 37 of 127 tasks are in the second group, and the environment tables render the two differently, so ``declared`` is now ``None`` only for the first. ``declared`` reports raw registry names, with BACKEND_MIRROR_NAMES exported so callers filter to suit what they are reporting.
The environment tables and the discovery API walked the Gym registry separately, so the same knowledge - which tasks are trainable, which presets they declare, which names mirror a backend selector - was maintained twice and had already drifted. Delegate the walk. The documentation policies stay here, because they are about how the tables read rather than what the registry contains: the implicit-PhysX inference, the manual RL-library overrides, and the render-time preset filtering are unchanged. Verified by generating environments.rst and environment-browser.js twice in one environment, once per implementation, in separate processes: both files are byte-identical. Note that `update_environments_rst.py --check` is not a valid test for this, because the generator's output depends on which optional backends are installed - a task shows `ovphysx` only when ovphysx is importable - so it conflates a code change with environment drift. The A/B comparison isolates the code change. Run with teleop and mimic installed so that all 127 tasks load; without them 31 configs are unloadable and never exercise the comparison.
Resolving a combination already builds its env config, so combinations can be compared on what they produce rather than on how they were spelled. Fingerprint the resolved config and collapse the matches: aliases fold into the backend they resolve to, and the no-token run folds into whichever preset the config already defaults to. Across the registry that turns 2024 declared combinations into 1045 distinct runs. Collapsing on the resolved config rather than on the backend is deliberate -- the Reach controller presets share a backend and are four different runs. What a task does with no tokens is reported separately as ``default``, since the collapse would otherwise hide it. This subsumes three name-based filters, all of which were wrong per task: * ``BACKEND_MIRROR_NAMES`` dropped every DOMAIN name matching a backend. On a task that exposes its backends only as ``presets=`` tokens, that was the sole handle: ``Isaac-Open-Drawer-Franka`` reported one mode instead of four. A DOMAIN name is now a duplicate only when the task also declares it on a typed axis. * ``_SKIP_PHYSICS`` dropped ``newton_mjwarp_vbd_proxy`` as a proxy variant. It is a real backend, and on ``Isaac-Lift-Cloth-Franka`` the only one that resolves. * ``_selector_names`` existed to let callers subtract aliases. It also missed selectors nested inside a variant, so it reported none for the cabinet tasks. Also: * An Isaac Lab API failure now drops the combination it was judging and logs it, instead of ending the whole walk; pass ``strict=True`` for the old behaviour. * ``collapse=False`` keeps every validated spelling, which is what documentation needs -- a preset naming the default of its own axis is still a token a reader can type. * ``RL_LIBRARY_PRIORITY`` and ``is_training_task`` are now owned here and imported by the docs generator, so the tables and the matrix cannot drift. ``rlinf`` was missing from the discovery copy. * ``Mode.presets`` documented that ``presets=`` composes across config paths and that discovery validates each name alone, so ``modes`` under-approximates a task with several independent preset axes. * ``modes`` documented as assuming a headless launch, since a visualizer, livestream or explicit experience is a Kit source that narrows the legal set. The changelog fragment is dropped: no source package changed, and nothing under tools/ is released.
The module was 49% prose: 240 lines of docstring and comment against 201 of code, much of it explaining bugs that no longer exist. That history belongs in the commit log, not in every reader's way -- _domain_presets carried 25 lines of docstring over 2 lines of code. Keep the contract and the non-obvious constraints: the headless-launch assumption, that presets compose across config paths but are validated one at a time, and why the collapse keys on the resolved config rather than the backend. Drop the narratives. Also folds the two single-use helpers into their call sites; the explicitness score now rides in the dedup map instead of being recomputed.
Sort resolution failures by what they mean instead of by which builtin they
happened to raise. Discovery's own imports -- two of them private sim_launcher
symbols -- now sit in their own try and raise DiscoveryError, so losing one is
reported as the API drift it is rather than escaping as a bare ImportError past
the per-combination handler and aborting the whole walk. That was the exact
failure strict= was added to make survivable, and it was not covered.
A missing extra is now caught as ModuleNotFoundError rather than ImportError,
which makes the ImportError entry in _INFRASTRUCTURE_ERRORS reachable and gives
it a meaning: a module that imports but no longer exports what a config asks for
is drift, not a rejection. Previously that entry was dead and its comment
described the opposite of the behaviour.
A task whose config will not load now reports no modes. It reported declared=None
("nothing is known") beside one all-None mode ("runs on its defaults"), which
contradict each other; a cross product of nothing known is not a cross product of
nothing declared. No task in a complete install hits this, but a partial one does.
Drop the field defaults on DiscoveredTask. They allowed a default-constructed
instance whose declared={} raises KeyError in the docs generator, and the one
construction site passes every field anyway.
Documentation corrections, all of them claims that did not survive checking:
Mode.physics/renderer None means "no token passed", not "declares none"/"headless";
only --visualizer kit is a Kit source, and the headless assumption is not hermetic
because LIVESTREAM is read from the environment and validation stops before the
Isaac Sim availability check; contrib tasks come from isaaclab_tasks, not from the
experimental package; -Eval tasks are separate registrations, not aliases; strict=
drops a combination, not a task; only the RL library ordering is shared with the
docs generator, so rlinf can differ between the two.
Add resolution tests against real configs, kept in their own module so the unit
tests stay runnable on pytest alone, and wire that unit module into tools-tests --
it was never running in CI, having been excluded as needing a full Isaac Lab
install, which is not true of it.
Two fixtures replace the monkeypatch boilerplate that five tests each repeated, and three merges remove cases that were re-asserting a neighbour: the declared cross product becomes one parametrized test over its three shapes, the two uncollapsed cases become one (a rejected token in the same table covers both), and the fingerprint's stability and discrimination are one property, not two. The argv guarantee moves into the error-taxonomy test, where it should hold on every branch rather than on one. Prose is cut where it restated itself rather than where it carried a fact -- the launch-baseline caveat keeps both leaks, LIVESTREAM and the missing Isaac Sim check, in half the words. Also corrects this file's own docstring, which still claimed selector detection was exercised by running a tool. Selector detection is gone and there is no tool; the resolver is covered by the companion module.
The module was 42% prose against 207 lines of code, because I had been putting reasoning in docstrings that belongs in the commit log and the PR. Every fact a caller needs to use the API correctly is kept -- the declared None-vs-empty distinction, the LIVESTREAM and Isaac Sim leaks in the headless assumption, the one-preset-at-a-time under-approximation, why ModuleNotFoundError is narrower than ImportError -- and the justifications around them are gone. 143 docstring lines to 107, 39 comment lines to 31.
Docstrings state the contract, not the reasoning behind it. Also fixes a docstring indent left mangled by an earlier edit, which ruff does not reflow.
The env constructor calls cfg.validate() before it builds anything, and tasks put their cross-axis rules there: Reach rejects the newton_ik action preset unless physics is Newton, and Lift rejects camera data types the Warp renderer cannot produce. Resolution stopped short of that call, so discovery reported those pairings as runnable and they failed on the first reset instead -- six OSMO jobs for Reach alone. Registry-wide this drops 1180 resolved combinations to 1027. Spot-checked: the Warp renderer keeps rgb and depth and loses albedo and simple_shading, which is exactly what lift_env_cfg allows, and albedo still resolves on isaacsim_rtx. No task loses every mode.
AntoineRichard
force-pushed
the
antoiner/task-discovery-api
branch
from
August 24, 2026 09:45
16dfb9d to
4c74226
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The Gym registry is currently walked in two places that answer two different questions:
tools/environ_docs.pyreads what a task declares. That is what theenvironments.rsttables publish.The gap is not theoretical. A combination can be documented while being impossible to run — that is how the AnymalC-Direct rows advertised presets the task does not have (fixed by hand in #6914). Because CI never runs
update_environments_rst.py --check, nothing catches that class of drift.This adds
tools/task_discovery.py:Resolved mode is the useful part: the cross product is not all legal. OVRTX is kitless and cannot share a process with Kit physics, so
isaacsim_physx + ovrtxis declared yet unusable. Discovering that costs one config resolution; discovering it on a GPU costs a whole benchmark run.Runs, not spellings
Resolving already builds each combination's env config, so combinations can be compared on what they produce rather than on how they were spelled. Fingerprinting the resolved config collapses the ones that produce the same run:
physics=physxfolds into whichever concrete backend it resolves to — no table of alias names needed, and it stays separate on tasks where it genuinely resolves differently.default, which the collapse would otherwise hide.Across the registry that turns 2277 declared combinations into 1180 distinct runs. On
Isaac-Cartpole-Camera, 200 → 90.Collapsing on the resolved config rather than on the backend is deliberate: the four Reach controller presets share a backend and are four different runs.
collapse=Falsekeeps every validated spelling, which is what documentation needs — a preset naming the default of its own axis is still a token a reader can type.Equivalence with the current docs walk
collect_environment_doc_rows()now delegates its registry walk todiscover_tasks(resolve=False). The generated output is unchanged: bothenvironments.rstandenvironment-browser.jsrender byte-identical todevelopacross all 136 tasks.environ_docskeeps its own display policy —apply_rl_library_overrides,_physics_names_for_docsand the selector filtering stay where they are.Scope
The module plus the docs-generator wiring. Deliberately not here: feeding resolved data to any published output. That is a separate change with its own cost and CI shape, sketched below.
Follow-up: validated combinations in the interactive environment browser
docs/source/_static/css/environment-browser.jscarries a generatedtaskRowsblock — 45 core tasks, five comma-joined strings each,[task, rl, physics, renderer, presets], consumed as three independent axis lists.Because the axes are independent, the picker offers combinations that cannot run. On
Isaac-Cartpole-Camerait presents 20 physics×renderer pairs; 2 are rejected by the runtime validator (isaacsim_physx + ovrtx,ovphysx + isaacsim_rtx) and 8 more are duplicate spellings. A user selecting one gets a failure at launch. This matters more than in the RST tables — the browser is a chooser, not prose.Measured cost of fixing it properly:
Two tasks are 45% of that —
Isaac-Lift-KukaAllegro-CameraandIsaac-Reorient-KukaAllegro-Cameraat 205 modes each (24 camera modality/resolution presets crossed with backends).It needs three coordinated changes:
discover_tasks(resolve=True, collapse=True), plus each task'sdefault.ovphysx, andisaacsim_rtxmust disappear), rather than filtering three axes independently.update_environments_rst.py --check, which gates every PR touchingtools/*.py. It needs a scheduled job that regenerates and commits the artifact, with--checkstill verifying the cheap declared parts.One constraint to design around: resolved results are environment-dependent.
LIVESTREAM=1in the environment silently rejects every kitless combination, and validation stops before the Isaac Sim availability check, so a kitless machine reports Kit-requiring combinations as runnable. The regenerating job has to be pinned and documented, or an ad-hoc run from a laptop would quietly publish a wrong matrix.A cheaper first step, if the full version is not worth it: keep the three-list model and only subtract the rejected pairs. No UI change, the page keeps working, it just stops offering combinations that fail.
Also open
tools/matchesenviron_docs.py, but if this should be a supported API for out-of-tree consumers,source/isaaclab_tasks/isaaclab_tasks/utils/may be the better place.RL_LIBRARY_OVERRIDESstays inenviron_docs, sorl_librariesand the rendered table still disagree forIsaacContrib-Assemble-Trocar-G129-Dex3(()vs{'rlinf': ['PPO']}). Only the ordering constant is shared.Type of change
Checklist
pre-commitchecks with./isaaclab.sh --formattools/, and no source package is touched