[Assets] Add SimReady catalogue objects and a lift environment that uses them - #3
Draft
hujc7 wants to merge 32 commits into
Draft
[Assets] Add SimReady catalogue objects and a lift environment that uses them#3hujc7 wants to merge 32 commits into
hujc7 wants to merge 32 commits into
Conversation
github-actions
Bot
added
documentation
hujc7
force-pushed
the
jichuanh/simready-multiobject
branch
from
July 28, 2026 06:23
2773365 to
8162e22
Compare
hujc7
force-pushed
the
jichuanh/simready-usd-search
branch
from
July 28, 2026 06:37
e216ed3 to
2cb8260
Compare
hujc7
force-pushed
the
jichuanh/simready-multiobject
branch
from
July 28, 2026 06:37
8162e22 to
c48d003
Compare
Spawn configurations previously required hardcoded USD file paths. This adds the isaaclab.sim.spawners.simready sub-module, which resolves the usd_path of a spawner configuration from a SimReady USD-Search semantic query at configuration construction time: SimReadyUsdFileCfg uses the top-ranked result and SimReadyMultiUsdFileCfg freezes the top-k results for heterogeneous cloning. No spawner changes are needed since the resolved paths feed the existing UsdFileCfg/MultiUsdFileCfg machinery. The required simready-search package is optional and lazy-imported, available through the new 'simready' extra (./isaaclab.sh -i simready). Search failures raise typed errors, results are tie-broken deterministically for multi-process runs, and already-resolved configurations skip the query on copy()/replace().
The constant tests assert the literal contents of VALID_EXTRA_FEATURES and MANUAL_EXTRA_FEATURES, so the new 'simready' extra must be added to their expected sets. Also mirror the manual-extra dispatch test used by 'ov' for the new selector-less 'simready' token.
The search function is spawner-agnostic: it takes a query and returns USD asset paths. Move it and the endpoint constant to isaaclab.utils.assets, next to the other asset retrieval helpers, so non-spawner consumers can use it without importing spawner modules. The simready spawner sub-module keeps only the two cfg subclasses. Split the tests to mirror the layout: resolver tests move to the utils asset tests together with the fake simready.search module; the spawner cfg tests stub the resolver directly and gain a parameter-forwarding test.
Per SimReady team guidance, treat semantic search as an authoring-time discovery step and consume the resolved USD files directly for reproducible training. Drop the spawner cfg subclasses (query-per-run ergonomics, import-time network calls, silent Hydra override no-op) in favor of the plain resolver plus pin-first guidance and the search API contract in its docstring. Add scripts/demos/simready_lift.py integrating the search with the Franka lift task: resolved assets replace the cube one variant per environment, with zero-action smoke, PPO training, and checkpoint playback modes. Pinned --usd_path runs need no service credentials.
The demo previously took a fixed list of asset paths and rescaled them to fit the gripper. That does not extend to a large object set: the search service returns only asset paths, so nothing in a query says whether a given robot can actually pick an object up. Add SimReadyObjectLibrary, which searches the catalogue, opens each candidate once to record the facts the service does not expose (rigid-body presence, authored mass, bounding box, latest validation verdict), caches those audits, and selects objects the robot can handle. Assets are left as published apart from extracting their rigid body into a uniform /Object root, which heterogeneous cloning requires because assets nest that body under differently-named paths. Asset access needs no Isaac Sim: each asset's layer closure is mirrored over HTTPS and opened as local files. Point the demo at Isaac-Lift-KukaAllegro, which already supplies point-cloud observations, contact-shaped rewards and a per-environment table, so the demo only supplies objects.
The search service filters server-side on bounding-box height and on path substrings, but the helper exposed neither, so callers had to re-implement the search to use them. Excluding locally also wastes the top_k budget on results that are thrown away. Sweeping many phrases additionally needs an empty result set to be a non-event rather than an error, since one barren phrase must not sink a whole sweep. Point the default endpoint at production. The dev deployment's asset index lags and returns stale results, so it was never a usable default.
Make room for a configuration class alongside the implementation, matching how the other utils sub-modules (noise, modifiers) are laid out: a lazily exported package whose overview lives in __init__ and whose symbols are declared in the type stub. No behaviour change.
Selection knobs were spread across four method signatures, so a caller had to know which argument each method wanted and repeat the ones that had to agree between calls. Collect them into SimReadyObjectFilterCfg and SimReadyObjectLibraryCfg and have the implementation read only from those, matching how the rest of the repository separates configuration from behaviour. Every filter bound is documented as a statement about robot capability, which is what the values actually encode: an object is dropped because the gripper could not hold it, never because its authored mass looks surprising. Delegate the service call to search_simready_usd_paths rather than re-implementing it, add resolve() for the common search-select-prepare sequence, and report progress through the module logger instead of print.
Selection now reads from cached audits, so the whole filter, family grouping, and mass stratification can be exercised without the simulation app or the search service by pre-populating the cache. Cover each rejection reason, the mass band reserved for objects at the edge of the gripper's capability, family grouping, and the cache round trip, where JSON has no tuple and the bounding-box extents could silently come back as a list.
Add isaaclab.utils.simready to the utils API reference and record the new configuration classes, the extended search filters, and the endpoint change in the changelog fragment. Also re-check the path exclusions locally after the search returns. They are applied by the service, and the redundant check costs nothing.
The audit cache persists ObjectSpec field-by-field, which couples the on-disk format to the field names: a rename turned every existing entry into a TypeError on load. That is the worst place to fail, because the cache is what keeps resolution off the slow path in the first place. Cache contents are regenerable by definition, so treat an entry whose layout no longer matches as a miss and audit the asset again.
simready/simready.py was the only module in utils/ sharing its package's name. Every other sub-module is named for what it holds -- noise_model.py, modifier_base.py, circular_buffer.py -- and a package/package.py pair also makes attribute access on the package ambiguous between the exported symbol and the sub-module. No behaviour change.
Importing it through the lazily exported isaaclab.utils package left every public symbol in this sub-module undocumented: while autodoc walks isaaclab.utils, resolving a member re-enters that package's lazy lookup, which raises, so autodoc records the member as unavailable and skips it. The API page rendered the module docstring and nothing else. Every other configuration module imports the symbol directly from isaaclab.utils.configclass. Do the same.
The config exposed three of the service's eleven filters and hardcoded a fourth, so anything else the service could answer had to be reached by bypassing the library. Worse, the workspace size bound was silently reused as the service's height query, mixing a robot-capability number into a catalogue request. Expose every filter the service supports, and keep the ones it does not -- mass, the two bounding-box axes beyond height, rigid-body presence, the newest-dated validation verdict, product-family collapsing -- as fields that the implementation completes after opening the asset. Each field records which of the two it is, so the cost of a filter is visible where it is set rather than buried in the query builder. Also check validation against the configured features instead of a hardcoded one, and document the four-stage workflow with its per-stage cost on the library itself.
The catalogue objects were only reachable through a demo script that rebuilt the task configuration by hand, so they could not be trained with the standard workflow and the overrides were invisible to anyone reading the task. Add a mixin that changes the one thing that differs from the stock lift task -- which objects are spawned -- and register Isaac-Lift-KukaAllegro- SimReady and its play variant alongside the existing entries, matching how the camera variants are configured. Training now runs through the usual train.py with no bespoke script. Spawn through MultiUsdFileCfg rather than assembling a list of per-asset spawners, since that wrapper already exists for exactly this case.
The registered Isaac-Lift-KukaAllegro-SimReady environment covers what the script did, through the standard training workflow, so keeping the script would leave two entry points that have to agree about how objects are resolved. Repoint the two references that named it: the search helper's docstring now points at the sub-module, and the changelog records the removal with the command that replaces it.
The filter shipped a table-top phrase list, workspace size bounds, gripper mass bounds and a heavy-object share, none of which describe the catalogue or the search service -- they describe one task on one robot. No other core configuration embeds a domain value list as a default, and must-choose fields elsewhere use MISSING. Make the fields that define a task MISSING, give the optional ones a neutral disabled default, and move the table-top values to the environment that wants them. Also drop the play variant, leaving a single registered environment.
heavy_from and heavy_fraction encoded one opinion about what a good object set looks like -- that some share should be too heavy to lift -- as two coupled knobs on a general library. The pair only meant anything together, neither described the catalogue or the robot, and no other criterion could be expressed that way. Drop both and add filter_func, an optional predicate over the audited spec, so a caller can express what the named fields cannot: an aspect-ratio bound, a density, a naming convention. The task keeps its mass range and lets the catalogue's own spread decide how often lifting is easy. The spec passed to the predicate carries what the built-in filters need and no more, since the audit is cached in that form; the field documents the limit.
Only the per-asset audits were cached, so every construction still issued one search request per phrase. Under distributed training that is once per rank, and a catalogue update between ranks would hand them different object sets -- a correctness problem, not just a slow start. Record the resolved asset list against a fingerprint of the effective query, so a repeat resolve issues no search at all and changing any filter resolves afresh. The file is JSON, sorted and indented, small enough to commit when a run has to be reproducible. Keep it separate from the audit cache: that one is keyed by asset, grows with the catalogue and is disposable, while this is keyed by query and is worth keeping.
Auditing is the one slow stage -- on a cold cache every candidate is an HTTPS fetch of a whole layer closure plus a stage open -- and it ran silently, so a resolve looked hung for twenty minutes with no way to tell progress from a stall short of counting files on disk. The earlier implementation logged every 25 candidates; that was lost when printing was replaced with the module logger.
Auditing 210 candidates took 32 minutes, and profiling put 100% of that in the network: mirroring an asset's layers costs ~20 s while opening the stage and measuring it costs ~0.01 s. The layers are fetched one request at a time, so the cost is per-file round-trips rather than bandwidth. Audit candidates through a thread pool. The work is independent per asset and bound by latency, so it scales close to linearly. Assets share material and texture layers, so concurrent audits can race for the same path; download to a private name and rename, which is atomic on the same filesystem.
Nothing about the asset host requires the layers of a single asset to be mirrored in sequence -- they are plain HTTPS objects. The sequencing was an artefact of walking the closure one URL per iteration, which made an asset cost the sum of its layers' round-trips: about thirteen requests at a second and a half each. A layer's own references are only known once it has been parsed, so the closure still has to be walked breadth-first. That ordering is per level, not per file, so fetch every layer of a level at once.
…package The task resolves its objects while its configuration is built, so the first launch legitimately needs the network. Every later one must not: it has to work on a machine with no connectivity and without the optional simready-search package installed. Assert it by making any search attempt raise, the way an uninstalled package would, and requiring the resolve to succeed regardless.
hujc7
force-pushed
the
jichuanh/simready-usd-search
branch
from
July 29, 2026 22:21
2cb8260 to
80608db
Compare
hujc7
force-pushed
the
jichuanh/simready-multiobject
branch
from
July 29, 2026 22:21
3d5ca98 to
c107314
Compare
hujc7
changed the title
The configuration described where each filter ran -- at the search service or after opening the asset -- which is an implementation detail that can change if the service starts returning more. From a caller's side there is one idea: assets have properties, and you constrain the ones you care about. Drop the service/local framing, make every field optional so an unset property is simply not constrained, and remove the fields nobody could populate: the catalogue vocabulary filters (classes, tags, profiles, countries, scene locations) work only if you already know a valid value, and the service publishes no list and returns none on results. Record the resolved asset set to a file meant to be committed beside the task. A configuration then resolves to a fixed set of objects that everyone fetches without contacting the service, rather than each machine -- or each rank -- querying a catalogue that may have moved in between. Also: num_objects belongs to the query rather than the scene; distinct_families becomes max_per_product_family, which says what it does; and _filter becomes _search_filter, since it returns a filter object.
Generated against the live catalogue: 210 candidates, of which 61 carry no rigid body, 44 exceed the workspace and 6 are too thin, leaving 99 usable assets across 39 product families. The record is used as-is on a later run -- 39 objects in 0.1 s with the search path made to raise -- so the task spawns the same objects on any machine without contacting the service. Warn when fewer objects satisfy the filter than were asked for. The task requests 100 and the catalogue holds 39 distinct table-top families, which was previously visible only as an INFO line.
Capping variants at one per product left 39 objects from 99 usable assets. The wider set is worth more to training than the visual repetition costs, and 99 is what the catalogue holds under these bounds.
Capping variants at one per product left 39 objects where 99 are usable. The wider set matters more to training than the visual repetition costs, so the task takes all of them and the field -- along with the name-derived family grouping behind it -- goes with it. num_objects stays at 100 as an upper bound; the catalogue yields 99 under these filters, and the warning added alongside says so rather than leaving the shortfall to be noticed.
Generated against the live service: 210 candidates, of which 61 carry no rigid body, 44 exceed the workspace and 6 are too thin, leaving 99. Replaying the record resolves those 99 in 0.1 s with the search path made to raise, so a run reproduces the same objects with no service call. Doubling the phrase list returns the same 210 candidates, so 99 is what the catalogue holds under these bounds rather than a limit of the query.
Capping variants is worth having when the objects are seen together -- a demo grid or a recorded video, where four golf balls read as one object -- and costs object count when they are not. The lift task wants the wider set, so it leaves the field unset and takes all 99. Restore the field, the grouping behind it, and its tests; the docstring now states the tradeoff with the measured numbers (99 uncapped, 39 at a cap of one) so the choice is visible where it is made.
UsdPhysicsMassAPI may be applied to the rigid body or to the collision prims beneath it, and the two are not alternatives: a value on the body is the body's mass, while values on descendants are contributions that sum. Catalogue assets use the second form -- a coffee cup declares body and lid separately, which is what places its centre of mass correctly. Reading a single attribute therefore recorded a fragment of the real mass. Measured across 120 downloaded assets, 52 declare mass on more than one prim, so the mass filter has been comparing against one part of the object for roughly two in five. A coffee cup weighing 0.0398 + 0.1216 kg was recorded as whichever prim the traversal reached last. Aggregate instead, preferring a value on the body when one exists so assets that state the same mass on both body and mesh are not counted twice.
A mesh collider that applies no UsdPhysicsMeshCollisionAPI still collides, but USD defaults its approximation to none -- a raw triangle mesh, legal for static geometry only -- so a simulator asked to move the body substitutes a convex hull. A cup loses its interior and a mug its handle gap, with nothing to show for it but a log line: the asset passes validation and the run succeeds. The catalogue is well authored here, so rejecting them is cheap. Of 211 candidates, 199 declare an approximation and 12 do not; none declares an unusable one, so this is an omission in a pipeline rather than a choice. Three of the ninety-nine objects the lift task spawns are affected, and they are exactly the three PhysX reported substituting at load. Record the effective approximation on the spec as well, using none where the schema is absent, so what the simulator will do is visible rather than inferred.
Regenerated against the live catalogue: 210 candidates, of which 61 carry no rigid body, 44 exceed the workspace, 6 declare no collider approximation and 3 are too thin, leaving 96. The three objects this removes from the previous record -- a mug, a cup and a chips bag -- are exactly the three PhysX reported substituting a convex hull for at load, so every object the task now spawns collides as its author authored it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds SimReady catalogue assets to Isaac Lab: a configurable library that turns USD-Search results
into task-ready rigid objects, and a lift environment that spawns 96 distinct catalogue objects
instead of primitive shapes.
Most of this PR exists to work around what the search client cannot answer. That is worth stating
first, because if the client closes those gaps, a large part of this code should be deleted.
1. What the search client would need for Lab to use it well
simready-searchsets 13 of the 54 fields its own HTTP endpoint accepts, so Lab downloads andopens 3.1 GB of assets to recover facts the service already holds.
Reachable over HTTP today, missing from the client
min/max_bbox_x,min/max_bbox_zSearchFilterHeightmaps only tobbox_y, so two axes are checked after downloadreturn_usd_dimensionsreturn_usd_propertiesfilter_by_properties_numericsimilarity_threshold/deduplicate_by_hash/search/stats/usd_propertiesNot available at either level — service-side asks
passed=~Truematches assets that passed once, so regressed assets still match.physics:mass4–5 levels down in a sublayer.massis indexed for 182 of 4411 assets;physics:approximationfor 1;__polygon_countreports 0–2 polygons for 25% of the catalogue because only the wrapper layer is counted.physics:mass<1000000returns 4 results out of 261. Filtering on mass would silently reduce this task's object set from 96 to about 4.If validation filtering used the newest verdict and physics attributes were indexed at their authored
depth,
require_rigid_body,require_declared_collisionand the validation re-read in this PR wouldall become unnecessary, and the audit would shrink to mass alone.
2. Lab-side decisions, given those gaps
Open every candidate once, and record the result
The audit is the only place these facts exist, so it opens each asset once and caches the outcome.
The resolved asset list is written to
simready_objects.jsonbeside the task config and committed,so later runs contact no service: a cold resolve costs ~60 s, a replay 0.1 s.
Filter on asset properties, not on where the answer comes from
SimReadyObjectFilterCfgnames properties; unset fields are unconstrained. Whether a property isanswered by the index or by opening the asset is an implementation detail.
Trust what the asset declares
Authored mass and scale are used as-is. An object is dropped because this robot could not handle it,
never because the number looks surprising. Measured on the live catalogue, 210 candidates yield 96:
Aggregate mass rather than reading one attribute
UsdPhysicsMassAPImay sit on the body or on the colliders beneath it, and catalogue assets use thesecond form — a coffee cup declares body and lid separately. Reading a single attribute returned a
fragment: a drinks can was recorded at exactly half its mass.
Reject silently-degraded collision
A mesh collider applying no
UsdPhysicsMeshCollisionAPIstill collides, but USD defaults itsapproximation to
none, which is legal for static geometry only, so PhysX substitutes a convex hulland a cup loses its interior. Of 211 candidates, 199 declare an approximation (almost all
sdf) and12 declare none; none declares an unusable one, so this is a pipeline omission rather than a
choice, and the three objects it removes are exactly the three PhysX reported substituting at load.
Change one thing about each asset
The rigid body is extracted to a uniform
/Objectroot, because assets nest it underdifferently-named paths and a per-environment view cannot be built across varying paths. Collider,
mass and origin are left as published.
Keep the pipeline Kit-free
UsdUtils.LocalizeAssetcannot resolve anhttps://path without a resolver plugin, andComputeAllDependenciesreports zero dependencies rather than failing, so the layer closure ismirrored explicitly. Fetching is parallel at two levels — across assets, and across each asset's
layers — because the cost is per-file round-trips rather than bandwidth: 25.4x over the
sequential form, with an identical result.
Validation
Live service and Isaac Sim PhysX on a 4xL40 box.
/ObjectKnown gaps in this PR
defaults. Friction decides whether an object slips from the fingers, so this is the most
consequential remaining gap for a grasping task.
usd_dimensionsexposesscene_mpu, which is not checked.MuJoCo's homogeneous-worlds requirement, and
sdf— used by 157 of 159 colliders here — is a PhysXtoken absent from Newton's approximation mapping.
(
UnsupportedValueType: ResolvableString), so they cannot be launched throughscripts/reinforcement_learning/train.py. Reproduces on the stockIsaac-Lift-KukaAllegro.Review guide
Read commit by commit.
utils/simreadyis moved in one commit and rewritten in the next, so thesquashed view shows a large delete-plus-add; per commit, git reports the move as a rename.