We evaluate VLA, diffusion, and world-model policies on RoboGate's 68-scenario industrial Pick&Place suite (NVIDIA Isaac Sim, Franka Panda) with real on-GPU inference and publish the result here — truthfully, whatever the score.
- Open a submission issue and fill in:
- Model name + HuggingFace ID (or a direct weights URL)
- Architecture family, parameter count, action space + control rate
- How to run inference: the conda/pip env and the call signature that maps an
(image, joint_state, instruction)observation to a 7-DoF action. A link to the model's official eval script is the single most helpful thing. - License (weights must be publicly downloadable; gated HF repos are OK if you note access).
- We triage, run it on our GPU host (Isaac Sim ↔ ZMQ bridge), and post the result back to your issue + add a row to LEADERBOARD.md.
The queue is manual today. A self-serve
pip install robogateharness is on the roadmap so you can reproduce the run yourself.
A policy passes a scenario only by actually completing the pick-and-place. Suite-level PASS criteria mirror the product's validation gate: grasp success ≥ 0.92, zero collisions, drop rate ≤ 0.03, cycle time within 1.1× baseline.
- Real inference only. No mock, no scripted fallback, no hand-edited numbers.
- Identical suite for every model — same 68 scenarios, same success logic as the scripted control baseline (83.8%).
- Truthful publication. We publish the measured result even when it's 0/68 — including for our own and NVIDIA's models. We will not quietly drop a bad result.
- Action-space / control-rate mismatches (a known confound for cross-embodiment eval) are noted on the model's row rather than hidden.
If you believe a run mis-handled your model (wrong env, action-space mapping, etc.), reply on the submission issue with specifics and we'll re-run. Re-runs and their diffs are logged publicly.