Repository navigation
Feat/sharpa state verifier - #52
Closed
rebecca-wu-applovin wants to merge 14 commits into
Closed
rebecca-wu-applovin wants to merge 14 commits into
rebecca-wu-applovin wants to merge 14 commits into
Conversation
The closed-loop sim eval builds and loads GWP-0.5 Sharpa EEF checkpoints through the training
script, so these pieces are needed on a fresh checkout:
- scripts/parallel_sharpa_forward_checks/run_giga_sharpa_train_job.py: model config, view
composition, image preprocessing and action sampling (also the training entry point)
- scripts/sharpa_eef/{action_codecs,canonical,eef_kinematics}.py and
scripts/lingbot_sharpa78/sharpa78_schema.py: giga codec, canonical wrist frame, FK/IK
- openwam/dataloader/{sharpa_eef,sharpa_hand}.py, registry entries, and video_io decoder
threading: the LeRobot readers behind SharpaGigaEEFDataset (used by the parity check)
- assets/robot_descriptions: YAM Ultra and SharpaWave URDFs (kinematics only, meshes are not
included) and the reference rollout used by the schema
- assets/giga_sharpa_train_job/prep_eef_full_v1: the norm stats and T5 task embeddings the
policy was trained with
- submodules third_party/giga-world-policy (model, sampler, ZMQ inference server) and
third_party/WebXR-Teleop (TacoSim, scene reset)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
benchmarks/sharpa_webxr runs the Sharpa EEF policy closed loop in the WebXR-Teleop MuJoCo sim, following the GWP-0.5 wam.cpp recipe: - observe, predict a 32-step chunk, execute it fully, re-observe - 10 flow steps, shift 5, no CFG - the training view composition, norm stats and prompts - wrist poses solved to arm joints with IK; hand joints pass through Validated against the recorded data: - renders reach PSNR 37.7 / 42.1 dB - replaying recorded setpoints reproduces the outcome to 1.2 mm - policy inputs are bit-identical to training Pieces: - scenes.py: materialize live-store scenes to data/webxr_scenes - env.py: TacoSim with teleop defaults, seeded prepare_scene reset, dataset cameras - policy_server.py: checkpoint behind the giga-models ZMQ server (--prep-dir) - rollout.py: closed-loop episodes against a policy server - postprocess.py: reach/grasp/move/fallen signals, rule verifier, judge keyframes - judge.py: Claude judge over keyframes - render_video.py: atomic video.mp4 writes - run_suite.sh: one checkpoint x 58 scenes x seeds; RUN selects the output dir - watch_run.sh: automatic eval of every checkpoint of a run - copies raw weights before rotation - evaluates EMA and raw on their own GPUs - renders videos from one shared low-priority queue - runs final seeds and best_val once training exits - options: VIDEO, VIDEO_SEEDS, VIDEO_P, PORT_BASE, PREP_DIR - aggregate.py: rates with Wilson CIs - build_review_page.py: side-by-side video page - check_policy_parity.py: offline input/loss/chunk-error gate - make_gt_rollouts.py: replay demos as reference rollouts - tasks.json: 58 scenes with prompts and step limits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
notebooks/_build_gwp_sim_eval_nb.py builds and executes notebooks/gwp_sim_eval.ipynb. It covers: - stage rates per checkpoint, with 95% CIs - all-three rate vs training step for EMA and raw, with the v1 range as reference - where rollouts stop - gpt_fleet_depth vs simteleop-only scenes - a per-scene heatmap - IK, fallen objects and episode length - keyframe grids and embedded videos for the best and worst scenes The executed copy holds gwp05_full_v1 (steps 3250-5000) and gwp05_full_v2 step 5000. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eneric runner The env no longer knows about any model. It takes one action per 50 ms control step and returns the observations it was configured with: - action_space "joint" (56 joint targets) or "eef" (wrist pose per arm plus hand joints; the env runs the IK, seeded from the previous solve) - ObsConfig selects cameras with their resolution, and state keys (joints, eef, object_poses, time) - step(action, observe=...) renders only when the caller asks for an observation Everything model-specific moves to policies/<name>.py: - what to observe, the input encoding and the model call - the execution strategy: act(obs) returns the actions to run before the next observation (a full chunk, its first k steps, or one step) rollout.py is now a generic loop over any registered adapter (--policy, --policy-kw). policies/gwp05.py keeps the wam.cpp GWP-0.5 recipe and reproduces the previous rollouts bit for bit: on v2 step 5000 EMA, scene 00178 seed 3, qpos, IK results and stage signals are identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
score.py <folder> judges each rollout under the folder with an OpenAI model (default gpt-6-luna): - input: the 16 keyframes, ego and chest views, with the task prompt - rubric and JSON schema: the same as judge.py - output: reach / grasp / move / complete / success Each verdict is cached as <rollout>/judge_<model>.json. The script writes these into <folder>: - scores_<model>.json: overall and per-task success rate (with Wilson CI), the GWP-0.5 graded score (0.25 per stage), judge and sim stage rates with their agreement, and per-seed rows - scores_<model>.tsv: one line per task The key comes from OPENAI_API_KEY or ~/.config/openai/key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scenes.py wrote manifest file names under the scene folder without checking them, and guarded zip members (and the sha256 checks) with assert, which python -O strips. Every destination now goes through _within(), which resolves the path and raises ValueError if it escapes the scene directory (absolute names, .., symlinks). Hash mismatches raise as well. Addresses review comments on benchmarks/sharpa_webxr/scenes.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_suite.sh: - each xargs worker now exits 1 if its rollout or postprocess failed - the suite prints FAILED and exits 1 instead of ALL_DONE watch_run.sh: - DONE_<seeds> is written only when the suite exited 0 and every scene x seed has signals.json - otherwise the tag is retried on the next pass, up to MAX_ATTEMPTS (default 2) - after that it is marked FAILED_<seeds> and skipped, so a broken checkpoint cannot stall the watcher; delete the marker to retry Tested with a stub interpreter: one failing scene gives suite exit 1 and FAILED; all passing gives exit 0 and ALL_DONE. Addresses review comments on run_suite.sh and watch_run.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- add sharpa_hand_stats_computation.py and its sharpa_hand.yaml config (placeholder dataset_dir) - the missing-stats error now gives the full command that writes meta/normalization_stats.npy in the schema the reader checks Addresses the review comment on openwam/dataloader/sharpa_hand.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Episode limit: - limits.limit(scene) = min(ceil(1.5 x p90 demo length), 20 s) - the runner, signals, judge frames and videos all use it - the policy is causal, so older rollouts are re-scored by truncation without re-running Judge (score.py --rubric r2, now the default): - postprocess.py --frames-only writes 24 times x (head, chest) frames at 480x640 to judge_frames/ - RUBRIC_R2 needs visible lift or carry for grasp, and purposeful displacement toward the goal for move; touching, nudging and knocking count as neither - r1 is kept for comparison On the sandwich rollout (v2 step 5000 EMA seed 0), r1 gave grasp+move (graded 0.75); r2 gives grasp only (0.50). Success is false in both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, one runner Two designs had been written for the same layer: - client-side adapters in this PR (policies/gwp05.py, which decoded on the sim side) - per-model servers on the policies/common.py contract (in the working tree) They collided on policies/__init__.py and on rollout.py helpers. This keeps the server contract as the only model interface: - policies/common.py + policies/gwp_server.py: the server owns image layout, state encoding, sampler and decode. Its meta publishes execute_steps (GWP: 20 of 32, as upstream's 30 of 48) and obs_offsets. gwp_server gains --prep-dir. - rollout.py: one runner for every server, built on the model-agnostic env (action_space "eef", IK in the env). It reads the meta and observes only at the rows the next request needs, so servers that want past frames (LDA-1B's t-5) work. - removed: the client-side adapter registry and gwp05.py - watch_run.sh / run_suite.sh: SERVER selects the server script, EXECUTE_STEPS overrides meta Parity, bit for bit (qpos and IK identical, 271 steps): - runner + gwp_server.py --execute-steps 32 vs the old policy_server.py client-decode rollout (v2 step 5000 EMA, scene 00178 seed 3) - runner at execute_steps 20 vs rollout_v2.py (v2 step 15000 EMA, 00178 seed 0) The other model servers (cosmos3, genie, lda1b, motus, xwam) follow in a separate PR with their training-script and submodule dependencies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A background judge loop writes judge frames and runs score.py (rubric r2) on every tag with a new DONE marker, so one watcher now waits for checkpoints, evaluates, renders videos and rates. JUDGE, JUDGE_WEIGHTS, JUDGE_MODEL and JUDGE_WORKERS control it; it is skipped when no OpenAI key is configured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ame completeness - API.md: the policy-server contract (meta / inference request and response, canonical pose layout, control loop) and how to add a model. - policies/hold_server.py: no-model reference server and template. - check_server.py: conformance check for any server (meta fields, shapes, rotations, frame, absolute first-step pose, latency). - task_breakdown.py: per-category and per-task judge success (k/n) across score files. - postprocess.py / score.py: judge_frames now writes COMPLETE last and drops cached verdicts when it re-renders; score.py refuses frames without it. Before, a render killed mid-write left a partial judge_frames dir that postprocess skipped forever, and score.py judged only the frames it found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Model-agnostic version of watch_run.sh (which stays as the GWP-specific watcher). The model side provides only its policy server script (--server) and a command that lists complete checkpoints as "<global_step> <ckpt> [extra server args]" (--list-ckpts). Same eval queue, retries, DONE/FAILED markers, video renderer, judge pass and final seed top-up as watch_run.sh, plus --done-file / --once, --stride, --only-steps, --sync-cmd and --stop-file (checked between every evaluation, so a capped container can be restarted during a backlog). API.md documents it with checkpoint-listing examples; hold_server.py accepts --ckpt so watchers can start it like any server. Tested with hold_server: a server that fails to start is retried then marked FAILED, the next checkpoint runs 58/58 rollouts to DONE + summary, WATCH_ALL_DONE, and a rerun skips both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d teleop episodes Adds a deterministic grader for the 58 Sharpa WebXR scenes. Each scene gets an ordered stage chain (task_specs.json, built by _build_task_specs.py) of simulator state checks: reach, grasp, hold_lifted, place_in, move_out, joint_goal, relative_pose, orientation, oscillation, free_flight, tilt_over, contact_events, sweep. 44 scenes are decided entirely from state; the 14 pour, tool-use and unsimulated-target scenes verify the prefix in sim and end in a judge stage, with a judge_context summary of the sim facts. verify.py replays trajectory.npz through the MuJoCo scene and writes verifier.json per rollout with any-time and end-state results for every stage. pull_episodes.py fetches the human-labelled teleop episodes (success / recover / failure) into the rollout format; calibrate.py scores the verifier against those labels (sim-only scenes: recall 0.959, specificity 0.849, n=2183); compare_judge.py compares it with the gpt-6-luna judge on policy rollouts (agreement 0.806 on 870 rollouts; judge over-accepts articulated and grasp-only scenes, under-accepts placements). score.py is unchanged; integration follows once the specs are reviewed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.