Skip to content

HANDOFF_ROWVE.md

Latest commit

 

History

History
210 lines (163 loc) · 15.3 KB

HANDOFF_ROWVE.md

File metadata and controls

210 lines (163 loc) · 15.3 KB
Raw

Handoff — Next developer prompt (Rowve / Marco)

Copy-paste everything under Prompt into a new chat to continue.

Mapping next session: use HANDOFF_MAPPING.md (stereo error + occupancy inflate; wheels locked).

Prompt

You are continuing the Marco-Dora / Rowve co-founder demo on the physical robot Marco. Marco is powered on and ready for this repo’s AR stack to be synced onto its Pi and for the remaining plan phases.

What the project is

Marco is a household differential-drive trike + SO-101 arm robot, Dora-first (not ROS-first), hostname LeBot @ 192.168.1.140. Repo: /home/jeff/Marco-Dora on the Orange Pi (orangepi5 @ 192.168.1.212). On Marco the live tree is /home/jeff/Marco-Dora/dora-lerobot. Same chassis as the old tank; treads replaced 2026-08-19 by two rear driven wheels + an unpowered caster.

Rowve goals

We are shipping a short hardware demo for a potential Rowve co-founder (farm weeder, similar tank + SO-101). Prove we can ship:

  1. Dora-first robot stack
  2. Phone/web AR so an operator sees live intent (“going to …”) and world labels
  3. Perception → energy-on-target (OAK-D arrows; laser stretch is stop-and-align, see docs/research/WEEDING_PRIOR_ART.md)

Must ship: standing behind Marco, phone AR locks to the rear nameplate and shows tread speeds + path; tap a named place and Marco heads that way down the hallway (wall-avoid on).
Secondary: OAK-D detection arrows in AR.
Stretch: laser stop-and-align (halt → aim → dwell); creep-while-firing is bonus only.

Read in this order:

  1. HANDOFF_ROWVE.md (this file)
  2. README-Rowve.md (thesis + progress)
  3. README.md (platform / trike recail)
  4. robots/so101/docs/AR_OPERATOR.md (HTTPS, actions, frames)
  5. robots/so101/docs/WALL_AVOID.md (do not redo heading-balance)
  6. docs/research/WEEDING_PRIOR_ART.md (OWL + LiteWeed — do not invent weed vision from COCO)
  7. Plan canvas: .cursor project canvases/marco-ar-interface-plan.canvas.tsx

Machines

Role Host Notes
Dev (Cursor) Orange Pi orangepi5 @ 192.168.1.212 /home/jeff/Marco-Dora
Robot Marco Pi 5 LeBot @ 192.168.1.140 ssh marco (key installed). On as of 2026-08-13.
Phone Android + Tailscale app Chrome needs HTTPS for camera
  • Pi code: /home/jeff/Marco-Dora/dora-lerobot — rsync this repo’s robots/so101 before any hardware test if Orange Pi edits are not yet on Marco
  • Dora venv on Pi: ~/Marco-Dora/dora-lerobot/.venv
  • Tank bus scripts: LeRobot venv ~/LeRobot/le-robot/.venv
  • Git remote: https://github.com/Landstanda/Marco-Dora (main). Phone AR + OAK-D YOLO + object avoid landed 2026-08-15. Do not force-push.

Hardware locked (do not reverse)

  • Drive bus: /dev/serial/by-id/usb-1a86_USB_Single_Serial_5AE6054870-if00
  • Motors ID 7 left, ID 8 right; left_sign=-1, right_sign=+1 (positive yaw = CCW / left)
  • Drivetrain (2026-08-19): trike — two large rear wheels on those servos + unpowered freely-pivoting front caster. Same Marco body / SO-101. tank_* is still the software namespace.
  • OAK-D Lite — one consumer only (the wall-avoid graph). Do not start a second DepthAI process.
  • Stop dora-so101 teleop for wheel-only tests (it waits on leader USB)
  • Leader 5V; follower/drive ~12V
  • Hard floor DEFAULT: LOCKED trike — robots/so101/docs/DRIVE_CAL.md / tank_drive.json: radius 0.055029, skid track 0.22612, one-wheel 0.26029, ICC 1.021 @ 12 cm / 1.050 @ 24 cm, cruise 0.20, accel 0.32, max_wheel_raw=2800. Banded-tread 0.052235 / 0.441318 and carpet 0.304899 are retired / tread-era — do not use on the trike.
  • --x 0.1875 on the distance script is m/s, not 187.5 cm.

Heading-balance — NAILED (do not redo)

Cue is depth just left vs right of image center, not sticky side strips. Gains in configs/wall_avoid.json: yaw_imbalance_kp=42, yaw_max_degps=40, front_clear_m=1.45, imbalance_deadband_m=0.10, engage_wall_m=1.20. Graph default cap is still 1.8 m; live object-avoid test used MAX_PATH_M=3.00. Named-place go-to should replace the fixed cap.

AR interface — where we are (plan phases)

Phone does camera + AprilTag WASM + Three.js. Pi serves static UI + telemetry + command arbiter. Do not import the old MindAR project.

Phase Status Notes
0 Foundations Done Rear tag36h11 id 532, black square 133.5 mm, robot_T_tag + tread anchors in configs/ar_marker.json
1 Telemetry Done GET /state marco.state.v1: meas+cmd wheel m/s, track_width_m, stale, deadman
2 Speed overlay Done on hardware Labels in cm/s at tread anchors; side mapping corrected for rear view
3 Instantaneous path Done on hardware Length/style tuned for demo; always-visible short stub at stop; projects from robot front
4 Safe controls Done on hardware POST /action marco.action.v1; STOP + 0.8 s lease; tank 0.6 s dead-man unchanged. goto currently rejected (named_places_not_enabled)
5 Named places Tags generated; go-to not built Sofa 101, bathroom door 201 A4 PDFs. No waypoint store / buttons yet. Graph default still 1.8 m; 3 m live-tested via MAX_PATH_M
6 Robot vision Done on hardware (YOLO + JPEG tabs, not WebRTC) Same OAK process: yolov6-nano + stereo + ~5 fps JPEG. Wrist JPEG via ffmpeg. OAK points straight forward (camera_yaw_deg=0). Keep a single OAK-D owner
7 World localization Landmarks only Room tags exist; map_pose is null until measured + surveyed

Graph (important): wall-avoid-controller/cmd_vel → robot-state-http/wall_cmd → arbiter → robot-state-http/cmd_vel → tank-driver. Default arbiter mode is wall_avoid with no lease so hallway runs still work without a phone. After the UI set_mode, lost heartbeats → idle.

UI: robots/so101/web-ar/ (vanilla JS, vendored Three r160 + AprilTag WASM). Preview without camera: python3 robots/so101/scripts/serve_ar_local.py then http://127.0.0.1:8787/?demo=1. Tests: test_tank_telemetry.py, test_ar_trajectory.py, test_command_arbiter.py, test_oakd_detect.py.

Frames (critical): robot +x forward, +y left, +z up. AprilTag convention is still (+X right, +Y down, +Z toward camera), but on Marco the rear tag is physically mounted in a horizontal plane. For AR intent path on this build, treat tag -Y as robot-forward in the plate plane and keep tag Z fixed near floor plane so the line does not point upward.

Session update — 2026-08-13 (late)

  • Tailscale installed and authenticated on LeBot; HTTPS AR served from Marco:
    • https://lebot.tail766b61.ts.net/
  • UI cleaned for filming:
    • Removed verbose camera/status line by default
    • Smaller bottom HUD
    • Speed HUD simplified to Left/Right measured cm/s
    • Tread floating labels are white text with black outline (no green boxes)
  • Path styling and behavior:
    • Medium blue, thicker, longer visible forecast
    • Keeps a 10 cm stub when stopped
    • Grows to about 65 cm by ~10 cm/s
    • Cache-busted script build currently used in field tests: ?v=20260813-1613
  • Wall-avoid live tuning changes:
    • Effective speed target: 10 cm/s cruise, 8 cm/s creep
    • max_wheel_raw raised to 1250 on hard-floor configs for this demo pass
    • Turn gating now starts only when nearer side depth < 65 cm
    • Stronger turn region when nearer side depth < 40 cm
    • OAK-D warmup/no-cue now holds velocity at zero
  • Known operational gotchas:
    • OAK-D can get stuck in X_LINK_BOOTED; kill stale DepthAI/Dora processes before relaunch
    • If you used UI set_mode and stop heartbeats, mode may fall to idle; restart graph or re-engage wall_avoid

Session update — 2026-08-15 (OAK-D YOLO + object avoid live)

Operator confirmed the live run worked great.

  • Same DepthAI process as wall-avoid: stereo depth_mm + on-device yolov6-nano (Hub 512×288; luxonis/yolov6-nano:coco-416x416 404s). Host filter of COCO 80; confidence_min 0.35. Allow-list includes chair (living-room tests) plus person, couch, backpack, …
  • Object avoid is a cone override on heading-balance: in-cone + inside object_engage_m (1.40) → creep + yaw away (AVOID_<LABEL>); inside object_stop_m (0.55) → vx=0. Walls still run when the cone is empty.
  • Live: DRY_RUN=0 WALL_AVOID_ENABLE=1 MAX_PATH_M=3.00 from /home/jeff/Marco-Dora/dora-lerobot. Graph: detect=on, dry_run=False, enabled=True. Path ~2.38 m then AVOID_PERSON / AVOID_BACKPACK — creeps and yaws instead of ramming.
  • Same day, banded treads: recail LOCKED (wheel_radius_m=0.052235, track_width_m=0.441318). Avoid rerun at 10 cm/s then +30% (vx_mps=0.13, vx_creep_mps=0.104) — both worked great; Rowve video captured. Speeds left at +30%.
  • Stop leftovers: ./robots/so101/scripts/stop_tank_wall_avoid.sh (the run script calls this first). After a take, the graph SIGINTs itself 60 s after motors go to zero (IDLE_STOP_S; detections do not keep it up). IDLE_STOP=0 to leave YOLO running. Do not pkill -f "dora run" over SSH.
  • AR detection arrows use the same horizontal-tag plane as the blue path (overlayBasePoint: robot +x → tag −Y). Client EMA (DET_EMA=0.28) + 800 ms hold; labels sit midway along the ray. Cache-bust: app.js?v=20260815-wrist (phone hard-refresh).
  • Sit-still viewing: DRY_RUN=1 WALL_AVOID_ENABLE=0 → SENSE_ONLY, vx=0, YOLO still on.
  • UI STOP or lease expiry after set_mode → arbiter idle (wheels zero) even while the controller still publishes AVOID_*. Default no-lease wall_avoid works without a phone; after the UI takes a lease, keep heartbeats or re-tap Play. STOP is software only.

Session update — 2026-08-15 (AR HUD tabs + arrow yaw)

Operator confirmed OAK-D and Wrist camera tabs look right.

  • HUD: graph name, tag lock, motion, Play (set_mode wall_avoid), STOP, tabs AR / OAK-D / Wrist. Camera auto-starts on HTTPS. No Enable-camera / Preview-overlays / Wall-avoid / Idle / L-R HUD speeds.
  • OAK-D tab: JPEG from the same DepthAI process (preview_jpeg 512×288 @ 5 fps) + YOLO boxes from /state. Wrist tab: nodes/wrist_preview.py ffmpeg MJPEG (OpenCV V4L wedges the Centerm UVC). Do not fps-filter the V4L source.
  • Shoulder OAK points straight forward (camera_yaw_deg=0 in configs/oakd_detect.json). spatial_mm_to_base(..., camera_yaw_deg) in lib/oakd_detect.py. AR and object-avoid share this frame.
  • Config/lib load at process start — rsync then restart the graph. Phone hard-refresh after.

HTTPS: Android Chrome will not open the camera on http://192.168.1.140. Serve on Marco: tailscale serve --bg 8787, open the https://…ts.net URL in Chrome. Phone Tailscale app should already be on the tailnet. An Orange Pi Docker Tailscale (orangepi5-ar) was attempted for practice only — do not treat it as the robot gateway.

Room print files (A4, 100%, do not fit-to-page):

  • Sofa id 101: robots/so101/assets/ar_marker/room_sofa_tag36h11_id101_A4.pdf
  • Bathroom door id 201: robots/so101/assets/ar_marker/room_bathroom_door_tag36h11_id201_A4.pdf
    Printed black square is 146.3 mm (configs/ar_room_markers.json, tag_size_verified). Mount sofa tag on sofa back facing the room; bathroom tag on hallway side of the door, ~camera height.

Session update — 2026-08-17 (marker hallway: sofa 101 → bath 201)

First live hallway graph: tank_marker_hall_local.yml (YOLO off; depth + host AprilTag + JPEG). Runner: DRY_RUN=0 IDLE_STOP=0 ./robots/so101/scripts/run_tank_marker_hall_local.sh. Same stop script as wall-avoid.

  • Printed tags 146.3 mm. Distance for standoff is how big the tag is in the image, not PnP (pnp_range_m lied ~0.47 m across the room and started the sofa arc immediately). Must see the tag far first, then hold inside standoff.
  • Live: sofa lock at ~1.16 m then right 45 cm arc; then bathroom 201. Operator: basically worked.
  • Homing uses image bearing: steer on image_bearing_deg (tag vs image center), not body-frame bearing_deg. If the tag is lost, hold heading 0.7 s then search the last-seen side, not always left. Initial no-tag hunt is still left.
  • AR tag arrows use image_bearing_deg + range (same as homing). YOLO object arrows use robot-frame detections. Cache-bust app.js?v=20260817-tag-ar.
  • Bathroom turn starts at 110 cm (bath_standoff_m: 1.10). Sofa still 120 cm.

HTTPS: https://lebot.tail766b61.ts.net (Tailscale Serve → :8787). Hard-refresh the phone after a graph restart.

Session update — 2026-08-19 (trike drivetrain LOCKED)

Tank treads removed. Two rear driven wheels + caster. LOCKED: wheel_radius_m=0.055029, track_width_m=0.219992, cruise_vx_mps=0.20, accel_mps2=0.32, max_wheel_raw=2800. Pass 8 tape 199.8 cm (2 mm shy). Next live session: sofa→bath AprilTag hallway — HANDOFF_MAPPING.md.

What to do next (this order)

Next chat: copy-paste HANDOFF_MAPPING.md. Occupancy must inflate stereo (~20 cm) and skip oblique tag views; then stereo-at-tag on the odom loop. Do not recail wheels.

Constraints

  • Browser never writes wheel velocities — only POST /action
  • STOP is software stop, not a certified E-stop
  • Do not mix visual wheel/tread overlay width (0.2535 m) with kinematic track_width_m
  • Do not clobber verified fields in ar_marker.json (tag_size_verified, robot_T_tag, overlays)
  • Wheel odom is a short-horizon prior; named-place AR should not trust dead reckoning alone
  • After editing on the Orange Pi, rsync to Marco again

Start here this session

  1. Confirm latest robots/so101 is on Marco (rsync if Orange Pi is ahead)
  2. Phone locks id 532 over Tailscale HTTPS; arrows should sit on the object (hard refresh). Shoulder OAK points forward (camera_yaw_deg=0).
  3. Then phase 5 go-to (named places)

Key paths:

robots/so101/web-ar/                 # operator UI
robots/so101/lib/command_arbiter.py
robots/so101/lib/tank_telemetry.py
robots/so101/lib/ar_trajectory.py
robots/so101/lib/oakd_detect.py          # RDF → base_link + camera_yaw_deg
robots/so101/nodes/robot_state_http.py   # GET /state /marker /places /preview; POST /action; mux cmd_vel
robots/so101/graphs/tank_wall_avoid_local.yml
robots/so101/configs/ar_marker.json
robots/so101/configs/ar_room_markers.json
robots/so101/configs/oakd_detect.json    # camera_yaw_deg (0°, OAK points forward)
robots/so101/docs/AR_OPERATOR.md

Quick commands

# Dev repo (Orange Pi)
cd /home/jeff/Marco-Dora
git status -sb

rsync -av --delete --exclude '__pycache__' --exclude '.venv' \
  /home/jeff/Marco-Dora/robots/so101/ \
  marco:/home/jeff/Marco-Dora/dora-lerobot/robots/so101/

# Robot
ssh marco
cd /home/jeff/Marco-Dora/dora-lerobot && source .venv/bin/activate
# sit still, YOLO on:
DRY_RUN=1 WALL_AVOID_ENABLE=0 ./robots/so101/scripts/run_tank_wall_avoid_local.sh
# live 3 m wall + object avoid:
DRY_RUN=0 WALL_AVOID_ENABLE=1 MAX_PATH_M=3.00 ./robots/so101/scripts/run_tank_wall_avoid_local.sh
curl -s http://127.0.0.1:8787/state | python3 -m json.tool
# after graph is healthy:
# tailscale serve --bg 8787

python robots/so101/scripts/test_wall_avoid.py --mode avoid --distance-m 1.80
python robots/so101/scripts/test_oakd_detect.py
python robots/so101/scripts/test_command_arbiter.py