Marco

A Dora-first mobile manipulator: phone AR so an operator can see what the robot intends, stereo perception for named places and obstacles, and a small arm for pick-and-place. The household build is the development twin for an agricultural platform — weeding and harvesting in rows, with a human in the loop who can trust the plan.

Marco on the differential-drive trike

Hardware today: SO-101 arm on a custom base, Raspberry Pi 5, OAK-D Lite + wrist camera, GY-521 MPU-6050 on the chassis side (rigid mount; Pi header VCC=1 SDA=3 SCL=5 GND=9), and a KY-008 650 nm pointer co-mounted with the wrist camera (GPIO17 via a 2N3904 low-side switch; hold-to-fire only). Experimenting with locomotion. Moved from a LeKiwi base, to differential drive with tank tread, and now a “Big Wheel” trike. A quadruped is the likely field chassis.

This repo is a public subset of that stack: robots/so101 graphs and nodes, reusable Dora components in node-hub, optional gym_dora tooling. DimOS is not vendored here — install it from dimensionalOS/dimos. This tree ships the Marco adapter: HTTP skills, MCP server, and prompts that sit on top of Dora.

For the next developer

Start with robots/so101/docs/DIMOS_HANDOFF.md — a one-page intro plus the paste-in prompt. Then pick the doc for the job:

Job	Doc
What we will / will not claim	docs/DIMOS.md
Run DimOS (tunnel, skills, STOP)	robots/so101/docs/DIMOS.md
House wander / floor overlay	robots/so101/docs/WANDER.md
Phone AR	robots/so101/docs/AR_OPERATOR.md
Wrist laser (KY-008, hold-to-fire)	robots/so101/docs/LASER.md
IMU pins + yaw fuse (straight = wheels, spin = gyro)	robots/so101/docs/IMU.md
Locked wheel numbers	robots/so101/docs/DRIVE_CAL.md
Occupancy retry	HANDOFF_MAPPING.md
Farm bottles	HANDOFF_ROW_FOLLOW.md

Pi 5 is LeBot. GPU desktop is Hiphas. Dora on the robot; DimOS only over HTTP. Phone STOP wins. Never pkill -f "dora run" over SSH.

DimOS (the agent layer — use this for demos)

English goes into DimOS. Wheels stay on Dora. That split is the product.

What is live: type in DimOS humancli → a local Unsloth model calls named skills (look, creep, turn, approach_label, …) → Raspberry Pi GET /state / POST /action → Marco moves a little, then looks again. Look JPEGs for Gemma can include a floor distance overlay. MARCO_WANDER=1 is a house-explore prompt (no map). Phone AR STOP still wins. Wall-avoid stays on the Pi.

What this is not: DimOS lidar SLAM, Unitree command-center, or a copy of DimOS inside this GitHub repo. Their mapping videos assume a lidar dog. Marco is a ~33 cm indoor trike with stereo + YOLO. We use DimOS as agent OS (ReAct / MCP / skills). We do not claim house-scale mapping.

Clip status: look-and-nudge (sofa tag, then chair) was filmed with local Unsloth + humancli + a moving robot.

Piece	Where it runs
Dora graph, OAK-D, wheels, avoid, POST /action	Raspberry Pi (robot)
Unsloth OpenAI-compatible API	GPU desktop
DimOS + Marco MCP (marco_dimos)	WSL on that desktop
Operator chat	DimOS humancli (browser on localhost)
Operator veto	Phone AR over HTTPS (Tailscale Serve on the Pi)

Bring-up (operator, after power-on):

1. On the Pi: Dora graph with DRY_RUN=0 IDLE_STOP=0 WALL_AVOID_ENABLE=1 (robots/so101/scripts/run_tank_wall_avoid_local.sh).
2. On the GPU PC: SSH tunnel Pi :8787 → a local port; Unsloth serving; WSL wsl_restart_marco_agent.sh then wsl_humancli.sh web.
3. Chat at http://127.0.0.1:8000. Keep STOP on the phone.

Secrets stay off GitHub: copy robots/so101/configs/unsloth.env.example and dimos.env.example to gitignored *.env files. Never commit API keys or Unsloth tokens.

Full honesty notes: docs/DIMOS.md. Runbook: robots/so101/docs/DIMOS.md. Wander plan: robots/so101/docs/WANDER.md. Next-session paste: robots/so101/docs/DIMOS_HANDOFF.md.

AR first

Field robots fail the trust test when the operator cannot see what the machine is planning. Marco’s operator UI is a phone web AR overlay, currently locked to Marco's rear AprilTag: live wheel speeds, an instantaneous path, detection arrows (label, confidence, depth), plus Play / STOP, and camera tabs (AR / OAK-D / wrist / map). The Wrist tab shows a live 640×480 MJPEG feed and a HOLD FOR LASER button: the 5 mW pointer is on only while the operator presses it; release, STOP, or motion turns it off.

Phone AR: path, wheel speeds, and labeled detections

Operator chrome: tag lock, Play / STOP, and camera tabs

The browser never writes wheel velocities. Commands go through a typed POST /action arbiter with a lease and a software STOP. Live camera lock needs HTTPS (Tailscale Serve on the robot). Doc: [robots/so101/docs/AR_OPERATOR.md](robots/so101/docs/AR_OPERATOR.md). Local preview without a phone camera:

python3 robots/so101/scripts/serve_ar_local.py
# http://127.0.0.1:8787/?demo=1

Same idea in a field: stand behind the machine, see the planned row, the labeled weeds or crops, and cancel if the overlay is wrong.

Agriculture platform

The mission evolved from a household helper to an agricultural platform that could lower the cost of organic food. Weeding and harvesting from an inexpensive robotics stack is the long bet.

Indoor (Marco, now)	Field (where this is going)
Named places (sofa, bathroom door) via AprilTags	Named rows, beds, and bins — in-frame row follow first, not a stored occupancy map
Heading-balance wall avoid + object cone	Row following + plant/obstacle cones
Phone AR path + detection arrows	Operator trust layer on a weeder/harvester
SO-101 + 5 mW KY-008 hold-to-fire on the wrist camera (aiming stack, not a weeder)	Effector: mechanical, 4 W laser, steam, or pulse — low energy + effective kill. Field laser dwell is tens of seconds at 4 W, not boom speed. Prior art: docs/research/WEEDING_PRIOR_ART.md

Marco is the stack we can iterate daily. A quadruped (or other field chassis) would reuse the same perception → labeled world → planned motion → AR loop. DimOS stays the language layer; Dora (or the equivalent on that chassis) still owns real-time control.

ROS is retired — Dora is the dataflow

Bring-up is a Dora dataflow, not a ROS 2 workspace. Graphs wire cameras, cmd_vel, occupancy, and HTTP in one place. DimOS does not replace that graph: it only calls HTTP skills.

Perception

Shoulder OAK-D Lite (stereo depth + on-device YOLO) and a wrist color camera with a co-mounted KY-008 pointer. Indoor, the OAK stream feeds hallway wall-avoid, object-cone avoid, and AprilTag homing. Occupancy from stereo is parked (not a navigable map). The wrist JPEG is operator aiming, not DimOS look. Field weeding should look down (wrist or nadir cam): OWL-style green-on-brown / crop-mask, then a tiny plant detector — not this COCO YOLO.

OAK-D view with live detections

Room landmarks are printed tag36h11 tags (sofa 101, bathroom door 201). Homing uses where the tag sits in the image. Tag size in the image is only a range ruler when the tag is nearly face-on; off-axis it reads short. PnP range is not used. Shoulder OAK-D points straight forward. Occupancy inflates stereo hits 20 cm and does not stamp when the lock tag is more than 15° off center (docs/DRIVE_CAL.md).

What works on hardware (2026)

• Differential trike drive, tape-calibrated odom (see robots/so101/docs/DRIVE_CAL.md): radius 0.055029, skid track 0.24638, cruise 20 cm/s, accel 0.32
• Dora cmd_vel driver, dead-man, surface profiles
• Heading-balance wall avoid + YOLO object avoid
• Marker hallway: seek sofa tag → standoff → right arc into the hall → bathroom tag → left arc
• Phone AR + command arbiter over Tailscale HTTPS
• Wrist camera MJPEG (~5 fps) + KY-008 hold-to-fire (UI-only; STOP and motion force off) — robots/so101/docs/LASER.md
• DimOS look-and-nudge (local Unsloth + MCP skills → Pi /action)
• Tag-anchored occupancy Map tab — failed live 2026-08-22, IMU heading retry 2026-08-27 still not navigable, parked (robots/so101/docs/MAPPING_POSTMORTEM.md). GY-521 fused yaw is live (robots/so101/docs/IMU.md); that is not a map product.

Still open: arm STOW / LASER_READY poses, beam-to-pixel calibration, supervised targeting, close-range YOLO drop after approach_label, hallway language, named-place button go-to, field chassis, weeding effector. Next farm demo: bottle-row follow (HANDOFF_ROW_FOLLOW.md). Field perception: OWL green-on-brown / crop-mask + LiteWeed stop-and-align — not hallway occupancy. The KY-008 is a 5 mW Class 3R pointer, not a 4 W weeder.

Quick start (Pi 5)

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -r robots/so101/requirements.txt

Arm teleop:

dora run ./robots/so101/graphs/mono_teleop_real_so101.yml

Hallway + map (on the robot, after rsync): DRY_RUN=0 IDLE_STOP=0 ./robots/so101/scripts/run_tank_marker_hall_local.sh

DimOS on a GPU box: pip install -e robots/so101 (package marco_dimos) into a DimOS venv; see the DimOS runbook. API keys: copy robots/so101/configs/openrouter.env.example and the DimOS *.env.example files — never commit the filled copies.

Layout

Path	Role
robots/so101/	Robot package: graphs, nodes, configs, web-ar/
robots/so101/marco_dimos/	DimOS HTTP / MCP adapter (skills over Pi HTTP)
robots/so101/docs/AR_OPERATOR.md	Phone AR, HTTPS, /state, /action
robots/so101/docs/DIMOS.md	DimOS operator runbook
robots/so101/docs/WANDER.md	House explore / overlay / peek-and-chain plan
robots/so101/docs/DIMOS_HANDOFF.md	Paste-in for the next Cursor session
robots/so101/docs/DRIVE_CAL.md	Locked wheel + tag/stereo range cals
robots/so101/docs/IMU.md	GY-521 wiring, I2C, yaw fuse (not SLAM)
robots/so101/docs/LASER.md	KY-008 pins, GPIO, Wrist hold-to-fire, aiming next
README-Rowve.md	Demo / agriculture narrative and live notes
docs/research/	OWL + LiteWeed prior art (weeding architecture)
docs/DIMOS.md	DimOS honesty + why; runbook [robots/so101/docs/DIMOS.md](robots/so101/docs/DIMOS.md)
HANDOFF_MAPPING.md	Occupancy parked (IMU helped heading; sketch unusable)
HANDOFF_ROW_FOLLOW.md	Next demo: bottles as crop rows
node-hub/	Reusable Dora nodes
gym_dora/	Optional sim

Drive configs still use the tank_* names (two-wheel differential). Hardware is the trike.

Credits

• Dora
• Dora-LeRobot
• LeRobot
• OpenRouter
• DimOS — agent OS above Dora (not vendored; adapter in this repo)
• OpenWeedLocator — green-on-brown / green-on-green detect → actuate
• LiteWeed — stop-and-align 4 W laser weeder (~$500)

License

Apache License 2.0.