Say what you need. It figures out the rest.
Argus is a voice-controlled SO-101 robotic arm that hands you what you actually need — not just what you literally say. Ask it for water, mention a loose screw, or just ask it a technical question, and it does the right thing. Two custom vision policies, one natural-language dispatcher, and zero laptops required at demo time: everything runs standalone on an Arduino Uno Q.
Built for the Arduino Physical AI Challenge India 2026.
- Understands need, not keywords — "I'm thirsty" and "hand me the bottle" both fetch the same object.
- Two trained pick-and-place policies — an Action Chunking Transformer (ACT) per task, trained from human teleoperation demonstrations, running fully on-device.
- Answers real questions — ask it something unrelated to fetching and it gives you a straight, spoken-friendly answer instead of a canned error.
- No laptop at demo time — the entire voice → intent → motion pipeline runs on the Arduino Uno Q itself. A phone browser is the only other device involved, and it only ever exchanges text.
- Concurrency-safe — interrupt it mid-task and it tells you to wait, instead of silently running two motions at once.
Full writeup: docs/ARGUS_technical_report.pdf · Slide deck: ARGUS_demo_deck.pptx — 271MB with embedded video, too large for this repo; link TODO
- Phone browser captures your voice and converts it to text entirely client-side (Web Speech API) — no audio ever leaves the phone.
- The transcript is POSTed over HTTPS to a small FastAPI server running on Terry, the project's Arduino Uno Q.
- Terry calls DeepSeek with a tightly constrained prompt to classify the request into one of a fixed set of intents — need-based, not keyword-matched — or to answer an open-ended question.
- On an object request, Terry runs the matching ACT policy directly, on-device, CPU-only, driving the SO-101 arm over its serial bus and reading its wrist camera.
- A short acknowledgment is spoken back immediately (client-side TTS) while the arm moves in the background.
Every design decision behind this — including two earlier approaches that didn't work and why — is documented in the full technical report.
.
├── arduino/ # Code that runs ON the Uno Q ("Terry")
│ ├── app.py # FastAPI server: intent routing + policy execution
│ ├── gestures.py # Hardcoded yes/no nod gestures (no training needed)
│ ├── static/index.html # Phone-facing voice UI (Web Speech API)
│ └── calibration/ # SO-101 servo calibration, as deployed
│
├── training/ # Laptop-side: data collection & policy training support
│ ├── gestures.py # Same gesture script, for local/dev testing
│ └── calibration/ # SO-101 servo calibration, laptop copy
│
├── docs/ # Documentation & submission material
│ ├── ARGUS_technical_report.pdf
│ ├── ARGUS_demo_deck.pptx
│ ├── demo_slides.html # Projector deck used in the demo video
│ ├── report_source/ # Report/diagram generation scripts
│ └── media/ # Raw demo footage (kept locally, not tracked)
│
├── datasets/ # Recorded teleoperation episodes (not tracked — see below)
├── outputs/ # Trained policy checkpoints (not tracked — see below)
└── Arduino_Challenge_Project_Report_Template.docx
Datasets and checkpoints are intentionally not tracked in this repo. Between them they run into multiple gigabytes, well past what a git repo should carry, and several individual files exceed GitHub's 100MB hard limit. See Reproducing the policies below.
| Component | Role |
|---|---|
| SO-101 (6-DOF) | Feetech STS3215 smart servos — absolute joint position over a serial bus, no external encoder needed |
| Arduino Uno Q ("Terry") | Qualcomm QRB2210 (quad-core Cortex-A53) + STM32U585, 3.6GB RAM — runs the whole voice/intent/inference stack, CPU-only |
| Wrist camera | USB, 640×480 @ 30fps, the policy's only visual input |
| A phone | Any modern Chrome-based browser — the voice I/O surface |
- LeRobot — data collection, training, and the SO-101 robot/motor interfaces
- ACT (Action Chunking Transformer) — ResNet-18 backbone + transformer encoder-decoder, ~51.6M params per policy
- PyTorch — CUDA on the training laptop, CPU-only wheels on Terry (aarch64)
- FastAPI + uvicorn — the on-device backend, served over a self-signed HTTPS certificate (required for the Web Speech API's secure-context rule)
- DeepSeek API — the only external network dependency; used purely for text-in/text-out intent classification and Q&A, never in the motion-control loop
This is what actually runs on Terry. Copy the arduino/ folder onto the board, drop your trained checkpoints under models/<task_name>/, and a .env file with deepseek_apikey="..." next to app.py (see .gitignore — never commit this).
python -m venv .venv
./.venv/bin/pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
./.venv/bin/pip install "lerobot[feetech]" fastapi uvicorn openai
openssl req -x509 -newkey rsa:2048 -nodes -keyout key.pem -out cert.pem -days 365 -subj "/CN=terry"
./.venv/bin/python app.pyThen open https://<terry-ip>:7000 on your phone, accept the self-signed cert warning once, and talk to it.
Data collection and training both go through LeRobot's own CLI — this repo doesn't wrap them in custom scripts:
lerobot-record --robot.type=so101_follower --teleop.type=so101_leader \
--dataset.repo_id=local/<task_name> --dataset.single_task="<task description>" \
--dataset.num_episodes=55 ...
lerobot-train --policy.type=act --dataset.repo_id=local/<task_name> \
--output_dir=outputs/<task_name> --steps=40000 --batch_size=8 ...Full hyperparameters, dataset statistics, and the distractor-object training methodology used for both tasks are in the technical report (§3–4).
Two earlier approaches were tried and abandoned before landing on this design — an open-loop hobby-servo arm that needed a vision-based encoder just to know its own position, and a position-only inverse-kinematics scheme that could reach a point in space but structurally couldn't aim a camera at one. Both are written up honestly in the technical report as part of the engineering record, not just the parts that worked.
MIT. Built on LeRobot (Hugging Face) and the SO-ARM100 hardware design. Submitted to the Arduino Physical AI Challenge India 2026.