@@ -60,32 +60,21 @@ about task semantics.
 
 ---
 
-## Two Research Paths
+## Unified Research Direction (2026-05)
 
-### Path B — The 4-Weekend PoC (ACTIVE, CURRENT FOCUS)
+Path A and Path B have been merged. See `EMERGENT_ROBOTICS_PLAN.md` for the full plan.
 
-**Goal**: Demonstrate the full loop — record demos, decompose into primitives, use them
-to improve LLM tool-call selection.
+**Three-layer architecture with learned interfaces:**
 
-- Object-relative primitives (move_to_object, not move_left 0.05)
-- Trivial demo parser (greedy axis segmentation — already built)
-- Imitation learning: demo → primitive sequence → fine-tune LLM tool calling
-- RAG retrieval: given task description, find similar demo sequences as context
+1. **Conditioned policy** (50-100Hz): `(state, latent_code) → delta`. Small MLP/GRU trained by behavioral cloning on segmented demos.
+2. **Action VQ-VAE** (self-supervised): trained on delta-sequence segments. Codebook (K≈8-12) IS the vocabulary — discovered, not designed.
+3. **LLM planner** (0.5-2Hz): emits sequences of latent codes. Names codes by observing scene-graph before/after of each code across demos. Reactive replan after each code execution.
 
-**Why this first**: It proves the thesis end-to-end without requiring research breakthroughs.
-If this works, even crudely, it validates the entire architecture.
+**RAG over discovered codes** replaces RAG over raw demos. Every successful run stores `(task, scene_features, code_sequence)`. Retrieval keyed on task embedding + scene graph (object positions, gripper state, starting EE pose) — not task string alone. Few-shot inject into LLM prompt. Cold-start by encoding the original 30 demos through the VQ-VAE.
 
-### Path A — Research Track (FUTURE)
+**Why this composition:** the VQ-VAE solves the abstraction mismatch — segmentation and vocabulary are co-discovered from the same self-supervised loss. The LLM stays in its strength zone (discrete reasoning over a small symbol set). The conditioned policy absorbs state-dependent execution detail without needing language. RAG compounds with use without retraining.
 
-**Goal**: Discover the right primitive vocabulary from data, bottom-up.
-
-- High-frequency axis-relative primitives (the current 20 tools)
-- Embed tool-call sequences using structured tokenization
-- Bottom-up vocabulary discovery: cluster sequences into higher-level skills
-- This is where the JEPA + differentiable MPC work eventually connects
-
-**Why later**: This requires solving the abstraction mismatch problem (see below).
-Path B proves the architecture; Path A optimizes it.
+**Why merged:** Path B's "demo → primitive sequence → fine-tune" assumed hand-designed primitives. Path A's "bottom-up vocabulary discovery" assumed bespoke clustering. The VQ-VAE does both jobs at once, with a proven technique (VQ-BeT, VQ-VAE for action discretization). No need to keep the paths separate.
 
 ---
 
@@ -132,48 +121,68 @@ we're researching — the planning/execution decomposition is. ArUco lets us sid
 vision uncertainty entirely for the PoC. The plan is to use a phone as camera via
 IP Webcam app, which avoids USB camera driver headaches on Linux.
 
-### 6. No IK — Data-Driven Joint-Space Lookup (CURRENT BLOCKER)
+### 6. Joint-Space Lookup Table — ABANDONED (resolved 2026-04-12)
 
-**What**: Replace the placo/PyBullet IK solver with a lookup table built from real
-teleoperation data. Teleoperate the arm to a grid of ~75 positions across the workspace,
-record (Cartesian_position, joint_angles) pairs. For arbitrary targets, interpolate between
-nearest recorded neighbors using KNN + inverse-distance weighting (optionally scipy RBF).
+**What was proposed**: Replace placo IK with a KNN lookup over a teleoperated calibration
+grid (~75 positions, ruler-measured xyz → joint angles).
 
-**Why**: The available IK solutions for SO-100/101 have inaccurate DH parameters — the
-arm moves to wrong positions. This is inherent to 3D-printed arms where the actual
-geometry doesn't match the URDF/DH model precisely. The lookup table bypasses this
-entirely: the mapping comes from the physical arm, not a kinematic model.
+**Why abandoned**: The root cause of "broken IK on hardware" turned out to be a software
+bug, not bad DH parameters. `joints_to_cartesian()` and `cartesian_to_joints()` were
+silently defaulting all input joints to 0° because LeRobot returns `"shoulder_pan.pos"`
+keys but `kinematics.py` looked up `"shoulder_pan"`. Fix: `_normalize_joint_dict()` in
+kinematics.py. Validated on hardware with 10cm XY square. placo IK is now accurate
+enough; the lookup table approach would be high effort for marginal gain.
 
-**Status**: This is the current blocker. The existing 20 MCP primitives (move_left, etc.)
-use placo IK via `kinematics.py`, which produces incorrect positions on real hardware.
-Until the lookup table replaces this, Cartesian-space primitives are unreliable, and
-segmenter output from teleoperation data maps to primitives that don't execute correctly.
+**Residual issue**: position-only IK changes wrist_flex during EE-space moves → small
+Z drift when moving in X. Tracked as a known limitation, not a blocker.
 
-**Control pipeline (target architecture)**:
-```
-MCP plan(skill="move_toward", target="cup")
-    → Resolve "cup" → Cartesian position from scene graph
-    → JointLookup.solve(target_xyz) → KDTree nearest neighbors
-                                    → inverse-distance interpolation
-                                    → target joint angles
-    → minimum_jerk_joint_trajectory(current, target, duration)
-    → TrajectoryExecutor: send joints at 50Hz via robot.send_action()
-```
+### 7. `move_to_delta` as the Canonical EE-Space Primitive (2026-04-19)
+
+**What**: One primitive — `move_to_delta(dx, dy, dz)` — handles all EE-space translation.
+Axis-aligned primitives (`move_left`, `move_up`, etc.) are thin aliases that call it with
+two zero components. Plans all sub-waypoints upfront from FK seed, chains IK seeds,
+uses convergence-based wait + active-hold the final pose against gravity droop.
+
+**Why**: Diagonal moves in a single IK call, no staircase artifacts in segmenter output,
+and the prior implementation's two bugs (mid-motion `get_observation()` reads + 200ms
+inner interpolation servos couldn't track) are fixed. Hardware-validated: 80-95% reach
+on horizontal/down moves, ~6mm gravity-sag floor.
+
+**Known hardware limit**: +Z up moves only reach 25-50% of commanded distance due to
+Feetech servo torque insufficient to lift the arm against gravity. Tracked in BACKLOG.
+
+### 8. Emergent Vocabulary via Action VQ-VAE (2026-05)
+
+**What**: Stop hand-designing the LLM's primitive vocabulary. Train a VQ-VAE on delta
+sequences from teleop demos. The discrete codebook IS the vocabulary. A conditioned
+policy executes any code from any state. The LLM names the codes by observing scene-graph
+outcomes and plans by sequencing them.
+
+**Why**: Hand-designed primitives create a ceiling on what the system can learn — the
+engineer's intuitions about action granularity may not match the robot's actual behavioral
+structure. VQ-VAE is well-understood in low-dim action spaces (VQ-BeT etc.). Self-supervised
+on cheap teleop data. Decouples reasoning (LLM) from structure discovery (VQ-VAE) from
+control (conditioned policy) — each can be improved independently.
+
+**What this replaces**: the previous plan of "fine-tune the LLM on (task, tool_call_sequence)
+pairs over hand-designed primitives." Fine-tuning becomes optional / Phase 7+ if it's even
+needed.
+
+### 9. RAG over Discovered Codes (2026-05)
+
+**What**: Retrieve past `(task, scene_features, code_sequence)` tuples by cosine similarity
+on task embedding + scene-graph features. Inject the top-K as few-shot in the LLM prompt.
 
-**Calibration grid spec**:
-- X: 0.15m–0.35m from base (5 points), Y: ±0.15m (5 points), Z: table to +0.15m (3 heights)
-- ~75 positions + 5-10 extra at key locations (home, grasp height)
-- Recording: teleoperate with leader arm, record follower joint angles
-- Position measurement: manual (ruler) initially, ArUco-based later
-- Safety: refuse targets more than ~5cm from any recorded point
+**Why this over fine-tuning**: code sequences are tiny (token IDs), self-improving (every
+successful run adds an entry), no retraining loop, and the reactive replan loop corrects
+when the retrieved sequence diverges from the current scene. Scene-aware retrieval is what
+matters — task-string-only retrieval can't tell "pick from far-left" from "pick from center."
 
-### 7. Minimum-Jerk Interpolation in Joint Space
+**Cold start**: bootstrap by encoding the original 30 demos through the trained VQ-VAE.
+Day one of Phase 4 already has 30 entries.
 
-**What**: Deterministic, closed-form trajectory generation. 20 lines of code.
-`x(t) = x₀ + (x_f - x₀) × [10(t/T)³ - 15(t/T)⁴ + 6(t/T)⁵]`, applied per-joint.
-**Why**: Produces smooth trajectories from coarse waypoints. No learned trajectory
-generation needed. Combined with the lookup table, this gives reliable, smooth motion
-from any current position to any target in the recorded workspace.
+**Reuses**: existing `decras/imitation/store.py` schema. A `Primitive` becomes
+`{code: int, name: str}` instead of `{tool: "move_to_delta", args: {...}}`.
 
 ---
 
 
@@ -91,11 +91,8 @@
 
 ### 6A.3 Test Zero
 
-- [ ] **Replay segmenter v2 output on hardware** — Take sticks_v2 episode 0, run through segmenter v2 (high density), replay the `move_to_delta` sequence via MCP tools on the real arm. Film it. Compare to the original teleop recording.
-  - *Done when*: Side-by-side comparison video. Honest assessment: does the arm do the same thing?
-  - *Time*: 30 min
-  - *Tag*: `hardware`
-  - **This is the critical gate.** If this fails, Path B needs rethinking.
+- [x] **Replay segmenter v2 output on hardware** — sticks_v3 ep0, tested at 19/21/31/49 primitives. Structure correct (approach → descent → grasp zone → transit → release → retreat). **PASSES.** Gravity sag causes drift in extended-arm sections — tracked separately.
+  - *Done 2026-05-17*: `scripts/replay_sequence.py` (new) reads starting joints from Parquet and replays via `LeRobotInterface` directly
 
 ---
 
@@ -129,33 +126,131 @@
 
 ---
 
-## Phase 6C — Demo Store & Retrieval (AFTER 6A + 6B)
+## Phase 6C — Demo Store (DONE, retriever/RAG superseded by Phase 6F)
 
-> Only worth building once Test Zero passes and the LLM loop exists.
+> Schema + writer are built. Retriever / RAG / fine-tuning items now live in Phase 6F
+> (RAG over discovered codes, not over raw `move_to_delta` sequences).
+> See `EMERGENT_ROBOTICS_PLAN.md` and ARCHITECTURE.md decisions #8 and #9.
 
 - [x] **Design demo store schema** — `decras/imitation/retrieval.py` — `Demo`, `Primitive`, `DemoMetadata` dataclasses.
+- [x] **Build demo store writer** — `decras.imitation.store` (save_demo/load_demo/list_demos/ingest_sequence), CLI `scripts/add_demo.py`, deterministic id, provenance fields.
+
+---
+
+## Phase 6D — Data Collection for Vocabulary Discovery (NEXT)
+
+> 30+ teleop demos with deliberate variation so the VQ-VAE can discover meaningful action
+> structure. Plan reference: Phase 1 in `EMERGENT_ROBOTICS_PLAN.md`.
+>
+> **Gate**: Test Zero (6A.3) must pass first. If delta replay fails on hardware, the
+> conditioned policy in 6E will fail for the same reason.
+
+### 6D.1 Recorder upgrades
+
+- [ ] **Gripper force in observation** — Plan wants `gripper.force`. Decide: read force from Feetech if supported, or drop force and use gripper position deltas as the open/close signal. Document.
+  - *Tag*: `research`, then `claude-code`
+  - *Time*: 30 min
+
+- [ ] **Wire camera frames into recorder** — Capture from `DECRAS_CAMERA` at 5Hz (every 6th control frame), store aligned with Parquet rows. Needed for later JEPA / scene graph annotation.
+  - *Tag*: `claude-code`
+  - *Time*: 30 min
+
+### 6D.2 Record 30 demos with variation
 
-- [x] **Build demo store writer** — Segmenter v2 output + task string → Demo JSON on disk. Store at `demos/` with deterministic id `<dataset>_ep<NNN>_<density>.json`. `decras.imitation.store` has `save_demo` / `load_demo` / `list_demos` / `ingest_sequence`. CLI: `uv run python -m scripts.add_demo <sequence.json> --task "..."`. Duplicates raise unless `--overwrite`. Provenance fields added at ingest: `created_at`, `source_sequence_path`, `segmenter_git_sha`.
+- [ ] **Pick 2-3 task types** — e.g. cup→plate, block→stack, stick→bin. 10 demos each. Document in `datasets/README.md`.
+  - *Tag*: `hardware`
+
+- [ ] **Vary starting positions deliberately** — Object positions, arm start poses, approach angles. At least 3 distinct starting regions per task. The plan is explicit: uniform starts kill conditioned-policy generalization.
+  - *Tag*: `hardware`
+  - *Time*: 30-45 min per batch of 10
+
+- [ ] **Coverage sanity check** — `scripts/visualize_trajectory.py` over the new dataset. EE paths should span the workspace.
+  - *Tag*: `claude-code`
+
+---
+
+## Phase 6E — Action VQ-VAE (after 6D)
+
+> ~300 LOC PyTorch. Plan reference: Phase 2.
+
+### 6E.1 Baseline segments
+
+- [ ] **Use segmenter v2 (medium) as segment source** — Already built. Per demo: ordered list of (delta_sequence, start_state, end_state) segments.
   - *Tag*: `claude-code`
 
-- [ ] **Build demo retriever** — TF-IDF or sentence-transformer cosine similarity on task descriptions. `retrieve("pick up the stick") → [demo_1, demo_3]`.
+- [ ] **Segment-distribution viz** — Histogram of segment lengths, directions, cause (gripper / direction change / speed dip). Informs choice of K.
   - *Tag*: `claude-code`
 
-- [ ] **RAG integration** — Inject retrieved demos into LLM prompt via `llm_controller/prompt.py`.
+### 6E.2 Train VQ-VAE
 
-- [ ] **Record more demos** — 5-10 episodes of different tasks (pick-place, push, stack).
+- [ ] **Build `decras/imitation/vqvae.py`** — 1D CNN or small GRU encoder, learnable codebook (K=8 first; try 12 and 16), decoder reconstructs delta sequence. Reconstruction + commitment losses.
+  - *Tag*: `claude-code`
+  - *Time*: 2-3h
+
+- [ ] **`scripts/train_vqvae.py`** — Read `demos/` segments, train, save checkpoint + codebook.
+  - *Tag*: `claude-code`
+
+### 6E.3 Validate vocabulary
+
+- [ ] **Codebook usage report** — Dead codes → reduce K. Wildly different members → increase K.
+- [ ] **Per-code member viz** — Overlay all delta trajectories mapping to code k. Should look similar in direction + speed profile.
+- [ ] **Encode all demos to code sequences** — Persist `segment_id → code` and per-demo `code_sequence` into the demo store.
+
+---
+
+## Phase 6F — Conditioned Policy + LLM Grounding + RAG-over-codes
+
+> ~200 LOC for the policy, rest is glue. Plan reference: Phases 3 + 4.
+
+### 6F.1 Conditioned policy
+
+- [ ] **Build `decras/imitation/policy.py`** — Small MLP/GRU. Input: current joints + gripper + one-hot code. Output: next delta + gripper action. BC training on (state_t, gripper_t, code) → (delta_t, gripper_action_t).
+  - *Tag*: `claude-code`
+
+- [ ] **`scripts/train_policy.py`**
+  - *Tag*: `claude-code`
+
+- [ ] **Held-out replay test** — Held-out demo, encoded to codes, executed by the policy from the recorded starting state. Does it match the demo?
+  - *Tag*: `hardware`
+
+- [ ] **Generalization test** — Same code from a different starting state. Sensible behavior?
   - *Tag*: `hardware`
 
-- [ ] **Export fine-tuning pairs** — `(task_description, tool_call_sequence)` JSONL for fine-tuning.
+### 6F.2 LLM grounding
+
+- [ ] **Collect before/after scene-graphs per code** — For each code, gather scene graphs before/after every segment that maps to it. Feed to LLM, save the assigned name in codebook metadata.
+  - *Tag*: `research`
+
+### 6F.3 MCP `execute_code(k)` tool
+
+- [ ] **Add `execute_code(code: int)` MCP tool** — Loads conditioned policy, runs at 50Hz from current state + code, returns post-execution scene graph.
+  - *Tag*: `claude-code`
+
+- [ ] **Keep `move_to_delta` registered** — Useful for debugging and fallback.
+
+### 6F.4 RAG over codes
+
+- [ ] **Extend `Primitive` dataclass** — Add code-mode variant `{code: int, name: str}` alongside the current tool-call variant.
   - *Tag*: `claude-code`
 
+- [ ] **Code-sequence retriever** — Cosine similarity over `(task_embedding, scene_features)` where scene_features = object positions + gripper state + starting EE pose. Top-K past code sequences.
+  - *Tag*: `claude-code`
+
+- [ ] **RAG integration** — Inject retrieved code sequences into `llm_controller/prompt.py`. Bootstrap by encoding all 30 Phase 6D demos.
+  - *Tag*: `claude-code`
+
+- [ ] **End-to-end demo** — Natural-language task → LLM plans code sequence (with RAG few-shot) → MCP executes via `execute_code` → reactive replan after each code. Film a full pick-and-place.
+  - *Tag*: `hardware`
+
 ---
 
-## Phase 7 — Memory & Context (FUTURE — don't start yet)
+## Phase 7 — Rotation, JEPA, Memory (FUTURE)
 
-- [ ] Design persistent memory format (what worked, what failed, object locations)
-- [ ] Implement task decomposition in LLM prompt
-- [ ] Build error recovery loop (grasp failed → re-observe → retry with adjustment)
+- [ ] Extend deltas to (dx, dy, dz, dθ); retrain VQ-VAE + policy with 4D action space
+- [ ] JEPA enters: encode (delta_sequence, latent_state_transition) pairs so vocabulary is grounded in what the world does, not just what the robot does
+- [ ] Persistent memory across sessions (what worked, what failed, object locations)
+- [ ] Error recovery loop (grasp failed → re-observe → retry with adjustment)
+- [ ] Optional: fine-tune LLM on accumulated (task, scene, code_sequence) data if RAG hits a ceiling
 
 ---