Skip to content

CLAUDE.md

Latest commit

 

History

History
39 lines (24 loc) · 3.42 KB

CLAUDE.md

File metadata and controls

39 lines (24 loc) · 3.42 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

Gesture Reels — a Manifest V3 Chrome/Edge extension for hands-free control of Instagram Reels and YouTube Shorts via webcam air gestures. All hand tracking runs locally (MediaPipe HandLandmarker, WASM + model bundled in vendor/); nothing is sent to a server. No build step, no git repo.

Commands

node test_gestures.mjs   # run the pure gesture-logic self-check

Load/reload in browser: chrome://extensions → Developer mode → Load unpacked (this folder). After editing, hit the reload icon on the extension card; content-script changes also need a page refresh on instagram.com/youtube.com.

There is no bundler, linter, or test framework — test_gestures.mjs is a plain Node script asserting against gestures.js. package.json only sets {"type":"module"} so the .mjs import resolves.

Architecture

Three isolated execution contexts, connected by postMessage / chrome.runtime messaging:

  1. content.js (injected into instagram.com / youtube.com) — owns page UI and drives the site. Adds the floating "👋 Hands-Free" toggle + a draggable/resizable camera iframe. Receives gesture actions and calls a per-site adapter to act on the page.
  2. camera.html + camera.js (extension-origin iframe, web_accessible_resources) — owns the webcam and the MediaPipe inference loop. Classifies each frame via gestures.js, posts {source:"gesture-reels", gesture} to the parent content script. Draws the fingertip ghost trail. Killed (camera + inference stop) the moment the iframe is removed.
  3. background.js (service worker) — toolbar icon click → sends gesture-reels-toggle message to the active tab's content script.

gestures.js is the one pure, DOM-free, MediaPipe-free module: classify(landmarks) → gesture name, and createEngine(onGesture) → stateful hold/repeat timers. This is what test_gestures.mjs exercises; keep it pure so it stays testable in Node.

Key boundaries when editing

  • Brittle site selectors live at the top of content.js in the youtube and instagram adapter objects. When either site changes its DOM and a gesture stops working, patch there — nowhere else. clickNear() scopes clicks to the active video's ancestor subtree so the wrong reel's button isn't hit; activeVideo() picks the most-visible <video>.
  • Gesture tuning (hold durations, which finger pose maps to which action) lives in gestures.js constants (POINT_HOLD_MS, etc.) and createEngine's holds map. Only point repeats while held; the rest fire once per hold.
  • vendor/ is the offline MediaPipe tasks-vision bundle (vision_bundle.mjs, wasm/, hand_landmarker.task). CSP allows 'wasm-unsafe-eval' for it; don't add remote script sources.

Manifest specifics

  • Content script matches instagram.com/* and youtube.com/*, but each adapter's matches() further gates the toggle to /reels and /shorts respectively (both are SPAs — content.js polls the URL every 500ms to show/hide the button).
  • rules.json (declarativeNetRequest) strips Instagram's permissions-policy response header so the extension iframe can use the camera; host_permissions is Instagram-only for this reason.
  • Camera permission is granted to the extension origin (the iframe), not the host site — so it survives across reels and both sites.