development-manual.md

Development Manual

简体中文

Built specifically for the Switch version of Tomodachi Life.

Updated: 2026-05-22 Status: active development

0. Local Verification Snapshot, 2026-05-22

This verification pass covered the desktop app, the web flasher, and ESP32 firmware compilation in the current repository:

• npm run ci:local:quick: passed, including desktop type-checking, 94 desktop tests, flasher type-checking, 8 flasher tests, and the flasher web build
• npm run build: passed, including root TypeScript compilation and static asset copying
• ~/.platformio/penv/bin/pio run -d firmware/esp32 -e esp32dev_wireless: passed
• ~/.platformio/penv/bin/pio run -d firmware/esp32 -e esp32dev_wireless_switch2: passed
• ~/.platformio/penv/bin/pio run -d firmware/esp32 -e esp32dev_wireless_switch_lite: passed
• npm run build --prefix site/flasher: passed and produced the web flasher release output
• npm run verify:pages --prefix site/flasher: passed

Notes:

• On this machine, python3 -m platformio is not available, but ~/.platformio/penv/bin/pio is available; firmware compilation was verified through the full pio path documented for repo-based usage
• These results only cover automated tests, builds, and firmware compilation; real flashing, Bluetooth pairing, timing tuning, and drawing still need device-by-device validation through the quick-start flow

Author notes

• Original source author: Xiaohongshu creator 惜羽拓麻镇
• The repository currently uses the GPL-3.0-or-later license
• If you publicly repost, mirror, or redistribute this project, it is recommended that you credit 惜羽拓麻镇
• If you publicly repost, mirror, or redistribute this project, it is also recommended that you include the original publication link

1. Current objective

The current objective is not to complete fully automatic, high-fidelity multicolor drawing for arbitrary images in one step. The current objective is to make this real-world usage chain stable first:

Packaged desktop app / repo-based workflow -> Firmware Flash -> Controller Test -> Timing Tune / Benchmark -> Script Studio -> serial ACK execution -> ESP32 Bluetooth controller output -> stable reproduction on the Switch drawing canvas

Current priorities:

1. stable Bluetooth connection and long-duration execution
2. stable mono, official-palette, and custom-multicolor drawing
3. keep calibrating color sources and offset assumptions
4. only then move on to more ambitious automation

2. Current runtime shape

The project currently has two real entry routes:

• Packaged desktop app: for macOS and Windows x64
• Repo-based workflow: for development, debugging, and protocol validation

In the current implementation:

• apps/desktop/src/electron/main.ts is responsible for the desktop shell, runtime path selection, local web-server startup, and main-window lifecycle
• package.json already contains scripts such as electron:dev, dist:mac, and dist:win:x64
• packaged builds do not run firmware directly from a read-only resource directory; they first copy firmware/esp32 into a writable location before handing it to PlatformIO
• the Windows package includes drivers/windows resources so the desktop UI can expose driver-helper entry points
• apps/desktop/src/web/recoverySessions.ts is responsible for recovery-session persistence, reload, cleanup, and state conversion
• recovery sessions are currently stored under the user's documents directory in FriendMaker/recovery-sessions
• apps/desktop/src/drawingTemplates.ts owns drawing-template definitions, and template masks / previews are loaded from static assets

So the real development shape is no longer just "CLI + webpage". It is:

Electron desktop app + embedded local web workspace + TypeScript development toolchain

3. Stable capabilities right now

As of the current version, these parts are already verified or basically usable:

• packaged desktop build scripts and a desktop entry already exist for macOS and Windows x64
• the web UI already covers Script Studio / Firmware Flash / Controller Test / Timing Tune / Benchmark
• the page can directly invoke PlatformIO to compile and flash firmware
• the page can directly run controller connection, Bluetooth reset, and button / D-pad / stick tests
• the page can directly tune inputDelay / buttonPressDuration and run loopback timing tests
• a 256x256 script-coordinate canvas
• the public UI currently exposes six square-pixel brush sizes: 1 / 3 / 7 / 13 / 19 / 27
• round pixel brushes are still reserved / guarded, and the main generation and execution path does not depend on them
• drawing starts from the center of the canvas after entering the drawing page
• after recovery, re-entering the drawing page is still modeled as starting again from the center
• A is used for drawing / confirming a stroke
• the D-pad is used for one-cell movement
• the page can import images, generate previews, inspect statistics, and execute commands
• the mono drawing path is already working
• the official-palette drawing path is already integrated through image-q
• the custom-multicolor path is already integrated as a formal feature and can write batches into the 9 custom color slots
• official colors are quantized into the 7 x 12 / 84-color base palette and then mapped into the game's right-side 9 palette slots
• the drawing-template system already supports categories, previews, masks, and command-time cropping
• automatic background removal, preview guides, and official-palette previews are already integrated
• the image import path currently supports PNG / JPG / WEBP / SVG
• execution logs, flash logs, and test logs are all observable in the page
• inputDelay currently behaves more like a stability knob, while buttonPressDuration behaves more like an input-strength knob
• recovery tasks are persisted after pause, stop, or abnormal exit; if the app exits while paused, that task is converted into a recoverable task on the next launch
• packaged desktop builds include firmware resources, app icons, and bundled Windows driver resources
• ESP32 already supports the base serial protocol
• ESP32 already supports the ACK execution path required by formal drawing
• ESP32 already supports test commands such as TAP <BUTTON> <COUNT>, HOLD <BUTTON> <MS>, and STICK <X> <Y> <MS>
• the web flasher can currently publish the two visible models, Switch 1 and Lite firmware and Switch 2, while hiding the old standard Switch firmware entry

4. Current modeling of the drawing page

The current target drawing page is modeled with these fixed rules:

• after entering the drawing page, the cursor is at the center of the main canvas
• there is currently no reliable reset semantic that can always move the main canvas cursor back to the top-left corner
• because of that, the recovery flow must also assume that the user re-enters the drawing page and starts again from the canvas center
• the current tool is the brush
• A is used for drawing
• the D-pad moves one cell per action
• the canvas is handled as 256x256 script coordinates

That means the current mainline for Script Studio is:

• fixed 256x256
• fixed center-start assumption
• three formal routes: mono, official-palette, and custom-multicolor
• fixed public-UI mainline around square pixel brushes; round pixel brushes are not a formal drawing entry yet

Important distinction:

• if the docs mention "reset to top-left" or "top-left start", that does not refer to the main canvas by default
• those descriptions only apply to internal navigation modeling in color lists, the base-color page, or the custom-color edit page

5. Multicolor split

There are currently two possible multicolor routes.

Route A: base colors

This is the current recommended mainline.

Known facts:

• the base colors are the system default palette
• the base color area is 7 rows x 12 columns
• there are 84 directly selectable base colors

Advantages of this route:

• the color set is limited
• it does not require precise tuning inside the custom-color edit page
• it is more suitable for open-loop control
• it is easier to stabilize for automated drawing

The long-term plan for this route should be:

1. record the complete 7 x 12 base color table
2. define a fixed row / col for each color
3. quantize input images into these 84 colors in the web UI
4. let ESP32 only open the color page and move to the correct cell

Currently implemented design:

• the web UI already includes official palette drawing
• official-palette drawing quantizes the input image into 84 base colors
• the script layer already has the BC <slot> <row> <col> command
• the firmware configures the 9 palette slots through slot -> base-color page row/col
• the current strategy no longer tries to "reset to top-left"; instead, it assumes all 9 slots begin from the game's default colors and then tracks each slot's current row / col
• the script starts with BC RESET to reset the firmware's internal slot-tracking state back to the default-slot assumption
• BC RESET only resets palette-slot tracking state; it does not mean the main canvas cursor is reset to top-left
• after moving to the target cell on the base-color page, pressing A returns directly to the canvas, so official-palette slot configuration no longer sends extra B / A / B

Current conclusion:

base colors 7x12 are the recommended mainline path for future multicolor drawing.

Route B: custom colors

This is the route already integrated formally as custom multicolor.

Known facts:

• the color page has two tabs: base colors and custom colors
• pressing Y on the drawing page opens the palette list
• the palette list has 9 palette slots
• there are other non-palette items above the list
• pressing Y again enters the palette edit page
• inside the edit page, R switches to the custom colors tab and L switches to the base colors tab
• B exits the palette page / edit page
• A selects the current palette slot
• inside the edit page, ZL / ZR control the hue bar at the bottom, while the large block above is a two-dimensional color area
• the direction semantics for M 1 0 / M -1 0 / M 0 -1 / M 0 1 are already confirmed

Why custom-color automation still needs more work:

• the displacement relationship between TAP DLEFT 20 and TAP DLEFT 40 is still unstable
• the number of taps and the actual cursor displacement are not strictly linear
• long holds and taps do not behave the same; holds can accelerate
• without visual feedback, there is no guarantee that #RRGGBB lands precisely on the intended location

Current conclusion:

custom color auto tuning is already connected as a formal part of custom multicolor, but both color precision and stability still need more work.

6. Confirmed facts about the custom-color page

Even though auto color editing is already connected, these confirmed facts still need to be kept because they are part of future optimization work.

6.1 Palette list

• Y opens the palette list
• repeatedly pressing Down can stably reach the bottom
• there are 9 palette slots between the bottom and the top

This means the stable selection strategy for the palette list should be:

1. open the list
2. move all the way down to the bottom
3. move back upward to the target slot

6.2 Direction semantics in the edit page

Confirmed:

• in the large color block above:
  • M 1 0 moves right
  • M -1 0 moves left
  • M 0 -1 moves up
  • M 0 1 moves down
• in the hue bar below:
  • ZL moves left
  • ZR moves right

6.3 Reset strategy in the edit page

Confirmed:

• after entering the edit page in the custom-color flow, pressing R first is required to land on the correct custom-color tab
• after entering the edit page in the official-color flow, pressing L first is required to land on the correct base-color tab
• diagonal movement is not stable
• it is more reasonable to reset vertically first, then horizontally, then reset the hue bar
• the current custom-color experiment branch now prefers testing the top-left start assumption
• one currently stable reset parameter set is: push up 1500ms, push left 3000ms, and hold ZL for 2500ms

Recommended reset order:

1. reset upward
2. reset leftward
3. hold ZL to the left edge of the hue bar

6.4 Range exploration results

Current observed behavior:

• the horizontal range in the large color block is obviously long
• the vertical range is about half the same order of magnitude as the horizontal one
• measuring the hue bar with HOLD time is unreliable because a long hold may accelerate
• based on current real-device testing, the custom-color edit page can be approximated as:
  • hue around 200 steps
  • saturation around 213 steps
  • brightness around 112 steps
• the hue forward direction under ZR is opposite to the usual increasing HSV hue direction, so conversion needs a reverse mapping
• after resetting to top-left, the bright end is closer to the starting point; lowering brightness requires moving downward, not upward

Current conclusion:

If custom-color automation is pushed further later, it must:

• prioritize TAP counts
• stop using raw HOLD duration as a direct color-coordinate conversion

7. Current protocol additions

In addition to the original drawing commands, the protocol now includes these test commands.

7.1 TAP

Format:

TAP <BUTTON> <COUNT>

Examples:

TAP DRIGHT 216
TAP DLEFT 110
TAP ZR 20

Meaning:

• tap a button repeatedly for a specified count
• this is a discrete step command
• it is suitable for calibrating step counts in the edit page or hue bar

7.2 HOLD

Format:

HOLD <BUTTON> <MS>

Examples:

HOLD ZL 10000
HOLD ZR 2000

Meaning:

• hold a button for a specified number of milliseconds
• suitable only for rough range observation
• not suitable as the final basis for precise color tuning

7.3 STICK

Format:

STICK <X> <Y> <MS>

Rules:

• X can only be -1 / 0 / 1
• Y can only be -1 / 0 / 1
• they cannot both be 0 0
• MS is the duration, from 1 to 60000

Direction convention:

• STICK 0 -1 <MS>: left stick upward
• STICK 0 1 <MS>: left stick downward
• STICK -1 0 <MS>: left stick left
• STICK 1 0 <MS>: left stick right

Examples:

STICK 0 -1 1500
STICK -1 0 3000
HOLD ZL 2500

Meaning:

• push the left stick toward a direction and keep holding for a duration
• suitable for directly testing simulated stick reset behavior in the custom-color edit page
• this is a raw input command with no position calibration

8. Current strategy decisions

The current recommendations should be fixed clearly as:

8.1 Mono

Continue using it as the mainline development and validation route.

8.2 Multicolor

Prioritize the base colors 7x12 route.

8.3 Custom colors

Already included as one of the formal automatic-drawing capabilities, but still needs more optimization in color precision and stability.

8.4 Recenter optimization

The code path keeps a time-saving recenter strategy, and it is covered by path-generation and /api/generate tests. The public UI currently keeps this entry hidden and off by default; the real-device mainline should still be validated as "no recenter, center-start" first.

9. Suggested next development order

1. keep improving Bluetooth connection stability and long-duration drawing stability
2. keep improving custom-multicolor color correction, color fidelity, and real-device appearance
3. keep optimizing drawing paths, command execution paths, and long-run efficiency
4. keep improving user experience, including logs, status cards, failure-recovery prompts, packaged desktop install flow, and internal validation guidance