Skip to content

Commit f03e8ae

Browse files
committed
docs(mcp): the export and record options, on the site and in the skill
Part of #1180
1 parent f550c2b commit f03e8ae

19 files changed

Lines changed: 129 additions & 87 deletions

File tree

‎skills/openscreen-demo/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ Write a storyboard and show it to the user in a few lines, then carry on unless
6060

6161
## 4. Record
6262

63-
1. `startRecording` with `window`. It returns once capture has started. Connected to the server in the app, the take goes through OpenScreen's recording bar: the editor closes, the bar counts down, and `stopRecording` opens the take in the editor.
63+
1. `startRecording` with `window`. It returns once capture has started. Connected to the server in the app, the take goes through OpenScreen's recording bar: the editor closes, the bar counts down, and `stopRecording` opens the take in the editor. Pass `camera` only when the user wants to appear on screen: the webcam then shows as a picture-in-picture.
6464
2. Play the storyboard. Move deliberately, and pause about one second after each result appears.
6565
3. `stopRecording` with a `title`. Keep what it returns: `projectId`, `checkpointId`, `videoPath`, `recordingStartedAt`.
6666

@@ -90,7 +90,7 @@ Keep the rest as it is: the background, padding, shadow and cursor style are cho
9090

9191
## 7. Export and review
9292

93-
1. `exportVideo` with the `projectId` and an absolute path: resolve, say, `demo/demo.mp4` against your working directory first, since the server in the app refuses a relative path. For a pull request video, pass `quality: "medium"` (720p) to keep the file small.
93+
1. `exportVideo` with the `projectId` and an absolute path: resolve, say, `demo/demo.mp4` against your working directory first, since the server in the app refuses a relative path. For a pull request video, pass `quality: "medium"` (720p) to keep the file small. For a GIF (a README, say), end the path in `.gif` and pass `gifSize: "small"` (480p).
9494
2. `getFrames` with the `projectId`, at the ruler seconds of your beats.
9595
3. Check every frame:
9696
- Is each callout readable and clear of the action?

‎technical-documentation/architecture/mcp-server.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -125,15 +125,16 @@ What differs between the two servers is who records, handed in as a `TakeRecorde
125125
| | In-app server | `openscreen mcp` |
126126
|---|---|---|
127127
| Recorder | `createAppRecorder` (app-recorder.ts) | `runnerRecorder` (media-tools.ts) |
128-
| `startRecording` | Points the recording bar at the request, as the editor's Record mode does before it hands over: the source, picked from the list `listSources` shows (`pickRecordingSource`, as `openscreen record`), and the microphone, system audio and editable cursor in the recording prefs. Then hands over as the editor's Record button does (`start-new-recording`): the editor closes, the HUD starts the take with its countdown and its camera setting. Returns once `set-recording-state` says it records. Refused while the editor has unsaved changes, since the editor closes. Where the system's picker owns the choice (the Linux portal, Apple's picker on macOS 15.2+), the request picks nothing: that picker asks the user. | A hidden `cli-record` runner of its own process, as `openscreen record`. |
129-
| `stopRecording` | Presses the HUD's Stop, as the tray does. The HUD saves the take and hands it to the editor (`set-current-recording-session`), which imports it as for any take: one clip, measured, and the zooms on its clicks when Record mode's *Auto-zoom after recording* is on. The recorder waits for the editor's document to hold the take with its real length, then for its revision to hold still for 1.5 s, and returns that project, open in the editor. A `title` is applied through the editor. | Stops the runner and makes the project here (`createProjectFromVideo`): a fresh id no editor can hold, the zooms planned from the clicks. |
130-
| The user's settings | The source, microphone, system audio and cursor mode the user had come back once the take is captured, or as the app quits mid-take. A request that fails before the take changes none of them. | Untouched: the runner never saves them. |
128+
| `startRecording` | Points the recording bar at the request, as the editor's Record mode does before it hands over: the source, picked from the list `listSources` shows (`pickRecordingSource`, as `openscreen record`), and the microphone, camera, system audio and cursor mode in the recording prefs (`takePrefs`; a device named by the client is looked up in the same list, for its id beside its name). Then hands over as the editor's Record button does (`start-new-recording`): the editor closes, the HUD starts the take with its countdown. Returns once `set-recording-state` says it records. Refused while the editor has unsaved changes, since the editor closes. Where the system's picker owns the choice (the Linux portal, Apple's picker on macOS 15.2+), the request picks nothing: that picker asks the user. | A hidden `cli-record` runner of its own process, as `openscreen record`, with the same request (`CliRecordRequest`: source, microphone, camera, system audio, cursor mode). |
129+
| `stopRecording` | Presses the HUD's Stop, as the tray does. The HUD saves the take and hands it to the editor (`set-current-recording-session`), which imports it as for any take: one clip, measured, and the zooms on its clicks when Record mode's *Auto-zoom after recording* is on. The recorder waits for the editor's document to hold the take with its real length, then for its revision to hold still for 1.5 s, and returns that project, open in the editor. A `title` is applied through the editor. | Stops the runner and makes the project here (`createProjectFromVideo`): a fresh id no editor can hold, the zooms planned from the clicks, and the webcam, when one was recorded, linked to the take's asset as the editor links it (its offset and size). |
130+
| The user's settings | The source, microphone, camera, system audio and cursor mode the user had (`TAKE_PREF_KEYS`) come back once the take is captured, or as the app quits mid-take. A request that fails before the take changes none of them. | Untouched: the runner never saves them. |
131131

132132
Then both servers carry on alike:
133133

134134
- `stopRecording` creates the take's checkpoint through the runner, and records when the take started, by its video, for `setPointerTrack`.
135135
- `setPointerTrack` gives a script-driven take a pointer track and plans zooms on its clicks. `pointerTrackEdit` computes the new sidecar and the zooms from the project the runner read, writes nothing itself, and hands both to the runner as a `RunnerEdit`. The runner then applies the same refusals as any edit (another process's editor, no checkpoint, a project changed mid-call) before writing anything, saves the project, and only then writes the sidecar, keeping its previous content for the checkpoints. Every refusal comes back as the tool's own text result. A sidecar write that fails after the save is reported as a partial edit: the project was saved, the pointer track was not. A `RunnerEdit` is a closure and cannot be forwarded, so the call carries what the edit needs (`events`, and `takes`, each take's start time by video): the app's channel serves `setPointerTrack` with those args and runs `pointerTrackEdit` in its own runner. That is the channel's own registration, not the in-app server's tool: the takes of `openscreen mcp` are known to it alone.
136136
- `exportVideo` renders in a hidden `cli-export` runner, the same in both processes: in the app, `listSources` and `exportVideo` run the CLI's runners (`createRunnerHost` with `createRunnerWindow`), whose IPC the app registers on first use. The runner reads the project by id (`CliExportRequest.projectId`, the editor's own `aiEdition.get`), not by path: opening a file by path moves the main process's current project and recording-session slot, which in the app is the hand-off from the recorder to the editor.
137+
- `exportVideo` takes the Export dialog's options and choices: `quality` and `fps` (24, 30, 60) for an MP4, `gifFrameRate`, `gifSize` and `gifLoop` for a GIF, the dialog's defaults when left out. An option of the other format is refused rather than ignored. Hiding the desktop icons is not a `startRecording` option: the HUD has no such control, so it follows the user's Record mode setting on both servers. Nor are the CLI's `--auto-zoom` and `--audio`: zooms and a voiceover are edits of the project, and an export that added them would not be the project `getFrames` maps through.
137138
- `exportVideo` and `getFrames` read projects; they write none. Over HTTP their paths must be absolute.
138139

139140
## `openscreen mcp` and the running app

‎technical-documentation/testing/manual-e2e-checklist.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -574,12 +574,13 @@ Two servers. The **in-app** one is HTTP, in *AI settings* → *MCP server*, off
574574

575575
- [ ] In *AI settings*, confirm *Enable MCP server* and *Allow MCP clients to edit your projects, record and export* are off by default; enable the server and confirm the status reads *Running* with a *Port*, an *Access token* (Show, Copy, Regenerate) and the Claude Code and Codex snippets. Change the port, *Apply*, and confirm a client reconnects on it.
576576
- [ ] With edits not allowed, confirm an edit tool, `startRecording` and `exportVideo` are refused, naming *MCP edits*; allow them and confirm the edit lands in the open editor and undoes with `Ctrl/Cmd+Z`.
577-
- [ ] From a client of the **in-app** server, `startRecording` on a window: the editor closes and the recording bar records, its countdown first, with no second recorder process. Click in the window, `stopRecording`: the editor opens on the take, laid out and zoomed as after a take started by hand, and the returned `projectId` is that project. An edit to it lands live and undoes with `Ctrl/Cmd+Z`; `exportVideo` to an absolute path and `getFrames` on it work. Confirm the recording bar's microphone and system audio are back to what they were before the take, and that `startRecording` is refused while the editor has unsaved changes.
578-
- [ ] Confirm `"<exe>" help` lists `mcp`, register the stdio server in an MCP client, and confirm `listSources` lists displays, windows and microphones.
577+
- [ ] From a client of the **in-app** server, `startRecording` on a window: the editor closes and the recording bar records, its countdown first, with no second recorder process. Click in the window, `stopRecording`: the editor opens on the take, laid out and zoomed as after a take started by hand, and the returned `projectId` is that project. An edit to it lands live and undoes with `Ctrl/Cmd+Z`; `exportVideo` to an absolute path and `getFrames` on it work. Confirm the recording bar's microphone, camera, system audio and cursor mode are back to what they were before the take, and that `startRecording` is refused while the editor has unsaved changes.
578+
- [ ] Confirm `"<exe>" help` lists `mcp`, register the stdio server in an MCP client, and confirm `listSources` lists displays, windows, microphones and cameras.
579579
- [ ] Run `startRecording` on a window, click inside it a few times, `stopRecording`, and confirm the returned project is in the editor's project list with one clip and the zooms its `autoZooms` count announces. Clicks close in time (3 s or less) and on screen share one zoom, and a click in the first seconds or the stopping click gets none, so there can be fewer zooms than clicks. On macOS 15.2+, confirm Apple's picker never appears; on Linux, confirm the portal asks.
580580
- [ ] Edit that project while it is closed, `restoreCheckpoint` with the returned checkpoint, and confirm the project is back to the raw take. Confirm an edit to a closed project with no checkpoint is refused.
581581
- [ ] Open the project in the editor, with the in-app MCP server turned off, then send an edit to it through the **stdio** server, a process apart from the app (#1174): `listProjects` reports the project `open: true`, the edit shows in the editor at once, Ctrl+Z takes it back in one step, and a later change in the editor (its autosave) keeps it. `restoreCheckpoint` with the checkpoint from `stopRecording` also lands in the editor as one undo step. Quit OpenScreen mid-session: the next stdio edit to that project, now closed, asks for a checkpoint (the app's left with it), and after `createCheckpoint` lands in its file. Relaunch OpenScreen and open the project: the next stdio edit goes through the editor again, with no restart of the MCP client.
582582
- [ ] `exportVideo`, then `getFrames` at a few ruler seconds after a trim, and confirm the frames show the moments asked for; edit the project again and confirm `getFrames` warns that the export is stale.
583+
- [ ] On each server, `startRecording` with `camera: true` (then a camera's name from `listSources`) and `cursor: "system"`: the take opens with the webcam as a picture-in-picture, the system pointer is in the video and no zoom was planned. `exportVideo` at `fps: 30` gives a 30 fps MP4; to a `.gif` with `gifSize: "small"` and `gifLoop: false` a GIF 480 pixels high that plays once; `gifLoop` on an `.mp4` path is refused.
583584
- [ ] Drive a take from a script instead of the mouse, hand its action log to `setPointerTrack`, and confirm the export draws a cursor and zooms on the logged clicks.
584585

585586
## Native compositor preview and export — v1.8.0

‎website/docs/ai-editing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ Checkpoints are kept in OpenScreen's memory: they are lost when it quits or when
7676

7777
## Letting an agent make the whole video
7878

79-
An agent can make a whole video on its own, from the recording to the export. Besides the editing tools, both ways of adding OpenScreen to it give it `listSources`, `startRecording`, `stopRecording`, `setPointerTrack`, `exportVideo` and `getFrames`.
79+
An agent can make a whole video on its own, from the recording to the export. Besides the editing tools, both ways of adding OpenScreen to it give it `listSources`, `startRecording`, `stopRecording`, `setPointerTrack`, `exportVideo` and `getFrames`. A take can also record your webcam and the system's own cursor, as the recording bar can, and an export takes the Export dialog's options: quality and frame rate for an MP4; frame rate, size and looping for a GIF.
8080

8181
**Through the server above**, a take is recorded by OpenScreen itself, as with its Record button: the editor closes, the recording bar counts down and records, and the take opens in the editor as a new project, where you watch each edit land. It needs **MCP edits** on, and the recording bar gets your own settings back once the take ends.
8282

0 commit comments

Comments
 (0)