You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f03e8ae
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: skills/openscreen-demo/SKILL.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -60,7 +60,7 @@ Write a storyboard and show it to the user in a few lines, then carry on unless
60
60
61
61
## 4. Record
62
62
63
-
1.`startRecording` with `window`. It returns once capture has started. Connected to the server in the app, the take goes through OpenScreen's recording bar: the editor closes, the bar counts down, and `stopRecording` opens the take in the editor.
63
+
1.`startRecording` with `window`. It returns once capture has started. Connected to the server in the app, the take goes through OpenScreen's recording bar: the editor closes, the bar counts down, and `stopRecording` opens the take in the editor. Pass `camera` only when the user wants to appear on screen: the webcam then shows as a picture-in-picture.
64
64
2. Play the storyboard. Move deliberately, and pause about one second after each result appears.
65
65
3.`stopRecording` with a `title`. Keep what it returns: `projectId`, `checkpointId`, `videoPath`, `recordingStartedAt`.
66
66
@@ -90,7 +90,7 @@ Keep the rest as it is: the background, padding, shadow and cursor style are cho
90
90
91
91
## 7. Export and review
92
92
93
-
1.`exportVideo` with the `projectId` and an absolute path: resolve, say, `demo/demo.mp4` against your working directory first, since the server in the app refuses a relative path. For a pull request video, pass `quality: "medium"` (720p) to keep the file small.
93
+
1.`exportVideo` with the `projectId` and an absolute path: resolve, say, `demo/demo.mp4` against your working directory first, since the server in the app refuses a relative path. For a pull request video, pass `quality: "medium"` (720p) to keep the file small. For a GIF (a README, say), end the path in `.gif` and pass `gifSize: "small"` (480p).
94
94
2.`getFrames` with the `projectId`, at the ruler seconds of your beats.
95
95
3. Check every frame:
96
96
- Is each callout readable and clear of the action?
|`startRecording`| Points the recording bar at the request, as the editor's Record mode does before it hands over: the source, picked from the list `listSources` shows (`pickRecordingSource`, as `openscreen record`), and the microphone, system audio and editable cursor in the recording prefs. Then hands over as the editor's Record button does (`start-new-recording`): the editor closes, the HUD starts the take with its countdown and its camera setting. Returns once `set-recording-state` says it records. Refused while the editor has unsaved changes, since the editor closes. Where the system's picker owns the choice (the Linux portal, Apple's picker on macOS 15.2+), the request picks nothing: that picker asks the user. | A hidden `cli-record` runner of its own process, as `openscreen record`. |
129
-
|`stopRecording`| Presses the HUD's Stop, as the tray does. The HUD saves the take and hands it to the editor (`set-current-recording-session`), which imports it as for any take: one clip, measured, and the zooms on its clicks when Record mode's *Auto-zoom after recording* is on. The recorder waits for the editor's document to hold the take with its real length, then for its revision to hold still for 1.5 s, and returns that project, open in the editor. A `title` is applied through the editor. | Stops the runner and makes the project here (`createProjectFromVideo`): a fresh id no editor can hold, the zooms planned from the clicks. |
130
-
| The user's settings | The source, microphone, system audio and cursor mode the user had come back once the take is captured, or as the app quits mid-take. A request that fails before the take changes none of them. | Untouched: the runner never saves them. |
128
+
|`startRecording`| Points the recording bar at the request, as the editor's Record mode does before it hands over: the source, picked from the list `listSources` shows (`pickRecordingSource`, as `openscreen record`), and the microphone, camera, system audio and cursor mode in the recording prefs (`takePrefs`; a device named by the client is looked up in the same list, for its id beside its name). Then hands over as the editor's Record button does (`start-new-recording`): the editor closes, the HUD starts the take with its countdown. Returns once `set-recording-state` says it records. Refused while the editor has unsaved changes, since the editor closes. Where the system's picker owns the choice (the Linux portal, Apple's picker on macOS 15.2+), the request picks nothing: that picker asks the user. | A hidden `cli-record` runner of its own process, as `openscreen record`, with the same request (`CliRecordRequest`: source, microphone, camera, system audio, cursor mode). |
129
+
|`stopRecording`| Presses the HUD's Stop, as the tray does. The HUD saves the take and hands it to the editor (`set-current-recording-session`), which imports it as for any take: one clip, measured, and the zooms on its clicks when Record mode's *Auto-zoom after recording* is on. The recorder waits for the editor's document to hold the take with its real length, then for its revision to hold still for 1.5 s, and returns that project, open in the editor. A `title` is applied through the editor. | Stops the runner and makes the project here (`createProjectFromVideo`): a fresh id no editor can hold, the zooms planned from the clicks, and the webcam, when one was recorded, linked to the take's asset as the editor links it (its offset and size). |
130
+
| The user's settings | The source, microphone, camera, system audio and cursor mode the user had (`TAKE_PREF_KEYS`) come back once the take is captured, or as the app quits mid-take. A request that fails before the take changes none of them. | Untouched: the runner never saves them. |
131
131
132
132
Then both servers carry on alike:
133
133
134
134
-`stopRecording` creates the take's checkpoint through the runner, and records when the take started, by its video, for `setPointerTrack`.
135
135
- `setPointerTrack` gives a script-driven take a pointer track and plans zooms on its clicks. `pointerTrackEdit` computes the new sidecar and the zooms from the project the runner read, writes nothing itself, and hands both to the runner as a `RunnerEdit`. The runner then applies the same refusals as any edit (another process's editor, no checkpoint, a project changed mid-call) before writing anything, saves the project, and only then writes the sidecar, keeping its previous content for the checkpoints. Every refusal comes back as the tool's own text result. A sidecar write that fails after the save is reported as a partial edit: the project was saved, the pointer track was not. A `RunnerEdit` is a closure and cannot be forwarded, so the call carries what the edit needs (`events`, and `takes`, each take's start time by video): the app's channel serves `setPointerTrack` with those args and runs `pointerTrackEdit` in its own runner. That is the channel's own registration, not the in-app server's tool: the takes of `openscreen mcp` are known to it alone.
136
136
-`exportVideo` renders in a hidden `cli-export` runner, the same in both processes: in the app, `listSources` and `exportVideo` run the CLI's runners (`createRunnerHost` with `createRunnerWindow`), whose IPC the app registers on first use. The runner reads the project by id (`CliExportRequest.projectId`, the editor's own `aiEdition.get`), not by path: opening a file by path moves the main process's current project and recording-session slot, which in the app is the hand-off from the recorder to the editor.
137
+
-`exportVideo` takes the Export dialog's options and choices: `quality` and `fps` (24, 30, 60) for an MP4, `gifFrameRate`, `gifSize` and `gifLoop` for a GIF, the dialog's defaults when left out. An option of the other format is refused rather than ignored. Hiding the desktop icons is not a `startRecording` option: the HUD has no such control, so it follows the user's Record mode setting on both servers. Nor are the CLI's `--auto-zoom` and `--audio`: zooms and a voiceover are edits of the project, and an export that added them would not be the project `getFrames` maps through.
137
138
-`exportVideo` and `getFrames` read projects; they write none. Over HTTP their paths must be absolute.
Copy file name to clipboardExpand all lines: technical-documentation/testing/manual-e2e-checklist.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -574,12 +574,13 @@ Two servers. The **in-app** one is HTTP, in *AI settings* → *MCP server*, off
574
574
575
575
-[ ] In *AI settings*, confirm *Enable MCP server* and *Allow MCP clients to edit your projects, record and export* are off by default; enable the server and confirm the status reads *Running* with a *Port*, an *Access token* (Show, Copy, Regenerate) and the Claude Code and Codex snippets. Change the port, *Apply*, and confirm a client reconnects on it.
576
576
-[ ] With edits not allowed, confirm an edit tool, `startRecording` and `exportVideo` are refused, naming *MCP edits*; allow them and confirm the edit lands in the open editor and undoes with `Ctrl/Cmd+Z`.
577
-
-[ ] From a client of the **in-app** server, `startRecording` on a window: the editor closes and the recording bar records, its countdown first, with no second recorder process. Click in the window, `stopRecording`: the editor opens on the take, laid out and zoomed as after a take started by hand, and the returned `projectId` is that project. An edit to it lands live and undoes with `Ctrl/Cmd+Z`; `exportVideo` to an absolute path and `getFrames` on it work. Confirm the recording bar's microphone and system audio are back to what they were before the take, and that `startRecording` is refused while the editor has unsaved changes.
578
-
-[ ] Confirm `"<exe>" help` lists `mcp`, register the stdio server in an MCP client, and confirm `listSources` lists displays, windowsand microphones.
577
+
-[ ] From a client of the **in-app** server, `startRecording` on a window: the editor closes and the recording bar records, its countdown first, with no second recorder process. Click in the window, `stopRecording`: the editor opens on the take, laid out and zoomed as after a take started by hand, and the returned `projectId` is that project. An edit to it lands live and undoes with `Ctrl/Cmd+Z`; `exportVideo` to an absolute path and `getFrames` on it work. Confirm the recording bar's microphone, camera, system audio and cursor mode are back to what they were before the take, and that `startRecording` is refused while the editor has unsaved changes.
578
+
-[ ] Confirm `"<exe>" help` lists `mcp`, register the stdio server in an MCP client, and confirm `listSources` lists displays, windows, microphones and cameras.
579
579
-[ ] Run `startRecording` on a window, click inside it a few times, `stopRecording`, and confirm the returned project is in the editor's project list with one clip and the zooms its `autoZooms` count announces. Clicks close in time (3 s or less) and on screen share one zoom, and a click in the first seconds or the stopping click gets none, so there can be fewer zooms than clicks. On macOS 15.2+, confirm Apple's picker never appears; on Linux, confirm the portal asks.
580
580
-[ ] Edit that project while it is closed, `restoreCheckpoint` with the returned checkpoint, and confirm the project is back to the raw take. Confirm an edit to a closed project with no checkpoint is refused.
581
581
-[ ] Open the project in the editor, with the in-app MCP server turned off, then send an edit to it through the **stdio** server, a process apart from the app (#1174): `listProjects` reports the project `open: true`, the edit shows in the editor at once, Ctrl+Z takes it back in one step, and a later change in the editor (its autosave) keeps it. `restoreCheckpoint` with the checkpoint from `stopRecording` also lands in the editor as one undo step. Quit OpenScreen mid-session: the next stdio edit to that project, now closed, asks for a checkpoint (the app's left with it), and after `createCheckpoint` lands in its file. Relaunch OpenScreen and open the project: the next stdio edit goes through the editor again, with no restart of the MCP client.
582
582
-[ ]`exportVideo`, then `getFrames` at a few ruler seconds after a trim, and confirm the frames show the moments asked for; edit the project again and confirm `getFrames` warns that the export is stale.
583
+
-[ ] On each server, `startRecording` with `camera: true` (then a camera's name from `listSources`) and `cursor: "system"`: the take opens with the webcam as a picture-in-picture, the system pointer is in the video and no zoom was planned. `exportVideo` at `fps: 30` gives a 30 fps MP4; to a `.gif` with `gifSize: "small"` and `gifLoop: false` a GIF 480 pixels high that plays once; `gifLoop` on an `.mp4` path is refused.
583
584
-[ ] Drive a take from a script instead of the mouse, hand its action log to `setPointerTrack`, and confirm the export draws a cursor and zooms on the logged clicks.
Copy file name to clipboardExpand all lines: website/docs/ai-editing.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -76,7 +76,7 @@ Checkpoints are kept in OpenScreen's memory: they are lost when it quits or when
76
76
77
77
## Letting an agent make the whole video
78
78
79
-
An agent can make a whole video on its own, from the recording to the export. Besides the editing tools, both ways of adding OpenScreen to it give it `listSources`, `startRecording`, `stopRecording`, `setPointerTrack`, `exportVideo` and `getFrames`.
79
+
An agent can make a whole video on its own, from the recording to the export. Besides the editing tools, both ways of adding OpenScreen to it give it `listSources`, `startRecording`, `stopRecording`, `setPointerTrack`, `exportVideo` and `getFrames`. A take can also record your webcam and the system's own cursor, as the recording bar can, and an export takes the Export dialog's options: quality and frame rate for an MP4; frame rate, size and looping for a GIF.
80
80
81
81
**Through the server above**, a take is recorded by OpenScreen itself, as with its Record button: the editor closes, the recording bar counts down and records, and the take opens in the editor as a new project, where you watch each edit land. It needs **MCP edits** on, and the recording bar gets your own settings back once the take ends.
0 commit comments