You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cd08cd8
Browse filesBrowse the repository at this point in the historyBrowse files
feat: add window enumeration, queries, and ScreenCaptureKit streaming
New Windows facade on Session (window list, current frame, topmost
window in z-order, owning-app activation) and push-based per-window
capture streams delivering zero-copy VideoFrame values with the
density/content-rect metadata needed to map frame pixels back to
desktop coordinates (toWindowPoint is the single implementation of
that formula; robot/mac/VideoFrame.h is the typed buffer bridge).
Streams guarantee serialized, ordered callbacks and a drain on stop():
when stop() returns no callback is executing or will start. macOS only
(ScreenCaptureKit); other platform builds report Unsupported through
the new canEnumerateWindows/canStreamWindows capabilities.
Copy file name to clipboardExpand all lines: README.md
+39-6Lines changed: 39 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,17 +20,18 @@ robot-cpp is built around the distinctions those libraries collapse:
20
20
- Move the cursor (absolute or smoothly interpolated), click, double-click, drag, and scroll, with buttons through X1/X2 and line- or pixel-unit scrolling.
21
21
- Press and release physical keys by position, build modifier chords, and type arbitrary Unicode text independent of the active keyboard layout.
22
22
- Enumerate every display with its own scale factor and both logical and physical bounds, capture any region or a whole monitor at native pixel resolution, sample individual pixels, and encode captures as PNG.
23
+
- Enumerate on-screen application windows, query their current bounds and z-order, activate their owning application, and stream a single window as zero-copy video frames carrying the density and content-rect metadata needed to map frame pixels back to desktop coordinates (macOS ScreenCaptureKit; reported as Unsupported elsewhere until those backends exist).
23
24
- Record global mouse and keyboard input and replay it with its original timing.
24
25
- Report per-environment capabilities and return typed errors instead of failing silently.
| Linux (Wayland) | uinput (opt-in) | keyboard + relative mouse only | no | no | no |
34
35
35
36
Wayland does not expose a protocol for an unprivileged client to inject input, warp the cursor, or capture the screen. Under a native Wayland session robot-cpp either runs through Xwayland (the X11 backend) or through the kernel-level uinput backend, with the limits above reported explicitly. See [Platform limitations](#platform-limitations) for the details.
36
37
@@ -207,6 +208,38 @@ if (auto color = screen.pixel({100, 200})) {
207
208
}
208
209
```
209
210
211
+
## Windows
212
+
213
+
Window enumeration and streaming live on `session->windows()`. Streaming is push-based: frames arrive on an internal capture thread, serialized and in order, as `VideoFrame` values that share (never copy) the platform pixel buffer. Each frame carries the scale and content-rect metadata needed to map its pixels back to window coordinates - `robot::toWindowPoint()` is the one implementation of that formula. On macOS, access the pixels through the typed bridge in `robot/mac/VideoFrame.h`.
214
+
215
+
```cpp
216
+
robot::Windows& windows = (*session)->windows();
217
+
218
+
// Pick a window (list() needs canEnumerateWindows):
219
+
auto list = windows.list();
220
+
const robot::Window& target = (*list)[0];
221
+
222
+
// Stream it. The callback must hand the frame off quickly and never call
`topmostWindowId()` reports the topmost normal-layer window in z-order (an ordering query - the honest primitive for "is my target frontmost"), and `activateApplicationFor(windowId)` brings a window's owning application forward; which of its windows ends up on top is the OS's decision.
242
+
210
243
## Recording and replay
211
244
212
245
A global event tap observes all mouse and keyboard activity and forwards it as normalized events, which a `Recorder` stamps with elapsed time. Recording is a privileged, platform-limited capability, so check `canRecordEvents` first. Key events are captured as physical keys, so a recording replays by position and is layout-independent.
0 commit comments