Skip to content

Commit cd08cd8

Browse files
committed
feat: add window enumeration, queries, and ScreenCaptureKit streaming
New Windows facade on Session (window list, current frame, topmost window in z-order, owning-app activation) and push-based per-window capture streams delivering zero-copy VideoFrame values with the density/content-rect metadata needed to map frame pixels back to desktop coordinates (toWindowPoint is the single implementation of that formula; robot/mac/VideoFrame.h is the typed buffer bridge). Streams guarantee serialized, ordered callbacks and a drain on stop(): when stop() returns no callback is executing or will start. macOS only (ScreenCaptureKit); other platform builds report Unsupported through the new canEnumerateWindows/canStreamWindows capabilities.
1 parent a9700a1 commit cd08cd8

22 files changed

Lines changed: 1012 additions & 12 deletions

‎CMakeLists.txt‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -62,19 +62,28 @@ set(ROBOT_COMMON_SOURCES
6262
src/common/Image.cpp
6363
src/common/Recorder.cpp
6464
src/common/EventTap.cpp
65+
src/common/Windows.cpp
6566
)
6667

6768
# ── Platform selection: exactly one directory is compiled ─────────────────────
6869
# This is the single compile boundary. No consumer translation unit contains a
6970
# platform #ifdef; the platform is chosen here, once.
7071
if(APPLE)
72+
# The window backend talks to ScreenCaptureKit, which has no C API.
73+
enable_language(OBJCXX)
74+
set(CMAKE_OBJCXX_STANDARD 23)
75+
set(CMAKE_OBJCXX_STANDARD_REQUIRED ON)
7176
set(ROBOT_PLATFORM_SOURCES
7277
src/platform/macos/MacBackendFactory.cpp
7378
src/platform/macos/MacKeyMap.cpp
7479
src/platform/macos/MacKeyboardBackend.cpp
7580
src/platform/macos/MacMouseBackend.cpp
7681
src/platform/macos/MacScreenBackend.cpp
7782
src/platform/macos/MacEventTapBackend.cpp
83+
src/platform/macos/MacWindowBackend.mm
84+
)
85+
set_source_files_properties(src/platform/macos/MacWindowBackend.mm
86+
PROPERTIES COMPILE_OPTIONS "-fobjc-arc"
7887
)
7988
elseif(WIN32)
8089
set(ROBOT_PLATFORM_SOURCES
@@ -136,9 +145,19 @@ set_target_properties(robot PROPERTIES POSITION_INDEPENDENT_CODE ON)
136145
# ── Platform dependencies, scoped per target ──────────────────────────────────
137146
if(APPLE)
138147
find_library(APPLICATION_SERVICES ApplicationServices REQUIRED)
139-
# ApplicationServices types appear only in .cpp files (behind the backend
140-
# interfaces), so the dependency is PRIVATE. Carbon is intentionally NOT used.
141-
target_link_libraries(robot PRIVATE ${APPLICATION_SERVICES})
148+
find_library(SCREEN_CAPTURE_KIT ScreenCaptureKit REQUIRED)
149+
find_library(CORE_MEDIA CoreMedia REQUIRED)
150+
find_library(CORE_VIDEO CoreVideo REQUIRED)
151+
find_library(APP_KIT AppKit REQUIRED)
152+
# Platform framework types appear only in the platform .cpp/.mm files
153+
# (behind the backend interfaces), so the dependencies are PRIVATE. Carbon
154+
# is intentionally NOT used.
155+
target_link_libraries(robot PRIVATE
156+
${APPLICATION_SERVICES}
157+
${SCREEN_CAPTURE_KIT}
158+
${CORE_MEDIA}
159+
${CORE_VIDEO}
160+
${APP_KIT})
142161
elseif(WIN32)
143162
# user32/gdi32 for SendInput, hooks, GDI capture. Shcore and the DPI context
144163
# API are resolved at runtime (see WinDpi.cpp / WinScreenBackend.cpp), so they

‎README.md‎

Lines changed: 39 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -20,17 +20,18 @@ robot-cpp is built around the distinctions those libraries collapse:
2020
- Move the cursor (absolute or smoothly interpolated), click, double-click, drag, and scroll, with buttons through X1/X2 and line- or pixel-unit scrolling.
2121
- Press and release physical keys by position, build modifier chords, and type arbitrary Unicode text independent of the active keyboard layout.
2222
- Enumerate every display with its own scale factor and both logical and physical bounds, capture any region or a whole monitor at native pixel resolution, sample individual pixels, and encode captures as PNG.
23+
- Enumerate on-screen application windows, query their current bounds and z-order, activate their owning application, and stream a single window as zero-copy video frames carrying the density and content-rect metadata needed to map frame pixels back to desktop coordinates (macOS ScreenCaptureKit; reported as Unsupported elsewhere until those backends exist).
2324
- Record global mouse and keyboard input and replay it with its original timing.
2425
- Report per-environment capabilities and return typed errors instead of failing silently.
2526

2627
## Supported platforms
2728

28-
| Platform | Backend | Injection | Capture | Recording |
29-
| ----------------- | ----------------------------- | ------------------------------- | ------- | --------- |
30-
| macOS | Quartz (CoreGraphics) | yes (needs Accessibility) | yes (needs Screen Recording) | yes (needs Accessibility) |
31-
| Windows | SendInput + GDI | yes | yes | yes |
32-
| Linux (X11) | XTest + XRandR + XRecord | yes | yes | yes |
33-
| Linux (Wayland) | uinput (opt-in) | keyboard + relative mouse only | no | no |
29+
| Platform | Backend | Injection | Capture | Window streaming | Recording |
30+
| ----------------- | ----------------------------- | ------------------------------- | ------- | ---------------- | --------- |
31+
| macOS | Quartz + ScreenCaptureKit | yes (needs Accessibility) | yes (needs Screen Recording) | yes (needs Screen Recording) | yes (needs Accessibility) |
32+
| Windows | SendInput + GDI | yes | yes | no | yes |
33+
| Linux (X11) | XTest + XRandR + XRecord | yes | yes | no | yes |
34+
| Linux (Wayland) | uinput (opt-in) | keyboard + relative mouse only | no | no | no |
3435

3536
Wayland does not expose a protocol for an unprivileged client to inject input, warp the cursor, or capture the screen. Under a native Wayland session robot-cpp either runs through Xwayland (the X11 backend) or through the kernel-level uinput backend, with the limits above reported explicitly. See [Platform limitations](#platform-limitations) for the details.
3637

@@ -207,6 +208,38 @@ if (auto color = screen.pixel({100, 200})) {
207208
}
208209
```
209210
211+
## Windows
212+
213+
Window enumeration and streaming live on `session->windows()`. Streaming is push-based: frames arrive on an internal capture thread, serialized and in order, as `VideoFrame` values that share (never copy) the platform pixel buffer. Each frame carries the scale and content-rect metadata needed to map its pixels back to window coordinates - `robot::toWindowPoint()` is the one implementation of that formula. On macOS, access the pixels through the typed bridge in `robot/mac/VideoFrame.h`.
214+
215+
```cpp
216+
robot::Windows& windows = (*session)->windows();
217+
218+
// Pick a window (list() needs canEnumerateWindows):
219+
auto list = windows.list();
220+
const robot::Window& target = (*list)[0];
221+
222+
// Stream it. The callback must hand the frame off quickly and never call
223+
// back into robot-cpp.
224+
auto stream = windows.stream(
225+
target.id, {.maxFps = 60.0},
226+
[&](const robot::VideoFrame& frame) { mailbox.publish(frame); },
227+
[&](const robot::Error& error) { std::println("stream died: {}", error.message); });
228+
229+
// ... later. After stop() returns no callback is running or will start.
230+
stream->stop();
231+
232+
// Input coordinate math: re-read the window's current bounds, then map a
233+
// frame pixel to desktop coordinates.
234+
if (auto point = robot::toWindowPoint(frame.info(), pixelX, pixelY)) {
235+
auto bounds = windows.frame(target.id);
236+
robot::LogicalPoint global{bounds->origin.x + point->x,
237+
bounds->origin.y + point->y};
238+
}
239+
```
240+
241+
`topmostWindowId()` reports the topmost normal-layer window in z-order (an ordering query - the honest primitive for "is my target frontmost"), and `activateApplicationFor(windowId)` brings a window's owning application forward; which of its windows ends up on top is the OS's decision.
242+
210243
## Recording and replay
211244

212245
A global event tap observes all mouse and keyboard activity and forwards it as normalized events, which a `Recorder` stamps with elapsed time. Recording is a privileged, platform-limited capability, so check `canRecordEvents` first. Key events are captured as physical keys, so a recording replays by position and is layout-independent.

‎include/robot/Capabilities.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@ struct Capabilities {
3333

3434
bool canCaptureScreen = false;
3535
bool canEnumerateMonitors = false;
36+
bool canEnumerateWindows = false; // Window list, frame, and focus queries.
37+
bool canStreamWindows = false; // Push-based per-window capture streams.
3638
bool canRecordEvents = false; // Global input tap / hook.
3739

3840
bool requiresAccessibilityPermission = false;

‎include/robot/Error.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ enum class ErrorCode : std::uint16_t {
2525
InvalidArgument,
2626
// A referenced monitor id does not exist.
2727
MonitorNotFound,
28+
// A referenced window id no longer exists on screen.
29+
WindowNotFound,
2830
// A character or key cannot be represented by the active keyboard layout or
2931
// injection method.
3032
UnmappableInput,
@@ -44,6 +46,7 @@ constexpr std::string_view toString(const ErrorCode code) {
4446
case ErrorCode::BackendUnavailable: return "BackendUnavailable";
4547
case ErrorCode::InvalidArgument: return "InvalidArgument";
4648
case ErrorCode::MonitorNotFound: return "MonitorNotFound";
49+
case ErrorCode::WindowNotFound: return "WindowNotFound";
4750
case ErrorCode::UnmappableInput: return "UnmappableInput";
4851
case ErrorCode::CaptureFailed: return "CaptureFailed";
4952
case ErrorCode::EncodeFailed: return "EncodeFailed";
@@ -79,6 +82,10 @@ struct Error {
7982
static Error monitorNotFound(const std::uint32_t id) {
8083
return {ErrorCode::MonitorNotFound, std::format("Monitor {} not found", id)};
8184
}
85+
static Error windowNotFound(const std::uint32_t id) {
86+
return {ErrorCode::WindowNotFound,
87+
std::format("Window {} no longer exists on screen", id)};
88+
}
8289
static Error unmappableInput(std::string_view detail) {
8390
return {ErrorCode::UnmappableInput,
8491
std::format("Cannot map input: {}", detail)};

‎include/robot/Robot.h‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,4 +22,8 @@
2222
#include "robot/Recorder.h"
2323
#include "robot/Scroll.h"
2424
#include "robot/Screen.h"
25-
#include "robot/Session.h"
25+
#include "robot/Session.h"
26+
#include "robot/VideoFrame.h"
27+
#include "robot/Window.h"
28+
#include "robot/Windows.h"
29+
#include "robot/WindowStream.h"

‎include/robot/Session.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
#include "robot/Monitor.h"
1313
#include "robot/Mouse.h"
1414
#include "robot/Screen.h"
15+
#include "robot/Windows.h"
1516

1617
namespace robot {
1718
namespace backend {
@@ -73,6 +74,10 @@ class Session {
7374
// implementation (see EventTap).
7475
[[nodiscard]] EventTap& eventTap() { return eventTap_; }
7576

77+
// Always present; calls report Unsupported if this platform build has no
78+
// window enumeration/capture implementation (see Windows).
79+
[[nodiscard]] Windows& windows() { return windows_; }
80+
7681
[[nodiscard]] const Capabilities& capabilities() const {
7782
return capabilities_;
7883
}
@@ -96,6 +101,7 @@ class Session {
96101
Mouse mouse_;
97102
Screen screen_;
98103
EventTap eventTap_;
104+
Windows windows_;
99105
};
100106

101107
} // namespace robot

‎include/robot/VideoFrame.h‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
#pragma once
2+
3+
#include <chrono>
4+
#include <cstdint>
5+
#include <memory>
6+
#include <optional>
7+
#include <utility>
8+
9+
#include "robot/Geometry.h"
10+
11+
namespace robot {
12+
13+
// Per-frame metadata for a streamed window frame. The scale fields and the
14+
// content rectangle map buffer pixels back to window points - see
15+
// toWindowPoint() below, which is the one implementation of that formula.
16+
// All fields are zero when the OS omitted the metadata; consumers must refuse
17+
// coordinate-dependent work rather than guess.
18+
struct VideoFrameInfo {
19+
std::uint32_t windowId = 0;
20+
PhysicalSize size; // buffer size in device pixels
21+
double scaleFactor = 0.0; // device pixels per surface point (Retina: 2.0)
22+
double contentScale = 0.0; // downscale applied to fit content in the buffer
23+
// Where the window content sits within the buffer's surface, in surface
24+
// points. The content can sit away from the buffer origin (shadows,
25+
// padding, resize letterboxing), so this is part of the contract.
26+
LogicalRect contentRect;
27+
// Presentation timestamp mapped onto steady_clock (not delivery time), so
28+
// frame age reflects how old the pixels actually are.
29+
std::chrono::steady_clock::time_point capturedAt;
30+
};
31+
32+
// Maps a buffer pixel coordinate to window-local logical coordinates:
33+
//
34+
// surfacePoint = pixel / scaleFactor
35+
// windowPoint = (surfacePoint - contentRect.origin) / contentScale
36+
//
37+
// Returns nullopt when the metadata is missing or the pixel lies outside the
38+
// window content area (shadow/letterbox padding) - callers must treat that as
39+
// a refusal, not round it into range.
40+
[[nodiscard]] inline std::optional<LogicalPoint> toWindowPoint(
41+
const VideoFrameInfo& info, const double pixelX, const double pixelY
42+
) {
43+
if (info.scaleFactor <= 0.0 || info.contentScale <= 0.0 ||
44+
info.contentRect.size.width <= 0.0 ||
45+
info.contentRect.size.height <= 0.0) {
46+
return std::nullopt;
47+
}
48+
49+
const double contentX = pixelX / info.scaleFactor - info.contentRect.origin.x;
50+
const double contentY = pixelY / info.scaleFactor - info.contentRect.origin.y;
51+
if (contentX < 0.0 || contentX >= info.contentRect.size.width ||
52+
contentY < 0.0 || contentY >= info.contentRect.size.height) {
53+
return std::nullopt;
54+
}
55+
56+
return LogicalPoint{
57+
.x = contentX / info.contentScale,
58+
.y = contentY / info.contentScale,
59+
};
60+
}
61+
62+
// One streamed frame: metadata plus a retained, shareable reference to the
63+
// platform pixel buffer (BGRA layout). Copies share the same buffer - a frame
64+
// is never deep-copied on its way through a consumer's pipeline.
65+
//
66+
// Access the pixels through the typed platform bridge (robot/mac/VideoFrame.h
67+
// on macOS) rather than casting nativeHandle() by hand. A portable pixel
68+
// accessor is deliberately absent until a cross-platform consumer exists.
69+
class VideoFrame {
70+
public:
71+
VideoFrame() = default;
72+
73+
// Constructed by backends; the buffer's deleter releases the platform
74+
// reference the backend retained.
75+
VideoFrame(const VideoFrameInfo& info, std::shared_ptr<void> nativeBuffer)
76+
: info_(info), nativeBuffer_(std::move(nativeBuffer)) {}
77+
78+
[[nodiscard]] const VideoFrameInfo& info() const noexcept { return info_; }
79+
80+
// Borrowed for this frame's lifetime. Prefer the typed platform bridge.
81+
[[nodiscard]] void* nativeHandle() const noexcept {
82+
return nativeBuffer_.get();
83+
}
84+
85+
explicit operator bool() const noexcept { return nativeBuffer_ != nullptr; }
86+
87+
private:
88+
VideoFrameInfo info_;
89+
std::shared_ptr<void> nativeBuffer_;
90+
};
91+
92+
} // namespace robot

‎include/robot/Window.h‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#pragma once
2+
3+
#include <cstdint>
4+
#include <string>
5+
6+
#include "robot/Geometry.h"
7+
8+
namespace robot {
9+
10+
// One on-screen application window, as reported by enumeration. Windows move
11+
// and resize after enumeration - query Windows::frame() for the current
12+
// bounds before doing coordinate math against one.
13+
struct Window {
14+
std::uint32_t id = 0;
15+
std::int32_t ownerPid = 0;
16+
std::string appName;
17+
std::string title;
18+
// Global logical desktop coordinates (top-left origin), at enumeration time.
19+
LogicalRect frame;
20+
};
21+
22+
} // namespace robot

‎include/robot/WindowStream.h‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#pragma once
2+
3+
#include <functional>
4+
#include <memory>
5+
6+
#include "robot/Error.h"
7+
#include "robot/VideoFrame.h"
8+
9+
namespace robot {
10+
namespace backend {
11+
class IWindowStreamBackend;
12+
}
13+
14+
struct WindowStreamOptions {
15+
double maxFps = 60.0;
16+
bool showsCursor = true;
17+
};
18+
19+
// Invoked on the stream's internal capture thread for every delivered frame,
20+
// serialized and in order. The callback must be brief (hand the frame off;
21+
// VideoFrame copies are cheap), must not throw, and must not call back into
22+
// robot-cpp APIs or touch this stream's handle.
23+
using VideoFrameCallback = std::function<void(const VideoFrame&)>;
24+
25+
// Invoked on the same capture thread (serialized with frame delivery) when
26+
// the stream dies outside stop(): the window closed, the OS revoked capture,
27+
// the display reconfigured. After it fires, isRunning() is false and no
28+
// further frames arrive. Same rules as VideoFrameCallback.
29+
using StreamErrorCallback = std::function<void(const Error&)>;
30+
31+
// A running window capture stream, returned by Windows::stream(). Move-only
32+
// value type owning the platform stream; destruction stops it.
33+
//
34+
// stop() contract: when stop() returns, no callback is executing and none
35+
// will start. That makes it safe to destroy callback-captured state right
36+
// after stopping. Corollary: never call stop() (or destroy the stream) from
37+
// inside one of its own callbacks - that would deadlock on the drain.
38+
class WindowStream {
39+
public:
40+
explicit WindowStream(std::unique_ptr<backend::IWindowStreamBackend> impl);
41+
~WindowStream();
42+
WindowStream(WindowStream&&) noexcept;
43+
WindowStream& operator=(WindowStream&&) noexcept;
44+
WindowStream(const WindowStream&) = delete;
45+
WindowStream& operator=(const WindowStream&) = delete;
46+
47+
// Idempotent. Blocks until in-flight callbacks have drained.
48+
void stop();
49+
50+
// False after stop() or a stream failure.
51+
[[nodiscard]] bool isRunning() const;
52+
53+
private:
54+
std::unique_ptr<backend::IWindowStreamBackend> impl_;
55+
};
56+
57+
} // namespace robot

0 commit comments

Comments
 (0)