Skip to content

Commit 8b3a2cc

Browse files
JasonVMoCopilot
andauthored
Fix macOS desktop-driver input accounting and application cleanup (#4482)
* Keep the macOS helper's input and application cleanup honest Count mouse buttons and drags in the effect ledger, verify that closed and abandoned applications exited, fail closed on an unreadable session, pointer or accessibility state, release input when the host ends, correlate capture windows by frame, negate the horizontal wheel delta, and never revive a closed window by title. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Self-test the macOS helper's input, lifecycle, termination and capture selection Run InputController through an injectable event poster, a Host over pipes, a fake process table with real children, and the pure capture window selection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Document the macOS helper's verified close, counted buttons and fail-closed state Describe application incarnation mode, performAccessibilityAction, drag counting, the run-loop findings and the wheel direction, and add the changeset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 3b2787e commit 8b3a2cc

18 files changed

Lines changed: 1461 additions & 178 deletions
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@fluentui-react-native/desktop-driver': patch
3+
---
4+
5+
Make the macOS helper count mouse-button events and drags in the effect ledger, so a cancelled or failed physical click reports a partial effect instead of none and cannot be retried into a second activation. End launched applications with a verified quit, wait, kill, and exit proof on close, dispose, and failed or late launches, keeping the lease when the exit cannot be proven. Release input and stop queued work when the client disconnects or a response cannot be written, ignore `SIGPIPE`, and fail the transport on invalid JSON. Fail closed on an unreadable login session or pointer position, take the cross-process input lock for focus and window geometry changes, require a window frame that matches before capturing it, and report unreadable Accessibility state as unsupported or `target-non-response` instead of a definite answer.

‎packages/agentic/desktop-driver/native/macos/README.md‎

Lines changed: 86 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ Accessibility.
1919
| `Host.swift` | FDR1 hello, request queue, framing, cancellation, and disposal |
2020
| `Driver.swift` | Command dispatch, capability reporting, and physical-input scopes |
2121
| `Application.swift` | Launch/attach leases, exact process/window identity, and ownership |
22+
| `ApplicationTermination.swift` | `terminateOwned`: quit, wait, kill, and verify the exit of an owned process |
23+
| `ApplicationIncarnation.swift` | One-shot `--application-incarnation`: whether a recorded process exited |
2224
| `Accessibility.swift` | AX tree walking, lookup, snapshots, focus, hit testing, actions, and source |
2325
| `Observation.swift` | Provider-neutral bounded sampler: request rules, budgets, and BFS result |
2426
| `AccessibilityObservation.swift` | AX source for `observe`: multi-attribute reads, shared ids, `observe` run |
@@ -32,6 +34,10 @@ Accessibility.
3234
| `SelfTest.swift` | Provider-local framing, identity, AX, input, capture, and permission checks |
3335
| `ObservationSelfTest.swift` | Synthetic-tree, randomized differential, and command checks for `observe` |
3436
| `ServiceIdentitySelfTest.swift` | URL, claim, listener-policy, and live describe/verify checks |
37+
| `InputIntegritySelfTest.swift` | Recorded input: counted buttons and drags, cancelled clicks, fail-closed |
38+
| `HostLifecycleSelfTest.swift` | In-process host: end of input, write failure, invalid JSON, queued work |
39+
| `ApplicationTerminationSelfTest.swift` | Termination verification against fake and real child processes |
40+
| `CaptureSelectionSelfTest.swift` | Pure capture-window selection and `visible` without geometry |
3541

3642
The Node build coordinator is
3743
`../../src/native/macos/buildMacOSDriver.ts`; runtime adaptation is under
@@ -141,6 +147,7 @@ furn-desktop-driver-host --self-test
141147
furn-desktop-driver-host --permissions [--prompt]
142148
furn-desktop-driver-host --release-input [--sweep]
143149
furn-desktop-driver-host --service-identity
150+
furn-desktop-driver-host --application-incarnation
144151
```
145152

146153
The long-lived `--stdio` process:
@@ -149,8 +156,20 @@ The long-lived `--stdio` process:
149156
2. processes one command at a time on its worker queue;
150157
3. handles correlated cancellation before accepting later side effects;
151158
4. holds a per-user `flock` while physical input is active;
152-
5. releases tracked keys and buttons on failure, cancellation, and disposal;
153-
6. exits after `dispose`.
159+
5. releases tracked keys and buttons on failure, cancellation, disposal, end of input, and a response it cannot
160+
write;
161+
6. exits after a `dispose` that closed every launched application.
162+
163+
A closed stdin ends the session: the helper drops queued requests, cancels the running one, releases held input and
164+
the lock, and returns. It ignores `SIGPIPE`, so a client that disappears surfaces as a failed write, which also
165+
releases input before the helper exits. A frame that is not valid JSON, or whose JSON is not an object, fails the
166+
transport: the helper writes a fixed diagnostic and exits with status 1 instead of skipping the frame.
167+
168+
The one-shot `--application-incarnation` mode reads exactly `{ "pid", "startedAt" }` from stdin and answers whether
169+
that exact process, a process id and its start time, has `exited`, is `running`, or cannot be told (`unknown`).
170+
`exited` is reported only with proof: the process id is gone (a zombie included) or now names a process that
171+
started more than a second later. Settlement recovery uses it to decide whether a launched application that the helper
172+
could not prove closed is really gone; [PROTOCOL.md](../PROTOCOL.md) defines its request and answer.
154173

155174
The input lock lives beneath
156175
`~/Library/Caches/com.microsoft.fluentui-react-native.desktop-driver`, so
@@ -170,6 +189,13 @@ resident memory for the life of the session (about 25 KB for a `probe`, 80 KB
170189
for a `tree`, and 250 KB for a 500-element `observe`). A new thread that serves
171190
requests must drain a pool per request too.
172191

192+
The main run loop is also what refreshes `NSWorkspace.runningApplications` and `NSRunningApplication.isTerminated`.
193+
Measured without it, a launched application registers within about 0.2 s, but after the process was killed
194+
`isTerminated` stayed false and the workspace list kept the dead process for the 15 s the probe watched; with the
195+
run loop pumped both updated within 0.25 s. Registration therefore needs no run loop, while exit never trusts that
196+
state: termination verification and lease checks read the kernel through `ApplicationIncarnation`, and attaching
197+
without a lease drops listed processes the kernel no longer knows.
198+
173199
## Application and window identity
174200

175201
The helper uses server-owned `bundleIdentifier`, `executablePath`,
@@ -188,15 +214,32 @@ timestamps allow only the bounded precision required by
188214
fail.
189215

190216
Launch honors an explicitly configured `launchTimeoutMs`, or uses the native
191-
90-second default within the server's 120-second default deadline. If LaunchServices
192-
completes after timeout or cancellation, the late completion closes only a
193-
new application causally owned by that request, never a preexisting app.
217+
90-second default within the server's 120-second default deadline. A launch that
218+
fails, times out, or is cancelled after the application started ends that
219+
application with the same quit, wait, kill, and verify sequence as `closeApplication`
220+
(`terminateOwned`), bounded to fit Node's cancellation deadline, instead of only
221+
asking it to quit. If LaunchServices completes after timeout or cancellation, the
222+
late completion ends only a new application causally owned by that request, never
223+
a preexisting app. When an application survives that cleanup, the `launch-failed`
224+
error carries its `processId`; a cancelled launch only writes a diagnostic. A bare executable is started with its standard
225+
streams on `/dev/null`, so it never inherits the helper's FDR1 pipes, and the
226+
helper polls until Launch Services registers it instead of reading the
227+
registration once.
194228
On October 22, 2026 (UTC), the helper built, handshook, and self-tested on
195229
macOS, and real FluentTester launches with a configured 180-second timeout
196230
succeeded. The late-completion branch remains covered only by code review.
197231

198232
Leases record whether the app was launched or attached. Session deletion
199233
preserves attached apps and closes only an exact process owned by the helper.
234+
`closeApplication` and `dispose` succeed only after the helper proves the exact
235+
process exited: they verify its incarnation (process id and start time), ask it to
236+
quit, wait, send `SIGKILL` if it is still running, and verify again. A process
237+
whose identity cannot be matched is never signalled. When the exit is not proven, the
238+
command fails with `automation-failed` and the lease, its window ids, and its
239+
element records stay in place, so a retry can finish the close and a failed
240+
`dispose` leaves the helper running. Window and element records are forgotten only
241+
after the close succeeds, and a lease registered before its window resolved is
242+
rolled back if the resolution fails. A closed window id is never revived by title.
200243
Opaque helper IDs represent windows and AX elements; native references never
201244
cross the public WebDriver boundary.
202245

@@ -263,6 +306,12 @@ list does not contain `AXEnabled`, matching how assistive technologies treat
263306
it. An element that lists `AXEnabled` without a readable value still reports
264307
an unsupported enabled state. Fabric views always publish the attribute.
265308

309+
An Accessibility state the helper cannot read is never reported as a definite answer. `visible` is unsupported when
310+
an element's position and size cannot be read, and `observe` then omits `geometry` instead of reporting an empty
311+
rectangle. A tree walk fails with `automation-failed` rather than treating an element whose children cannot be read
312+
as a leaf, a hit test, focus, or clear against an unresponsive target fails as `target-non-response`, and a
313+
cancellation is checked at every element of a walk. Reads that tolerate a React Native Fabric remount, where an
314+
element briefly answers `failure` or `illegalArgument`, still treat those two errors as an absent value.
266315
The `accessibilityActions` command reads `AXUIElementCopyActionNames` for a live
267316
element; an element whose actions are unsupported yields an empty list. AppKit
268317
lists an `NSAccessibilityCustomAction` as
@@ -281,13 +330,18 @@ actions that share a name without revealing their target address, and stops
281330
matching if the application replaces an action. `performAccessibilityAction`
282331
re-reads `AXUIElementCopyActionNames`, refuses unless exactly one raw action
283332
has the key (`action-changed`), and calls `AXUIElementPerformAction` with it.
284-
Like `activate`, it moves no pointer and takes no physical input lock.
333+
It moves no pointer and takes no physical input lock, unlike `activate`,
334+
`setWindowRect`, and `closeWindow`, which move keyboard focus or windows another helper may be typing into and
335+
therefore wait for the same cross-process lock as physical input. It is a tracked mutation: a failure or
336+
cancellation reports an `effect`.
285337

286338
Quartz Event Services supplies physical pointer, keyboard, text, and wheel
287339
input. Physical input requires Accessibility/PostEvent authority for the
288340
responsible process and an unlocked interactive Aqua session. Missing authority
289341
returns `input-unavailable`; the provider never substitutes unrelated
290-
accessibility activation and reports physical success.
342+
accessibility activation and reports physical success. A positive wheel
343+
`deltaX` scrolls right and a positive `deltaY` scrolls down, as on Windows. Quartz
344+
scrolls left and up for positive values, so both deltas are negated.
291345

292346
The helper advertises `strictClear`. `clear` writes an empty `AXValue` when
293347
`AXUIElementIsAttributeSettable` reports the value settable, and otherwise
@@ -305,8 +359,9 @@ applications from publishing complete accessibility windows and from receiving
305359
physical input. Launch, attach, window discovery, and new pointer, keyboard,
306360
text, or wheel input check the login session and fail immediately with
307361
`input-unavailable` and a `session` diagnostic instead of retrying into a
308-
window-enumeration error. Key and button releases remain allowed so recovery
309-
can complete while the session is locked.
362+
window-enumeration error. Physical input fails closed: a login session or pointer
363+
position the helper cannot read is a failure, not permission to proceed. Key and
364+
button releases remain allowed so recovery can complete while the session is locked.
310365

311366
Activation sets the application's `AXFrontmost` attribute, which a trusted
312367
client applies immediately, and falls back to `NSRunningApplication.activate`
@@ -389,7 +444,7 @@ read, through both `observe` and `tree`.
389444
## Mutation effects
390445

391446
Failed and cancelled mutating commands (`activate`, `clear`, `click`,
392-
`closeWindow`, `performActions`, `releaseActions`, `sendKeys`, `setWindowRect`)
447+
`closeWindow`, `performAccessibilityAction`, `performActions`, `releaseActions`, `sendKeys`, `setWindowRect`)
393448
report an FDR1 `effect`, so Node can tell a rejection from a mutation that may
394449
have reached the application (see [PROTOCOL.md](../PROTOCOL.md)). Reads and
395450
lifecycle commands carry none.
@@ -398,9 +453,10 @@ The worker owns an `EffectLedger` that `Driver.execute` resets at the start of
398453
each command. It counts only primary mutations, at the point they are
399454
dispatched:
400455

401-
- an `AXPress`, an `AXValue` write, the close-button press, and each window
402-
position or size set, bracketed by `dispatching()` and `completed(_:)`;
403-
- every posted key, button, and wheel event, through `delivered()`.
456+
- an `AXPress` (`click` and `performAccessibilityAction`), an `AXValue` write, the close-button press, and each
457+
window position or size set, bracketed by `dispatching()` and `completed(_:)`;
458+
- every posted key, button press and release, and wheel event, through `delivered()`, and each pointer move made
459+
while a button is held (a drag).
404460

405461
An AX call that succeeds is applied. One that returns an error the platform
406462
documents as a rejection (`actionUnsupported`, `attributeUnsupported`,
@@ -410,8 +466,10 @@ from a busy or modal target, stays unresolved, so a failure reports `unknown` an
410466
Node quarantines the target. That can quarantine a target that was not changed,
411467
but it never lets a possibly applied mutation be retried as a rejection.
412468

413-
Focus, window raise and activation, pointer moves, and key or button releases that
414-
clean up counted input are idempotent and not counted. `setWindowRect` checks that
469+
Focus, window raise and activation, bare pointer moves, and key or button releases that
470+
clean up counted input are idempotent and not counted. A click cancelled or failed between its
471+
press and release therefore reports `partial`, never `none`, so Node quarantines the target
472+
instead of retrying a second activation. `setWindowRect` checks that
415473
both position and size are settable before it sets either, so a non-resizable
416474
window rejects without moving. A queued mutating request that is cancelled before
417475
it starts reports `none`, a queued read or lifecycle command reports no `effect`,
@@ -422,18 +480,26 @@ mutation must bracket its platform call with the ledger and be listed in
422480

423481
## Capture
424482

425-
ScreenCaptureKit captures a uniquely correlated application window and encodes
426-
PNG output through ImageIO. Element screenshots crop the window image using the
483+
ScreenCaptureKit captures the application window whose frame agrees with the
484+
tracked AX window and encodes
485+
PNG output through ImageIO. A candidate must be an ordinary window (layer 0) whose
486+
four edges are within 2 points of the AX frame, which absorbs rounding but not a
487+
different window; a title match narrows the candidates first. The provider never
488+
captures the nearest window of the application, such as a panel or another
489+
document. Element screenshots crop the window image using the
427490
matched AppKit display's backing scale factor. Retina output therefore reports
428491
pixel dimensions and `scaleFactor` separately from point geometry.
429492

430493
Screen Recording permission is degradable. Without it, the helper does not
431494
advertise screenshot capabilities, while AX and input automation can continue
432-
when their own authority is present. Capture fails explicitly when:
495+
when their own authority is present. Capture fails explicitly with `capture-failed`,
496+
and a `reason` of `no-process-window`, `uncorrelated-window-geometry`, or
497+
`ambiguous-window-geometry` where the window is the cause, when:
433498

434-
- Screen Recording permission is absent;
499+
- Screen Recording permission is absent, which is reported as a permission
500+
problem from one guard and never as an operating-system limitation;
435501
- the target is minimized or off-screen;
436-
- ScreenCaptureKit cannot uniquely correlate the AX window;
502+
- ScreenCaptureKit has no window that correlates uniquely with the AX window;
437503
- crop geometry is empty or outside the capture.
438504

439505
The provider does not implicitly restore or move a window to make capture pass.

0 commit comments

Comments
 (0)