Skip to content

Commit bdeb0b2

Browse files
JasonVMoCopilot
andcommitted
Retain Windows input after failed release
Keep unreleased strokes and buttons under the physical-input lock, preserve uncertain host recovery, and refuse unsafe fallback or disposal. Fix Windows portability of desktop-test lifecycle and reporting contracts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 53a0c7cf-1330-4677-8700-d249b508c6ee
1 parent a1813ae commit bdeb0b2

21 files changed

Lines changed: 467 additions & 83 deletions
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@fluentui-react-native/desktop-test': patch
3+
---
4+
5+
Make the desktop-test lifecycle and reporting contracts portable to Windows, including process cleanup and path projection.

‎.changeset/win-helper-cancellation.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@
22
'@fluentui-react-native/desktop-driver': patch
33
---
44

5-
Check cancellation in more Windows helper waits: bounded UI Automation calls, the whole-subtree cache, window response probes, posted window restores and moves, activation retries, and the `closeApplication` termination wait, and a cancelled mutation reports effect `unknown`. Direct UI Automation calls and the activation calls themselves can still block. Window management and activation probe the target window first and fail with reason `target-non-response` when it does not answer. A restore that stalls after its initial probe reports effect `unknown` rather than `partial`. Held keys and buttons keep the physical input lock until released, and queued commands are dropped after the transport ends. A malformed or non-object JSON frame now fails the transport. Pointer presses and scrolls over another application's window are refused, failed pattern and focus reads are no longer reported as unsupported or unfocused, a job-contained launch root is no longer terminated after launch, a packaged launch root must report its AUMID, an unreadable start time fails the lease, and activation grants foreground rights only to the target process.
5+
Check cancellation in more Windows helper waits: bounded UI Automation calls, the whole-subtree cache, window response probes, posted window restores and moves, activation retries, and the `closeApplication` termination wait, and a cancelled mutation reports effect `unknown`. Direct UI Automation calls and the activation calls themselves can still block. Window management and activation probe the target window first and fail with reason `target-non-response` when it does not answer. A restore that stalls after its initial probe reports effect `unknown` rather than `partial`. Held keys and buttons keep the physical input lock while the helper runs; failed cleanup retains unreleased input for retry, and a failed physical mutation reports an unknown effect. The adapter retains uncertain input for recovery, and disposal reports failed cleanup. Queued commands are dropped after the transport ends. A malformed or non-object JSON frame now fails the transport. Pointer presses and scrolls over another application's window are refused, failed pattern and focus reads are no longer reported as unsupported or unfocused, a job-contained launch root is no longer terminated after launch, a packaged launch root must report its AUMID, an unreadable start time fails the lease, and activation grants foreground rights only to the target process.

‎packages/agentic/desktop-driver/native/PROTOCOL.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,11 @@ and `setWindowRect`. Reads and
194194
lifecycle commands (`launch`, `attach`, `closeApplication`, `dispose`) carry no
195195
`effect`.
196196

197+
On Windows, a failed physical-input mutation whose cleanup leaves a key stroke
198+
or button depressed reports `unknown` and retains the input and lock while the
199+
helper remains alive. A later release can retry it; `dispose` refuses to
200+
acknowledge if cleanup still fails.
201+
197202
Node fails closed: an absent `effect` on a failed or cancelled mutation is
198203
`unknown`. `none` is a rejection and leaves the target usable; `partial` and
199204
`unknown` are indeterminate, and Node quarantines the target until it is

‎packages/agentic/desktop-driver/native/windows/README.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -284,11 +284,14 @@ to the later window and element checks. A click is refused as
284284
point, and so is a `performActions` `pointerDown` or `scroll` whose point is
285285
covered by a window that neither belongs to a leased application nor is the
286286
top-level window around one. Keys or buttons that `performActions` leaves down
287-
keep the session's physical input lock until they are released, by
288-
`releaseActions`, a later command, cancellation, or the helper's exit, so
289-
another helper cannot interleave its input with them. A `SendInput` call that
290-
injects fewer events than requested fails as `input-failed` and names UIPI, a
291-
secure desktop, or blocked input.
287+
keep the session's physical input lock while the helper runs, until
288+
`releaseActions`, a later command, cancellation, or shutdown releases them.
289+
If cleanup fails, the helper retains the unreleased input and lock for a later
290+
retry; a failed physical mutation reports effect `unknown`, and `dispose`
291+
refuses to acknowledge while input remains down. If the worker exits first,
292+
an abandoned mutex can warn one waiting helper, but is not a persistent
293+
recovery barrier. A `SendInput` call that injects fewer events than requested
294+
fails as `input-failed` and names UIPI, a secure desktop, or blocked input.
292295

293296
`click` mode `auto` falls back to accessibility activation when physical input
294297
is unavailable, and an explicit `physical` request reports `input-unavailable`.

‎packages/agentic/desktop-driver/native/windows/src/driver.cpp‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -291,7 +291,7 @@ void Driver::ClickPhysically(const std::string& elementId, const CancellationTok
291291
POINT center{};
292292
center.x = (snapshot.physicalRect.left + snapshot.physicalRect.right) / 2;
293293
center.y = (snapshot.physicalRect.top + snapshot.physicalRect.bottom) / 2;
294-
RequireClickPointOwnedBy(window, center);
294+
RequireClickPointOwnedBy(window, center, hitTester_);
295295
input_.ClickAt(center, token);
296296
}
297297

@@ -501,16 +501,15 @@ void Driver::ReleaseInput() noexcept {
501501
const PhysicalInputScope scope(idle, kPhysicalInputCleanupWaitMs, AbandonPolicy::Adopt);
502502
input_.ReleaseAll();
503503
} catch (const HelperError&) {
504-
// Depressed input must never outlive a cancelled command, so recovery
505-
// proceeds even when the cross-process lock could not be acquired.
504+
// Continue attempting cleanup even if the cross-process lock could not
505+
// be acquired.
506506
input_.ReleaseAll();
507507
}
508508
}
509509
} catch (...) {
510510
// Releasing input must never mask the original failure.
511511
}
512-
// Input that could not be released keeps the lock, so the thread's exit
513-
// abandons it and the next helper reports the depressed input.
512+
// Input that could not be released keeps the lock while the worker lives.
514513
if (!input_.HasDepressedInput()) {
515514
heldInput_.Release();
516515
}
@@ -815,7 +814,7 @@ CommandResult Driver::ExecuteCommand(const std::string& command, const json::Val
815814
// The environment can change between the check and the click. When
816815
// it refused before any click was delivered, automatic mode still
817816
// has the accessibility click.
818-
if (mode != "auto" || error.code() != kErrorInputUnavailable) {
817+
if (mode != "auto" || error.code() != kErrorInputUnavailable || input_.HasDepressedInput()) {
819818
throw;
820819
}
821820
automation_.AccessibilityClick(elementId, token);
@@ -944,6 +943,9 @@ CommandResult Driver::ExecuteCommand(const std::string& command, const json::Val
944943
}
945944
if (command == "dispose") {
946945
ReleaseInput();
946+
if (input_.HasDepressedInput()) {
947+
Fail(kErrorInputFailed, "The helper could not release all held physical input during disposal.");
948+
}
947949
applications_.Dispose(token);
948950
automation_.Reset();
949951
result.result = json::Value::Null();

‎packages/agentic/desktop-driver/native/windows/src/driver.h‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -88,10 +88,9 @@ class Driver {
8888
private:
8989
CommandResult ExecuteCommand(const std::string& command, const json::Value& params, const CancellationToken& token);
9090
void ForgetRetiredWindows() noexcept;
91-
// Holds the cross-process physical input mutex for a whole action chain and
92-
// guarantees that a failed or cancelled chain releases its depressed input
93-
// before the next owner acquires the lock. Keys or buttons a chain leaves
94-
// down keep the lock held across commands until they are released.
91+
// Holds the cross-process physical input mutex for a whole action chain.
92+
// Failed cleanup retains both the unreleased input and lock while this
93+
// worker lives; successful cleanup releases the lock.
9594
template <typename Fn>
9695
void RunPhysical(const CancellationToken& token, Fn&& body) {
9796
PhysicalInputScope scope(token);
@@ -106,6 +105,10 @@ class Driver {
106105
// Preserve the original failure.
107106
}
108107
}
108+
if (input_.HasDepressedInput()) {
109+
// A failed cleanup leaves the command's physical-input state uncertain.
110+
RecordDispatching();
111+
}
109112
HoldInputWhileDepressed(scope);
110113
throw;
111114
}

‎packages/agentic/desktop-driver/native/windows/src/host.cpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -264,8 +264,8 @@ void Host::WorkerLoop() {
264264
ExitProcess(0);
265265
}
266266
}
267-
// The physical input lock belongs to this thread, so it is released here
268-
// rather than abandoned when the thread exits.
267+
// The physical input lock belongs to this thread; unresolved input retains
268+
// it until this worker exits.
269269
driver_.ReleaseInput();
270270
winrt::uninit_apartment();
271271
}

‎packages/agentic/desktop-driver/native/windows/src/input.cpp‎

Lines changed: 66 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
#include <algorithm>
44
#include <cmath>
5+
#include <exception>
56

67
#include "geometry.h"
78
#include "settlement.h"
@@ -352,6 +353,7 @@ void InputController::PointerDown(int button, const CancellationToken& token) {
352353
// Carry the coordinate so the press lands where the pointer was placed even if
353354
// another input source has since moved the cursor.
354355
std::vector<INPUT> inputs{MakeAbsoluteMouseInput(pointer_, MOUSEEVENTF_MOVE | downFlag, mouseData)};
356+
pressedButtons_.reserve(pressedButtons_.size() + 1);
355357
Send(inputs);
356358
pressedButtons_.push_back(button);
357359
}
@@ -372,19 +374,27 @@ void InputController::PointerUp(int button, const CancellationToken& token) {
372374
}
373375

374376
void InputController::PointerCancel() {
375-
for (auto iterator = pressedButtons_.rbegin(); iterator != pressedButtons_.rend(); ++iterator) {
377+
std::exception_ptr failure;
378+
auto iterator = pressedButtons_.end();
379+
while (iterator != pressedButtons_.begin()) {
380+
--iterator;
376381
DWORD downFlag = 0;
377382
DWORD upFlag = 0;
378383
DWORD mouseData = 0;
379384
ButtonFlags(*iterator, downFlag, upFlag, mouseData);
380385
std::vector<INPUT> inputs{MakeAbsoluteMouseInput(pointer_, upFlag, mouseData)};
381386
try {
382387
Send(inputs, false);
388+
iterator = pressedButtons_.erase(iterator);
383389
} catch (const HelperError&) {
384-
// Release as much as possible; the remaining ledger entries still drop.
390+
if (!failure) {
391+
failure = std::current_exception();
392+
}
385393
}
386394
}
387-
pressedButtons_.clear();
395+
if (failure) {
396+
std::rethrow_exception(failure);
397+
}
388398
}
389399

390400
void InputController::ClickAt(POINT target, const CancellationToken& token) {
@@ -480,12 +490,7 @@ void InputController::KeyDown(std::wstring_view value, const CancellationToken&
480490
}
481491
const std::vector<KeyStroke> strokes = StrokesForValue(value);
482492
GuardKeyPress();
483-
// The ledger records the press before injection so a partially injected
484-
// sequence is still released by the ledger.
485-
pressedKeys_.emplace_back(std::wstring(value), strokes);
486-
for (const KeyStroke& stroke : strokes) {
487-
SendStroke(stroke, true);
488-
}
493+
PressStrokes(std::wstring(value), strokes);
489494
}
490495

491496
void InputController::KeyUp(std::wstring_view value, const CancellationToken& token) {
@@ -502,8 +507,9 @@ void InputController::KeyUp(std::wstring_view value, const CancellationToken& to
502507
continue;
503508
}
504509
RequirePhysicalInput();
505-
for (auto stroke = iterator->second.rbegin(); stroke != iterator->second.rend(); ++stroke) {
506-
SendStroke(*stroke, false);
510+
while (!iterator->second.empty()) {
511+
SendStroke(iterator->second.back(), false);
512+
iterator->second.pop_back();
507513
}
508514
pressedKeys_.erase(std::next(iterator).base());
509515
return;
@@ -539,17 +545,34 @@ void InputController::TypeText(std::wstring_view text, const CancellationToken&
539545
}
540546
}
541547

548+
void InputController::PressStrokes(std::wstring ledgerKey, const std::vector<KeyStroke>& strokes) {
549+
std::vector<KeyStroke> recorded;
550+
// Recording an accepted stroke must not allocate after injection.
551+
recorded.reserve(strokes.size());
552+
pressedKeys_.emplace_back(std::move(ledgerKey), std::move(recorded));
553+
auto& held = pressedKeys_.back().second;
554+
try {
555+
for (const KeyStroke& stroke : strokes) {
556+
SendStroke(stroke, true);
557+
held.push_back(stroke);
558+
}
559+
} catch (...) {
560+
if (held.empty()) {
561+
pressedKeys_.pop_back();
562+
}
563+
throw;
564+
}
565+
}
566+
542567
void InputController::PressAndRelease(std::wstring ledgerKey, const std::vector<KeyStroke>& strokes,
543568
const CancellationToken& token) {
544569
token.ThrowIfCancelled();
545570
GuardKeyPress();
546-
pressedKeys_.emplace_back(std::move(ledgerKey), strokes);
547-
for (const KeyStroke& stroke : strokes) {
548-
SendStroke(stroke, true);
549-
}
550-
const std::vector<KeyStroke>& recorded = pressedKeys_.back().second;
551-
for (auto stroke = recorded.rbegin(); stroke != recorded.rend(); ++stroke) {
552-
SendStroke(*stroke, false);
571+
PressStrokes(std::move(ledgerKey), strokes);
572+
auto& recorded = pressedKeys_.back().second;
573+
while (!recorded.empty()) {
574+
SendStroke(recorded.back(), false);
575+
recorded.pop_back();
553576
}
554577
pressedKeys_.pop_back();
555578
}
@@ -563,8 +586,7 @@ void InputController::PressChord(const std::vector<WORD>& virtualKeys, const Can
563586
const KeyStroke stroke{false, key, false};
564587
// Chord keys enter the ledger under a synthetic name that can never
565588
// collide with a WebDriver key value, so a failed chord still releases.
566-
pressedKeys_.emplace_back(L"chord:" + std::to_wstring(key), std::vector<KeyStroke>{stroke});
567-
SendStroke(stroke, true);
589+
PressStrokes(L"chord:" + std::to_wstring(key), {stroke});
568590
}
569591
while (pressedKeys_.size() > base) {
570592
const std::vector<KeyStroke>& strokes = pressedKeys_.back().second;
@@ -636,17 +658,36 @@ std::size_t InputController::ReleaseDesktopModifiers() {
636658
}
637659

638660
void InputController::ReleaseAll() {
639-
for (auto iterator = pressedKeys_.rbegin(); iterator != pressedKeys_.rend(); ++iterator) {
640-
for (auto stroke = iterator->second.rbegin(); stroke != iterator->second.rend(); ++stroke) {
661+
std::exception_ptr failure;
662+
auto iterator = pressedKeys_.end();
663+
while (iterator != pressedKeys_.begin()) {
664+
--iterator;
665+
auto stroke = iterator->second.end();
666+
while (stroke != iterator->second.begin()) {
667+
--stroke;
641668
try {
642669
SendStroke(*stroke, false, false);
670+
stroke = iterator->second.erase(stroke);
643671
} catch (const HelperError&) {
644-
// Continue releasing the remaining ledger entries.
672+
if (!failure) {
673+
failure = std::current_exception();
674+
}
645675
}
646676
}
677+
if (iterator->second.empty()) {
678+
iterator = pressedKeys_.erase(iterator);
679+
}
680+
}
681+
try {
682+
PointerCancel();
683+
} catch (const HelperError&) {
684+
if (!failure) {
685+
failure = std::current_exception();
686+
}
687+
}
688+
if (failure) {
689+
std::rethrow_exception(failure);
647690
}
648-
pressedKeys_.clear();
649-
PointerCancel();
650691
}
651692

652693
} // namespace furn

‎packages/agentic/desktop-driver/native/windows/src/input.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,7 @@ class InputController {
112112
// Runs the keyboard guard. Called before a press is recorded in the ledger,
113113
// so a refused press leaves nothing to release.
114114
void GuardKeyPress();
115+
void PressStrokes(std::wstring ledgerKey, const std::vector<KeyStroke>& strokes);
115116
void PressAndRelease(std::wstring ledgerKey, const std::vector<KeyStroke>& strokes,
116117
const CancellationToken& token);
117118
// Counted sends are primary mutations in the command's effect ledger. Pointer

0 commit comments

Comments
 (0)