Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/windows-application-containment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@fluentui-react-native/desktop-driver': patch
---

Contain unpackaged Windows launches in a helper-owned Job Object so the application and its descendants end with the lease or the helper, report each lease's `containment` (`job-object`, `uncontained`, or `external`), and fail attach closed on unreadable or cross-session identity. A refused foreground activation now re-attaches to a changed foreground holder and names the holder and its cause.
7 changes: 7 additions & 0 deletions .changeset/windows-qualification-records.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
'@fluentui-react-native/desktop-driver-mcp': patch
'@fluentui-react-native/desktop-test': patch
'@fluentui-react-native/storybook-desktop': patch
---

Record Windows Fabric and Win32 Paper launch, containment, and MCP measurement evidence, and document the Win32 agent-mode bundle, the Windows foreground refusal, and the `serve --agent` interrupt behavior.
76 changes: 74 additions & 2 deletions packages/agentic/desktop-driver-mcp/references/current-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ owner_paths:

# Current adapter and measurement status

**Work in progress, updated 2026-11-05.** The adapter is a thin client, not a
**Work in progress, updated 2026-11-08.** The adapter is a thin client, not a
supervisor. Keep this package name until actual harness-owned management tools
land; a later `desktop-mcp` facade must retain driver-only use and plan published
compatibility. Done below describes implementation, not full native qualification.
Expand Down Expand Up @@ -73,7 +73,51 @@ tokenizer/version, serialized UTF-8 bytes and characters, text tokens,
instructions overhead, projection/budgets, sample count, cold/warm timing,
success/retry/cancellation/cleanup counts, and approved artifact locator.
Do not claim token reductions from reducing information scope or omitting
discovery. Windows and Paper measurements are currently **not recorded here**.
discovery. macOS Paper measurements are currently **not recorded here**.

### Windows Fabric and Win32 Paper - 2026-11-08 UTC

Revision `9d3e11516` plus the uncommitted Windows ownership and foreground
helper changes, which do not alter tool text. MCP SDK 1.32.2 stdio client
through the default descriptor; `gpt-tokenizer` 4.1.0 with `o200k_base` over
every text block. Windows 11 build 26300, AMD64; Storybook served by
`yarn desktop-test --target windows|win32 --reporter json serve --agent`. The
console session was disconnected, so `click` was not measured; the toggle
row uses `activate`. Each endpoint made 139 calls with no errors or retries,
then closed its session. Runs `caba9e12-d9da-42a0-8b06-7300940e8318`
(Windows) and `5f75aac2-d74c-4969-aba3-bd8c925d5725` (Win32), each with its
`run.json` under `apps/storybook/artifacts/desktop/<run-id>/`.

| Measurement | Windows Fabric | Win32 Paper |
| ------------------------------------------------------- | ----------------------------------- | ----------------------------------- |
| Six-tool `tools/list` | 4,691 tokens; 20,277 bytes (chars) | Same |
| Server instructions | 120 tokens; 591 bytes | Same |
| `open` (attach) | 45 ms | 31-40 ms |
| Checkbox `activate` + `checked` check + default observe | 1,923 tokens; 5,346 bytes; 2,288 ms | 1,645 tokens; 5,187 bytes; 2,302 ms |
| Of which `activate` | 2,190 ms | 2,196 ms |
| Observe default (100 of about 113 nodes, 20 calls) | 66.7-73.7 ms median, max 93.1 ms | 81.1-85.8 ms median, max 97.3 ms |
| Observe text per call | 1,801-1,843 tokens | 1,594-1,633 tokens |
| Delta (20 calls, idle) | 72.7-77.5 ms median, max 98.4 ms | 83.3-88.3 ms median, max 106.4 ms |
| Delta text per call | 368 tokens | 220 tokens |

Ranges span the Button, Checkbox, and Menu stories (`components-button--appearance`,
`components-checkbox--constrained-text`, `components-menu--default`); the
toggle row used `components-checkbox--default`. Observe used the default
budget, which emitted 100 nodes. On Windows an idle delta is never empty:
the React Native Test App's React, Debug, and System menu-bar items get new
references on every read (three removed, three added). The helper reuses a
reference only when the UI Automation runtime identity matches; whether these
items lack stable runtime identities is unconfirmed. Without those changes
the envelope is about 151 tokens. The Win32
idle delta was not decomposed. The first Win32 toggle attempt chose the
wrapping-label checkbox in `constrained-text` and returned a `stopped`
sequence; it is retained here, not diagnosed, and the default story was used
instead.

Discovery grew by 3 tokens and 8 bytes since `41fc7473f` from tool-text
changes. The Windows toggle rows are not comparable with the 386-token macOS
row: they target a different story and their closing observe is an unscoped
default observe, which accounts for most of their text.

## Before full submission

Expand Down Expand Up @@ -266,3 +310,31 @@ Win32 needed none. All sessions closed and cleanup removed owned processes,
listeners, and the workload claim; no default descriptor remained. These
runs issued no descriptor grant and do not change the descriptor matrix's
Partial or Fail outcomes.

## 2026-11-08 UTC connected descriptor-path checks

Source `fec94cec4995c7372b798877c5412ddcb156523b`, Windows 11 Enterprise
build 26300, AMD64. An MCP SDK stdio client used the default private
descriptor and attached to the real, instance-matched Storybook app under an
exclusive claim. These are manual operations, not selected suite tests.

| Endpoint | Driver run / MCP session | Physical checkbox | Target PNG |
| -------------- | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------- |
| Windows Fabric | `af97fdb3-68de-4ce9-836e-935331528c8e` / `6ab236be-dc25-4b53-97f9-4e6ce8875ab2` | Pass: 2 applied `physical-pointer` actions, unchecked -> checked -> unchecked | Pass: 42,436 bytes, resource SHA-256 verified |
| Win32 Paper | `3294d66d-9332-4639-95fe-8d3047f47e46` / `8d75052d-ddae-48f9-a1c1-2c8abcf8a019` | Pass: 2 applied `physical-pointer` actions, unchecked -> checked -> unchecked | Pass: 31,639 bytes, resource SHA-256 verified |

Each session selected `components-checkbox--default`, checked every state,
read the PNG through its MCP resource, verified its signature and digest,
and closed. The Windows case supersedes the earlier failed physical-click
attempt for this bounded descriptor-path operation. The first Windows
readiness attempt ended after 30 rejected, effect-none retries while Metro
was still building; a later attempt passed.

On Win32, a session-local injection closed the authenticated bridge after
selection. A newly observed preview checkbox was unavailable (`find: none`)
while the chrome root remained present; fresh-preview mutation fencing and
chrome mutation remain Partial, not passes. An initial 10-second wait timed
out before the observed bridge retirement; the subsequent attempt waited for
retirement. No raw image or grant is retained in this reference. Driver
shutdown, app cleanup, and the failed Win32 app-run reports are
recorded in `packages/agentic/desktop-test/references/current-status.md`.
69 changes: 69 additions & 0 deletions packages/agentic/desktop-driver/fixtures/win32/src/fixture.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@
// --offscreen <0|1> place the main window off every monitor (default 0)
// --lifetime-ms <count> exit the process after this long even when the UI thread is
// blocked, so an abandoned run cannot linger (default 600000, 0: never)
// --child-pid-file <path> start one offscreen descendant copy of this fixture and
// write "<descendant pid> <parent pid>" to this file, so a
// contract can prove which processes an owned launch reaches
#define NOMINMAX
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
Expand All @@ -19,6 +22,7 @@

#include <commctrl.h>
#include <shellapi.h>
#include <tlhelp32.h>
#include <uiautomation.h>

#include <algorithm>
Expand Down Expand Up @@ -57,6 +61,7 @@ struct Options {
DWORD stuckMilliseconds = 0;
DWORD lifetimeMilliseconds = 600000;
bool offscreen = false;
std::wstring childPidFile;
};

Options options;
Expand Down Expand Up @@ -89,6 +94,8 @@ Options ParseOptions() {
parsed.lifetimeMilliseconds = static_cast<DWORD>(std::max(0, _wtoi(value)));
} else if (name == L"--offscreen") {
parsed.offscreen = _wtoi(value) != 0;
} else if (name == L"--child-pid-file") {
parsed.childPidFile = value;
}
}
LocalFree(arguments);
Expand Down Expand Up @@ -397,6 +404,65 @@ void StartWatchdog() {
}).detach();
}

DWORD ReadParentProcessId() {
const HANDLE snapshot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
if (snapshot == INVALID_HANDLE_VALUE) {
return 0;
}
DWORD parent = 0;
PROCESSENTRY32W entry{};
entry.dwSize = sizeof(entry);
for (BOOL more = Process32FirstW(snapshot, &entry); more != FALSE; more = Process32NextW(snapshot, &entry)) {
if (entry.th32ProcessID == GetCurrentProcessId()) {
parent = entry.th32ParentProcessID;
break;
}
}
CloseHandle(snapshot);
return parent;
}

// Starts one offscreen copy of this fixture with the default title, then
// publishes its process id and this process's parent id through a rename, so a
// reader never sees a partial file. Returns false when either step fails.
bool StartDescendant() {
if (options.childPidFile.empty()) {
return true;
}
wchar_t modulePath[MAX_PATH * 4] = {};
const DWORD length = GetModuleFileNameW(nullptr, modulePath, static_cast<DWORD>(std::size(modulePath)));
if (length == 0 || length >= std::size(modulePath)) {
return false;
}
std::wstring commandLine = L"\"" + std::wstring(modulePath, length) +
L"\" --offscreen 1 --lifetime-ms " + std::to_wstring(options.lifetimeMilliseconds);
STARTUPINFOW startup{};
startup.cb = sizeof(startup);
PROCESS_INFORMATION information{};
if (CreateProcessW(modulePath, commandLine.data(), nullptr, nullptr, FALSE, 0, nullptr, nullptr, &startup,
&information) == FALSE) {
return false;
}
CloseHandle(information.hThread);
CloseHandle(information.hProcess);

const std::string contents =
std::to_string(information.dwProcessId) + " " + std::to_string(ReadParentProcessId()) + "\n";
const std::wstring temporary = options.childPidFile + L".tmp";
const HANDLE file = CreateFileW(temporary.c_str(), GENERIC_WRITE, 0, nullptr, CREATE_ALWAYS,
FILE_ATTRIBUTE_NORMAL, nullptr);
if (file == INVALID_HANDLE_VALUE) {
return false;
}
DWORD written = 0;
const bool complete = WriteFile(file, contents.data(), static_cast<DWORD>(contents.size()), &written, nullptr) !=
FALSE &&
written == contents.size();
CloseHandle(file);
return complete &&
MoveFileExW(temporary.c_str(), options.childPidFile.c_str(), MOVEFILE_REPLACE_EXISTING) != FALSE;
}

} // namespace

int WINAPI wWinMain(HINSTANCE instance, HINSTANCE, PWSTR, int) {
Expand All @@ -417,6 +483,9 @@ int WINAPI wWinMain(HINSTANCE instance, HINSTANCE, PWSTR, int) {
return 1;
}
ShowWindow(window, SW_SHOWNOACTIVATE);
if (!StartDescendant()) {
return 3;
}

MSG message{};
while (GetMessageW(&message, nullptr, 0, 0) > 0) {
Expand Down
11 changes: 11 additions & 0 deletions packages/agentic/desktop-driver/native/PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,17 @@ application descriptors may include an existing absolute `workingDirectory`
for unpackaged process launch; packaged activation rejects that override.
These fields are trusted supervisor configuration, not WebDriver capabilities.

A `launch` or `attach` result is the application lease `{ id, ownership,
processId, processStartedAt }`. The Windows provider adds `containment`, which
states how far its ownership reaches: `job-object` for an unpackaged launch it
started suspended inside a Job Object of its own, so closing the lease or the
helper exiting ends the application and every descendant; `uncontained` for a
packaged activation, where it owns only the identity-verified process whose
window it leased (the activated process or a verified descendant) and no job;
and `external` for an attached application it never ends. A provider that omits
`containment` makes no containment claim. The field is additive and does not
change the wire version.

Element snapshots can additionally include `required` as a supported Boolean
or an unsupported value with its reason. Older or non-supporting providers may
omit it; the Node boundary treats absence as unsupported. This is additive
Expand Down
54 changes: 47 additions & 7 deletions packages/agentic/desktop-driver/native/windows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,18 +86,41 @@ remain unsupported.

Attach accepts an owner-generated nonce-bound lease, or an exact title
qualified by the canonical executable or AUMID. Title-only attach, zero
matches, and multiple qualified matches fail.
Leases record ownership, PID, process start, executable, and the primary window.
Attached apps survive WebDriver deletion; launched apps are closed only by
their exact recorded process identity.
matches, and multiple qualified matches fail. Attach fails closed: a lease's
process must be in the helper's session and match the lease's executable or
AUMID, and an unleased attach fails as ambiguous when any titled candidate's
identity cannot be read rather than skipping it.
Leases record ownership, PID, process start, executable, the primary window,
and `containment`. Attached apps survive WebDriver deletion and helper
disposal (`external`); launched apps are closed only by their exact recorded
process identity.

Unpackaged arguments use the standard `CommandLineToArgvW` quoting rules.
Launch ownership requires a non-preexisting window from the exact created
process or a verified descendant, plus a matching canonical executable or
AUMID and a process start after the launch request. Reused packaged processes
fail with guidance to use attach mode. Cancellation, launch failure, session
cleanup, and helper disposal close only causally owned processes and retain
the lease when forced termination cannot be proven.
cleanup, and helper disposal close only owned processes and retain the lease
when forced termination cannot be proven.

Launch and attach require the helper's window station to be interactive. An
unpackaged launch is containment `job-object`: the helper creates the process
suspended, assigns it to a Job Object of its own with
`JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` and no breakaway, and only then resumes
it, so every descendant starts inside the job. A failed assignment ends the
suspended process and fails the launch as unsupported instead of running it
uncontained. Window discovery then accepts only windows of job members in the
helper's session. Closing the lease asks the root window to close, then ends
the job and waits up to 3 seconds for it to drain; an undrained job fails
`cleanup-failed` and keeps the lease so the close can be retried. If the helper
exits without closing, the operating system ends the job. Descendants are ended
by the job, not asked to close, and an application that starts children with
`CREATE_BREAKAWAY_FROM_JOB` fails inside the job. A packaged activation is
containment `uncontained`: the Application Activation Manager starts it outside
any job the helper owns, so closing the lease ends only the identity-verified
process whose window the helper leased (the activated process or a verified
descendant); other processes the application starts are not ended.
The helper never places an attached application in a job.

## Automation behavior

Expand Down Expand Up @@ -144,6 +167,14 @@ already active, as macOS does for application activation. Reactivating the
owner would light-dismiss an open popup with input that no user produced. A
physical press on the owner still activates it.

Activation shares the input queue of whichever thread holds the foreground,
moving to the new holder when the foreground changes during the attempt. When
Windows still keeps the foreground away, `window-activation-failed` names the
holder by opaque window handle and process id, never by title or class, and the
first matching cause: no foreground window, another window of the target
application, an elevated or unreadable holder, a holder that is not responding,
a refused input-queue attachment, or a holder that took the foreground back.

A subtree walk retries while its root remains available if a descendant
becomes unavailable, or if a closing popup island's provider reports a UI
Automation timeout or an unspecified failure. Only an unavailable root is a
Expand Down Expand Up @@ -483,7 +514,9 @@ handshakes, runs the native self-test, exercises malformed-frame recovery, and
verifies shared Windows/Win32 selection. A second suite builds the test-only
Win32 fixture (`fixtures/win32` in the repository, not published) and drives it
through the helper: window events, owned popups, hung and stuck providers, large
trees, and bounded retention. `yarn test --runInBand win32Fixture` runs only
trees, bounded retention, and application containment (a contained launch whose
descendant ends on close and when the helper is killed, and an external fixture
that survives attach, close, and disposal). `yarn test --runInBand win32Fixture` runs only
that suite. A third suite serves the same fixture through a grant-protected
Desktop Driver service with the helper behind it: unauthenticated requests reach
no helper call, sessions keep their owner, a revoked grant closes its
Expand Down Expand Up @@ -564,6 +597,13 @@ attached app survived. Shutdown used the CLI's stop handler through a
test-only IPC bridge, not a physical console Ctrl+C. These fixture results
do not qualify either React Native renderer's new agent-operation path.

Application containment passed the fixture contract and two launch probes
through `desktop-test serve --agent`: when the helper was killed, a REX
FluentTester launch on Win32 Paper (`job-object`) ended with it, and the
packaged Storybook app on Windows Fabric (`uncontained`) survived. The
foreground-refusal causes are self-tested only; a live Windows Fabric
physical click has not been rerun.

CI must build this helper explicitly into a job-local cache, run the opt-in
native contract once for the shared Windows/Win32 provider, verify each
endpoint with `desktop-driver doctor`, and set
Expand Down
Loading
Loading