diff --git a/.changeset/brave-popup-listings.md b/.changeset/brave-popup-listings.md new file mode 100644 index 0000000000..8ba2e2e657 --- /dev/null +++ b/.changeset/brave-popup-listings.md @@ -0,0 +1,5 @@ +--- +'@fluentui-react-native/desktop-driver': patch +--- + +Keep Windows and Win32 window listings available when a secondary window closes between discovery and native geometry serialization, without hiding failures for live windows. diff --git a/.changeset/native-callout-window-command-identities.md b/.changeset/native-callout-window-command-identities.md new file mode 100644 index 0000000000..5cddcd2c12 --- /dev/null +++ b/.changeset/native-callout-window-command-identities.md @@ -0,0 +1,5 @@ +--- +'@fluentui-react-native/native-core': patch +--- + +Give the legacy Callout window-command story's focus and blur buttons stable test identities for unique secondary-window input qualification. diff --git a/.changeset/quiet-menu-windows.md b/.changeset/quiet-menu-windows.md new file mode 100644 index 0000000000..23a69f8901 --- /dev/null +++ b/.changeset/quiet-menu-windows.md @@ -0,0 +1,5 @@ +--- +'@fluentui-react-native/desktop-driver': patch +--- + +Add a session-scoped, non-activating window tree query so consumers can distinguish legitimate secondary menus from unexpected dialogs without dismissing the menu. diff --git a/.changeset/windows-connected-session-qualification.md b/.changeset/windows-connected-session-qualification.md new file mode 100644 index 0000000000..75b2dfefdc --- /dev/null +++ b/.changeset/windows-connected-session-qualification.md @@ -0,0 +1,10 @@ +--- +"@fluentui-react-native/desktop-driver": patch +"@fluentui-react-native/desktop-driver-mcp": patch +"@fluentui-react-native/desktop-test": patch +"@fluentui-react-native/storybook-desktop": patch +--- + +Record Windows Fabric and Win32 Paper connected-session qualification, +physical MCP click costs, explicit untested display scenarios, and the +remaining secondary-window semantic activation and registration gaps. diff --git a/.changeset/windows-connected-session-refusals.md b/.changeset/windows-connected-session-refusals.md new file mode 100644 index 0000000000..2d8127caff --- /dev/null +++ b/.changeset/windows-connected-session-refusals.md @@ -0,0 +1,9 @@ +--- +'@fluentui-react-native/desktop-driver': patch +--- + +Refuse Windows legacy-only semantic activation before dispatch, preserving machine-readable reasons without recording a delivery or implicitly retrying with physical input. + +Register untitled Windows secondary tool popups only when their exact-process ownership and bounded UIA content are verified, retaining helper-window exclusions. + +Keep admitted popups listable through transient provider failures, share one bounded discovery deadline across new candidates, and include admitted popups in publication diagnostics. diff --git a/apps/E2E/README.md b/apps/E2E/README.md index 686c456e29..94270c34e0 100644 --- a/apps/E2E/README.md +++ b/apps/E2E/README.md @@ -138,6 +138,17 @@ Native setup retains initial window bounds: it does **not** emulate the old maximize hook. Popup lookup remains scoped to the selected window. +On Windows and Win32, the assertion-dialog safety check inspects each secondary +window without selecting or activating it, so it cannot dismiss an open menu. +Only a single accessibility `menu` root without nested dialogs or windows is +recognized as an expected popup. A tree read reporting `no such window` is +treated as a closed popup. Empty or unrecognized roots fail the check only +when a fresh window listing still includes that handle, avoiding false +assertions while a menu closes. Every other tree-read error still fails the +test rather than being treated as no assertion. +The native-task-library bypass, Android page check, and other runners' existing +window-count check are unchanged. + WebdriverIO 10 throws a `StrictSelectorError` when a selector matches several elements, where WebdriverIO 9 took the first match. A native tree can expose several elements for one `testID` (a container and its children), and the page diff --git a/apps/E2E/desktop-adapters.test.cjs b/apps/E2E/desktop-adapters.test.cjs index 84f39b1f03..0a3f5d50f9 100644 --- a/apps/E2E/desktop-adapters.test.cjs +++ b/apps/E2E/desktop-adapters.test.cjs @@ -1,13 +1,162 @@ const assert = require('node:assert/strict'); -const { afterEach, test } = require('node:test'); +const { afterEach, mock, test } = require('node:test'); const { BasePage } = require('./lib/common/BasePage.js'); const { Keys } = require('./lib/common/consts.js'); const { installLegacyDesktopAttributes, scrollNativeDesktopElement, sendNativeDesktopKeys } = require('./lib/common/nativeDesktop.js'); afterEach(() => { + mock.restoreAll(); delete globalThis.browser; }); +test('Windows assertion checks inspect menu windows without selecting or activating them', async () => { + const reads = []; + globalThis.browser = { + capabilities: { browserName: 'furn-native-desktop', platformName: 'windows', 'furn:endpoint': 'win32' }, + options: { protocol: 'http', hostname: '127.0.0.1', port: 1, path: '/' }, + sessionId: 'session', + getWindowHandles: async () => ['main', 'menu'], + getWindowHandle: async () => 'main', + switchToWindow: () => assert.fail('an assertion check must not change native focus'), + }; + mock.method(globalThis, 'fetch', async (url, init) => { + reads.push(String(url)); + assert.equal(init.method, 'GET'); + return new Response(JSON.stringify({ value: [{ role: 'menu', children: [{ role: 'menuitem', children: [] }] }] })); + }); + assert.equal(await new BasePage().didAssertPopup(), false); + // eslint-disable-next-line @microsoft/sdl/no-insecure-url -- mocked loopback transport; no service is contacted + assert.deepEqual(reads, ['http://127.0.0.1:1/session/session/furn/window/menu/tree']); + browser.getWindowHandles = async () => ['main']; + browser.getWindowHandle = () => assert.fail('one window needs no further inspection'); + assert.equal(await new BasePage().didAssertPopup(), false); + assert.equal(reads.length, 1); +}); + +test('Windows assertion checks fail closed for dialogs, unrecognized roots, and unreadable windows', async () => { + globalThis.browser = { + capabilities: { browserName: 'furn-native-desktop', platformName: 'windows', 'furn:endpoint': 'windows' }, + options: { protocol: 'http', hostname: '127.0.0.1', port: 1, path: '/' }, + sessionId: 'session', + getWindowHandles: async () => ['main', 'other'], + getWindowHandle: async () => 'main', + switchToWindow: () => assert.fail('an assertion check must not change native focus'), + }; + let value; + const reader = mock.method(globalThis, 'fetch', async () => new Response(JSON.stringify({ value }))); + const page = new BasePage(); + for (const roots of [ + [{ role: 'window', name: 'Assertion failed', children: [] }], + [{ role: 'group', children: [] }], + [], + [{ role: 'menu', children: [{ role: 'dialog', children: [] }] }], + [{ role: 'menu', children: [{ role: 'window', children: [] }] }], + [ + { role: 'menu', children: [] }, + { role: 'dialog', children: [] }, + ], + ]) { + value = roots; + assert.equal(await page.didAssertPopup(), true); + } + reader.mock.mockImplementation(async () => { + throw new Error('provider is unavailable'); + }); + await assert.rejects(page.didAssertPopup(), /provider is unavailable/); +}); + +test('Windows assertion checks do not let an open menu hide another unexpected window', async () => { + globalThis.browser = { + capabilities: { browserName: 'furn-native-desktop', platformName: 'windows', 'furn:endpoint': 'win32' }, + options: { protocol: 'http', hostname: '127.0.0.1', port: 1, path: '/' }, + sessionId: 'session', + getWindowHandles: async () => ['main', 'menu', 'assert'], + getWindowHandle: async () => 'main', + }; + mock.method( + globalThis, + 'fetch', + async (url) => + new Response( + JSON.stringify({ + value: [{ role: String(url).endsWith('/menu/tree') ? 'menu' : 'window', children: [] }], + }), + ), + ); + assert.equal(await new BasePage().didAssertPopup(), true); +}); + +test('Windows assertion checks skip a vanished menu and continue checking other windows', async () => { + let handles = ['main', 'closing-menu']; + globalThis.browser = { + capabilities: { browserName: 'furn-native-desktop', platformName: 'windows', 'furn:endpoint': 'win32' }, + options: { protocol: 'http', hostname: '127.0.0.1', port: 1, path: '/' }, + sessionId: 'session', + getWindowHandles: async () => handles, + getWindowHandle: async () => 'main', + switchToWindow: () => assert.fail('an assertion check must not change native focus'), + }; + mock.method(globalThis, 'fetch', async (url) => + String(url).endsWith('/closing-menu/tree') + ? new Response(JSON.stringify({ value: { error: 'no such window', message: 'The menu closed.' } }), { status: 404 }) + : new Response(JSON.stringify({ value: [{ role: 'window', name: 'Assertion failed', children: [] }] })), + ); + const page = new BasePage(); + assert.equal(await page.didAssertPopup(), false); + handles = ['main', 'closing-menu', 'assert']; + assert.equal(await page.didAssertPopup(), true); +}); + +test('Windows assertion checks recheck an empty or unrecognized root before reporting an assertion', async () => { + let lists = 0; + globalThis.browser = { + capabilities: { browserName: 'furn-native-desktop', platformName: 'windows', 'furn:endpoint': 'win32' }, + options: { protocol: 'http', hostname: '127.0.0.1', port: 1, path: '/' }, + sessionId: 'session', + getWindowHandles: async () => (++lists === 1 ? ['main', 'closing-menu'] : ['main']), + getWindowHandle: async () => 'main', + switchToWindow: () => assert.fail('an assertion check must not change native focus'), + }; + let value; + mock.method(globalThis, 'fetch', async () => new Response(JSON.stringify({ value }))); + const page = new BasePage(); + for (const roots of [[], [{ role: 'group', children: [] }], [{ role: 'menu', children: [{ role: 'dialog', children: [] }] }]]) { + lists = 0; + value = roots; + assert.equal(await page.didAssertPopup(), false); + } +}); + +test('Windows assertion checks do not treat other provider errors as closed windows', async () => { + globalThis.browser = { + capabilities: { browserName: 'furn-native-desktop', platformName: 'windows', 'furn:endpoint': 'win32' }, + options: { protocol: 'http', hostname: '127.0.0.1', port: 1, path: '/' }, + sessionId: 'session', + getWindowHandles: async () => ['main', 'unreadable-menu'], + getWindowHandle: async () => 'main', + }; + let code; + mock.method( + globalThis, + 'fetch', + async () => new Response(JSON.stringify({ value: { error: code, message: 'The provider could not read the tree.' } }), { status: 500 }), + ); + for (const error of ['unknown error', 'stale element reference']) { + code = error; + await assert.rejects(new BasePage().didAssertPopup(), { code: error }); + } +}); + +test('assertion checks retain non-native and macOS window-count behavior', async () => { + for (const capabilities of [ + { browserName: 'non-native' }, + { browserName: 'furn-native-desktop', platformName: 'macos', 'furn:endpoint': 'macos' }, + ]) { + globalThis.browser = { capabilities, getWindowHandles: async () => ['main', 'other'] }; + assert.equal(await new BasePage().didAssertPopup(), true); + } +}); + test('native checkbox and switch state never falls back to selected', async () => { const page = new BasePage(); const element = { elementId: 'toggle', isSelected: () => assert.fail('checked is not selected') }; diff --git a/apps/E2E/src/common/BasePage.ts b/apps/E2E/src/common/BasePage.ts index 77f7538c5a..f40b2377f2 100644 --- a/apps/E2E/src/common/BasePage.ts +++ b/apps/E2E/src/common/BasePage.ts @@ -12,7 +12,7 @@ import { TESTPAGE_BUTTONS_SCROLLVIEWER, TESTPAGE_CONTENT_SCROLLVIEWER, } from './consts.js'; -import { isNativeDesktop, scrollNativeDesktopElement, sendNativeDesktopKeys } from './nativeDesktop.js'; +import { hasUnexpectedNativeDesktopWindow, isNativeDesktop, scrollNativeDesktopElement, sendNativeDesktopKeys } from './nativeDesktop.js'; import type { PageElement } from './nativeDesktop.js'; const DUMMY_CHAR = ''; @@ -440,9 +440,9 @@ export abstract class BasePage { * WHY/HOW: Unfortunately, some Asserts don't take focus away from the test app. Because of this, the test execution is able * to continue without realizing an assert as popped up (unless the assert is a crashing one). * - * In order to detect an assert, this function gets the number of window handles currently open within the test app. - * Only one instance SHOULD be open at a time (the app itself). If another instance of the app is open, we know an assert dialogue - * has popped up. With this information, we know to fail the test. + * Unexpected secondary windows can indicate a non-focusing assert. Windows native menus are legitimate secondary + * windows, so inspect their accessibility roots without changing focus. Unrecognized windows still fail the check, + * and an unreadable window fails the test rather than silently claiming no assert. * * Additionally, you might think it's logical to simply place this in the afterEach() hook so it's called after every test (rather than duplicating this call in every test). * Unfortunately, afterEach() is designed for setup/teardown - not for determining if a test should fail or not. @@ -456,8 +456,10 @@ export abstract class BasePage { if (PLATFORM === 'android') { return !(await this.isPageLoaded()); } - // If more than 1 instance of the app is open, we know an assert dialogue popped up. const windowHandles = await browser.getWindowHandles(); + if (isNativeDesktop() && browser.capabilities.platformName === 'windows') { + return hasUnexpectedNativeDesktopWindow(windowHandles); + } return windowHandles.length > 1; } diff --git a/apps/E2E/src/common/nativeDesktop.ts b/apps/E2E/src/common/nativeDesktop.ts index 890cdd91a2..0ca3a103d5 100644 --- a/apps/E2E/src/common/nativeDesktop.ts +++ b/apps/E2E/src/common/nativeDesktop.ts @@ -1,3 +1,5 @@ +import { createDesktopDriverClient, DesktopSessionClient, WebDriverError } from '@fluentui-react-native/desktop-driver'; +import type { DesktopTreeNode } from '@fluentui-react-native/desktop-driver'; import { Keys } from './consts.js'; /** A page object getter, or the element it has already resolved to. */ @@ -7,6 +9,42 @@ export function isNativeDesktop(): boolean { return browser.capabilities.browserName === 'furn-native-desktop'; } +export async function hasUnexpectedNativeDesktopWindow(handles: readonly string[]): Promise { + if (handles.length <= 1) return false; + const current = await browser.getWindowHandle(); + const url = `${browser.options.protocol}://${browser.options.hostname}:${browser.options.port}${browser.options.path}`; + const session = new DesktopSessionClient( + createDesktopDriverClient({ url }), + browser.sessionId, + Object.fromEntries(Object.entries(browser.capabilities)), + ); + for (const handle of handles) { + if (handle === current) continue; + // Switching windows would activate the target and can dismiss the menu being tested. + let roots: DesktopTreeNode[]; + try { + roots = await session.getWindowTree(handle); + } catch (error) { + if (error instanceof WebDriverError && error.code === 'no such window') continue; + throw error; + } + // A closing popup can lose its content before its window disappears. + if (!isExpectedMenuTree(roots) && (await browser.getWindowHandles()).includes(handle)) return true; + } + return false; +} + +function isExpectedMenuTree(roots: readonly DesktopTreeNode[]): boolean { + if (roots.length !== 1 || roots[0].role !== 'menu') return false; + const pending = [...roots]; + while (pending.length > 0) { + const node = pending.pop()!; + if (node.role === 'dialog' || node.role === 'window') return false; + pending.push(...node.children); + } + return true; +} + export async function sendNativeDesktopKeys(element: PageElement, keys: readonly Keys[]): Promise { const target = await element; await browser.elementSendKeys(await target.elementId, ''); diff --git a/packages/agentic/desktop-driver-mcp/references/current-status.md b/packages/agentic/desktop-driver-mcp/references/current-status.md index 397a16d21f..6eae382e42 100644 --- a/packages/agentic/desktop-driver-mcp/references/current-status.md +++ b/packages/agentic/desktop-driver-mcp/references/current-status.md @@ -341,6 +341,101 @@ retirement. No raw image or grant is retained in this reference. Driver shutdown, app cleanup, and the failed Win32 app-run reports are recorded in `packages/agentic/desktop-test/references/current-status.md`. +## Windows Fabric and Win32 Paper — #4441 physical MCP click measurement — 2026-11-12 UTC + +**Observed:** revision `505a2187b0a2451aed7081c9260b343d55742f93`, +Windows 11 Enterprise build 26300 x64, Node 26.13.0. Packaged Windows +Fabric reused the native build at `bdeb0b291` after confirming no native +implementation differences; Win32 Paper used REX 0.81.1. Metro/catalogs/ +bundles came from the tested checkout. Each run held the W3 campaign and +machine lock with `-RequirePhysical`. Before/after gates were eligible, +unlocked **remote RDP**, one 1728 x 1084 monitor at origin zero, 96 DPI / +100%; this was not console-session qualification. + +### Method and missing click cost + +Real MCP SDK **1.32.2** stdio connection through the private descriptor and +native-verified service; tokenizer **gpt-tokenizer 4.1.0 / o200k_base**. +After selecting `components-checkbox--default`, each endpoint made six +actual `click` actions, each followed by a checked-state assertion and an +**unscoped default observe**. Every receipt was applied/physical-pointer; +no semantic fallback counted. All six state assertions passed and the +checkbox ended unchecked. Each connection made 23 tool calls, zero error +responses/readiness rejections, and closed its session. These runs did not +exercise retries or cancellation. + +Discovery serialized the six-tool array using the November 8 baseline method: +**4,691 tokens / 20,277 UTF-8 bytes (characters)**, once per connection. +The full `tools/list` result envelope was separately 4,693 tokens / 20,287 +bytes; server instructions added 120 tokens / 591 bytes. Discovery has not +been omitted from the accounting. + +| Measurement, six samples per endpoint | Windows Fabric | Win32 Paper | +| ------------------------------------------------------ | -------------------------------------- | -------------------------------------- | +| Click response text, all blocks | **99–106 tokens / 313–314 bytes** | **96–103 tokens / 311–312 bytes** | +| Click request JSON, separately | 56–63 tokens / 186 bytes | 57–64 tokens / 186 bytes | +| Request + response-text pair | 155–169 tokens / 499–500 bytes | 153–167 tokens / 497–498 bytes | +| Click RPC p50 | **299.107 ms** | **308.003 ms** | +| Click RPC p90 / max | 369.976 ms | 318.208 ms | +| Click + checked check + default observe, response text | 1,999–2,005 tokens / 5,537–5,546 bytes | 1,585–1,592 tokens / 5,302–5,309 bytes | +| Same three RPCs, summed elapsed time | 411.849–550.601 ms | 431.774–473.477 ms | +| Default observe completeness | Incomplete: maxNodes, 13 omitted | Complete, no truncation | +| Default observe emitted / matched / visited | 100 / 113 / 115 | 99 / 99 / 124 | + +Response text blocks were independently encoded; request counts encode +user-facing `JSON.stringify({name, arguments})`, **not** JSON-RPC wire or +model-billing totals. Opaque identifiers cause normal token variation. +Percentiles use nearest rank; with six samples p90 is the maximum. +`rpcElapsedMs` stops immediately after SDK `callTool` resolves, before +tokenization, excluding service/app startup and discovery. Raw artifacts +also retain encoding-inclusive elapsed time: p50/max 299.192/370.036 ms +(Fabric), 308.128/318.281 ms (Paper). Earlier encoding-inclusive samples +were not silently substituted for this RPC distribution. + +Default budgets on both were deadline 3000 ms, maxBytes 12288, maxDepth 24, +maxNodes 100, maxStringLength 120, maxVisited 1500. No reduced scope or +budget was used to claim a saving. These six local samples complete the +previously missing physical-click row; they are not a controlled speedup +comparison with November 8's `activate` tuple or a platform ranking. + +### Commands, evidence and qualification limits + +Exact supervisor invocations from the qualification root, inside +`with-native-lock.ps1 -Lane W3 -RequirePhysical`: + +```powershell +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows --mcp-only --with-secondary +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs win32 --mcp-only +``` + +The retained supervisor owns declared Storybook driver/bundle/run commands, +exact app identity, MCP child connection and cleanup. It is local +qualification tooling, not a new published CLI surface. Artifacts are +relative to that checkout: +`artifacts\desktop\w3-qualification\runs\\mcp-click-.json`. + +| Endpoint | Qualification ID | Driver run / MCP session | Physical session gates, UTC | +| -------- | -------------------------------------- | ------------------------------------------------------------------------------- | --------------------------- | +| Fabric | `f188fae5-67c9-47e4-9800-ea584185fed9` | `57d4f6cb-2dc3-4b00-bb95-4331efd8d34b` / `b739a296-4647-4c34-bb80-a8f898fddab3` | 15:15:13.606–15:15:47.712 | +| Paper | `6d474fd4-61cb-41f6-824e-7a80df68db45` | `918930ba-e498-4e20-ad98-52e4187b5618` / `6df40ce1-32db-44d1-a9ba-64cc2dc6ec90` | 15:14:30.979–15:14:57.843 | + +Both MCP cases passed independently. Fabric's enclosing run also retained +a **failed** secondary semantic-effect assertion; its overall exit 1 is +not relabeled passed. A later exact-secondary physical click passed, but +semantic activation reproduced applied/accessibility with actual status +`Idle`. Win32's visible untitled tool-window popup was not separately +registered. These are **Observed** open gaps being fixed on this branch; +accessibility-pattern dispatch and the untitled-toolwindow registration +filter are **Inferred** causes, not proven fixes. + +200% was **not run** (100% scale; changes unauthorized), mixed DPI was +**not run** (one monitor), negative-origin layout was **not run** (only an +origin-zero monitor), and physical console Ctrl+C was **not run** (separate +implementation owner). Full per-case native results are in +`@fluentui-react-native/desktop-test/references/current-status.md`. +Sessions, applications and owned services drained; the campaign ended and +pre/post-end audits passed: +`FURN-AUDIT: no FURN desktop processes found`. ## F15 macOS Paper measurement prerequisite — 2026-11-12 Source `505a2187b0a2451aed7081c9260b343d55742f93`; macOS 27.2.1 diff --git a/packages/agentic/desktop-driver/fixtures/win32/README.md b/packages/agentic/desktop-driver/fixtures/win32/README.md index 52b2b7da62..c81f202c4f 100644 --- a/packages/agentic/desktop-driver/fixtures/win32/README.md +++ b/packages/agentic/desktop-driver/fixtures/win32/README.md @@ -20,8 +20,8 @@ Remove-Item Env:\FURN_NATIVE_DRIVER_TEST ``` Without `FURN_NATIVE_DRIVER_TEST=1` the suite is skipped. Use `--runInBand`: the scenarios share one desktop and one -build cache, and each starts its own helper process because the mutation quarantine is process-wide. A full run takes -about one minute. The onscreen scenario needs an unlocked interactive desktop for hit testing and capture; every other +build cache, and each starts its own helper process because the mutation quarantine is process-wide. Hang scenarios +include bounded provider-recovery waits. The onscreen scenario needs an unlocked interactive desktop for hit testing and capture; every other scenario places the window off every monitor. `yarn test --runInBand protectedService` runs the protected-service suite, and `yarn test --runInBand operations.contract` runs the agent-operation suite described below. The latter activates the fixture and sends real @@ -29,14 +29,16 @@ pointer and keyboard input. ## Options -| Option | Default | Effect | -| ----------------------- | -------------------- | ---------------------------------------------------------------------------------- | -| `--title ` | `FURN Win32 Fixture` | Main window title; the contract passes a unique value per scenario. | -| `--nodes ` | `0` | Adds a tree view with this many `Item N` entries. | -| `--hang-ms ` | `20000` | How long the **Hang** button blocks the UI thread. | -| `--stuck-ms ` | `0` (no control) | Adds a control whose UI Automation `Invoke` blocks this long on a provider thread. | -| `--offscreen <0\|1>` | `0` | Places the main window off every monitor. | -| `--lifetime-ms ` | `600000` | Exits after this long even when the UI thread is blocked; `0` never exits. | +| Option | Default | Effect | +| ---------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------- | +| `--title ` | `FURN Win32 Fixture` | Main window title; the contract passes a unique value per scenario. | +| `--nodes ` | `0` | Adds a tree view with this many `Item N` entries. | +| `--hang-ms ` | `20000` | How long the **Hang** button blocks the UI thread. | +| `--stuck-ms ` | `0` (no control) | Adds a control whose UI Automation `Invoke` blocks this long on a provider thread. | +| `--no-default-action <0\|1>` | `0` | Adds MSAA and passive-UIA controls that acknowledge a legacy default action without an actionable UIA pattern. | +| `--tool-windows <0\|1>` | `0` | Adds a button opening owned application-content and excluded helper-like tool roots. | +| `--offscreen <0\|1>` | `0` | Places the main window off every monitor. | +| `--lifetime-ms ` | `600000` | Exits after this long even when the UI thread is blocked; `0` never exits. | The lifetime watchdog is the only cleanup guarantee for an abandoned run: the fixture is not in a Job Object. @@ -44,18 +46,22 @@ The lifetime watchdog is the only cleanup guarantee for an abandoned run: the fi The window is shown without activation. Ids are `AutomationId` values. -| Id | Control | Behavior | -| ----------- | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | -| 1004 | Status label, initially `Ready` | Reports results: `Activated N times`, `Hang finished`, `Stuck action applied N times`. | -| 1001 | **Activate** button | Counts activations in the status label. | -| 1005 | **Rename window** button | Appends ` - renamed N` to the title; the helper window id is unchanged. | -| 1006 | **Open dialog** button | Opens the owned popup `Fixture dialog` (label 3002 `Dialog open`, button 3001 `Dialog OK`). | -| 1007 | **Hang** button | Blocks the UI thread for `--hang-ms`. | -| 1008, 1002 | `Name:` label and edit | Keyboard and `clear` target. | -| 1003 | **Enabled** checkbox | Toggle pattern target. | -| 1009 | **Stuck action** (`--stuck-ms` only) | `Invoke` blocks on a provider thread, then applies once. | -| 1010 | `Items` tree view (`--nodes` only) | `Item N` entries for tree-size and retention scenarios. | -| 2001 - 2003 | Menu bar `File` (`Increment`, `Exit`), `Help` (`About`) | Menu items in the UI Automation tree. Opening a menu popup needs activation, so popups are not qualified. | +| Id | Control | Behavior | +| ----------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| 1004 | Status label, initially `Ready` | Reports results: `Activated N times`, `Hang finished`, `Stuck action applied N times`. | +| 1001 | **Activate** button | Counts activations in the status label. | +| 1005 | **Rename window** button | Appends ` - renamed N` to the title; the helper window id is unchanged. | +| 1006 | **Open dialog** button | Opens the owned popup `Fixture dialog` (label 3002 `Dialog open`, button 3001 `Dialog OK`). | +| 1007 | **Hang** button | Blocks the UI thread for `--hang-ms`. | +| 1008, 1002 | `Name:` label and edit | Keyboard and `clear` target. | +| 1003 | **Enabled** checkbox | Toggle pattern target. | +| 1009 | **Stuck action** (`--stuck-ms` only) | `Invoke` blocks on a provider thread, then applies once. | +| 1010 | `Items` tree view (`--nodes` only) | `Item N` entries for tree-size and retention scenarios. | +| 1011 | `No default action` (`--no-default-action` only) | Advertises no default action; an attempted dispatch changes the status to `Nonactionable default dispatched`. | +| 1012 | **Open tool windows** (`--tool-windows` only) | Opens seven untitled tool roots; only the main-owned, visible, positive-size content root is eligible for registration. | +| 1013 | `No action pattern` (`--no-default-action` only) | Advertises only `ScrollIntoView`; a legacy default-action acknowledgment must not count as activation. | +| 3101, 3102 | **Tool action**, **Close tool** | Content-window activation updates the main status; closing retires the secondary identity. | +| 2001 - 2003 | Menu bar `File` (`Increment`, `Exit`), `Help` (`About`) | Menu items in the UI Automation tree. Opening a menu popup needs activation, so popups are not qualified. | The default tree has 18 nodes; `--nodes N` yields about `N + 24` in total. @@ -68,6 +74,15 @@ The default tree has 18 nodes; `--nodes N` yields about `N + 24` in total. - **Title change:** an asynchronous rename keeps the window id and publishes `EVENT_OBJECT_NAMECHANGE`. - **Owned popup:** the owner rejects clicks as `element-not-interactable` while the dialog is open; closing the dialog publishes `window-closed` and `EVENT_OBJECT_DESTROY`. +- **Legacy-only activation:** both an MSAA control without default-action metadata and a native UIA button exposing + only `ScrollIntoView` are refused before dispatch, without quarantine; an ordinary Invoke remains usable. +- **Untitled tool roots:** only visible, positive-size application content owned by the main window is registered. + Unowned roots, unrelated owner chains, empty content, hidden roots, zero-size roots, and tooltip roots stay excluded. + Publication marks the admitted popup as both listed and tracked; closing it retires its identity without affecting the main window. +- **Popup liveness under a hung provider:** an admitted popup stays listable and its rectangle readable under the same identity. + New-popup discovery remains bounded across multiple candidates, and deferred content appears after the provider recovers. + The helper self-test separately injects failed probes and an exhausted shared deadline to prove that registered popups + bypass UIA and unprobed candidates retry on the next listing. - **Hung provider (`--hang-ms 14000`):** reads and clicks fail as `automation-failed`, windows stay listable, and the helper recovers once the app responds. - **Stuck provider (`--stuck-ms 8000`):** the first click settles as `mutation-indeterminate`, later guarded mutations @@ -101,6 +116,8 @@ operation routes rather than raw helper commands: and journals without action names or typed content. - A stuck named Invoke that quarantines later operations even after its late application effect is observed exactly once. +- A no-action activation refusal preserves its machine reason, proves no effect, and leaves input receipts and + subsequent legitimate activation intact. Physical click and crop alignment have been checked at 100% and 150% scale. Other DPI configurations and React Native applications remain separate diff --git a/packages/agentic/desktop-driver/fixtures/win32/Win32Fixture.vcxproj b/packages/agentic/desktop-driver/fixtures/win32/Win32Fixture.vcxproj index c2ada0f24d..604212bfe3 100644 --- a/packages/agentic/desktop-driver/fixtures/win32/Win32Fixture.vcxproj +++ b/packages/agentic/desktop-driver/fixtures/win32/Win32Fixture.vcxproj @@ -44,7 +44,7 @@ UNICODE;_UNICODE;%(PreprocessorDefinitions) - user32.lib;shell32.lib;comctl32.lib;uiautomationcore.lib;oleaut32.lib;ole32.lib;%(AdditionalDependencies) + user32.lib;shell32.lib;comctl32.lib;uiautomationcore.lib;oleacc.lib;oleaut32.lib;ole32.lib;%(AdditionalDependencies) Windows @@ -57,7 +57,7 @@ UNICODE;_UNICODE;NDEBUG;%(PreprocessorDefinitions) - user32.lib;shell32.lib;comctl32.lib;uiautomationcore.lib;oleaut32.lib;ole32.lib;%(AdditionalDependencies) + user32.lib;shell32.lib;comctl32.lib;uiautomationcore.lib;oleacc.lib;oleaut32.lib;ole32.lib;%(AdditionalDependencies) Windows diff --git a/packages/agentic/desktop-driver/fixtures/win32/src/fixture.cpp b/packages/agentic/desktop-driver/fixtures/win32/src/fixture.cpp index 131a6f2401..084ce3c06f 100644 --- a/packages/agentic/desktop-driver/fixtures/win32/src/fixture.cpp +++ b/packages/agentic/desktop-driver/fixtures/win32/src/fixture.cpp @@ -14,12 +14,15 @@ // --child-pid-file start one offscreen descendant copy of this fixture and // write " " to this file, so a // contract can prove which processes an owned launch reaches +// --no-default-action <0|1> add an MSAA control whose no-op default method returns S_OK +// --tool-windows <0|1> add a button that opens content and non-content tool roots #define NOMINMAX #define WIN32_LEAN_AND_MEAN #include #include #include +#include #include #include @@ -31,12 +34,15 @@ #include #include #include +#include namespace { constexpr wchar_t kMainClass[] = L"FurnWin32Fixture"; constexpr wchar_t kDialogClass[] = L"FurnWin32FixtureDialog"; constexpr wchar_t kStuckClass[] = L"FurnWin32FixtureStuck"; +constexpr wchar_t kNoActionClass[] = L"FurnWin32FixtureNoAction"; +constexpr wchar_t kToolClass[] = L"FurnWin32FixtureTool"; constexpr int kActivateButton = 1001; constexpr int kNameEdit = 1002; @@ -48,12 +54,19 @@ constexpr int kHangButton = 1007; constexpr int kNameLabel = 1008; constexpr int kStuckControl = 1009; constexpr int kTree = 1010; +constexpr int kNoActionControl = 1011; +constexpr int kToolButton = 1012; +constexpr int kPassiveControl = 1013; constexpr int kMenuIncrement = 2001; constexpr int kMenuExit = 2002; constexpr int kMenuAbout = 2003; constexpr int kDialogOk = 3001; constexpr int kDialogLabel = 3002; +constexpr int kToolAction = 3101; +constexpr int kToolClose = 3102; constexpr UINT kStuckApplied = WM_APP + 1; +constexpr UINT kNoActionDispatched = WM_APP + 2; +constexpr UINT kToolApplied = WM_APP + 3; struct Options { std::wstring title = L"FURN Win32 Fixture"; @@ -62,6 +75,8 @@ struct Options { DWORD stuckMilliseconds = 0; DWORD lifetimeMilliseconds = 600000; bool offscreen = false; + bool noDefaultAction = false; + bool toolWindows = false; std::wstring childPidFile; std::wstring stuckStartedFile; }; @@ -73,6 +88,8 @@ HWND dialogWindow = nullptr; int activations = 0; int renames = 0; int stuckApplications = 0; +int toolActivations = 0; +std::vector toolWindows; Options ParseOptions() { Options parsed; @@ -100,6 +117,10 @@ Options ParseOptions() { parsed.offscreen = _wtoi(value) != 0; } else if (name == L"--child-pid-file") { parsed.childPidFile = value; + } else if (name == L"--no-default-action") { + parsed.noDefaultAction = _wtoi(value) != 0; + } else if (name == L"--tool-windows") { + parsed.toolWindows = _wtoi(value) != 0; } } LocalFree(arguments); @@ -158,9 +179,9 @@ void CreateMenuBar(HWND window) { // Automation thread, so the UI thread stays responsive, and applies its effect // only when the wait ends: a mutation whose outcome the client cannot know, // like a wedged or very slow application. -class StuckProvider : public IRawElementProviderSimple, public IInvokeProvider { +class StuckProvider : public IRawElementProviderSimple, public IInvokeProvider, public IScrollItemProvider { public: - explicit StuckProvider(HWND window) : window_(window) {} + explicit StuckProvider(HWND window) : window_(window), passive_(GetDlgCtrlID(window) == kPassiveControl) {} ULONG STDMETHODCALLTYPE AddRef() override { return ++references_; } @@ -180,6 +201,8 @@ class StuckProvider : public IRawElementProviderSimple, public IInvokeProvider { *object = static_cast(this); } else if (id == __uuidof(IInvokeProvider)) { *object = static_cast(this); + } else if (id == __uuidof(IScrollItemProvider) && passive_) { + *object = static_cast(this); } else { *object = nullptr; return E_NOINTERFACE; @@ -201,9 +224,12 @@ class StuckProvider : public IRawElementProviderSimple, public IInvokeProvider { return E_POINTER; } *provider = nullptr; - if (pattern == UIA_InvokePatternId) { + if (pattern == UIA_InvokePatternId && !passive_) { *provider = static_cast(this); AddRef(); + } else if (pattern == UIA_ScrollItemPatternId && passive_) { + *provider = static_cast(this); + AddRef(); } return S_OK; } @@ -218,10 +244,10 @@ class StuckProvider : public IRawElementProviderSimple, public IInvokeProvider { value->lVal = UIA_ButtonControlTypeId; } else if (property == UIA_NamePropertyId) { value->vt = VT_BSTR; - value->bstrVal = SysAllocString(L"Stuck action"); + value->bstrVal = SysAllocString(passive_ ? L"No action pattern" : L"Stuck action"); } else if (property == UIA_AutomationIdPropertyId) { value->vt = VT_BSTR; - value->bstrVal = SysAllocString(std::to_wstring(kStuckControl).c_str()); + value->bstrVal = SysAllocString(std::to_wstring(passive_ ? kPassiveControl : kStuckControl).c_str()); } return S_OK; } @@ -234,6 +260,10 @@ class StuckProvider : public IRawElementProviderSimple, public IInvokeProvider { } HRESULT STDMETHODCALLTYPE Invoke() override { + if (passive_) { + PostMessageW(mainWindow, kNoActionDispatched, 0, 0); + return S_OK; + } if (!options.stuckStartedFile.empty()) { const HANDLE started = CreateFileW(options.stuckStartedFile.c_str(), GENERIC_WRITE, FILE_SHARE_READ, nullptr, CREATE_NEW, FILE_ATTRIBUTE_NORMAL, nullptr); @@ -247,9 +277,12 @@ class StuckProvider : public IRawElementProviderSimple, public IInvokeProvider { return S_OK; } + HRESULT STDMETHODCALLTYPE ScrollIntoView() override { return S_OK; } + private: std::atomic references_{1}; HWND window_; + bool passive_; }; LRESULT CALLBACK StuckProc(HWND window, UINT message, WPARAM wparam, LPARAM lparam) { @@ -269,7 +302,8 @@ LRESULT CALLBACK StuckProc(HWND window, UINT message, WPARAM wparam, LPARAM lpar GetClientRect(window, &bounds); FrameRect(context, &bounds, static_cast(GetStockObject(BLACK_BRUSH))); SetBkMode(context, TRANSPARENT); - DrawTextW(context, L"Stuck action", -1, &bounds, DT_CENTER | DT_VCENTER | DT_SINGLELINE); + DrawTextW(context, GetDlgCtrlID(window) == kPassiveControl ? L"No action pattern" : L"Stuck action", -1, &bounds, + DT_CENTER | DT_VCENTER | DT_SINGLELINE); EndPaint(window, &paint); return 0; } @@ -279,6 +313,150 @@ LRESULT CALLBACK StuckProc(HWND window, UINT message, WPARAM wparam, LPARAM lpar return DefWindowProcW(window, message, wparam, lparam); } +// Keep the normal MSAA properties without an actionable pattern. The default +// action acknowledges a call without effect, exercising the driver's refusal +// of legacy-only activation. +class NoActionAccessible : public IAccessible { + public: + explicit NoActionAccessible(IAccessible* standard) : standard_(standard) {} + ~NoActionAccessible() { standard_->Release(); } + + ULONG STDMETHODCALLTYPE AddRef() override { return ++references_; } + ULONG STDMETHODCALLTYPE Release() override { + const ULONG remaining = --references_; + if (remaining == 0) { + delete this; + } + return remaining; + } + HRESULT STDMETHODCALLTYPE QueryInterface(REFIID id, void** object) override { + if (object == nullptr) return E_POINTER; + *object = nullptr; + if (id != IID_IUnknown && id != IID_IDispatch && id != IID_IAccessible) return E_NOINTERFACE; + *object = static_cast(this); + AddRef(); + return S_OK; + } + HRESULT STDMETHODCALLTYPE GetTypeInfoCount(UINT* count) override { return standard_->GetTypeInfoCount(count); } + HRESULT STDMETHODCALLTYPE GetTypeInfo(UINT index, LCID locale, ITypeInfo** info) override { + return standard_->GetTypeInfo(index, locale, info); + } + HRESULT STDMETHODCALLTYPE GetIDsOfNames(REFIID id, LPOLESTR* names, UINT count, LCID locale, DISPID* ids) override { + return standard_->GetIDsOfNames(id, names, count, locale, ids); + } + HRESULT STDMETHODCALLTYPE Invoke(DISPID member, REFIID id, LCID locale, WORD flags, DISPPARAMS* params, + VARIANT* result, EXCEPINFO* exception, UINT* argument) override { + return standard_->Invoke(member, id, locale, flags, params, result, exception, argument); + } + HRESULT STDMETHODCALLTYPE get_accParent(IDispatch** parent) override { return standard_->get_accParent(parent); } + HRESULT STDMETHODCALLTYPE get_accChildCount(long* count) override { return standard_->get_accChildCount(count); } + HRESULT STDMETHODCALLTYPE get_accChild(VARIANT child, IDispatch** result) override { + return standard_->get_accChild(child, result); + } + HRESULT STDMETHODCALLTYPE get_accName(VARIANT child, BSTR* result) override { return standard_->get_accName(child, result); } + HRESULT STDMETHODCALLTYPE get_accValue(VARIANT child, BSTR* result) override { return standard_->get_accValue(child, result); } + HRESULT STDMETHODCALLTYPE get_accDescription(VARIANT child, BSTR* result) override { + return standard_->get_accDescription(child, result); + } + HRESULT STDMETHODCALLTYPE get_accRole(VARIANT, VARIANT* result) override { + if (result == nullptr) return E_POINTER; + VariantInit(result); + result->vt = VT_I4; + result->lVal = ROLE_SYSTEM_STATICTEXT; + return S_OK; + } + HRESULT STDMETHODCALLTYPE get_accState(VARIANT child, VARIANT* result) override { return standard_->get_accState(child, result); } + HRESULT STDMETHODCALLTYPE get_accHelp(VARIANT child, BSTR* result) override { return standard_->get_accHelp(child, result); } + HRESULT STDMETHODCALLTYPE get_accHelpTopic(BSTR* file, VARIANT child, long* topic) override { + return standard_->get_accHelpTopic(file, child, topic); + } + HRESULT STDMETHODCALLTYPE get_accKeyboardShortcut(VARIANT child, BSTR* result) override { + return standard_->get_accKeyboardShortcut(child, result); + } + HRESULT STDMETHODCALLTYPE get_accFocus(VARIANT* result) override { return standard_->get_accFocus(result); } + HRESULT STDMETHODCALLTYPE get_accSelection(VARIANT* result) override { return standard_->get_accSelection(result); } + HRESULT STDMETHODCALLTYPE get_accDefaultAction(VARIANT, BSTR* result) override { + if (result == nullptr) return E_POINTER; + *result = nullptr; + return S_FALSE; + } + HRESULT STDMETHODCALLTYPE accSelect(long flags, VARIANT child) override { return standard_->accSelect(flags, child); } + HRESULT STDMETHODCALLTYPE accLocation(long* x, long* y, long* width, long* height, VARIANT child) override { + return standard_->accLocation(x, y, width, height, child); + } + HRESULT STDMETHODCALLTYPE accNavigate(long direction, VARIANT from, VARIANT* result) override { + return standard_->accNavigate(direction, from, result); + } + HRESULT STDMETHODCALLTYPE accHitTest(long x, long y, VARIANT* result) override { return standard_->accHitTest(x, y, result); } + HRESULT STDMETHODCALLTYPE accDoDefaultAction(VARIANT) override { + PostMessageW(mainWindow, kNoActionDispatched, 0, 0); + return S_OK; + } + HRESULT STDMETHODCALLTYPE put_accName(VARIANT child, BSTR value) override { return standard_->put_accName(child, value); } + HRESULT STDMETHODCALLTYPE put_accValue(VARIANT child, BSTR value) override { return standard_->put_accValue(child, value); } + + private: + std::atomic references_{1}; + IAccessible* standard_; +}; + +LRESULT CALLBACK NoActionProc(HWND window, UINT message, WPARAM wparam, LPARAM lparam) { + if (message == WM_GETOBJECT && static_cast(lparam) == OBJID_CLIENT) { + IAccessible* standard = nullptr; + if (SUCCEEDED(CreateStdAccessibleObject(window, OBJID_CLIENT, IID_IAccessible, reinterpret_cast(&standard)))) { + auto* accessible = new NoActionAccessible(standard); + const LRESULT result = LresultFromObject(IID_IAccessible, wparam, accessible); + accessible->Release(); + return result; + } + } + return DefWindowProcW(window, message, wparam, lparam); +} + +LRESULT CALLBACK ToolProc(HWND window, UINT message, WPARAM wparam, LPARAM lparam) { + if (message == WM_COMMAND) { + if (LOWORD(wparam) == kToolAction) { + PostMessageW(mainWindow, kToolApplied, 0, 0); + return 0; + } + if (LOWORD(wparam) == kToolClose) { + DestroyWindow(window); + return 0; + } + } + return DefWindowProcW(window, message, wparam, lparam); +} + +void OpenToolWindows(HWND owner) { + if (!toolWindows.empty()) return; + INITCOMMONCONTROLSEX controls{sizeof(INITCOMMONCONTROLSEX), ICC_WIN95_CLASSES}; + InitCommonControlsEx(&controls); + RECT bounds{}; + GetWindowRect(owner, &bounds); + const auto create = [&](HWND parent, bool content, bool visible, bool empty, const wchar_t* className = kToolClass) { + const HWND tool = CreateWindowExW(WS_EX_TOOLWINDOW | WS_EX_NOACTIVATE | WS_EX_TOPMOST, className, L"", + WS_POPUP | WS_BORDER, bounds.left + Scale(30, owner), bounds.top + Scale(80, owner), + empty ? 0 : Scale(280, owner), empty ? 0 : Scale(120, owner), + parent, nullptr, GetModuleHandleW(nullptr), nullptr); + if (tool == nullptr) return static_cast(nullptr); + toolWindows.push_back(tool); + if (content) { + AddChild(tool, L"BUTTON", L"Tool action", WS_TABSTOP | BS_PUSHBUTTON, kToolAction, 10, 10, 120, 28); + AddChild(tool, L"BUTTON", L"Close tool", WS_TABSTOP | BS_PUSHBUTTON, kToolClose, 10, 50, 120, 28); + } + if (visible) ShowWindow(tool, SW_SHOWNOACTIVATE); + return tool; + }; + create(owner, true, true, false); + const HWND unowned = create(nullptr, true, true, false); + create(unowned, true, true, false); + create(owner, false, true, false); + create(owner, true, false, false); + create(owner, true, true, true); + create(owner, true, true, false, TOOLTIPS_CLASSW); + SetStatus(L"Opened " + std::to_wstring(toolWindows.size()) + L" tool windows"); +} + void CreateMainControls(HWND window) { CreateMenuBar(window); statusLabel = AddChild(window, L"STATIC", L"Ready", SS_LEFT, kStatusLabel, 10, 10, 440, 20); @@ -296,6 +474,13 @@ void CreateMainControls(HWND window) { if (options.nodes > 0) { CreateTree(window); } + if (options.noDefaultAction) { + AddChild(window, kNoActionClass, L"No default action", 0, kNoActionControl, 10, 140, 200, 28); + AddChild(window, kStuckClass, L"", 0, kPassiveControl, 10, 180, 200, 28); + } + if (options.toolWindows) { + AddChild(window, L"BUTTON", L"Open tool windows", WS_TABSTOP | BS_PUSHBUTTON, kToolButton, 10, 140, 180, 28); + } } // The dialog is owned by the main window and modal in the Win32 sense: it @@ -364,6 +549,9 @@ LRESULT CALLBACK MainProc(HWND window, UINT message, WPARAM wparam, LPARAM lpara case kDialogButton: OpenDialog(window); return 0; + case kToolButton: + OpenToolWindows(window); + return 0; case kHangButton: // A busy or suspended application: nothing is pumped until this returns. Sleep(options.hangMilliseconds); @@ -380,6 +568,13 @@ LRESULT CALLBACK MainProc(HWND window, UINT message, WPARAM wparam, LPARAM lpara stuckApplications += 1; SetStatus(L"Stuck action applied " + std::to_wstring(stuckApplications) + L" times"); return 0; + case kNoActionDispatched: + SetStatus(L"Nonactionable default dispatched"); + return 0; + case kToolApplied: + toolActivations += 1; + SetStatus(L"Tool activated " + std::to_wstring(toolActivations) + L" times"); + return 0; case WM_DESTROY: PostQuitMessage(0); return 0; @@ -404,6 +599,12 @@ void RegisterClasses(HINSTANCE instance) { windowClass.lpfnWndProc = StuckProc; windowClass.lpszClassName = kStuckClass; RegisterClassExW(&windowClass); + windowClass.lpfnWndProc = NoActionProc; + windowClass.lpszClassName = kNoActionClass; + RegisterClassExW(&windowClass); + windowClass.lpfnWndProc = ToolProc; + windowClass.lpszClassName = kToolClass; + RegisterClassExW(&windowClass); } void StartWatchdog() { diff --git a/packages/agentic/desktop-driver/native/PROTOCOL.md b/packages/agentic/desktop-driver/native/PROTOCOL.md index 3759c4f6a4..87c8a9b9f3 100644 --- a/packages/agentic/desktop-driver/native/PROTOCOL.md +++ b/packages/agentic/desktop-driver/native/PROTOCOL.md @@ -380,6 +380,27 @@ events: Queries remain authoritative. Events can invalidate state or wake waits but do not replace a liveness or state query. +For Windows/Win32, an untitled tool popup needs verified native ownership and +UIA content for initial admission to its exact-process lease. Subsequent +`windows` queries retain its opaque identity using native lease/process, +thread/class, visibility, bounds, and ownership checks, without calling UIA +again. A provider timeout or transient content failure is not window closure. +All new popup-content probes share one 1,000 ms discovery deadline per +`windows` request. Unproved or unprobed candidates are omitted and can be +retried by a later request; admitted windows remain listable even if discovery +uses the whole budget. A read deadline does not quarantine mutations, and +cancellation is still honored. Publication diagnostics mark admitted live +popups as both `listed` and `tracked` without another content probe. + +If a Windows/Win32 window closes between enumeration and geometry +serialization, `windows` omits it instead of failing the remaining live +listing. This suppresses only `no-such-window` when native liveness confirms +the handle has gone; unrelated errors and failures for still-live handles +propagate. Direct commands for the closed window still fail. + +This is internal Windows admission/liveness policy: no request field, response +shape, command, or FDR1 1.0 constant in `protocol.json` changes. + Helpers that advertise the `windowEvents` feature also accept a `windowEvents` command with the target `lease` and return a bounded diagnostic log of recent window notifications and helper commands, and publish a `window-closed` event @@ -427,6 +448,22 @@ one input turn and refuses a display name the element does not publish exactly once, so a key never reaches a different action than the one the agent named. +### Windows accessibility activation + +For the shared Windows/Win32 provider, accessibility `click` requires an +actionable `Invoke`, `Toggle`, `SelectionItem`, or `ExpandCollapse` pattern. +LegacyIAccessible default-action metadata, even when nonempty, and a +`ScrollIntoView` pattern alone do not prove an activation can be dispatched. +Without an actionable pattern, the helper fails `unsupported-operation` with +`data.reason` `activation-unsupported` and effect `none` before mutation. Node +retains the native code and reason, records no delivery, and does not +implicitly retry with physical input. An explicit permitted physical request +is a separate operation; an indeterminate mutation must not be replayed. + +This is a provider dispatch-policy correction using the existing error and +effect fields. FDR1 framing, commands, and the 1.0 wire constants in +`protocol.json` are unchanged. + ### Clear mechanisms The mutating `clear` command takes `{ "elementId": "" }`. Helpers that diff --git a/packages/agentic/desktop-driver/native/windows/README.md b/packages/agentic/desktop-driver/native/windows/README.md index 35f6604a3a..46275f5053 100644 --- a/packages/agentic/desktop-driver/native/windows/README.md +++ b/packages/agentic/desktop-driver/native/windows/README.md @@ -309,6 +309,37 @@ either differs. A recycled handle of the same class on the same thread whose destroy notification was never delivered is indistinguishable. Event delivery across integrity levels is untested. +Global title discovery still excludes untitled tool-window roots. Within an +exact-process application lease, `windows` also admits an untitled tool popup +when it is visible, has positive native bounds, and its same-process owner +chain reaches the verified primary window. Known tooltip and IME helper +classes stay excluded. Initial admission requires a bounded UIA content-view +probe to verify the exact native root handle/process and find an exact-process +content child; tooltip content and unreadable content do not qualify. The +helper rechecks ownership, process, visibility, bounds, and the +primary/candidate fingerprints after the provider read before registering the +opaque window identity. + +An admitted popup remains listable through Win32 liveness checks while its +lease, process, thread, class, visibility, bounds, and owner chain still match. +It is not probed again, so a provider timeout or temporary content re-layout +cannot masquerade as window closure. All new candidates share one 1,000 ms +discovery deadline per `windows` call, including `ElementFromHandle` and content +reads; candidates not proved before that deadline are omitted and retried on a +later listing. Exhausting discovery never removes an already-admitted window. +The probe retains no element records or whole-subtree cache and never +quarantines the session; cancellation still propagates. + +If a window closes after enumeration but before its geometry is serialized, +the listing omits it and still returns the other live windows. Only a +`no-such-window` failure confirmed by native handle liveness is absorbed; +failures for live handles and unrelated errors still propagate. Direct +commands for the closed window continue to fail as `no-such-window`. + +`windowEvents.publication.windows[].listed` includes admitted tool popups +using those same native identity/ownership checks. It does not repeat UIA +discovery or report an admitted live popup as `listed: false, tracked: true`. + ### Input authority Before physical input the helper evaluates the desktop it would inject into and @@ -446,7 +477,7 @@ read of a large tree. ### Unsettled mutations UI Automation mutations (`SetFocus`, `Invoke`, `Toggle`, `Select`, -`Expand`/`Collapse`, `DoDefaultAction`, and `SetValue`) run on a bounded worker +`Expand`/`Collapse`, and `SetValue`) run on a bounded worker with a five-second deadline because a suspended or busy application can stall its UI thread indefinitely. A deadline does not cancel the provider call: the application can still apply the effect later. The helper therefore reports @@ -466,8 +497,8 @@ transaction timeouts. A cancelled read simply stops waiting. A cancelled mutation is abandoned and quarantined exactly like one that timed out, so its effect is `unknown` and the refusal says it was cancelled before it settled. -Click tries `Invoke`, `Toggle`, `SelectionItem`, `ExpandCollapse`, then -`LegacyIAccessible`. On Win32 controls `Invoke` is posted: the provider accepts +Accessibility click tries `Invoke`, `Toggle`, `SelectionItem`, then +`ExpandCollapse`. On Win32 controls `Invoke` is posted: the provider accepts the call and the application applies the effect afterwards, so callers poll for the visible result. Only a pattern the element does not implement moves on to the next one, and only an element without any of them is unsupported; a pattern @@ -475,6 +506,15 @@ query that fails otherwise fails as `automation-failed`, and one for an element that went away as `stale-element`. Named accessibility actions and a `semantic` clear classify their pattern queries the same way. +Legacy-only activation is refused before dispatch as `unsupported-operation` +with `data.reason: activation-unsupported` and effect `none`, even when the +LegacyIAccessible bridge advertises a nonempty default action. The bridge can +acknowledge `DoDefaultAction` without delivering an actionable UIA pattern. +`ScrollIntoView` alone is not activation. Node preserves the reason and native +code, records no input delivery, and never retries physically on its own. A +caller may explicitly request physical click under the normal input policy; +an indeterminate mutation must never be replayed as a fallback. + While quarantined, `click`, `clear`, `sendKeys`, `performActions`, `performAccessibilityAction`, `activate`, and `setWindowRect` fail with `target-quarantined` before reaching the diff --git a/packages/agentic/desktop-driver/native/windows/src/application.cpp b/packages/agentic/desktop-driver/native/windows/src/application.cpp index 07cdd010e8..aea5f8b5a7 100644 --- a/packages/agentic/desktop-driver/native/windows/src/application.cpp +++ b/packages/agentic/desktop-driver/native/windows/src/application.cpp @@ -52,6 +52,7 @@ constexpr unsigned int kCloseTimeoutMs = 5000; constexpr unsigned int kTerminateWaitMs = 2000; constexpr unsigned int kActivationTimeoutMs = 2000; constexpr unsigned int kWindowResponseTimeoutMs = 2000; +constexpr DWORD kWindowContentDiscoveryBudgetMs = 1000; constexpr UINT kWindowProbeMessageTimeoutMs = 200; constexpr unsigned int kWindowRestoreTimeoutMs = 500; constexpr DWORD kLaunchCleanupGraceMs = 250; @@ -71,6 +72,12 @@ struct EnumerationContext { std::vector windows; }; +bool IsVisibleSizedRoot(HWND window) { + RECT bounds{}; + return IsWindowVisible(window) != FALSE && GetAncestor(window, GA_ROOT) == window && + GetWindowRect(window, &bounds) != FALSE && !IsEmptyRect(bounds); +} + BOOL CALLBACK CollectWindow(HWND window, LPARAM parameter) { auto* context = reinterpret_cast(parameter); DWORD processId = 0; @@ -78,20 +85,13 @@ BOOL CALLBACK CollectWindow(HWND window, LPARAM parameter) { if (context->processId != 0 && processId != context->processId) { return TRUE; } - if (IsWindowVisible(window) == FALSE) { - return TRUE; - } - if (GetAncestor(window, GA_ROOT) != window) { + if (!IsVisibleSizedRoot(window)) { return TRUE; } const LONG_PTR exStyle = GetWindowLongPtrW(window, GWL_EXSTYLE); if ((exStyle & WS_EX_TOOLWINDOW) != 0 && ReadWindowTitle(window).empty()) { return TRUE; } - RECT bounds{}; - if (GetWindowRect(window, &bounds) == FALSE || IsEmptyRect(bounds)) { - return TRUE; - } context->windows.push_back(window); return TRUE; } @@ -130,6 +130,30 @@ WindowFingerprint ReadWindowFingerprint(HWND window) { return fingerprint; } +bool IsOwnedContentCandidate(HWND window, HWND primaryWindow, DWORD expectedProcessId) { + DWORD processId = 0; + GetWindowThreadProcessId(window, &processId); + if (primaryWindow == nullptr || window == primaryWindow || processId != expectedProcessId || + !IsVisibleSizedRoot(window) || (GetWindowLongPtrW(window, GWL_STYLE) & WS_POPUP) == 0) { + return false; + } + const WindowFingerprint fingerprint = ReadWindowFingerprint(window); + if (fingerprint.className.empty()) return false; + const wchar_t* const helperClasses[] = {L"tooltips_class32", L"IME", L"MSCTFIME UI", L"CiceroUIWndFrame"}; + for (const wchar_t* helperClass : helperClasses) { + if (CompareStringOrdinal(fingerprint.className.c_str(), -1, helperClass, -1, TRUE) == CSTR_EQUAL) return false; + } + HWND owner = GetWindow(window, GW_OWNER); + for (std::size_t depth = 0; owner != nullptr && depth < 32; ++depth) { + DWORD ownerProcessId = 0; + GetWindowThreadProcessId(owner, &ownerProcessId); + if (ownerProcessId != expectedProcessId) return false; + if (owner == primaryWindow) return true; + owner = GetWindow(owner, GW_OWNER); + } + return false; +} + std::string ReadFileWithRetry(const std::wstring& path, const CancellationToken& token) { for (int attempt = 0; attempt < 20; attempt += 1) { token.ThrowIfCancelled(); @@ -1153,6 +1177,19 @@ bool ApplicationManager::IsWindowClosed(const WindowRecord& record) const { return record.window == nullptr || IsWindow(record.window) == FALSE || windowEvents_.IsClosed(record.id); } +bool ApplicationManager::HasRegisteredWindowIdentity(HWND window, const ApplicationLease& lease) const { + const auto id = windowIdsByHandle_.find(window); + if (id == windowIdsByHandle_.end()) return false; + const auto record = windowsById_.find(id->second); + if (record == windowsById_.end() || record->second.leaseId != lease.id || + record->second.processId != lease.processId || IsWindowClosed(record->second)) return false; + DWORD processId = 0; + GetWindowThreadProcessId(window, &processId); + const WindowFingerprint fingerprint = ReadWindowFingerprint(window); + return processId == lease.processId && record->second.threadId == fingerprint.threadId && + record->second.className == fingerprint.className; +} + void ApplicationManager::RetireWindow(const std::string& windowId) { const auto entry = windowsById_.find(windowId); if (entry != windowsById_.end()) { @@ -1897,13 +1934,16 @@ json::Value ApplicationManager::RecoverApplication(const json::Value& params, co return exited(); } -std::vector ApplicationManager::Windows(const std::string& leaseId, const CancellationToken& token) { +std::vector ApplicationManager::Windows( + const std::string& leaseId, const CancellationToken& token, + const std::function& hasContent) { const auto entry = leases_.find(leaseId); if (entry == leases_.end()) { Fail(kErrorNoSuchLease, "The requested application lease is not held by this helper."); } ApplicationLease& lease = entry->second; token.ThrowIfCancelled(); + const ULONGLONG contentDeadline = GetTickCount64() + kWindowContentDiscoveryBudgetMs; if (!IsProcessAlive(lease.process)) { Fail(kErrorNoSuchLease, "The leased application process has exited."); } @@ -1912,11 +1952,30 @@ std::vector ApplicationManager::Windows(const std::string& leaseId, // Closed windows leave the table before the live ones register, so a handle // reused by a later window can never keep its predecessor's identifier. RetireClosedWindows(leaseId); - for (const HWND window : EnumerateTopLevelWindows(lease.processId)) { + for (const HWND window : EnumerateAllTopLevelWindows(lease.processId)) { + token.ThrowIfCancelled(); + DWORD processId = 0; + GetWindowThreadProcessId(window, &processId); + if (processId != lease.processId || !IsVisibleSizedRoot(window)) continue; + const std::wstring title = ReadWindowTitle(window); + if ((GetWindowLongPtrW(window, GWL_EXSTYLE) & WS_EX_TOOLWINDOW) != 0 && title.empty()) { + if (!HasRegisteredWindowIdentity(lease.primaryWindow, lease) || + !IsOwnedContentCandidate(window, lease.primaryWindow, lease.processId)) continue; + // Content proves initial admission, not whether an admitted window closed. + if (!HasRegisteredWindowIdentity(window, lease)) { + if (!hasContent || GetTickCount64() >= contentDeadline) continue; + const WindowFingerprint before = ReadWindowFingerprint(window); + if (!hasContent(window, lease.processId, contentDeadline)) continue; + const WindowFingerprint after = ReadWindowFingerprint(window); + if (!HasRegisteredWindowIdentity(lease.primaryWindow, lease) || + !IsOwnedContentCandidate(window, lease.primaryWindow, lease.processId) || + before.threadId != after.threadId || before.className != after.className) continue; + } + } WindowInfo info; info.window = window; info.processId = lease.processId; - info.title = ReadWindowTitle(window); + info.title = title; info.primary = window == lease.primaryWindow || (lease.primaryWindow == nullptr && info.title == lease.windowTitle); info.id = RegisterWindow(window, lease.processId, leaseId, info.primary, info.title); results.push_back(std::move(info)); @@ -2278,7 +2337,10 @@ json::Value ApplicationManager::PublicationDiagnostic(const ApplicationLease& le const WindowDescriptor descriptor = DescribeWindow(window); json::Value entry = json::Value::Object(); entry.Set("className", json::Value::String(descriptor.className)); - entry.Set("listed", json::Value::Bool(std::find(listed.begin(), listed.end(), window) != listed.end())); + const bool admittedPopup = HasRegisteredWindowIdentity(window, lease) && + HasRegisteredWindowIdentity(lease.primaryWindow, lease) && + IsOwnedContentCandidate(window, lease.primaryWindow, lease.processId); + entry.Set("listed", json::Value::Bool(std::find(listed.begin(), listed.end(), window) != listed.end() || admittedPopup)); entry.Set("owned", json::Value::Bool(descriptor.owned)); if (descriptor.hasTitle) { entry.Set("title", json::Value::String(descriptor.title)); diff --git a/packages/agentic/desktop-driver/native/windows/src/application.h b/packages/agentic/desktop-driver/native/windows/src/application.h index 0ec6232857..ce9c64b1ca 100644 --- a/packages/agentic/desktop-driver/native/windows/src/application.h +++ b/packages/agentic/desktop-driver/native/windows/src/application.h @@ -84,7 +84,9 @@ class ApplicationManager { json::Value RecoverApplication(const json::Value& params, const CancellationToken& token); void CloseApplication(const std::string& leaseId, const CancellationToken& token); void Dispose(const CancellationToken& token); - std::vector Windows(const std::string& leaseId, const CancellationToken& token); + std::vector Windows( + const std::string& leaseId, const CancellationToken& token, + const std::function& hasContent = {}); WindowInfo RequireWindow(const std::string& windowId); const ApplicationLease* FindLease(const std::string& leaseId) const; @@ -153,6 +155,7 @@ class ApplicationManager { void RetireWindow(const std::string& windowId); void RetireClosedWindows(const std::string& leaseId); bool IsWindowClosed(const WindowRecord& record) const; + bool HasRegisteredWindowIdentity(HWND window, const ApplicationLease& lease) const; bool ForegroundBelongsToLease(const ApplicationLease& lease, HWND foreground) const; json::Value PublicationDiagnostic(const ApplicationLease& lease) const; ApplicationLease& CreateLease(std::string ownership, DWORD processId, HANDLE process, HWND primaryWindow, diff --git a/packages/agentic/desktop-driver/native/windows/src/automation.cpp b/packages/agentic/desktop-driver/native/windows/src/automation.cpp index ce93d4b705..1d5376b427 100644 --- a/packages/agentic/desktop-driver/native/windows/src/automation.cpp +++ b/packages/agentic/desktop-driver/native/windows/src/automation.cpp @@ -467,11 +467,69 @@ void Automation::Initialize() { automation->get_RawViewWalker(rawWalker_.put())); CheckHresult(kErrorAutomationFailed, "Reading the UI Automation control walker", automation->get_ControlViewWalker(controlWalker_.put())); + CheckHresult(kErrorAutomationFailed, "Reading the UI Automation content walker", + automation->get_ContentViewWalker(contentWalker_.put())); CheckHresult(kErrorAutomationFailed, "Reading the UI Automation not-supported sentinel", UiaGetReservedNotSupportedValue(notSupportedValue_.put())); automation_ = automation; } +bool Automation::HasWindowContent(HWND window, DWORD processId, ULONGLONG expiresAt, const CancellationToken& token) { + token.ThrowIfCancelled(); + const ULONGLONG now = GetTickCount64(); + if (now >= expiresAt) return false; + const DWORD timeoutMs = static_cast(std::min(expiresAt - now, kProviderCallTimeoutMs)); + const auto automation = automation_; + const auto walker = contentWalker_; + const auto content = std::make_shared(false); + try { + const HRESULT result = RunBoundedProviderCall( + ProviderCall::Read, [automation, walker, window, processId, content, expiresAt]() { + if (GetTickCount64() >= expiresAt) return kUiaTimeoutResult; + // A late read must not start another transaction after the deadline. + const auto checked = [expiresAt](HRESULT result) { + return SUCCEEDED(result) && GetTickCount64() >= expiresAt ? kUiaTimeoutResult : result; + }; + winrt::com_ptr root; + HRESULT result = checked(automation->ElementFromHandle(window, root.put())); + if (FAILED(result) || !root) return FAILED(result) ? result : E_FAIL; + int rootProcessId = 0; + result = checked(root->get_CurrentProcessId(&rootProcessId)); + if (FAILED(result)) return result; + if (rootProcessId != static_cast(processId)) return S_FALSE; + UIA_HWND rootWindow{}; + result = checked(root->get_CurrentNativeWindowHandle(&rootWindow)); + if (FAILED(result)) return result; + if (rootWindow != window) return S_FALSE; + CONTROLTYPEID rootType = 0; + result = checked(root->get_CurrentControlType(&rootType)); + if (FAILED(result)) return result; + if (rootType == UIA_ToolTipControlTypeId) return S_FALSE; + winrt::com_ptr child; + result = checked(walker->GetFirstChildElement(root.get(), child.put())); + if (FAILED(result) || !child) return FAILED(result) ? result : S_FALSE; + int childProcessId = 0; + result = checked(child->get_CurrentProcessId(&childProcessId)); + if (FAILED(result)) return result; + BOOL isContent = FALSE; + result = checked(child->get_CurrentIsContentElement(&isContent)); + if (FAILED(result)) return result; + CONTROLTYPEID type = 0; + result = checked(child->get_CurrentControlType(&type)); + if (FAILED(result)) return result; + *content = childProcessId == static_cast(processId) && isContent != FALSE && + type != UIA_ToolTipControlTypeId; + return S_OK; + }, + timeoutMs, "Reading the owned window content", kErrorAutomationFailed); + token.ThrowIfCancelled(); + return SUCCEEDED(result) && *content; + } catch (const HelperError&) { + token.ThrowIfCancelled(); + return false; + } +} + winrt::com_ptr Automation::ParentOf(IUIAutomationElement* element) const { winrt::com_ptr parent; if (!rawWalker_ || FAILED(rawWalker_->GetParentElement(element, parent.put()))) { @@ -1402,7 +1460,8 @@ void Automation::AccessibilityClick(const std::string& elementId, const Cancella BoundedRead([expand, state]() { return expand->get_CurrentExpandCollapseState(state.get()); }, "Reading the expand state")); if (*state == ExpandCollapseState_LeafNode) { - Fail(kErrorUnsupported, "The element is a leaf node and cannot be activated through accessibility."); + throw HelperError(kErrorUnsupported, "The element is a leaf node and cannot be activated through accessibility.", + "activation-unsupported"); } const bool collapse = *state == ExpandCollapseState_Expanded; CheckClickResult("Expanding or collapsing the element", @@ -1410,18 +1469,11 @@ void Automation::AccessibilityClick(const std::string& elementId, const Cancella "Expanding or collapsing the element")); return; } - const auto legacyQuery = - QueryPattern(element.get(), UIA_LegacyIAccessiblePatternId); - if (RequirePatternQuery(legacyQuery, "Reading the element's LegacyIAccessible pattern")) { - const winrt::com_ptr legacy = legacyQuery.pattern; - CheckClickResult("Invoking the legacy accessibility default action", - CountedMutation([legacy]() { return legacy->DoDefaultAction(); }, - "Invoking the legacy accessibility default action")); - return; - } - Fail(kErrorUnsupported, - "The element does not implement an Invoke, Toggle, SelectionItem, ExpandCollapse, or LegacyIAccessible pattern, so an " - "accessibility click is unavailable."); + // The legacy bridge can synthesize a nonempty default action and acknowledge + // it without dispatching any actionable UIA pattern. + throw HelperError(kErrorUnsupported, + "The element does not implement an actionable Invoke, Toggle, SelectionItem, or ExpandCollapse pattern.", + "activation-unsupported"); } json::Value Automation::AccessibilityActions(const std::string& elementId, const CancellationToken& token) { diff --git a/packages/agentic/desktop-driver/native/windows/src/automation.h b/packages/agentic/desktop-driver/native/windows/src/automation.h index 163a586873..5eb48e401c 100644 --- a/packages/agentic/desktop-driver/native/windows/src/automation.h +++ b/packages/agentic/desktop-driver/native/windows/src/automation.h @@ -79,6 +79,10 @@ class Automation { void Initialize(); + // A bounded content-view probe for an already identity-checked secondary + // window. It retains no element records; unreadable content is not proven. + // expiresAt is the shared windows-call deadline in GetTickCount64 units. + bool HasWindowContent(HWND window, DWORD processId, ULONGLONG expiresAt, const CancellationToken& token); std::vector SnapshotWindowTree(const WindowContext& context, const CancellationToken& token); std::vector Find(const WindowContext& context, const std::string& rootElementId, const Selector& selector, const CancellationToken& token); @@ -158,6 +162,7 @@ class Automation { winrt::com_ptr rawViewCondition_; winrt::com_ptr rawWalker_; winrt::com_ptr controlWalker_; + winrt::com_ptr contentWalker_; winrt::com_ptr notSupportedValue_; ElementRegistry registry_; }; diff --git a/packages/agentic/desktop-driver/native/windows/src/driver.cpp b/packages/agentic/desktop-driver/native/windows/src/driver.cpp index f7a2ae79c2..cfd78388f1 100644 --- a/packages/agentic/desktop-driver/native/windows/src/driver.cpp +++ b/packages/agentic/desktop-driver/native/windows/src/driver.cpp @@ -342,6 +342,9 @@ json::Value Driver::ProbeResult(const json::Value& params) { } json::Value Driver::WindowSummary(const WindowInfo& info) { + if (windowSummaryHook_) { + windowSummaryHook_(info.window); + } const WindowMetrics metrics = MeasureWindow(info.window); json::Value value = json::Value::Object(); value.Set("id", json::Value::String(info.id)); @@ -638,8 +641,19 @@ CommandResult Driver::ExecuteCommand(const std::string& command, const json::Val Fail(kErrorInvalidParams, "The windows request is missing its lease."); } result.result = json::Value::Array(); - for (const WindowInfo& info : applications_.Windows(lease->StringField("id"), token)) { - result.result.Append(WindowSummary(info)); + for (const WindowInfo& info : applications_.Windows( + lease->StringField("id"), token, + [&](HWND window, DWORD processId, ULONGLONG expiresAt) { + return automation_.HasWindowContent(window, processId, expiresAt, token); + })) { + try { + result.result.Append(WindowSummary(info)); + } catch (const HelperError& error) { + // A popup can finish closing after enumeration, before its geometry is read. + if (error.code() != kErrorNoSuchWindow || IsWindow(info.window) != FALSE) { + throw; + } + } } return result; } diff --git a/packages/agentic/desktop-driver/native/windows/src/driver.h b/packages/agentic/desktop-driver/native/windows/src/driver.h index 0fe5e7c039..6892546739 100644 --- a/packages/agentic/desktop-driver/native/windows/src/driver.h +++ b/packages/agentic/desktop-driver/native/windows/src/driver.h @@ -84,6 +84,8 @@ class Driver { } // Replaces the hit test that checks where a pointer press or scroll lands. void UseTestHitTester(ClickHitTester hitTest) { hitTester_ = hitTest; } + // Runs after enumeration, immediately before a listed window's geometry is read. + void UseTestWindowSummaryHook(std::function hook) { windowSummaryHook_ = std::move(hook); } private: CommandResult ExecuteCommand(const std::string& command, const json::Value& params, const CancellationToken& token); @@ -166,6 +168,7 @@ class Driver { ForegroundReader foregroundReader_; WindowActivator activator_; ClickHitTester hitTester_{&WindowFromPoint}; + std::function windowSummaryHook_; }; } // namespace furn diff --git a/packages/agentic/desktop-driver/native/windows/src/selftest.cpp b/packages/agentic/desktop-driver/native/windows/src/selftest.cpp index c62fb718df..f1f4fa9c0e 100644 --- a/packages/agentic/desktop-driver/native/windows/src/selftest.cpp +++ b/packages/agentic/desktop-driver/native/windows/src/selftest.cpp @@ -974,12 +974,12 @@ class WindowThread { WindowThread(const WindowThread&) = delete; WindowThread& operator=(const WindowThread&) = delete; - HWND Create(const wchar_t* className, const wchar_t* title, WNDPROC procedure = DefWindowProcW) { + HWND Create(const wchar_t* className, const wchar_t* title, WNDPROC procedure = DefWindowProcW, HWND owner = nullptr) { HWND window = nullptr; - Run([&window, className, title, procedure]() { + Run([&window, className, title, procedure, owner]() { RegisterTestWindowClass(className, procedure); window = CreateWindowExW(WS_EX_NOACTIVATE | WS_EX_TOOLWINDOW, className, title, WS_POPUP, -32000, -32000, 64, 64, - nullptr, nullptr, GetModuleHandleW(nullptr), nullptr); + owner, nullptr, GetModuleHandleW(nullptr), nullptr); if (window != nullptr) { ShowWindow(window, SW_SHOWNOACTIVATE); } @@ -1537,6 +1537,131 @@ void TestWindowEventHooks() { "a closed lease has no event log"); } +void TestRegisteredToolWindowLiveness() { + WindowThread windows; + const HWND primary = windows.Create(L"FurnSelfTestToolPrimary", L"Tool primary"); + const HWND popup = windows.Create(L"FurnSelfTestToolPopup", L"", DefWindowProcW, primary); + ApplicationManager applications; + const CancellationToken token; + const std::string leaseId = applications.AdoptWindowForTesting(primary, L"Tool primary").id; + const auto findPopup = [popup](const std::vector& listed) { + const auto found = std::find_if(listed.begin(), listed.end(), + [popup](const WindowInfo& info) { return info.window == popup; }); + return found == listed.end() ? std::string{} : found->id; + }; + std::size_t probes = 0; + const auto initial = applications.Windows(leaseId, token, [&](HWND window, DWORD processId, ULONGLONG) { + probes += 1; + return window == popup && processId == GetCurrentProcessId(); + }); + const std::string popupId = findPopup(initial); + Check(!popupId.empty() && probes == 1, "an owned tool popup is admitted once after content proof"); + if (popupId.empty()) return; + + probes = 0; + const auto unavailable = applications.Windows(leaseId, token, [&](HWND, DWORD, ULONGLONG) { + probes += 1; + return false; + }); + Check(findPopup(unavailable) == popupId && probes == 0, + "a registered tool popup stays listable without repeating a failed content probe"); + + std::vector failed; + const std::string code = FailureCode([&]() { + failed = applications.Windows(leaseId, token, [&](HWND, DWORD, ULONGLONG) -> bool { + probes += 1; + throw HelperError(kErrorAutomationFailed, "Injected content probe failure."); + }); + }); + Check(code == "none" && findPopup(failed) == popupId && probes == 0, + "a registered tool popup never enters a failing provider probe"); + Check(findPopup(applications.Windows(leaseId, token)) == popupId, + "a registered tool popup remains listable when content discovery is unavailable"); + + const json::Value log = applications.WindowEventLog(leaseId); + const json::Value* publication = log.Find("publication"); + const json::Value* published = publication == nullptr ? nullptr : publication->Find("windows"); + const auto containsListedPopup = [](const json::Value& entry) { + return entry.StringField("className") == "FurnSelfTestToolPopup" && + entry.BoolField("listed") && entry.BoolField("tracked"); + }; + Check(published != nullptr && + std::any_of(published->Items().begin(), published->Items().end(), containsListedPopup), + "publication marks an admitted tool popup as both listed and tracked"); + + const HWND deferredFirst = windows.Create(L"FurnSelfTestDeferredTool", L"", DefWindowProcW, primary); + const HWND deferredSecond = windows.Create(L"FurnSelfTestDeferredTool", L"", DefWindowProcW, primary); + probes = 0; + const ULONGLONG started = GetTickCount64(); + const auto bounded = applications.Windows(leaseId, token, [&](HWND, DWORD, ULONGLONG expiresAt) { + probes += 1; + const ULONGLONG now = GetTickCount64(); + if (now < expiresAt) Sleep(static_cast(expiresAt - now + 10)); + return false; + }); + Check(probes == 1 && GetTickCount64() - started < 2500 && bounded.size() == 2 && findPopup(bounded) == popupId, + "one exhausted discovery deadline defers new popups but keeps registered windows listable"); + + probes = 0; + ULONGLONG sharedDeadline = 0; + bool shared = true; + const auto retried = applications.Windows(leaseId, token, [&](HWND window, DWORD, ULONGLONG expiresAt) { + probes += 1; + if (sharedDeadline == 0) sharedDeadline = expiresAt; + shared = shared && sharedDeadline == expiresAt; + return window == deferredFirst || window == deferredSecond; + }); + Check(probes == 2 && shared && retried.size() == 4 && findPopup(retried) == popupId, + "a later listing retries deferred candidates under one fresh shared discovery deadline"); + applications.CloseApplication(leaseId, token); +} + +void TestClosingWindowListing() { + WindowThread windows; + const HWND primary = windows.Create(L"FurnSelfTestClosingListPrimary", L"Listing primary"); + const HWND popup = windows.Create(L"FurnSelfTestClosingListPopup", L"Closing popup", DefWindowProcW, primary); + Driver driver; + const CancellationToken token; + json::Value request = json::Value::Object(); + request.Set("lease", driver.AdoptWindowForTesting(primary, L"Listing primary")); + const CommandResult initial = driver.Execute("windows", request, token); + std::string primaryId; + std::string popupId; + for (const json::Value& info : initial.result.Items()) { + if (info.StringField("title") == "Listing primary") primaryId = info.StringField("id"); + if (info.StringField("title") == "Closing popup") popupId = info.StringField("id"); + } + Check(!primaryId.empty() && !popupId.empty(), "the closing-window test initially lists both native windows"); + if (primaryId.empty() || popupId.empty()) return; + + driver.UseTestWindowSummaryHook([&](HWND window) { + if (window == popup) windows.Destroy(popup); + }); + CommandResult listed; + const std::string code = FailureCode([&]() { listed = driver.Execute("windows", request, token); }); + Check(code == "none" && listed.result.IsArray() && listed.result.Items().size() == 1 && + listed.result.Items().front().StringField("id") == primaryId, + "a secondary closing after enumeration does not fail the live-window listing"); + + driver.UseTestWindowSummaryHook({}); + json::Value closed = json::Value::Object(); + closed.Set("windowId", json::Value::String(popupId)); + Check(FailureCode([&]() { driver.Execute("getWindowRect", closed, token); }) == kErrorNoSuchWindow, + "the omitted closed secondary still rejects direct window commands"); + + driver.UseTestWindowSummaryHook([](HWND) { + throw HelperError(kErrorNoSuchWindow, "Injected geometry failure for a live window."); + }); + Check(FailureCode([&]() { driver.Execute("windows", request, token); }) == kErrorNoSuchWindow, + "a geometry error for a still-live window is not mistaken for closure"); + driver.UseTestWindowSummaryHook([&](HWND window) { + windows.Destroy(window); + throw HelperError(kErrorAutomationFailed, "Injected unrelated listing failure."); + }); + Check(FailureCode([&]() { driver.Execute("windows", request, token); }) == kErrorAutomationFailed, + "closing a window does not suppress an unrelated listing error"); +} + void TestDriverWindowEvents() { WindowThread windows; const HWND window = windows.Create(L"FurnSelfTestDriver", L"Driver window"); @@ -3951,6 +4076,8 @@ int RunSelfTest() { TestLeaseRegistrationFailure(); TestAttachedCloseKeepsQuarantine(); TestWindowEventHooks(); + TestRegisteredToolWindowLiveness(); + TestClosingWindowListing(); TestDriverWindowEvents(); TestInputEnvironment(); TestInjectionIntegrity(); diff --git a/packages/agentic/desktop-driver/references/current-status.md b/packages/agentic/desktop-driver/references/current-status.md index 7448b0f0d4..7eea077aac 100644 --- a/packages/agentic/desktop-driver/references/current-status.md +++ b/packages/agentic/desktop-driver/references/current-status.md @@ -478,6 +478,763 @@ probe timed out after 10 seconds; bridge retirement occurred about 19 seconds after injection. A later attempt waited for retirement and retained the same Partial result. Physical Ctrl+C and DPI changes were not exercised. +## Windows Fabric and Win32 Paper — #4441 connected-session qualification — 2026-11-12 UTC + +**Observed baseline:** `505a2187b0a2451aed7081c9260b343d55742f93`, +Windows 11 Enterprise build 26300, x64, Node 26.13.0. All native runs held +the machine-wide campaign/lock; physical gates before and after each run +reported an active, unlocked **remote RDP** session, not a console session. +The single attached monitor was 1728 x 1084, origin `(0,0)`, 96 DPI / 100%. +No display settings changed. The packaged Fabric app's registered native +build was `bdeb0b291`; comparison of the Windows app project and native-core +sources to the tested SHA showed only native-core +version/changelog changes. **Inferred:** native-build reuse was valid. +Metro, generated catalogs and protected bundles came from the qualification +checkout. Paper used the prebuilt REX 0.81.1 host. + +### Matching-OS F16 clear contracts + +F16 is issue #4467, implemented by merged PR #4468, +`ac20660ff039f4e0dc914706ff24f548610f30aa`. From the qualification checkout, +with `$CampaignTools` identifying the private campaign helper directory: + +```powershell +$QualificationRoot = (Get-Location).Path +$env:TEMP = Join-Path $QualificationRoot 'artifacts\native-working' +$env:TMP = $env:TEMP +$env:TMPDIR = $env:TEMP +$env:FURN_NATIVE_DRIVER_TEST = '1' +pwsh -NoProfile -File "$CampaignTools\with-native-lock.ps1" ` + -Lane W3 -Label 'W3 qualification native strict-clear and fixture contracts' ` + -RequirePhysical ` + -WorkingDirectory (Join-Path $QualificationRoot 'packages\agentic\desktop-driver') -- ` + yarn test --runInBand nativeDriver win32Fixture operations.contract +``` + +**Passed:** 3 suites, 27 tests; **skipped:** 2 macOS-only signed/unsigned +build cases. This is one shared Windows-provider/Win32-fixture run, not 27 +tests per renderer. The real helper self-test negotiated `strictClear` and +covered semantic/keyboard mechanism selection and no-dispatch refusal; +the Windows native provider serves both endpoints. Helper artifact SHA-256: +`d8ee05e4796fb3aa22f7b5dbf481bd8cdfb0c20b6d5d24676d689dc7900d5b3c`. +The self-test's keyboard-denial instrumentation is not itself successful +React Native keyboard deletion. + +Separate live endpoint probes **passed** actual nonempty RN keyboard clear +(keyboard receipts 1 -> 2), accessible clear of an **initially empty** +TextInput, and readonly accessible clear refusal with effect `none` and +unchanged receipts. Nonempty semantic removal is native self-test evidence, +not a claim about the initially-empty RN probe. + +### Secondary targeting, capture and remaining gaps + +All rows below are **Observed** at the baseline SHA: + +| Case | Windows Fabric packaged | Win32 Paper REX | +| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| Separately registered secondary | Passed: exact new opaque handle, explicit switch, complete 11-node native observation, unique `modern-callout-focus`, physical-pointer click, `focus: confirmed`, then handle removal | Not run / partial: real Callout shown, but only main registered; no secondary targeting claim | +| Additional secondary semantic activation | Failed: applied/accessibility on the exact unique button, but six checks / 2090 ms and final native observation stayed `Idle` | Not run: no registered secondary | +| Minimized capture | Passed: `unsupported operation`, still minimized, no implicit restore; explicit restore then valid 586 x 793 capture | Passed: same refusal/restore contract; restored capture 1010 x 761 | +| 200% scale | Not run: 100% RDP scale; changing it was not authorized | Not run: same | +| Mixed DPI | Not run: only one attached monitor | Not run: same | +| Negative-origin layout | Not run: only an origin-zero monitor; no approved eligible layout | Not run: same | +| Physical console Ctrl+C | Not run: owned by the separate implementation lane | Not run: same | + +The semantic failure was reproduced **without** switching foreground back +to main: main status was read through its explicitly cited native root. +The same selector and status assertion passed with physical click in one +attempt / 79 ms. **Inferred:** the renderer/accessibility-pattern dispatch +path may acknowledge activation without reaching Pressable `onPress`. +`native/windows/src/automation.cpp` tries several patterns and accepts a +successful HRESULT; the precise selected pattern was not instrumented. +An applied receipt alone is not application-level effect proof. + +On Win32, exact-process native inventory **observed** a visible, untitled, +root `Net UI Tool Window` with ToolWindow true. **Inferred:** the deliberate +untitled `WS_EX_TOOLWINDOW` exclusion in `CollectWindow` explains the missing +registration. Modern Callout was absent from the actual Win32 catalog. +No native title/style manipulation was attempted. Both open findings are +being fixed on this connected-session branch; this entry records the +unfixed baseline, not successful fix qualification. + +Exact endpoint supervisor invocations, each inside the same W3 +`with-native-lock.ps1 -RequirePhysical` wrapper: + +```powershell +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs win32 +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows --secondary-only --modern-only +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows --secondary-only --modern-only --physical-secondary +``` + +Retained artifacts are relative to that qualification checkout, not shipped +package files: `artifacts\desktop\w3-qualification\runs\\`. + +| Evidence | Qualification ID | Report in that directory | +| --------------------------------------------------------------------------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | +| Fabric clear / minimized capture | `5c97fc26-71ef-483c-aa2a-b454d6ff4458` | `descriptor-cases-windows-clear.json`, `descriptor-cases-windows-minimized-capture.json` | +| Paper clear / minimized capture / missing secondary | `22332065-6fa3-4061-a2b6-93f83295ca2e` | `descriptor-cases-win32-clear.json`, `descriptor-cases-win32-minimized-capture.json`, `descriptor-cases-win32-secondary.json` | +| Fabric semantic failure, driver run `a9f2e71e-71ab-40f0-a375-fd6d76a0b971` | `675160f7-e069-413c-a6c8-486de0c37b69` | `descriptor-cases-windows-secondary---modern.json` | +| Fabric physical secondary pass, driver run `16aac92b-bb36-4742-bd29-01df381d9594` | `7fbe3b0b-df33-476b-80e3-95b97bf19f02` | `descriptor-cases-windows-secondary---modern---physical.json` | + +The physical secondary pass does not erase the semantic failure or earlier +receipt-only/investigation failures. The full matrix and injected Win32 +fresh-preview fence are recorded in +`@fluentui-react-native/desktop-test/references/current-status.md`; real MCP +click costs are in +`@fluentui-react-native/desktop-driver-mcp/references/current-status.md`. +Exact owned applications, helpers, Metro and service processes drained. +The campaign ended; pre/post-end audits both exited 0: +`FURN-AUDIT: no FURN desktop processes found`. No physical Ctrl+C claim is +made from the supervisor's cooperative programmatic shutdown. + +## Windows Fabric and Win32 Paper — connected-session fixes — 2026-11-12 UTC + +**Observed test base:** `a1cee02547b7aae4c72134902fe43ed007215069` plus the +then-uncommitted fixes described here. This section supplements, rather than +rewrites, the unfixed qualification above. No macOS qualification is added. +The native runs used the serialized W3 campaign/lock. Eligible runs reported +an active, unlocked **remote RDP** session on `Default` before and after; +they were not console-session tests. The packaged Fabric build at +`bdeb0b291` was reused: neither the Windows app native project nor native-core +Windows native sources differed. Metro and bundles came from the fix +checkout. Paper used the prebuilt REX 0.81.1 host; its native build remains +unsupported. + +### Observed causes and corrections + +Instrumented Fabric and Paper runs +`6488435c-6222-4c35-a670-d50655753da4` and +`ecd1cdff-6b77-4c42-ad57-89733dc154d0` found no Invoke, Toggle, +SelectionItem, or ExpandCollapse pattern. LegacyIAccessible was selected: +its default-action read returned `S_OK` and a **nonempty** action, and +`DoDefaultAction` returned `S_OK`, but the application did not change. +This occurred on main-window Pressables and agentic Buttons on both +endpoints, and on the Fabric secondary button. The saved fixed-format traces +are `native-trace-21424.jsonl` and `native-trace-38384.jsonl` under +`artifacts\desktop\w3-qualification`. + +**Inferred from that dispatch trace and the independent effect checks:** a +nonempty legacy default action is not sufficient activation evidence. +An empty-action-only guard would not repair the observed defect. The helper +therefore refuses legacy-only activation before mutation with +`unsupported-operation`, reason `activation-unsupported`, and effect `none`. +The four explicit actionable pattern paths remain unchanged. The public +error preserves its class, native code, and reason; it records no delivery, +does not quarantine, and never implicitly retries physically. Temporary +native tracing was removed. + +The Paper inventory also proved an exact-process, visible, untitled, +280 x 144 `Net UI Tool Window` popup owned/root-owned by the main window. +The strict untitled-tool exclusion originated in provider commit +`5b3f74c7b`; the available history supplied no more specific rationale. +The correction leaves global discovery conservative and qualifies only +lease-local popup roots using verified primary identity, exact-process +ownership, positive visible bounds, helper-class exclusions, and a bounded +UIA exact-root/content check, followed by native identity rechecks. + +The final helper registered that real Paper secondary. Its unique Callout +content observation was complete at eight nodes; its public `find` proved +the actual focus **button** unique, cited activation refused honestly, and +story cleanup removed the secondary handle. The host's whole-window UIA tree +also exposes the owner subtree, so this is scoped popup-content evidence, +not a claim that the default whole-window observation was complete. +Stable focus/blur test IDs were added to the legacy Callout story rather +than weakening the public uniqueness-proof requirement. + +### Red/green checks and helper identity + +All commands below ran from `packages\agentic\desktop-driver` unless stated +otherwise. Native commands used the W3 lock/campaign; interactive runs also +used `-RequirePhysical`. Test working directories were confined beneath the +ignored checkout `artifacts\desktop\w3-fix\native-working`. + +```powershell +$env:FURN_NATIVE_DRIVER_TEST = '1' +yarn test --runInBand operations.contract win32Fixture.contract --testNamePattern 'nonactionable legacy|legacy-only|untitled tool roots' +yarn test --runInBand operations.contract win32Fixture.contract nativeDriver +yarn desktop-driver build-driver --platform windows +``` + +- **Expected red:** five behavioral failures, 18 skipped, in + `native-passive-red.log`: MSAA no-action and ScrollIntoView-only UIA + activation both falsely succeeded through raw/public routes, and the + owned content tool root was missing. +- **Passed on the initial fixed helper:** five new cases, 18 filtered cases + skipped (`native-fix-focused-green.log`); then three suites, 32 tests + passed and two macOS build/signing cases skipped + (`native-fix-regressions.log`). This includes existing operation, fixture, + cancellation, retention, capture, and helper build/handshake checks. +- That initial helper was artifact + `29b67c2c775f51cd360016c6abd09eb2ca377b7868b0cd7c6feb7432799e871e`, + source digest `e99e3246963a0fa4848a31c5753a026200492d9ad92c036a132bba16f9d1b469`. +- **Passed final helper build:** after adding an exact UIA native-root + handle check, artifact + `b47212dcd9e63aba388907b930da19d9a6d602f7d3de32e364f1584890f4d6dc`, + source digest `16ecdda1e41112cf2dafaf57eaad843054fbeeb2e9add0518f9c41886e8c867c`, + build fingerprint `3628b82e4280727dce2b265db184baa52285cc9f88f571621182d3893b051397`. + The build log is `native-fix-final-build.log`. The real Paper pass below + used this final helper and story blob + `f96de651a8433fc446f9c02b7b5ee3f5075c945c`. +- The **final-helper native-contract rerun is Not run**, not a replay of + the initial helper's 32 passes: the desktop was paused after disconnect. + Logs in this list are relative to `artifacts\desktop\w3-fix`. + +### Endpoint results and session boundary + +The copied qualification supervisor ran these exact commands from the +checkout root, inside the W3 `-RequirePhysical` wrapper: + +```powershell +node artifacts\desktop\w3-qualification\run-endpoint.mjs windows --secondary-only --modern-only --activation-diagnostics +node artifacts\desktop\w3-qualification\run-endpoint.mjs win32 --secondary-only --activation-diagnostics +``` + +| Case | Outcome and evidence | +| --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Fabric main Pressable and agentic Button | **Passed** honest refusal on the initial fixed helper; original statuses remained `Closed` and `0`, with unchanged receipts. | +| Fabric modern secondary semantic action | **Passed** honest refusal on the initial fixed helper: exact secondary handle, complete 11-node observation, unique button, no delivery/quarantine, main status stayed `Idle`, handle retired. | +| Paper main Pressable and agentic Button | **Passed** honest refusal on the final helper; original statuses remained `Native window: Shown` and `0`, with unchanged receipts. | +| Paper secondary registration, observation, semantic action, retirement | **Passed** on the final helper: main plus one secondary, complete eight-node content sample, unique focus button, `activation-unsupported`/effect `none`, no receipt/quarantine/fallback, then main alone. | +| Final-helper Fabric semantic rerun | **Not run**: first startup failed before readiness with `spawn rnx-cli ENOENT`; dependency resolution confirmed the CLI was installed. Its one retry was session-gated before the supervisor started. | +| Final-helper Fabric and Paper secondary physical clicks | **Not run (session disconnected)**. No physical input was attempted after the gate. The earlier baseline Fabric physical pass remains valid; it is not a final-helper rerun. | +| DPI, mixed-monitor, negative-origin monitor layout, physical console Ctrl+C | **Not run** by this fix campaign; no new claim beyond the baseline section. | + +Artifacts are beneath +`artifacts\desktop\w3-qualification\runs\\`: + +| Evidence | Qualification ID | Driver run ID | +| -------------------------------------------------------- | -------------------------------------- | -------------------------------------- | +| Initial fixed Fabric main/secondary semantic pass | `a583840b-7ad5-4691-9e60-f4f83d49fd48` | `0644505e-3642-4fed-b6aa-157275fb519f` | +| Final fixed Paper main/secondary semantic pass | `7edaa34a-fefd-4a8b-846b-dc8aa8c3880e` | `2d342e8c-dd72-4553-bc90-af070716add2` | +| Final Fabric startup failure, no app-level case executed | `c807faf4-31e8-4aa1-a3de-5b78e720e20f` | `07fe422f-7c94-4bbc-b6e8-379b9ee989c7` | + +Each successful directory retains `qualification.json` and +`descriptor-cases--main-activation.json` / +`descriptor-cases--secondary[---modern].json`, with cleanup +acknowledgments. Earlier failed Paper probes were qualification-harness +failures, not passes: the owner subtree exhausted the observation byte +budget, searching _inside_ the unnamed button did not prove that button, +and an observation sample was correctly refused as `sample-not-unique`. +The successful run uses a real unique `find` with the stable story test ID. +The optional managed UIA probe now serializes nonfinite geometry as null; +its bounded 32-node content result is explicitly incomplete, not a complete +window-tree claim. + +At **2026-11-12T17:03:45Z** the wrapper observed session 1 as remote, +Disconnected, locked, LogonUI running, with no input desktop. It returned +75 / `NOT RUN` before starting the retry. All native/GUI/physical work was +then paused; no reconnect or session move was attempted. The campaign ended +and both audits passed: +`FURN-AUDIT: no FURN desktop processes found`. + +### Portable checks and remaining qualification + +**Passed:** desktop-driver `yarn test --runInBand` with native opt-in unset: +73 suites / 1,201 tests, with eight native suites / 93 native tests +**skipped**, not passed. Native-core `yarn test` passed its build, story +typecheck, 18 Node checks, 56 Jest checks, and one snapshot; native-core +format/lint passed. These do not replace native qualification. + +From the checkout root, the final portable checks **passed**: + +- `yarn workspace @fluentui-react-native/desktop-driver format` +- `yarn workspace @fluentui-react-native/desktop-driver lint` +- `yarn workspace @fluentui-react-native/desktop-driver build` +- `yarn oxfmt --check` and `git diff --check` +- `yarn markdown-link-check -c .github\markdown-link-check-config.json ` + for all ten changed Markdown files, including both patch changesets: + 72 links checked. + +One permitted session check at **2026-11-12T17:21:31Z** reconfirmed remote +session 1 as Disconnected, locked, LogonUI running, and without an input +desktop. No native or GUI retry followed. + +The implementation is committed as +`cd036a2d06ae450f9c26ac0837c4fe140877ffb9`. Its committed +`yarn change:check` **passed**. A post-commit process audit also passed: +`FURN-AUDIT: no FURN desktop processes found`; campaign status reported no +campaign or holder. The branch has not been pushed. + +The post-review clean-cache repository aggregate and final native/physical +reruns are still pending. Outstanding qualification stays tracked in +[remaining work](remaining-work.md) / #4441; no issue is closed and no +final-helper physical-success claim is made here. + +## Windows Fabric and Win32 Paper — popup liveness review fixes — 2026-11-12 UTC + +**Observed source base:** `96b91661e2886aa4a22bd3f08b485e51d11c90d7` +plus the then-uncommitted review fixes below. This section supplements the +earlier Windows evidence; it does not rerun or relabel that evidence. The +earlier section's final helper was the pre-review `b47212dc…` artifact. +No macOS code, ledger section, or qualification changed. + +### Admission, liveness, and publication + +Registered untitled popups now bypass UIA content discovery when their lease, +process, thread/class, primary identity, and owned-content native predicates +still match. A failed content read or temporary re-layout therefore cannot +remove a live, admitted popup from `windows`. New candidates share one +1,000 ms deadline per listing; UIA waits use its remaining time, late reads +stop before another transaction, and unprobed candidates retry later. +Exhausting discovery does not skip registered windows later in the listing. +Publication diagnostics now include admitted live popups as `listed` as well +as `tracked`, without another content probe. + +**Test-first authoring:** the gated fixture regressions and native self-test +were added before production edits. The self-test injects false, throwing, +and unavailable content probes, an exhausted shared deadline, a later retry, +and listed/tracked publication assertions. The fixture checks hung-provider +popup retention/rectangle reads and bounded new-popup discovery/recovery; +its existing admission case also checks publication. Native red/green +execution is **Not run**, not inferred as passed: the desktop remains paused. + +### Compile-only and portable validation + +The helper-only build used the shared W3 lock, no campaign, and no launch. +The following build command is shown with its cache-root argument relative to +the driver package; the recorded invocation resolved that path absolutely: + +```powershell +yarn desktop-driver build-driver --platform windows --cache-root ..\..\..\artifacts\desktop\w3-review\native-cache +``` + +**Passed** x64 release compilation, including the new native self-test. +Lock acquisition was at `2026-11-12T17:45:53Z`, and the wrapper released +successfully with exit 0. No helper, fixture, REX, packaged app, or Metro was +launched. This is compilation, not native runtime qualification. + +- Artifact: `eb91ef8a4164080bcae787aacce3fd5aa5556d8c3d0b22d0cd0c661a50934066` +- Source digest: `89cc81fbb72cb8a79852e5d4d90ce96d8b2982283086236b860289f2664ac557` +- Build fingerprint: `7b38faa5c0eeba1ace7441d613a3f0c126915f78eff58c4a32466825c4e11854` +- Build log: `artifacts\desktop\w3-review\helper-build.log` + +From `packages\agentic\desktop-driver`, with native opt-in unset: + +```powershell +Remove-Item Env:FURN_NATIVE_DRIVER_TEST -ErrorAction SilentlyContinue +yarn test --runInBand win32Fixture nativeDriver +yarn test --runInBand +``` + +**Passed:** focused portable checks, eight tests; 20 native tests **skipped**. +The full portable suite passed 73 suites / 1,201 tests, with eight native +suites / 95 tests **skipped**, not passed. Logs are +`tests-authored-portable.log`, `focused-portable-final.log`, and +`portable-tests.log` beneath `artifacts\desktop\w3-review`. + +From the checkout root, **passed**: + +```powershell +yarn workspace @fluentui-react-native/desktop-driver format +yarn workspace @fluentui-react-native/desktop-driver lint +yarn workspace @fluentui-react-native/desktop-driver build +``` + +The README, protocol, helper/fixture references, and patch changeset describe +the same policy. FDR1 fields, commands, and `native/protocol.json` remain +unchanged. Root `yarn oxfmt --check`, `git diff --check`, and +`yarn markdown-link-check -c .github\markdown-link-check-config.json ` +**passed** for all seven changed Markdown files (66 links). +The review-fix implementation is committed as +`4d39582f2292ff6436dafaba646ee63cb9bdc3cc`. +Its committed `yarn change:check` **passed**; the log is +`artifacts\desktop\w3-review\change-check.log`. +The process audit passed with +`FURN-AUDIT: no FURN desktop processes found`; no W3 campaign was started, +and shared campaign status reported no campaign or holder. No push was made. + +### Remaining session-gated runs + +One permitted check at **2026-11-12T17:52:11Z** observed remote session 1 as +Disconnected, locked, LogonUI running, with no input desktop. No GUI or native +contract was attempted; no reconnect or session move was attempted. + +Still **Not run (session disconnected / W3 desktop grant required)**: + +- Final-helper Fabric main/secondary semantic case. +- Fabric and Paper secondary physical clicks, under `-RequirePhysical`. +- Final native selector below, including the new failed/hung-probe, + shared-deadline/retry, and publication regressions plus the native self-test. + +```powershell +$env:FURN_NATIVE_DRIVER_TEST = '1' +yarn test --runInBand operations.contract win32Fixture.contract nativeDriver +``` + +The clean-cache repository aggregate is **Not run** in this review-fix +segment; W1 holds the buildci-only grant. It remains required before +publication. The native cases remain in [remaining work](remaining-work.md) +and #4441; no new native pass, physical success, or issue closure is claimed. + +## Windows Fabric and Win32 Paper — default-environment publication validation — 2026-11-12 UTC + +**Observed tested source:** `cad25f8c53855c30f2c45f3ce6655f668d09bdc9`. +The coordinator authorized buildci-only use of the shared W3 lock, followed +by draft publication. No app, native contract, helper runtime, or physical +input was launched in this segment. Native opt-in remained unset. TEMP and +concurrency used their defaults; no workaround, guidance, or timeout edits +were made. + +The exact root validation command, inside the W3 machine-wide lock: + +```powershell +cmd.exe /d /c "yarn clean-lage-cache && yarn lage buildci" +``` + +### Failure comparison and unchanged-source retry + +- **Failed, first branch attempt:** lock acquired at `18:33:59Z`, released + at `18:36:20Z`; 136 tasks succeeded and Storybook desktop tests failed. + Four checks failed: the WDIO worker's 7,000 ms deadline, two 10,000 ms + diagnostic timeouts, and the server-integration stop result. +- **Observed cleanup:** Lage exited before its desktop-test descendants. + No independent WDIO command had been started outside the lock. The + descendants finished naturally; their recorded root PID was already gone + when checked. A subsequent process audit exited 0. +- **Failed on clean baseline too:** the same default-environment command + ran on isolated detached `505a2187b0a2451aed7081c9260b343d55742f93`. + It reproduced the WDIO deadline and server-stop readiness race; baseline + diagnostics passed. There were 138 successful tasks and one failed task. + All baseline children drained before the lock released at `18:45:14Z`. + The disposable checkout remained clean and was removed afterward. +- Storybook source/config, desktop-test source/config, scripts, lockfile, + and Lage configuration were byte-unchanged between the baseline and this + branch. **Inferred:** these load-sensitive failures were not caused by + the Windows activation/popup changes; no unrelated source fix was made. +- **Passed, unchanged-source branch retry:** the lock was acquired at + `18:48:10Z`, validation started at `18:48:11Z`, and all **178 tasks passed** + at `18:50:53Z` (zero failed, aborted, pending, or skipped Lage tasks). + Owned descendant drain completed before lock release. This is a task-level + aggregate result, not a claim that opt-in native cases executed. + +The local ignored supervisor `run-buildci-and-drain.ps1` runs the exact +command above and keeps the wrapper alive until owned Node descendants +finish. It does not alter the test environment or send input. Logs are +retained beneath `artifacts\desktop\w3-buildci`: +`buildci-default.log`, `baseline-buildci-default.log`, and +`buildci-default-rerun.log`. + +At `2026-11-12T18:52:16Z`, the post-run audit exited 0: +`FURN-AUDIT: no FURN desktop processes found`. Shared state reported no +campaign or holder. W3 started no campaign; its build-only lock was released. + +### Publication scope and remaining native work + +The final local Fabric semantic case, both final-helper secondary physical +clicks, and final native contracts remain **Not run**. The earlier disconnect +prevented them; after the session became Active/unlocked again, the desktop +was reserved for W1, not a W3 native campaign. No reconnect, session move, +display change, or native retry occurred here. The current review helper +remains compile-only as recorded above. + +The default-environment aggregate requirement is now satisfied. Draft +publication does not close #4441 or replace its outstanding native, 200%, +mixed-DPI, negative-origin, or separate #4491 console-Ctrl+C qualification. + +## Windows Fabric and Win32 Paper — hosted popup compatibility follow-up — 2026-11-12 UTC + +**Observed published revision:** `4208ab36d641bcfe3675836634f18dee2072ef49`, +draft PR #4498, workflow run `43284991535` (attempt 1). +**Tested portable correction:** `28c7690085342a3bc2ee114b2c4c2032d7d050c4`. +No native source or FDR1 schema changed in this follow-up. + +### Hosted evidence at the published revision + +- **Passed:** Windows Storybook's opt-in package command, + `yarn workspace @fluentui-react-native/desktop-driver test --runInBand` + with `FURN_NATIVE_DRIVER_TEST=1`: + 79 suites / 1,251 tests; two suites / 45 tests **skipped**. The Windows + fixture, operations, native-driver self-test, and other matching-OS native + suites actually ran. This includes the registered-popup hung-provider, + deferred-discovery, and listed/tracked publication regressions, rather than + merely compiling them. These are hosted fixture results, not the remaining + local real-application or physical-secondary reruns. +- **Passed:** Windows Storybook accessible smoke: 255 executed / 255 passed, + no failures, skips, unsupported cases, or retries. Internal navigation + diagnostics were retained; this count is the final selected-test summary, + not a claim that every diagnostic navigation attempt succeeded. +- **Passed:** Win32 Storybook physical smoke: 288 passed and one + unsupported/quarantined case; no failures or retries. The helper source + digest was `89cc81fbb72cb8a79852e5d4d90ce96d8b2982283086236b860289f2664ac557`, + matching the review helper. +- **Passed:** Windows legacy E2E, JS PR, both Win32 Storybook jobs, publishing, + links, changesets, and CodeQL. Other hosted jobs were still completing + during this follow-up; no whole-workflow pass is claimed. +- **Failed, deterministic:** Win32 legacy E2E executed 201 tests: 191 passed, + ten menu tests failed only at the final “An assert popped up” safety check. + Their menu-open/state checks had passed. The existing guard treated every + second window as an assertion dialog. All 20 retained failure captures + (including suite retries) showed the secondary accessibility root as `menu`. + Correctly registering native menu windows exposed that consumer assumption; + no flake rerun was requested. + +Logs and the bounded downloaded E2E artifact are retained under +`artifacts\desktop\w3-buildci`: `ci-windows-storybook.log`, +`ci-win32-physical.log`, `ci-win32-job.log`, and `ci-win32-e2e`. + +### Test-first correction and boundaries + +`DesktopSessionClient.getWindowTree` now reads a currently listed session +window without selecting or activating it. The server refreshes authoritative +window membership and rejects unknown/closed IDs before tree inspection. +Native focus, selected-window identity, and input state are unchanged. + +The Windows/Win32 legacy guard exempts only a single `menu` root without +nested dialogs or windows. Unrecognized secondary roots still fail; unreadable +windows reject rather than report no assertion. A menu cannot hide another +unexpected window. macOS, non-native runners, Android, and the native-task +library bypass keep their earlier behavior. Diagnostic window events are +not used as authority. The public route is documented in +[the WebDriver contract](protocol.md); it is a full tree query, not a budgeted +agent operation. + +From `apps\E2E`, `yarn test` first produced **two expected failures** for a +legitimate menu and an unreadable secondary. From `packages\agentic\desktop-driver`, +the two focused new routing cases first failed because the read-only route +did not exist. Logs: `e2e-assert-red.log`, `window-tree-red.log`. + +| Exact command | Result | +| ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | +| `yarn test --runInBand desktopDriver.test DesktopDriverClient.test` (desktop-driver) | **Passed:** 52 tests, including non-activation and unknown/closed-window refusal. | +| `yarn test --runInBand` (desktop-driver, native opt-in unset) | **Passed:** 73 suites / 1,203 tests; eight suites / 95 native tests **skipped**. | +| `yarn workspace @fluentui-react-native/e2e-testing test` | **Passed:** all 11 adapter tests, including menus, possible assertions, failed reads, and unchanged other-runner behavior. | +| `yarn workspace @fluentui-react-native/desktop-driver build` | **Passed**. | +| `yarn workspace @fluentui-react-native/e2e-testing build` | **Passed**. | +| `yarn workspace @fluentui-react-native/desktop-driver lint` | **Passed**. | +| `yarn workspace @fluentui-react-native/e2e-testing lint` | **Passed** after marking the mocked loopback URL as test-only. | +| `yarn workspace @fluentui-react-native/desktop-driver format` | **Passed**. | +| `yarn oxfmt --check` | **Passed**. | +| `git diff --check` | **Passed**. | +| `yarn markdown-link-check -c .github\markdown-link-check-config.json ` | **Passed:** E2E README and public protocol, 25 links. | +| `yarn change:check` | **Passed** after committing the implementation and patch changeset `quiet-menu-windows.md`. | + +The new correction is committed locally and awaits coordinator review before +push. Its clean-cache aggregate and affected hosted E2E rerun are **not run +yet**; the earlier 178-task aggregate does not qualify this new source. +W1 holds the live desktop campaign. W3 has not launched an app, native helper, +fixture, WDIO suite, or physical action in this portable correction segment. +The final local Fabric semantic case, both final-helper secondary physical +clicks, and requested local native selector remain **not run**. Hosted +fixture passes do not substitute for those real-app/physical cases. + +The shared process audit returned 2: it listed W5's test tree and the W3 +formatter running concurrently with the audit. Both recorded W3 formatter +PIDs were confirmed exited afterward. No W3 native runtime or campaign is +left running; a globally clean audit is not claimed while peer work is active. + +## Windows Fabric and Win32 Paper — closing-menu review correction — 2026-11-12 UTC + +**Reviewed base:** `ad8b67c9720a00d7d8df4d27c2ca7436c78d6198`. +**Tested implementation:** `36efc5a88dc507c626f2d29ac487cfc37e9e4bf3`. +The coordinator's review accepted the scoped, non-activating tree query and +conservative menu classification, and identified a closing-popup race in the +legacy E2E adapter. + +The adapter now skips a tree read only when its typed WebDriver error is +`no such window`, then continues checking other handles. Before reporting an +empty, non-menu, or dialog-containing tree as an assertion, it refreshes the +authoritative window list and confirms that handle is still present. +Persistent unexpected windows still fail the check. Transport, provider, +and stale-element errors still reject; no diagnostic event or input fallback +is used. + +**Observed test-first:** `yarn test` in `apps\E2E` initially failed two new +cases: a `no such window` read rejected, and an empty closing-menu tree +reported an assertion. The retained log is +`artifacts\desktop\w3-buildci\popup-race-red.log`. +After the fix, all 14 adapter tests passed, including a vanished menu followed +by a real unexpected window, empty/non-menu roots followed by disappearance, +persistent unexpected roots, and unreadable still-listed windows. + +| Exact command | Result | +| --------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | +| `yarn workspace @fluentui-react-native/e2e-testing build` | **Passed**. | +| `yarn workspace @fluentui-react-native/e2e-testing test` | **Passed:** 14 tests, zero skips or failures. | +| `yarn workspace @fluentui-react-native/e2e-testing lint` | **Passed**. | +| `yarn oxfmt --write apps\E2E\src\common\nativeDesktop.ts apps\E2E\desktop-adapters.test.cjs apps\E2E\README.md` | **Passed**. | +| `yarn oxfmt --check` | **Passed**. | +| `git diff --check` | **Passed**. | +| `yarn markdown-link-check -c .github\markdown-link-check-config.json apps\E2E\README.md` | **Passed:** 19 links. | +| `yarn change:check` | **Passed** after the implementation commit. | + +This correction changes only the private E2E adapter, its focused tests, and +related documentation; no native helper or public protocol changed. The +existing driver patch changeset remains committed. +Per coordinator instruction, the locked clean-cache aggregate, native +contracts, GUI/physical runs, and push are **not run** in this segment. +W1 retains the desktop campaign. W3 awaits its explicit grant for the final +Fabric semantic case, both secondary physical clicks, and final native +contracts/probe tests, then the clean-cache aggregate and publication. + +## Windows Fabric and Win32 Paper — final sweep and native close-race correction — 2026-11-12 UTC + +**App qualification revision:** `3957cd6aec1cf83c8dc62e4d56403fe942505aa4`. +**Close-race implementation:** `a53451513eb84f5ff62dc7763bd419521c617d5b`. +The session had returned to Active/unlocked before this campaign. All native +commands below used the W3 campaign and serialized lock with +`-RequirePhysical`; both preflight and postflight reported an eligible +**remote** session with the Default input desktop. No session change was +reported. + +The registered Fabric native app was reused only after checking its install +identity and verifying no native app-source differences from its build +revision. Metro and bundles came from this worktree. Paper used the exact +REX executable with SHA-256 +`3ecd1f58690b029eccf06838dc23411f08b1f3765b99950091f8246aedd74725`. +Apps, Metro, helpers, and fixtures were closed between runs. + +The following results used helper source +`89cc81fbb72cb8a79852e5d4d90ce96d8b2982283086236b860289f2664ac557`, +artifact `eb91ef8a4164080bcae787aacce3fd5aa5556d8c3d0b22d0cd0c661a50934066`. +Run artifacts are under `artifacts\desktop\w3-qualification\runs\`. + +| Endpoint / case | Observed result | Run ID | +| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | +| Fabric main/secondary semantic action | **Passed:** honest `activation-unsupported` refusal with effect `none`; no fabricated receipt, fallback, or app change. | `98d0ae8f-e84b-43c1-b332-7136dd676f23` | +| Fabric secondary physical click | **Passed:** separately registered window, complete 11-node sample, physical-pointer receipt and public `focus: confirmed`, then retirement. | `537cc479-b3d3-4ef0-93b2-d39c35177653` | +| Paper secondary physical click | **Passed:** exact owned tool window, complete eight-node sample, physical-pointer delivery and retirement. The legacy story has no native-command outcome, so this is not proof of the RN focus command. | `e29ff4aa-62f5-4f53-8a5e-450a5e799466` | + +### Observed failure and test-first correction + +With `$env:FURN_NATIVE_DRIVER_TEST='1'`, the requested package command +`yarn test --runInBand operations.contract win32Fixture.contract nativeDriver` +**failed**: 33 passed, one failed, two macOS-only cases skipped. The tool-content +fixture failed with `no-such-window`, not a session or permission refusal. +Its log remains `artifacts\desktop\w3-buildci\final-native-contracts.log`. +This failed sweep is not claimed as a final native pass. + +A bounded diagnostic replay reproduced the failure on its first fixture. +The command journal isolated it to `windows` immediately after the close +button's successful accessibility click; the destroy event confirmed the +secondary had closed. Native enumeration can return a window that closes +before `WindowSummary` reads its geometry. This is separate from semantic +dispatch and the E2E adapter's closing-tree race. + +The correction omits a geometry failure only when its code is +`no-such-window` **and** native liveness confirms the handle has gone. A +still-live handle's error and every unrelated error propagate. Direct +commands for the omitted window still refuse. No input fallback, +cross-process admission, or wire-schema change was added. + +A deterministic native self-test closes a secondary exactly between +enumeration and geometry serialization. Before the guard, this command +**failed** its new live-window-listing assertion: + +```powershell +yarn test --runInBand nativeDriver --testNamePattern 'handshakes with the Windows helper' +``` + +After the guard, the following command **passed**, two native cases with 26 +unselected cases skipped. It includes the deterministic self-test and the +original real-fixture registration/action/closure scenario: + +```powershell +yarn test --runInBand nativeDriver win32Fixture.contract --testNamePattern 'handshakes with the Windows helper|registers only owned untitled tool roots' +``` + +The self-test also verifies direct closed-window refusal, propagation of an +error for a live handle, and propagation of an unrelated error even when its +window closes. The unchanged diagnostic replay then **passed six isolated +fixture lifecycles**, including close polling and subsequent main-window +actions. Retained logs under `artifacts\desktop\w3-buildci` are +`trace-tool-contract-replay.log`, `closed-window-listing-red.log`, +`closed-window-listing-green.log`, and `trace-tool-contract-green.log`. + +The fixed helper has source +`7119f42fb101452990b7bea13daf685e90a67ad75840fba60968a5efe42f62d1`, +artifact `e0e6fc988fc9214b91e02bd0909c099b203d4f9bfd89c45db8f8c9f9397a05e0`, +and build fingerprint +`afca889e68ccd07b9c1874b6d46b0c2a3a3f635e3e7b09c1b43eca20b9fb6f9c`. +The package `yarn build` and `yarn lint`, root `yarn oxfmt --check`, +`git diff --check`, and Markdown link checks for the native provider README +and protocol **passed**. Patch changeset: `brave-popup-listings.md`. + +The new implementation needs coordinator review before push. The complete +native selector, Fabric semantic case, and both secondary physical clicks +must be rerun against this **new** helper; the earlier app results above do +not qualify its new source digest. The new-source clean-cache aggregate and +hosted CI are **not run** yet. The 200% DPI, mixed-DPI, and negative-origin +gaps remain unchanged. Both campaigns ended, and the final audit reported +`FURN-AUDIT: no FURN desktop processes found`. + +## Windows Fabric and Win32 Paper — reviewed-helper final qualification — 2026-11-12 UTC + +**Tested revision:** `6c626602cf1eedd10425c8a63d794b1e12b1ff95`, clean worktree +for every real-app run. The coordinator's Opus review of the close-race +correction reported no findings. W3 received a new desktop grant after W2's +recovery verification; the session was **remote**, Active/unlocked, with +the Default input desktop and `physicalEligible: true`. + +All native and real-app commands below ran inside the W3 campaign and +serialized machine lock with `-RequirePhysical`. Every wrapper's before/after +session check stayed eligible. No hosted input skip or semantic action is +counted as physical evidence. + +The source-verified helper cache selected source +`7119f42fb101452990b7bea13daf685e90a67ad75840fba60968a5efe42f62d1`, +artifact `e0e6fc988fc9214b91e02bd0909c099b203d4f9bfd89c45db8f8c9f9397a05e0`, +and build fingerprint +`afca889e68ccd07b9c1874b6d46b0c2a3a3f635e3e7b09c1b43eca20b9fb6f9c`. +The app supervisor used prebuilt-only policy, not an implicit helper build. +The exact native app identity was checked before each run. The registered +Fabric app had no native source differences from its build revision; +Metro and bundles came from this worktree. Paper used the exact REX executable +with SHA-256 +`3ecd1f58690b029eccf06838dc23411f08b1f3765b99950091f8246aedd74725`. +The plain Win32 fixture executable had SHA-256 +`9e082f5a9a50e1a6f93f8d91a0c01f8e880820030a7e9d8b5c1917993fc8bc13`. +Apps, Metro, helpers, and fixtures were cleaned up between runs. + +The native package command below ran from `packages\agentic\desktop-driver` +with `$env:FURN_NATIVE_DRIVER_TEST='1'`. Endpoint supervisor commands ran +from the worktree root; their source and immutable run artifacts remain +under ignored `artifacts\desktop\w3-qualification`. + +| Exact command | Observed result | +| -------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `yarn test --runInBand operations.contract win32Fixture.contract nativeDriver --verbose` | **Passed:** three suites, 34 tests; two macOS-only cases **skipped**. The previously failing tool-content closure case passed. The log is `artifacts\desktop\w3-buildci\final-native-contracts-fixed.log`. | +| `node artifacts\desktop\w3-qualification\run-endpoint.mjs windows --secondary-only --modern-only --activation-diagnostics` | **Passed:** main Pressable/Button and modern secondary refuse honestly with `activation-unsupported`, effect `none`, no delivery receipt, quarantine, physical fallback, or application change. Run `8f6d457c-c0d3-4ab5-9204-6dbf66383dbd`. | +| `node artifacts\desktop\w3-qualification\run-endpoint.mjs windows --secondary-only --modern-only --physical-secondary` | **Passed:** a distinct secondary is registered, its 11-node sample is complete, the click is applied through `physical-pointer`, the public `focus: confirmed` effect is observed, and its handle retires. Run `94c0b321-cebd-4e9b-99e2-3404bbbd80c3`. | +| `node artifacts\desktop\w3-qualification\run-endpoint.mjs win32 --secondary-only --physical-secondary` | **Passed for native delivery:** the exact owned tool secondary is registered, its eight-node sample is complete, a physical-pointer click is applied, and its handle retires. Run `f46d008a-d310-4e6e-98fd-e85941879edd`. The legacy story has no native-command outcome, so this is not proof of the RN focus command. | + +The full native selector executed the review-specific coverage, not just +compiled it: failed-probe retention and shared-deadline/retry self-tests; +the real hung-popup listing and bounded new-popup discovery/recovery cases; +truthful listed/tracked publication; the close-after-enumeration self-test +with live-error and unrelated-error propagation; and both legacy-only +activation refusals. Legitimate Invoke/Toggle and the existing observation, +capture, mutation settlement, containment, cancellation, and retention +contracts also passed. Cancellation completed in 32 ms; retained elements +stayed at 20 after 500 fixture cycles. The earlier failed sweep and +deterministic red/green evidence remain recorded above. + +### Final-source aggregate and hygiene + +After ending the native campaign and obtaining a clean audit, the exact root +command `cmd.exe /d /c "yarn clean-lage-cache && yarn lage buildci"` ran under +the W3 machine lock with **default TEMP and concurrency**, native opt-in +unset, and no policy, timeout, or guidance workaround. It **passed on the +first attempt**: 178 successful tasks, zero failed, aborted, pending, or +skipped tasks. Native tests remain opt-in in this aggregate and are qualified +by the separate 34-test run above. + +The aggregate ran from 22:30:40 to 22:33:27 UTC. The operational supervisor +retained the lock until its owned descendants drained; drain completed +before lock release. Its retained log is +`artifacts\desktop\w3-buildci\final-buildci-fixed.log`. No readiness-flake +rerun was needed. No code changed after the reviewed and qualified source; +this publication update changes only Windows evidence/tracking. + +The campaign ended. Both its post-campaign audit and the post-aggregate audit +reported `FURN-AUDIT: no FURN desktop processes found`. No W3 app, helper, +fixture, Metro, or WDIO process survived. + +The current-source local reruns are complete. Hosted results for the +publication head will be recorded in draft PR #4498, separately from +earlier-head hosted evidence. The 200% DPI, mixed-DPI, and negative-origin +cases remain **not run** for the previously recorded authorization/topology +reasons; console Ctrl+C remains separate work in #4491. ## 2026-11-12: bounded refresh timing observations Part of #4435. Story selection can return allowlisted, service-local timing diff --git a/packages/agentic/desktop-driver/references/native-helpers.md b/packages/agentic/desktop-driver/references/native-helpers.md index d91630efe6..893566ca19 100644 --- a/packages/agentic/desktop-driver/references/native-helpers.md +++ b/packages/agentic/desktop-driver/references/native-helpers.md @@ -283,6 +283,29 @@ verify a service's process and listener before a credential is sent, and to prove a quarantined application's recorded process exited. See the [Windows provider guide](../native/windows/README.md#authority-helper-modes). +## Windows activation and secondary content windows + +The shared Windows/Win32 provider activates through actionable UIA `Invoke`, +`Toggle`, `SelectionItem`, or `ExpandCollapse` patterns. A legacy default-action +acknowledgment is not sufficient dispatch evidence. Legacy-only controls, +including controls exposing only `ScrollIntoView`, are refused before dispatch +with `unsupported-operation`, reason `activation-unsupported`, and effect +`none`. The public error retains the reason and native code, without an input +receipt, quarantine, or implicit physical retry. Callers may explicitly choose +physical click where policy permits; an unsettled mutation is not a safe +fallback opportunity. + +An application lease can register an untitled, visible, positive-size tool +popup only when its exact-process owner chain reaches the verified primary +window and a bounded UIA read proves exact-process content. Known IME/tooltip +helper classes, unrelated owner chains, hidden or zero-size roots, and +unreadable or empty content stay excluded from initial admission. Once admitted, +a popup stays listable using native identity and ownership rather than another +UIA read. New candidates share one 1,000 ms discovery budget per listing and +are retried later when that budget is exhausted. This does not broaden global title discovery. +See the [Windows identity rules](../native/windows/README.md#window-events-and-identity) +and [Windows qualification evidence](current-status.md). + ## Diagnostics `doctor` uses prebuilt-only resolution: diff --git a/packages/agentic/desktop-driver/references/operations.md b/packages/agentic/desktop-driver/references/operations.md index dbb6741c50..efe2828100 100644 --- a/packages/agentic/desktop-driver/references/operations.md +++ b/packages/agentic/desktop-driver/references/operations.md @@ -288,6 +288,12 @@ mutation. Provider messages quote window titles and element ids, so they are not passed on. A provider that no longer has the element or window drops the reference, as it does for a read. +An unsupported native activation keeps its `nativeCode` and any reported +machine `reason`. A rejection with effect `none` records no input delivery and +does not quarantine the target. `activate` never retries with physical input; +after such a rejection, a caller may explicitly choose `click` if the session +permits it. An indeterminate outcome is not permission to replay the action. + A session's `furn:inputMode` ([protocol](protocol.md#sessions-and-targets)) limits the mechanisms `act` may use. In an `accessible` session the runtime reports no `physicalClick` or `keyboard`, so `click` and `type` fail with diff --git a/packages/agentic/desktop-driver/references/protocol.md b/packages/agentic/desktop-driver/references/protocol.md index c31184bb08..040bfb0ad6 100644 --- a/packages/agentic/desktop-driver/references/protocol.md +++ b/packages/agentic/desktop-driver/references/protocol.md @@ -84,6 +84,8 @@ Implemented `furn` extensions: - `POST /session/{id}/furn/story/reset`; - `POST /session/{id}/furn/story/args`; - `GET /session/{id}/furn/tree`; +- `GET /session/{id}/furn/window/{window-id}/tree`, reading a currently listed + session window without selecting or activating it (`DesktopSessionClient.getWindowTree`); - `GET /session/{id}/furn/window-events`, when the `windowEvents` feature is advertised in `furn:features`; - `GET /session/{id}/furn/ops` and `POST /session/{id}/furn/ops/{name}`, the @@ -98,6 +100,12 @@ with role, subrole, title, frame, and the matching WindowServer window so transient popups and sheets remain identifiable after they close. The log is evidence for diagnosis only; window and element queries remain authoritative. +The explicit-window tree query refreshes the session's window list and refuses +an unlisted or closed ID with `no such window` before asking for its tree. It +does not change the selected window, native focus, input state, or preview +generation. Like `furn/tree`, it returns a full compact tree rather than a +budgeted agent observation; use agent operations for bounded observations. + Navigation, history, cookies, frames, shadow roots, prompts, printing, arbitrary JavaScript execution, browser CSS values, and new-window creation return `unsupported operation`. diff --git a/packages/agentic/desktop-driver/references/remaining-work.md b/packages/agentic/desktop-driver/references/remaining-work.md index fd3442eee4..a7793bf8df 100644 --- a/packages/agentic/desktop-driver/references/remaining-work.md +++ b/packages/agentic/desktop-driver/references/remaining-work.md @@ -1103,6 +1103,36 @@ have evidence at the final revision, implementation changesets are present, and the final clean-cache repository aggregate passes. Only then prepare/push the PR; a hosted capability skip is not completion. +### Windows connected-session follow-up — 2026-11-12 UTC + +The legacy-only activation refusal and lease-local content-popup registration +corrections and close-during-listing guard have reviewed test-first evidence. +At clean revision `6c626602c`, with the new helper and an eligible remote +session, Fabric main/secondary semantic refusal and both endpoints' +secondary physical delivery passed. The complete native selector passed +34 tests with two macOS-only skips, including failed/hung-probe retention, +the shared discovery deadline and deferred retry, truthful publication, +and the deterministic close-after-enumeration self-test. The +default-environment clean-cache aggregate passed all 178 tasks on its first +attempt. See the separate [Windows evidence entry](current-status.md). + +Keep #4441 open for the unrun 200% DPI, mixed-DPI, and negative-origin cases +and other issue-owned gaps. Native delivery receipts do not prove the +legacy Callout's unreported RN focus-command outcome. Current-head hosted +Windows/Win32 coverage, including the legacy menu assertion guard and its +closing-window correction, is tracked in draft PR #4498; earlier-head CI +does not qualify later code. + +The W3 campaign ended, owned aggregate descendants drained before lock +release, and post-campaign/post-aggregate process audits were clean. +Further native/GUI/physical work still requires an +Active, unlocked session, an explicit W3 desktop grant, and the serialized +campaign; physical work additionally requires `-RequirePhysical`. No +reconnect, session move, or skip-to-pass substitution is authorized. +Earlier eligible physical results remain evidence for their tested +revisions, not new-helper reruns. Authorized draft publication leaves +these qualification gaps and the unrun display-topology cases open. + ## 9. Qualification and efficiency gates | Lane | Required scope | diff --git a/packages/agentic/desktop-driver/src/client/DesktopDriverClient.ts b/packages/agentic/desktop-driver/src/client/DesktopDriverClient.ts index fb496aa2fb..585a34910e 100644 --- a/packages/agentic/desktop-driver/src/client/DesktopDriverClient.ts +++ b/packages/agentic/desktop-driver/src/client/DesktopDriverClient.ts @@ -277,6 +277,11 @@ export class DesktopSessionClient { return this.command('GET', '/furn/tree', undefined, options?.signal); } + /** Reads a live window in this session without selecting it or changing native focus. */ + getWindowTree(windowId: string, options?: DesktopClientCallOptions): Promise { + return this.command('GET', `/furn/window/${encodeURIComponent(windowId)}/tree`, undefined, options?.signal); + } + getWindowEvents(options?: DesktopClientCallOptions): Promise { return this.command('GET', '/furn/window-events', undefined, options?.signal); } diff --git a/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.test.ts b/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.test.ts index 710571be26..850b3edd32 100644 --- a/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.test.ts +++ b/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.test.ts @@ -89,6 +89,24 @@ describe('NativeDesktopHost error translation', () => { }); }); + test('preserves a no-dispatch activation refusal without making it an uncertain mutation', () => { + const translated = translateNativeError( + attachFailureEffect( + new NativeDriverError('unsupported-operation', 'No default action is available.', { reason: 'activation-unsupported' }), + 'none', + ), + ); + + expect(translated).toMatchObject({ name: 'HostUnsupportedError' }); + expect(toWebDriverError(translated)).toMatchObject({ + code: 'unsupported operation', + data: { nativeCode: 'unsupported-operation', reason: 'activation-unsupported' }, + }); + expect(readFailureEffect(translated)).toBe('none'); + expect(dispositionOfFailure(translated)).toBe('rejected'); + expect(readProviderQuarantine(translated)).toBe(false); + }); + describe('provider settlement codes', () => { test.each<[nativeCode: string, effect: string, disposition: string]>([ ['mutation-indeterminate', 'unknown', 'indeterminate'], diff --git a/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts b/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts index 67a97af040..0ff56833df 100644 --- a/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts +++ b/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts @@ -792,7 +792,7 @@ export function translateNativeError(error: unknown): unknown { } const effect = readFailureEffect(error); if (error.code === 'unsupported-operation') { - return attachFailureEffect(new HostUnsupportedError(error.message), effect); + return attachFailureEffect(new HostUnsupportedError(error.message, { ...error.data, nativeCode: error.code }), effect); } if (error.code === 'stale-element') { return attachFailureEffect(new HostStaleError(error.message), effect); diff --git a/packages/agentic/desktop-driver/src/native/windows/operations.contract.test.ts b/packages/agentic/desktop-driver/src/native/windows/operations.contract.test.ts index 37b25e7a6a..db400b3148 100644 --- a/packages/agentic/desktop-driver/src/native/windows/operations.contract.test.ts +++ b/packages/agentic/desktop-driver/src/native/windows/operations.contract.test.ts @@ -87,6 +87,30 @@ describe('Windows agent operations against the Win32 fixture', () => { }); }); + windowsTest.each(['1011', '1013'])( + 'rejects nonactionable legacy activation without a delivery receipt or quarantine (%s)', + async (automationId) => { + await withFixture( + async (session) => { + const before = await session.getInput(); + await expect(session.act({ action: 'activate', find: { automationId } })).rejects.toMatchObject({ + code: 'unsupported operation', + data: { + mutation: { disposition: 'rejected', effect: 'none', quarantined: false }, + nativeCode: 'unsupported-operation', + reason: 'activation-unsupported', + }, + }); + expect(await session.getInput()).toEqual(before); + expect((await session.check({ expect: { name: 'Ready' }, find: status })).result.status).toBe('passed'); + expect((await session.act({ action: 'activate', find: activate })).result.mechanism).toBe('accessibility'); + await expectStatus(session, 1); + }, + ['--no-default-action', '1'], + ); + }, + ); + inputTest('physically clicks with a hit-tested proof, types, and clears without journaling the text', async () => { await withFixture(async (session) => { const { result: found } = await session.find(activate); diff --git a/packages/agentic/desktop-driver/src/native/windows/win32Fixture.contract.test.ts b/packages/agentic/desktop-driver/src/native/windows/win32Fixture.contract.test.ts index c33b0112f2..a9f4c55bcc 100644 --- a/packages/agentic/desktop-driver/src/native/windows/win32Fixture.contract.test.ts +++ b/packages/agentic/desktop-driver/src/native/windows/win32Fixture.contract.test.ts @@ -32,6 +32,7 @@ type Retention = { privateBytes: number; retainedElementLimit: number; retainedE type WindowEventReport = { events: { descriptor?: { title?: string }; notification: string; windowId?: string }[]; observing: boolean; + publication: { windows: { className: string; listed: boolean; tracked: boolean }[] }; retention: Retention; source: string; }; @@ -237,6 +238,102 @@ describe('Windows helper against a plain Win32 application', () => { } }); + windowsTest.each(['1011', '1013'])('refuses legacy-only activation of a nonactionable control (%s)', async (automationId) => { + const session = await FixtureSession.start('no-default-action', { args: ['--no-default-action', '1'] }); + try { + const target = await session.control(session.main.id, automationId); + const { actions } = await session.call<{ actions: string[] }>('accessibilityActions', { elementId: target.id }); + expect(actions).toEqual(automationId === '1013' ? ['ScrollIntoView'] : []); + await expect(session.host.request('click', { elementId: target.id, mode: 'accessibility' })).rejects.toMatchObject({ + code: 'unsupported-operation', + data: { reason: 'activation-unsupported' }, + effect: 'none', + }); + expect(await session.status()).toBe('Ready'); + await session.click('1001'); + await waitFor('the actionable control to remain usable', async () => (await session.status()) === 'Activated 1 times'); + } finally { + await session.dispose(); + } + }); + + windowsTest('registers only owned untitled tool roots containing application content', async () => { + const session = await FixtureSession.start('tool-content', { args: ['--tool-windows', '1'] }); + try { + expect(await session.windows()).toHaveLength(1); + await session.click('1012'); + await waitFor('all content and excluded tool roots to open', async () => (await session.status()) === 'Opened 7 tool windows'); + const secondary = await waitFor('the owned content tool window', async () => { + const windows = await session.windows(); + return windows.length === 2 ? windows.find((window) => window.id !== session.main.id) : undefined; + }); + expect(secondary.title).toBe(''); + const tools = (await session.eventReport()).publication.windows.filter((window) => window.className === 'FurnWin32FixtureTool'); + expect(tools.filter((window) => window.listed)).toEqual([expect.objectContaining({ listed: true, tracked: true })]); + expect(tools.filter((window) => !window.tracked).every((window) => !window.listed)).toBe(true); + expect(await session.control(secondary.id, '3101')).toMatchObject({ name: 'Tool action', role: 'button' }); + await session.click('3101', secondary.id); + await waitFor('the secondary action to reach the application', async () => (await session.status()) === 'Tool activated 1 times'); + await session.click('3102', secondary.id); + await waitFor('the content tool window to close', async () => (await session.windows()).length === 1); + await expect(session.host.request('getWindowRect', { windowId: secondary.id })).rejects.toMatchObject({ code: 'no-such-window' }); + await session.click('1001'); + await waitFor('the main window to remain usable', async () => (await session.status()) === 'Activated 1 times'); + } finally { + await session.dispose(); + } + }); + + windowsTest('keeps an admitted owned popup listable while its provider is hung', async () => { + const session = await FixtureSession.start('tool-hang', { args: ['--tool-windows', '1', '--hang-ms', '22000'] }); + try { + const hang = await session.control(session.main.id, '1007'); + await session.click('1012'); + await waitFor('the tool roots to open', async () => (await session.status()) === 'Opened 7 tool windows'); + const secondary = await waitFor('the owned tool popup to register', async () => + (await session.windows()).find((window) => window.id !== session.main.id), + ); + await session.press(hang); + await delay(300); + + for (let attempt = 0; attempt < 2; attempt += 1) { + const startedAt = Date.now(); + const listed = await session.windows(); + expect(Date.now() - startedAt).toBeLessThan(2_500); + expect(listed.map((window) => window.id)).toEqual(expect.arrayContaining([session.main.id, secondary.id])); + expect(await session.call('getWindowRect', { windowId: secondary.id })).toMatchObject(secondary.rect); + } + } finally { + await session.dispose(); + } + }); + + windowsTest('bounds new-popup discovery while its provider is hung and recovers later', async () => { + const session = await FixtureSession.start('tool-probe-budget', { args: ['--tool-windows', '1', '--hang-ms', '22000'] }); + try { + const hang = await session.control(session.main.id, '1007'); + await session.click('1012'); + await waitFor('the tool roots to open without listing them', async () => (await session.status()) === 'Opened 7 tool windows'); + await session.press(hang); + await delay(300); + + const startedAt = Date.now(); + const listed = await session.windows(); + expect(Date.now() - startedAt).toBeLessThan(2_500); + expect(listed).toContainEqual(session.main); + // Providers that still answer may prove content even while the UI thread is busy. + expect(listed.length).toBeLessThanOrEqual(2); + + await waitFor('the provider to respond again', async () => (await session.status().catch(() => undefined)) === 'Hang finished', { + intervalMs: 250, + timeoutMs: 40_000, + }); + await waitFor('the deferred content popup to register', async () => (await session.windows()).length === 2); + } finally { + await session.dispose(); + } + }); + windowsTest('refuses input to an owner while its popup is open and reports the popup closing', async () => { const session = await FixtureSession.start('popup'); try { diff --git a/packages/agentic/desktop-driver/src/operations/envelope.test.ts b/packages/agentic/desktop-driver/src/operations/envelope.test.ts index 2536038f4d..70fbf9380d 100644 --- a/packages/agentic/desktop-driver/src/operations/envelope.test.ts +++ b/packages/agentic/desktop-driver/src/operations/envelope.test.ts @@ -13,6 +13,15 @@ import { } from './envelope'; import { maxRequestIdLength } from './limits'; +describe('WebDriverError', () => { + test('does not expose stack traces in its serialized response', () => { + const error = new WebDriverError('unknown error', 'Failed.'); + error.stack = 'Error: Failed.\n at /private/source.js:1:1'; + + expect(error.toJSON().stacktrace).toBe(''); + }); +}); + describe('parseOperationRequest', () => { test('defaults a missing body or input to an empty input', () => { expect(parseOperationRequest(undefined)).toEqual({ input: {} }); diff --git a/packages/agentic/desktop-driver/src/protocol/errors.ts b/packages/agentic/desktop-driver/src/protocol/errors.ts index 1bdcdbf65b..f3f5125231 100644 --- a/packages/agentic/desktop-driver/src/protocol/errors.ts +++ b/packages/agentic/desktop-driver/src/protocol/errors.ts @@ -40,7 +40,7 @@ export class WebDriverError extends Error { return { error: this.code, message: this.message, - stacktrace: this.stack ?? '', + stacktrace: '', ...(this.data ? { data: this.data } : {}), }; } @@ -76,7 +76,7 @@ function convertToWebDriverError(error: unknown): WebDriverError { return new WebDriverError('stale element reference', error.message); } if (error instanceof HostUnsupportedError) { - return new WebDriverError('unsupported operation', error.message); + return new WebDriverError('unsupported operation', error.message, error.data); } return new WebDriverError('unknown error', error instanceof Error ? error.message : String(error)); } @@ -101,8 +101,11 @@ export class HostStaleError extends Error { } export class HostUnsupportedError extends Error { - constructor(message: string) { + readonly data?: Record; + + constructor(message: string, data?: Record) { super(message); this.name = 'HostUnsupportedError'; + this.data = data; } } diff --git a/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts b/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts index e4e7632c4a..7739b5a749 100644 --- a/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts +++ b/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts @@ -629,6 +629,19 @@ async function route(context: RouteContext, sessions: SessionManager, targets: T ); return buildDesktopTree(snapshots); } + if (method === 'GET' && command[1] === 'window' && command[2] && command[3] === 'tree' && command.length === 4) { + const windowId = command[2]; + session.windows = await hostCommand(session, 'Reading application windows', (signal) => + session.target.host.windows(session.lease, signal), + ); + if (!session.windows.some(({ id }) => id === windowId)) { + throw new WebDriverError('no such window', `Window "${windowId}" does not exist in this session.`); + } + const snapshots = await hostCommand(session, 'Reading a window accessibility tree', (signal) => + session.target.host.tree(windowId, signal), + ); + return buildDesktopTree(snapshots); + } if (method === 'GET' && command[1] === 'window-events' && command.length === 2) { const windowEvents = session.target.host.windowEvents?.bind(session.target.host); if (!windowEvents) { diff --git a/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts b/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts index 4829fa8478..9fbdc0e26a 100644 --- a/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts +++ b/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts @@ -9,6 +9,63 @@ import { createDesktopDriverTestHarness } from '../testing/protocolHarness.js'; import type { DesktopStoryManifest, StoryOrchestrator, StoryReadyResult } from '../storybook.js'; describe('Desktop Driver W3C remote end', () => { + test('reads a specified live window without activation or changing the selected window', async () => { + const harness = await createDesktopDriverTestHarness({ + windows: [ + { id: 'main', title: 'Main', elements: [] }, + { + id: 'menu', + title: '', + elements: [ + { + id: 'menu-root', + role: 'menu', + scope: 'secondary-window', + windowId: 'menu', + rect: { x: 0, y: 0, width: 100, height: 100 }, + }, + ], + }, + ], + }); + try { + const client = createDesktopDriverClient({ url: harness.server.url }); + const session = await client.newSession({ + alwaysMatch: { platformName: 'windows', 'furn:target': harness.target.id }, + }); + const actions = [...harness.host.actions]; + const tree = await session.getWindowTree('menu'); + expect(tree).toEqual([expect.objectContaining({ role: 'menu' })]); + expect(await session.getWindowHandle()).toBe('main'); + expect(harness.host.actions).toEqual(actions); + await session.delete(); + } finally { + await harness.close(); + } + }); + + test('refuses unlisted and closed windows before trying to inspect their trees', async () => { + const harness = await createDesktopDriverTestHarness(); + try { + const client = createDesktopDriverClient({ url: harness.server.url }); + const session = await client.newSession({ + alwaysMatch: { platformName: 'windows', 'furn:target': harness.target.id }, + }); + const tree = jest.spyOn(harness.host, 'tree'); + const [window] = await session.getWindowHandles(); + for (const handle of ['unlisted', window]) { + if (handle === window) await harness.host.closeWindow(window); + await expect(session.getWindowTree(handle)).rejects.toMatchObject({ + code: 'no such window', + }); + } + expect(tree).not.toHaveBeenCalled(); + await session.delete(); + } finally { + await harness.close(); + } + }); + test.each([ undefined, { supported: false as const, reason: 'Not exposed.' }, diff --git a/packages/agentic/desktop-driver/src/server/operations.test.ts b/packages/agentic/desktop-driver/src/server/operations.test.ts index e308562b87..30aa57d084 100644 --- a/packages/agentic/desktop-driver/src/server/operations.test.ts +++ b/packages/agentic/desktop-driver/src/server/operations.test.ts @@ -8,6 +8,7 @@ import type { FakeDesktopHostOptions } from '../hosts/fake/FakeDesktopHost.js'; import { translateNativeError } from '../hosts/native/NativeDesktopHost.js'; import { NativeDriverError } from '../native/NativeDriverError.js'; import type { OperationCatalogEntry } from '../operations/index.js'; +import { attachFailureEffect } from '../protocol/settlement.js'; import { createFakeTree, fakeTreeSize } from '../testing/fakeTree.js'; import type { FakeTreeShape } from '../testing/fakeTree.js'; import { createDesktopDriverTestHarness } from '../testing/protocolHarness.js'; @@ -493,6 +494,39 @@ describe('Desktop Driver agent operations', () => { }); describe('act', () => { + test('keeps a native activation refusal without a receipt or implicit physical fallback', async () => { + await withFixture({ windows: actWindow }, async ({ harness, session }) => { + const click = jest.spyOn(harness.host, 'click').mockImplementationOnce(async () => { + throw translateNativeError( + attachFailureEffect( + new NativeDriverError('unsupported-operation', 'No default action is available.', { + reason: 'activation-unsupported', + }), + 'none', + ), + ); + }); + const before = await session.getInput(); + + await expect(session.act({ action: 'activate', find: { automationId: 'save' } })).rejects.toMatchObject({ + code: 'unsupported operation', + data: { + mutation: rejected(), + nativeCode: 'unsupported-operation', + reason: 'activation-unsupported', + }, + }); + expect(await session.getInput()).toEqual(before); + expect(click.mock.calls.map(([, mode]) => mode)).toEqual(['accessibility']); + expect(harness.host.actions.filter((action) => action.type === 'click')).toEqual([]); + + await expect(session.act({ action: 'click', find: { automationId: 'save' } })).resolves.toMatchObject({ + result: { disposition: 'applied', mechanism: 'physical-pointer' }, + }); + expect(click.mock.calls.map(([, mode]) => mode)).toEqual(['accessibility', 'physical']); + }); + }); + test('clicks, activates, types into, and clears a proven element, and says how each was carried out', async () => { await withFixture({ windows: actWindow }, async ({ harness, session }) => { await expect(session.act({ action: 'click', ...(await proven(session, 'agree')) })).resolves.toMatchObject({ diff --git a/packages/agentic/desktop-test/references/current-status.md b/packages/agentic/desktop-test/references/current-status.md index d3b61790a7..a642784634 100644 --- a/packages/agentic/desktop-test/references/current-status.md +++ b/packages/agentic/desktop-test/references/current-status.md @@ -705,6 +705,135 @@ by hand only after its owner was proved dead and its resources were proved settled, or after the fenced launch probe's survivor was ended. Nothing reclaims a claim automatically. +## Windows Fabric and Win32 Paper — #4441 connected-session qualification — 2026-11-12 UTC + +**Observed baseline:** `505a2187b0a2451aed7081c9260b343d55742f93`, +Windows 11 Enterprise build 26300 x64, Node 26.13.0. Native/GUI work ran +serially in the W3 campaign, 14:50:24–15:22:51 UTC, under the machine-wide +lock. Every before/after physical gate reported an active unlocked +**remote RDP** session. The attached display was one 1728 x 1084, +origin-zero, 96-DPI / 100% monitor; no display setting changed. + +The registered Fabric native build was `bdeb0b291`; the compared app/native +sources differed only in native-core version/changelog metadata, not Windows app/native +implementation. **Inferred:** reuse was valid. Own-checkout Metro, +generated manifests and protected bundles were used. Paper ran the prebuilt +REX 0.81.1 host. Each supervisor matched the newly launched app's exact +image, start time, title and window before probing it. + +### Per-case outcomes + +These are manual native/descriptor probes, not selected authored suite +cases. **Observed** results must not be inferred from supervisor exit alone: + +| Case | Windows Fabric packaged | Win32 Paper REX | +| ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Matching-OS F16 native contracts | Passed: shared Windows-provider/fixture run, 3 suites / 27 tests | Same shared run includes the real Win32 fixture; 2 macOS-only build cases skipped, not passes | +| Nonempty RN physical keyboard clear | Passed, keyboard receipts 1 -> 2 | Passed, keyboard receipts 1 -> 2 | +| Initially-empty accessible clear | Passed, semantic-clear receipt | Passed, semantic-clear receipt | +| Readonly accessible clear refusal | Passed, effect none, receipts unchanged | Passed, effect none, receipts unchanged | +| Separately registered secondary observation / interaction / cleanup | Passed: one exact new opaque handle, complete 11-node observation, unique control, physical-pointer click, `focus: confirmed`, then popup handle removed | Not run / partial: popup shown, but no separately registered handle | +| Additional secondary semantic activation effect | Failed: applied/accessibility receipt but actual main operation status stayed `Idle` | Not run: no registered secondary | +| Fresh native preview after lost provider context | Not requested | Passed: old preview rejected `ref-region-changed`; fresh unique native preview rejected `story-context-stale`; both effect none, not quarantined; state unchanged and old chrome ref survived | +| Physical MCP click measurement | Passed: 6 clicks and post-click state checks | Passed: 6 clicks and post-click state checks | +| Minimized capture and explicit restore | Passed: unsupported-operation refusal without implicit restore, then valid restored capture | Passed: same safety contract | +| 200% scale | Not run: 100% RDP scale; changing it not authorized | Not run: same | +| Mixed DPI | Not run: one attached monitor | Not run: same | +| Negative-origin layout | Not run: one origin-zero monitor; no approved eligible layout | Not run: same | +| Physical console Ctrl+C | Not run: separate implementation lane owns it | Not run: same | + +The native strict-clear self-test supplies nonempty **semantic** clear +evidence; the RN semantic endpoint probe started empty. Provider +keyboard-denial instrumentation is not substituted for the separate real +RN keyboard deletion. The native run skipped only the two macOS-specific +build/signing cases. + +### Commands and retained evidence + +Run from the qualification checkout. `$CampaignTools` is the separately +managed private campaign helper directory, not committed configuration: + +```powershell +$QualificationRoot = (Get-Location).Path +$env:FURN_NATIVE_DRIVER_TEST = '1' +pwsh -NoProfile -File "$CampaignTools\with-native-lock.ps1" ` + -Lane W3 -Label 'W3 qualification native strict-clear and fixture contracts' ` + -RequirePhysical ` + -WorkingDirectory (Join-Path $QualificationRoot 'packages\agentic\desktop-driver') -- ` + yarn test --runInBand nativeDriver win32Fixture operations.contract +``` + +Endpoint lifecycle commands below each ran within the same +`with-native-lock.ps1 -Lane W3 -RequirePhysical` scope. The local supervisor +awaited driver readiness, ran the declared bundle stage before app launch, +then owned probes and cleanup: + +```powershell +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs win32 +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs win32 --fence-only +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows --mcp-only --with-secondary +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs win32 --mcp-only +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows --secondary-only --modern-only +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows --secondary-only --modern-only --physical-secondary +``` + +Artifacts are retained locally under the qualification checkout at +`artifacts\desktop\w3-qualification\runs\\`, including +`qualification.json`, command logs and the named probe JSON. They are not +published package files. `artifacts\desktop\w3-qualification\qualification-summary.json` +is the machine-readable matrix. + +| Case / endpoint | Qualification ID | Probe JSON / driver run | +| ----------------------------------------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | +| Clear and minimized, Fabric | `5c97fc26-71ef-483c-aa2a-b454d6ff4458` | `descriptor-cases-windows-clear.json`, `descriptor-cases-windows-minimized-capture.json` | +| Clear, minimized, unregistered secondary, Paper | `22332065-6fa3-4061-a2b6-93f83295ca2e` | `descriptor-cases-win32-clear.json`, `descriptor-cases-win32-minimized-capture.json`, `descriptor-cases-win32-secondary.json` | +| Fresh-preview fence, Paper | `60751352-c158-40a2-b4f9-9251e15fec95` | `descriptor-cases-win32-fence.json`; driver `64d2de39-df69-4a2b-a164-fc842f0ef64e` | +| MCP physical clicks, Fabric | `f188fae5-67c9-47e4-9800-ea584185fed9` | `mcp-click-windows.json`; driver `57d4f6cb-2dc3-4b00-bb95-4331efd8d34b` | +| MCP physical clicks, Paper | `6d474fd4-61cb-41f6-824e-7a80df68db45` | `mcp-click-win32.json`; driver `918930ba-e498-4e20-ad98-52e4187b5618` | +| Secondary semantic failure, Fabric | `675160f7-e069-413c-a6c8-486de0c37b69` | `descriptor-cases-windows-secondary---modern.json`; driver `a9f2e71e-71ab-40f0-a375-fd6d76a0b971` | +| Secondary physical pass, Fabric | `7fbe3b0b-df33-476b-80e3-95b97bf19f02` | `descriptor-cases-windows-secondary---modern---physical.json`; driver `16aac92b-bb36-4742-bd29-01df381d9594` | + +### Baseline failures, scope and cleanup + +**Observed:** Fabric semantic activation targeted the unique native popup +button and returned applied/accessibility, but six status checks over +2090 ms and a final native observation remained `Idle`. Main was read +through its cited root without changing foreground. The same test with +physical click passed in one attempt / 79 ms. **Inferred:** native +accessibility-pattern/renderer dispatch may acknowledge without executing +Pressable `onPress`; no exact pattern trace was captured. + +**Observed:** Paper's shown Callout produced a visible, untitled root +`Net UI Tool Window` with ToolWindow true, but no registered secondary. +Modern Callout was absent from its catalog. **Inferred:** the provider's +intentional untitled-toolwindow filter explains the registration gap. +Both findings are being fixed on this branch; neither is claimed fixed +by this baseline evidence. + +The Win32 fence deliberately invoked the **real** provider's bridge +retirement while preserving the socket/native preview. In one uninterrupted +session, old ref `hkbo1` was fenced, fresh unique ref `hkbo3` was fenced +in the next preview epoch, and chrome ref `hkbo2` remained usable for +presence checks. This proves injected provider-context loss, not a native +crash, real network disconnect/reconnect, or chrome mutation. + +Earlier missing-catalog/setup failures, invalid probe selectors/facts and +secondary receipt-only attempts remain in the run history; they are not +erased by independent case passes. Fabric's final MCP case passed inside +an enclosing run that failed its separate semantic-secondary assertion. +Win32 window close hid REX without ending it; the uniquely owned run CLI's +cooperative stop drained its process tree. This is not physical Ctrl+C +verification. No other lane's portable test/build workers were stopped. + +All owned apps, helpers, Metro/services, sessions and child grants drained. +Endpoint listeners were absent. Campaign end and both audits **passed**: +`FURN-AUDIT: no FURN desktop processes found`. Display-layout and console +qualification remain not run. MCP costs/budgets are recorded in +`@fluentui-react-native/desktop-driver-mcp/references/current-status.md`; +Storybook context details are in +`@fluentui-react-native/storybook-desktop/references/current-status.md`. + ## Connected Windows physical-input follow-up - 2026-11-08 UTC Source `fec94cec4995c7372b798877c5412ddcb156523b`; Windows 11 Enterprise diff --git a/packages/agentic/native-core/references/contributing.md b/packages/agentic/native-core/references/contributing.md index f0e813a66a..9f28d3ddfc 100644 --- a/packages/agentic/native-core/references/contributing.md +++ b/packages/agentic/native-core/references/contributing.md @@ -55,6 +55,11 @@ full catalog before running the selected native tests. Add `--paper` to run the macOS Paper lane; a successful Paper build alone does not establish Paper runtime or input parity. +The legacy Callout `WindowCommands` story exposes +`agentic-storybook-callout-focus` and `agentic-storybook-callout-blur` on its +actual command buttons. Use those identities for unique secondary-window +input proofs; a text child or an arbitrary unnamed button is not a substitute. + Executable cases are declared with `desktopIt` and its `requires` list (see `@fluentui-react-native/storybook-desktop/references/story-tests.md`), not with run-time `skip` guards. Cases that click or type declare `physical-click` and `keyboard`, which diff --git a/packages/agentic/native-core/src/legacy/callout/Callout.stories.tsx b/packages/agentic/native-core/src/legacy/callout/Callout.stories.tsx index 864f43add9..f96de651a8 100644 --- a/packages/agentic/native-core/src/legacy/callout/Callout.stories.tsx +++ b/packages/agentic/native-core/src/legacy/callout/Callout.stories.tsx @@ -76,10 +76,20 @@ const CalloutExample = ({ defaultVisible = false, onDismiss, onShow, showWindowC This content is hosted in a separate native window. {showWindowCommands && ( - calloutRef.current?.focusWindow()} style={styles.command}> + calloutRef.current?.focusWindow()} + style={styles.command} + testID="agentic-storybook-callout-focus" + > Focus window - calloutRef.current?.blurWindow()} style={styles.command}> + calloutRef.current?.blurWindow()} + style={styles.command} + testID="agentic-storybook-callout-blur" + > Blur window diff --git a/packages/agentic/storybook-desktop/references/README.md b/packages/agentic/storybook-desktop/references/README.md index 0734ff1a4c..c1c5ecf622 100644 --- a/packages/agentic/storybook-desktop/references/README.md +++ b/packages/agentic/storybook-desktop/references/README.md @@ -18,4 +18,4 @@ owner_paths: | `storybook-desktop.win32` | [Win32](platforms/win32.md) | Bundling or running the REX host | | `storybook-desktop.troubleshooting` | [Troubleshooting](troubleshooting.md) | Diagnosing build, extraction, readiness, or smoke failures | | `storybook-desktop.contributing` | [Contributing](contributing.md) | Editing package code or tests | -| `storybook-desktop.current-status` | [Current evidence](current-status.md) | Checking bounded measurements and qualification gaps | +| `storybook-desktop.current-status` | [Current evidence](current-status.md) | Checking endpoint evidence, bounded measurements, and qualification gaps | diff --git a/packages/agentic/storybook-desktop/references/current-status.md b/packages/agentic/storybook-desktop/references/current-status.md index 31f24a1192..ee4d3d4cd4 100644 --- a/packages/agentic/storybook-desktop/references/current-status.md +++ b/packages/agentic/storybook-desktop/references/current-status.md @@ -5,8 +5,100 @@ owner_paths: - packages/agentic/storybook-desktop --- -# Current qualification evidence +# Native qualification status + +## Windows Fabric and Win32 Paper — #4441 connected-session qualification — 2026-11-12 UTC + +**Observed baseline:** `505a2187b0a2451aed7081c9260b343d55742f93`, +Windows 11 Enterprise build 26300 x64, Node 26.13.0. Packaged Windows +Fabric reused native build `bdeb0b291` after comparison found only +native-core metadata/version differences; **Inferred:** reuse was valid. +Win32 Paper used REX 0.81.1. Own-checkout Metro, manifests and protected +bundles served both endpoints. Native/GUI runs were serialized in the W3 +campaign with `-RequirePhysical`; before/after gates stayed eligible, +unlocked **remote RDP**, not console. One 1728 x 1084 origin-zero monitor +at 96 DPI / 100% was attached; settings were not changed. + +### Win32 fresh-preview fencing + +**Passed, Observed:** after a verified Checkbox selection, an injected +stimulus called the real orchestrator's bridge-retirement path while +deliberately preserving the socket and native preview. Old preview ref +`hkbo1` at epoch `(1,1)` rejected mutation with `ref-region-changed`. +A newly found, unique **native** checkbox ref `hkbo3` at epoch `(1,2)` +then rejected mutation with `story-context-stale`. Both were effect-none, +not quarantined; the checkbox remained unchecked. Pre-established chrome +ref `hkbo2` survived, and old/fresh chrome presence checks passed. + +This closes the earlier bounded fresh-preview-fence gap for **injected +provider-context loss**. It does not claim a native crash, actual socket +disconnect/reconnect, transparent upstream reconnection, chrome mutation, +or quarantine recovery. + +Exact local qualification invocation, inside the machine-wide +`with-native-lock.ps1 -Lane W3 -RequirePhysical` scope: + +```powershell +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs win32 --fence-only +``` + +Qualification ID `60751352-c158-40a2-b4f9-9251e15fec95`, driver run +`64d2de39-df69-4a2b-a164-fc842f0ef64e`, session +`08419564-7f43-48d2-9e08-b0f971590a95`. Artifacts relative to the tested +qualification checkout: +`artifacts\desktop\w3-qualification\runs\60751352-c158-40a2-b4f9-9251e15fec95\descriptor-cases-win32-fence.json` +and `qualification.json` in the same directory. The harness explicitly read +current-story state before injecting loss. These retained local scripts +are not a new published Storybook CLI or supported fault-injection API. + +### Secondary-window observations and open fixes + +| Case | Observed outcome | +| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Fabric separately registered modern Callout | Passed: exact distinct opaque handle, switch, complete 11-node observation, unique `modern-callout-focus`, actual physical-pointer click, `focus: confirmed`, then popup removal | +| Fabric secondary semantic activation | Failed: exact same control returned applied/accessibility, but main status stayed `Idle` over six checks / 2090 ms and final native observation | +| Paper separately registered Callout | Not run / partial: legacy popup shown, but only main registered; modern Callout absent from actual catalog | +| 200% | Not run: current scale 100%; changing it unauthorized | +| Mixed DPI | Not run: one attached monitor | +| Negative-origin layout | Not run: origin-zero monitor only; no approved eligible layout | +| Physical console Ctrl+C | Not run: separate implementation lane owns it | + +The Fabric status assertion read main through its explicit native root +without switching foreground; physical click passed once / 79 ms. +**Inferred:** accessibility-pattern/renderer dispatch may acknowledge an +activation without reaching Pressable `onPress`; the selected pattern was +not instrumented. Paper native inventory **observed** a visible untitled +root `Net UI Tool Window`, ToolWindow true. **Inferred:** the driver's +untitled `WS_EX_TOOLWINDOW` filter prevents its registration. These two +gaps are being fixed on this branch; this entry does not qualify the fixes. + +Exact bounded follow-ups, under the same physical-gated campaign wrapper: + +```powershell +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows --secondary-only --modern-only +node .\artifacts\desktop\w3-qualification\run-endpoint.mjs windows --secondary-only --modern-only --physical-secondary +``` +Semantic failure: qualification `675160f7-e069-413c-a6c8-486de0c37b69`, +driver `a9f2e71e-71ab-40f0-a375-fd6d76a0b971`; report +`artifacts\desktop\w3-qualification\runs\675160f7-e069-413c-a6c8-486de0c37b69\descriptor-cases-windows-secondary---modern.json`. +Physical pass: qualification `7fbe3b0b-df33-476b-80e3-95b97bf19f02`, +driver `16aac92b-bb36-4742-bd29-01df381d9594`; same artifact-root layout, +`descriptor-cases-windows-secondary---modern---physical.json`. +Paper inventory: qualification `22332065-6fa3-4061-a2b6-93f83295ca2e`, +`descriptor-cases-win32-secondary.json`. + +The supervisor ran supported driver/bundle/run stages in order, bundling +before launch and using actual readiness identity. Earlier setup, +unsupported-story, invalid-selector/fact and receipt-only failures remain +retained, not silently counted as passes. Session/child grants drained, +the exact owned apps/services stopped, and the W3 campaign ended. +Pre/post-end audits passed: +`FURN-AUDIT: no FURN desktop processes found`. +The complete native clear/capture/MCP matrix is in +`@fluentui-react-native/desktop-test/references/current-status.md`; +MCP response costs and default-observe completeness are in +`@fluentui-react-native/desktop-driver-mcp/references/current-status.md`. ## 2026-11-12: refresh-reduction timing baseline Part of #4435. This section records bounded measurements, not a reliability