Repository navigation
Desktop driver: bounded agent operations, protected service, and MCP adapter - #4434
Merged
Merged
Conversation
…Windows Windows provider: run UIA mutations through a bounded provider-call runner. A mutation that times out is recorded as abandoned and reported as mutation-indeterminate; later guarded commands fail with target-quarantined before reaching the provider, and quarantine persists after a late completion. Reads time out without quarantining. Cleanup and recovery commands stay exempt. Node: map the two additive error codes to unknown-error HostWebDriverErrors carrying a disposition (indeterminate or rejected). Treat empty DESKTOP_REPORTER and DESKTOP_EVENTS as unset, and tolerate a consumer closing stdout in the three one-shot CLIs instead of crashing with EPIPE. Tests: fix a POSIX-only temp path in desktop-test results.test.ts, retry the Windows temp-directory cleanup in the owned-service contract, and add native self-test, NativeDesktopHost, reporter, and real-pipe CLI coverage. Update the protocol, Windows README, and remaining-work validation record. The macOS provider has not adopted the new codes yet. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 79f52cdd-cbee-4c8e-9fd1-a1e0ea629959
A mutation that fails, times out, or is cancelled may still have reached the application, so replaying it or sending more input can compound the change. Native helpers now report an additive FDR1 `effect` (none, partial, unknown) on failed and cancelled mutating commands, and the service quarantines a target whose mutation may have landed instead of advancing its queue. Node: settlement helpers classify an effect into an applied, indeterminate, or rejected disposition and fail closed when a provider reports none. A late success after a Node timeout is applied. SessionManager guards every mutating route, pins the reservation, exempts releaseActions, clears a launched target's quarantine only when its exit is verified, and keeps an attached target quarantined. Failed mutations report data.mutation and /status lists quarantines. The fake host gains fault injection for late, partial, and unobserved effects. macOS: an EffectLedger counts primary mutations (AX press, value write, window close and geometry, key, button, and wheel events), classifies AX errors as rejected or unresolved, and annotates tracked commands on error and cancel. setWindowRect checks both geometry attributes before setting either so a non-resizable window cannot move without resizing. Tests: Node settlement, quarantine, host-process, and native-host cases, Swift self-test ledger and residue checks (sabotage verified), and a gated real helper test. Windows UIA settlement is a separate parallel change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reconcile the two mutation-settlement designs so both providers settle identically at the service. The Windows helper reports mutation-indeterminate (effect unknown) and target-quarantined (effect none) and quarantines itself; the macOS helper reports a per-command effect. Node reads the Windows codes as those effects and mirrors the provider quarantine on the target (origin provider), so a refused mutation still stops the target. data.mutation stays the single canonical report and data.nativeCode keeps the helper code. macOS keeps reporting effect rather than adopting the two codes. PROTOCOL.md documents the unified contract. remaining-work.md lists the Windows follow-ups: report effect on failed and cancelled mutating commands, scope the helper ledger to the application lease and clear it when a launched exit is verified, and add self-tests for the reported effects. The fake host gains a providerQuarantined fault with a test that fails without the mirroring. Validation on macOS: driver 137 passed and 3 skipped, gated native run 138 passed and 2 skipped, Swift self-test 148 of 148, desktop-test 32, storybook-desktop 182 passed and 2 skipped, yarn lage buildci 176 succeeded with 0 failed. Not run: the merged Node mapping against the real Windows helper, real hung applications, GitHub Actions rendering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the additive `boundedObservation` FDR1 feature: `observe` reads one budgeted, breadth-first sample of a window's accessibility tree and `releaseElements` drops the records Node created for agents. Budgets are rejected rather than clamped, every sample names why it stopped, and each node says whether its children are none, complete, truncated, or unread. Node validates each request before it reaches a helper and each response against the request that produced it, so a provider cannot pass unrequested or inconsistent data on. The fake host is the reference provider, with one identity registry shared across find, snapshot, observe, and release, and a conformance grid exercises every truncation reason. No helper advertises the feature yet. The remaining-work plan carries the Windows work package for the contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add `observe` and `releaseElements` to the Swift helper and advertise the `boundedObservation` feature in `hello` and `probe`. - Observation.swift holds the provider-neutral part: request parsing, budget ranges, fact shaping, and the breadth-first sampler, generic over an ObservationSource so the self-test drives the same code with a synthetic tree and a controllable clock. - AccessibilityObservation.swift supplies the AX source: one AXUIElementCopyMultipleAttributeValues read per element, shared element ids with tree/find/snapshot, record release, and the `references` truncation at the 10,000-record capacity instead of `tree-too-large`. - A live run against Calculator found a defect no synthetic tree had: AXUIElementCopyActionNames answers a general failure for AppKit split dividers, so the `actions` fact dropped those elements and reported `unreadable`. The `actions` fact and `accessibilityActions` now publish an empty list for that answer; `click` keeps its strict handling. - PROTOCOL.md records the rules the provider settled: an unreadable element is skipped and leaves its parent truncated, duplicate and stale children are not counted, `unsupportedFacts` is a provider capability, and an element without an action list reports an empty list. Validation (macOS): Swift self-test 314/314 (148 before), including a randomized differential check of 500 trees; gated native Jest passes and Node's protocol oracle accepts all 110 samples the self-test dumps; Calculator (37 nodes) and Font Book (272 nodes) match the legacy tree on every element and stable fact, and every budget run, rooted sample, release, and `no-such-window` case conforms. `observe` without facts is 3.5 to 4 times smaller and about 3 times faster than `tree`; with every fact it is about 1.3 times the bytes because unsupported facts repeat their reason per node, which the Node projection must drop. Not run: the Windows provider, a hung application, trees near the 5,000-node limit, a real secure text field, the FluentUI Storybook app, and memory across repeated observations. Windows handoff: - Implement `observe` and `releaseElements` in native/windows/** from the M3 work package in references/remaining-work.md section 7, including the new "Rules the macOS provider settled" list. sampleObservation and ObservationSelfTest.swift are a worked reference for the loop and the randomized differential test; src/hosts/fake/observeTree.ts is the TypeScript model. - Advertise `boundedObservation` in `hello` and `probe` only after the self-tests pass. - When FURN_DRIVER_SELF_TEST_OBSERVATIONS names a file, have the Windows self-test dump its samples, and replay them from the windowsTest block in src/native/nativeDriver.test.ts through expectSelfTestObservationsToConform, as the macOS block does. - Ownership: native/windows/** is yours; src/**, native/PROTOCOL.md, native/macos/**, and references/remaining-work.md stay with the macOS side. Propose a contract change in your commit message rather than editing PROTOCOL.md, so the two providers do not drift. - Merge user/jasonvmo/driver-mcp-stage-two into driver-mcp first; it is a fast-forward of 2357547. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pure, host-independent building blocks for the S2 operation layer: random server-unique references with a classified failure mapping, the release ledger, untrusted-text hardening, the redacting operation journal, the bounded outline/nodes projection, and the tri-state `find` matcher that only claims uniqueness from a complete sample. Not wired into the server yet. Validation: yarn tsc -b, yarn lint, yarn test operations (94 tests) passed. Not run: server wiring, live macOS validation, Windows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
withCommandTimeout now accepts an optional parent signal. Aborting it stops the command the same way a deadline does: the child signal aborts, the failure waits for the command to settle so the session queue never advances over running native work, and the failure keeps the effect the settled command reported. A signal that is already aborted never starts the command, and a signal that aborts before the command begins reports no effect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Builds the transport-neutral operation layer on the bounded-observation contract: a request envelope, a registry that publishes JSON Schemas, and the first three operations (observe, find, journal). Operations depend on an OperationRuntime rather than the server, so the HTTP routes and a later MCP adapter run the same code. - sampling: one bounded sample per call. References are issued only for the elements an operation returns; every other element the provider read is queued for release and flushed on a command that survives caller disconnect, whether the operation succeeds or throws. - observe: compact outline (or nodes) with explicit budgets, a content/state fact selection, and truncation that says what was left out. - find: proves unique/multiple/none, or says inconclusive and why (incomplete walk, clipped field, unsupported fact) instead of guessing. - failures leave as WebDriver errors with a closed machine reason, a bounded neutralized message, no stack, and only allowlisted data; the mutation report is rebuilt from the closed effect set. - the journal keeps declared scalar summary fields and error codes, never values an application displayed or a caller searched for. - operator-controlled content policy withholds values, text, and help. Validation: yarn tsc -b, yarn lint, yarn format clean; yarn test operations passes (13 suites, 237 tests). Mutation checks confirmed the within-root wiring and the native-fact exhaustiveness assertion fail when broken. Not run: server wiring, MCP, live macOS or Windows validation (follow-up commits); Windows native code is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Binds the operation executor to a session and exposes it at
GET /session/<id>/furn/ops (the catalog) and POST /session/<id>/furn/ops/<name>.
- Operations run on the session's command queue, so they serialize with W3C
commands and the queue advances only after a command settles.
- A caller that disconnects aborts its host commands; release work still runs
to completion on a non-cancelled command so no provider record leaks.
- Elements a W3C client holds are marked external so an observation never
releases them.
- Operation bodies are capped at 64 KiB; the operator can deny reading
displayed content with `operations: { content: 'deny' }`.
- The session client gains operations(), operation(), observe(), find() and
journal(). The test harness accepts server options.
- A queued deleteAll waits for in-flight commands instead of racing them.
Validation (packages/agentic/desktop-driver): yarn format, yarn tsc -b, yarn
lint, and yarn test for src/server and src/operations (286 tests, including 16
new server-level cases). The new cases were mutation-checked: dropping
markExternal, the disconnect abort, the body cap, the stack-trace drop, the
quarantine flag, the content policy, record release, and the route-token fix
each fail the test that targets them.
Not run: Windows native, MCP client tests, live macOS validation of the
server-level path, large-tree behavior, and a hung-application deadline.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add references/operations.md for the versioned observe, find, and journal operations: routes and envelope, catalog, budgets, the outline grammar, reference and provider-record lifetimes, the closed reason vocabulary, the content policy and journal, and known limitations. Index it, link it from the architecture, service, protocol, contributing, README, and AGENTS guidance, and note in the native protocol that providers drop a window's records at closeWindow, closeApplication, and dispose. Per R-PKGDOC-009, trim the remaining-work plan to what is still open: update the status, record the operation-layer progress under M1, replace the implemented observation and reference sections with their open items, and state the requirements the read operations set for M4 actions. Add the changeset. Validated: yarn format, root yarn oxfmt --check, yarn check-agent-instructions, and markdown-link-check over the changed files passed. The validation record for the operation layer is added with the live macOS checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The helper's stdin reader and worker threads run no run loop, so nothing drained their autorelease pools. Live validation against Calculator showed resident memory growing without a plateau: about 25 KB per probe, 80 KB per legacy tree read, and 250 KB per 500-element observation. Run each inbound frame and each request in its own autorelease pool. With the fix the helper holds at its warm-up size across 3,000 tree and observe calls, and a gated native test now fails when 1,000 probe round trips grow the process by more than 8 MiB (it grew by about 24 MiB without the pools). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…identity Adds WinEvent-based window events and identity, input authority checks, element identity with bounded LRU retention, UIA failure diagnostics, and a hit-test ownership fix. Adds a test-only plain Win32 fixture with an opt-in native contract (7 scenarios), and updates the protocol, Windows README, references, and the remaining-work validation record. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 79f52cdd-cbee-4c8e-9fd1-a1e0ea629959
A live run against a stopped Calculator showed observe and find failing after about 1.5 seconds as a bare unknown error: the helper's automation-failed carried no data, and Node passed the native error through without it. An agent could not tell a hung application from a defect. The helper now marks an Accessibility call the target did not complete with reason target-non-response, the token window discovery already uses. Node maps automation-failed to unknown error, keeps the provider's data, and adds nativeCode, so the operation response and the session journal both carry the reason. The session is not quarantined and recovers when the application resumes. Record the measured hung-application, memory, and token results in the validation record, and document the reason in the protocol and operations references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Brings the Windows agent's c939189 into the integrator branch: WinEvent window events, input authority, the never-reused element registry with a fingerprint check, UIA failure diagnostics, and the opt-in plain Win32 fixture contract. The commit predates the macOS bounded observation provider and the Node operation layer, so the plan now records what is still open on Windows: reporting `effect`, scoping the ledger to the lease, the bounded observation work package, and reconciling the registry's LRU eviction and subtree release with the frozen contract. PROTOCOL keeps the mutation settlement text and adds the provider-timeout sentence and the windowEvents paragraph. The validation record keeps both platforms' evidence and states that macOS reports `effect` where Windows reports its settlement codes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on macOS The native protocol reserves window-closed for windows the platform reports destroyed, and says closeWindow publishes none for its id. The macOS helper pressed the close button and only then forgot the window, so the observer thread published the event whenever the destroyed notification arrived first. Against Calculator that was 4 of 4 closes. The helper now marks the window before the press, still records the destroyed notification in the event log, and removes the mark if the press fails. The fake host no longer publishes the event for closeWindow either, and the plan records the macOS answers to the Windows hit-test, reclamation, and event parity questions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…part A service given a DesktopAuthority requires a grant on every route, including ordinary WebDriver routes, unknown routes, and /status, and decides that before it reads the request body. Each session belongs to the grant that created it and is invisible to every other grant, a grant opens only the targets it names, and contention and quarantine data reach other grants without any grant, secret, or session identifier. Revoking a grant closes its sessions, including one still launching. A service without an authority behaves as before. Recovers the work a crashed session left uncommitted: restores the pre-queue ownership check that a mutation experiment had commented out, and fixes a revocation race that left a launching session holding its target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ants A create that authenticated before its grant was revoked could still reach the session manager and register a session owned by a dead grant. The manager now refuses creates for a revoked grant. A foreign session now answers with the same stack as a missing one, the client no longer prints its grant when serialized, a null authority is rejected instead of disabling authentication, and a flaky wrong-secret test credential is fixed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement the versioned descriptor and required trusted identity-verifier seam, macOS owner-validated storage, and grant transport for connected, attached, and WDIO-managed clients. Reject redirects and redact standard request-header inspection. Keep Windows storage and native process/listener verification explicitly fail-closed pending their adapters. Extend the opt-in native helper contract to prove protected-service probe routing without claiming real-app lifecycle or input qualification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…service contract The helper recorded every unsettled mutation process-wide, so one stalled call kept refusing mutations for every later session on the same helper even after the application that raised it was closed. It now records each abandoned mutation against the lease of the application it acted on, and closeApplication clears those records once it has verified the launched process exited. Attached applications keep running, so their records stay, and the refusal remains session-wide while any record exists. The opt-in protectedService native contract serves the Win32 fixture through a grant-protected service with the real helper behind it: grants, session ownership, revocation, and quarantine. Its stuck-provider scenario failed before this change. win32FixtureSupport shares the helper and fixture builders with the win32Fixture contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 79f52cdd-cbee-4c8e-9fd1-a1e0ea629959
A record name ending in a dot, or whose base is a Windows device name such as NUL or COM1, is not a safe regular file on Windows. The shared name assertion now rejects them on every platform, so a name is portable and the Windows ownership adapter does not need its own rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Brings in 0c2b636 so the Windows native private-storage work starts from the shared name rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 79f52cdd-cbee-4c8e-9fd1-a1e0ea629959
Windows clean-cache buildci timed out the CommonJS prebuild fallback child at 10 s twice while the full graph ran, although it takes about 300 ms alone. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add two restricted one-shot modes to the Windows desktop-driver helper, --private-storage and --service-identity. They are not FDR1 sessions: each reads one JSON request on stdin and writes one JSON line, and exits 0 only when the answer has "ok":true. Private storage opens and pins an owner-only root. It validates the ancestors, owner, protected DACL, reparse points, hard links, volume, and root identity from open handles, and publishes records through a temporary file and a non-replacing rename, so a record is never replaced and a racing publisher cannot win twice. Service identity describes a process and verifies a service claim: the live process start time, image, user, and interactive session, and that the claimed process owns every listener that can accept the loopback address. The self-test grows by 84 checks (548 in total) and covers the name, base64, root, ancestor-ACL, URL, and listener rules, tamper and race cases, and describe and verify against real processes and listeners. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 79f52cdd-cbee-4c8e-9fd1-a1e0ea629959
Add createWindowsPrivateStorage, describeWindowsService, and createWindowsServiceVerifier, which implement the shared PrivateStorage and service-verifier contracts on top of the helper's --private-storage and --service-identity modes. The adapters refuse every other platform, verify the helper artifact before the first request, pin the root and its identity on every operation, and resolve a verification to true only for a helper answer of exactly ok and verified. The only shared file changed is src/authority/index.ts, which gains the export lines. Unit tests run against a scripted helper on any host. Two opt-in native contracts (privateStorage and serviceIdentity) drive the real helper with no application: exclusive and concurrent publication, replaced, removed, and junction roots, an extra hard link, an ACL that grants another principal access, and claims that differ in start time, image, session, listener owner, or address. Document the Windows storage and verifier rules in the security, service, native helper, and Windows provider references, update the remaining-work notes, and add a changeset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 79f52cdd-cbee-4c8e-9fd1-a1e0ea629959
Add the macOS-approved paragraph for --private-storage and --service-identity to "Restricted helper modes": one-shot, one bounded JSON request on stdin, one JSON line out, exit 0 exactly when ok is true, and fixed failure reasons. FDR1, protocol.json, and src/native/types.ts are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 79f52cdd-cbee-4c8e-9fd1-a1e0ea629959
Re-authenticate queued session requests before dispatch, cancel opening launches and active host commands when a grant is revoked or the service closes, and wait for native settlement before releasing target reservations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the Windows side of the boundedObservation contract: a breadth-first sampler that pages UI Automation children one sibling at a time so every budget applies mid-level, pinned observe records that capacity never evicts (the references truncation instead), non-cascading releaseElements, and password-safe facts. Covered by synthetic, registry, and real-window self-tests, the Node validator over the self-test dump, and a 10,000-item Win32 fixture tree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 79f52cdd-cbee-4c8e-9fd1-a1e0ea629959
Add a --service-identity mode to the macOS native helper and the describeMacOSService/createMacOSServiceVerifier adapters, so a client can check that a published service descriptor names a live process of the current user and audit session that alone serves the claimed loopback listener (no same-user wildcard sharing and no SO_REUSEPORT). Extract the one-shot authority helper runner shared with the Windows adapters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add act's perform action: the service proves the element publishes the named action exactly once, then performs it by an opaque provider key. The macOS helper keys each AX action with an HMAC under a per-process secret, so duplicate custom-action names stay distinct and target addresses never leave the helper. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
serve --agent protects the registered target with a grant authority, issues one target-scoped grant, and publishes the private connection descriptor the Desktop Driver MCP adapter reads. Stopping withdraws the descriptor before revoking the grant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apple moved the Mac Automation Scripting Guide's UI chapter, so the old AutomatingUI.html address now returns 404 and fails test-links. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Qualify operation routes and the agent service/MCP adapter against the Win32 fixture at 100% and 150% scaling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fe83a2ec-9d6f-46de-aa59-002d495eca48
Add the stories and story operations, which list a Storybook target's catalog and render a story with marker verification, preview-region expiry, and indeterminate, non-quarantining failure reports. The WebDriver story routes share the same change path. The MCP adapter exposes them as desktop_story. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
storybook-desktop `driver --agent` protects the embedded Desktop Driver with a grant authority, publishes the private agent connection from the channel service, disables the channel's unauthenticated MCP endpoint, and keeps the macOS application lease current so agents attach without `storybook test`. desktop-test `serve --agent` now routes to integrations that declare `supportsAgent` and passes them the output format for their readiness document. Qualified live on macOS Fabric through the MCP adapter's desktop_story tool. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep bounded AX event logs truthful after their ring evicts old events, matching the Windows provider. Extend the native self-test to cover tail truncation, cleanup, and a deterministic 10,000-node observation bounded to 5,000 nodes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cache up to two sampled observations per session with signed page and delta cursors, expose observePage/observeDelta through the client and the MCP desktop_observe read/delta variants, and pin cached native IDs through the release ledger. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Long-lived serve, driver, and server commands now write their bounded summary when they fail before readiness. Integrations receive onReady. Storybook CLI tests use an isolated project root so they no longer overwrite a live app's generated lease and ready files. Records the live macOS M3 paging, delta, identity, and retention qualification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…port driver --agent on Windows and Win32 had no application lease to verify, so agents could not attach. The agent connection now names the app's verified AUMID or REX executable, resolved through resolveDesktopApplication's new attach option, and the command fails before starting services when no identity can be verified. The RN Test App Windows Debug host always loads its bundle from Metro on 8081, so driver now serves Metro on a platform's declared fixed smoke Metro port and fails clearly when that port is in use. Records Windows Fabric and Win32 Paper M3 real-app and Storybook agent qualification in the desktop-driver remaining-work plan. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fe83a2ec-9d6f-46de-aa59-002d495eca48
Both helpers' self-tests cover large trees and duplicates, and replacement, paging, deltas, repeated context, and the retention plateau passed live on macOS Fabric, Windows Fabric, and Win32 Paper. 200% and mixed-DPI scale stay with the M4 geometry work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Bundle size reportTree-shaken, minified production esbuild bundles with React and React Native runtimes externalized.
The job is advisory: size changes are reported but do not fail the pull request. Bundle or analysis errors still fail. |
Mark what changed since c84acb7: the MCP adapter and bounded operations, measured token and latency costs, corrected baseline claims, per-suggestion status, and the open Windows Storybook PR native-contract failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hosted Windows runners disable keyboard, physical click, and wheel input and cannot create a Graphics Capture item, so the Win32 fixture and operations contract scenarios that need them now skip explicitly under that policy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement the approved P1/P2 foundation and bounded P3 safety fences. Preserve historical measurements in package-owned status references and keep unfinished orchestration and platform qualification explicit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 5, 2026
Open
Follow-up work from the desktop layering plan now lives in #4435 and its sixteen sub-issues, which record dependencies and parallel waves. Link the tracker and the gate-specific sub-issues from the package status ledgers and the driver remaining-work plan, and remove the root planning document. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mes out The Windows helper's tree and find cache a window's whole subtree in one UI Automation transaction before enforcing the 5,000-node limit. A 10,000-item tree takes about 4 seconds to cache, so on slower hosted runners it outlasts the 5-second transaction timeout and was reported as automation-failed, which made the large-tree contract test flaky. When the subtree cache fails, count the subtree one child at a time, checking cancellation before each provider call, and refuse it with tree-too-large once it passes the limit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Morse (JasonVMo)
marked this pull request as ready for review
November 5, 2026 18:59
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Vladimir Morozov (vmoroz)
approved these changes
Nov 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Platforms Impacted
Description of changes
This implements the desktop driver agent plan in
remaining-work.mdand lays the foundation for theapproved desktop layering (approach A:
desktop-driverstays the neutral control plane,desktop-testis the neutralworkload host, and Storybook contributes to it). M3 (bounded native observations and identities) is complete on macOS
Fabric, Windows Fabric, and Win32 Paper. M0–M2, MR, M4, M5, and M6 remain open and are tracked in that file.
The packages are work in progress. Each package's
references/current-status.mdseparates what is done from what isstill in progress and lists the evidence required before full submission. Follow-up work is tracked in #4435 and its
16 sub-issues, which record dependencies and the waves that can proceed in parallel.
@fluentui-react-native/desktop-driversrc/operations/), served from the driver service atfurn/ops.observereads a bounded sample, with paged immutable samples throughobservePageand deltas throughobserveDelta.findsearches for elements.journalreturns the session's recorded operations.act: click, activate, type, clear, or perform a named accessibility action. It acts only on an element a search proved unique, and retried request ids return the first outcome.check: reads state, name, value, or text, or counts matches, with optional waits.sequence: runs up to eight act and check steps in one call.captureandartifact: hash-verified window and element PNGs.storiesandstory: list Storybook stories and render one.effect. The service quarantines a target whose mutation may have landed, persists quarantines, and offers operator recovery. Both helpers can probe whether an application incarnation has exited.src/authority/):story-args-unsupported, because they are not yet governed by a grant;launchModes.treeandfindrefuse a tree past 5,000 nodes withtree-too-largeeven when its whole-subtree cache outlasts the 5-second UIA transaction on a slow host. They count the subtree one child at a time instead of reportingautomation-failed; this was the cause of an intermittent large-tree contract failure on hosted Windows.native/PROTOCOL.md); the new fields are additive and negotiated.@fluentui-react-native/desktop-driver-mcp(new): an MCP stdio adapter with six tools over the operation routes:desktop_session,desktop_observe,desktop_act,desktop_check,desktop_capture, anddesktop_story. It connects only through a verified private descriptor.session-uncertain.@fluentui-react-native/desktop-test:serve --agentprotects a registered target and publishes the agent connection.supportsAgentare served through it, and they receiveonReadyin their run context.resolveDesktopApplicationgains anattachoption.integration.run,supportsAgent, and adapter-written readiness) is documented as transitional; Infrastructure: Convert Storybook lifecycle takeover into desktop-test resource contributions #4449 replaces it with resource contributions.@fluentui-react-native/storybook-desktop:driver --agentserves an agent connection for the Storybook app.Docs and guidance:
references/current-status.mdindesktop-driver,desktop-test, anddesktop-driver-mcp, linked from each README. They hold capability tables, native evidence, and the measurement ledgers (including the measured token-cost tables) that replace the rootdriver-capabilities.mdresearch report.Verification
Automated. After
0569e1b86, the branch changes only reference docs and the Windows helper's oversized-tree path, so the JavaScript results from0569e1b86still apply. The helper change was compiled and its native contract run only by Windows CI; no local Windows host was used.bc3f1811d)68e30f11f)bounds a large tree and stays responsive.a6080c165, runs 42080241662 and 42080241660)bounds a large tree and stays responsive(automation-failedinstead oftree-too-large); the same failure occurred at1f27dcc94. Fixed in68e30f11f.0569e1b86, runs 41996864667 and 41996864832)yarn clean-lage-cache && yarn lage buildci(macOS, at0569e1b86)yarn lage test-links --verbose --grouped(macOS, at0569e1b86)yarn change:checkyarn markdown-link-check -c .github/markdown-link-check-config.jsonon the four edited references and the Windows helper README (head)yarn oxfmt --checkandyarn check-agent-instructions(head)yarn test(desktop-driver, at0569e1b86)yarn test(desktop-driver-mcp, at0569e1b86)yarn test(desktop-test, at0569e1b86)yarn test(storybook-desktop, at0569e1b86)--self-test(at600235c35)yarn clean-lage-cache && yarn lage buildci(Windows, at600235c35)Manual and live (before the layering-foundation commit):
storybook driver --agent):storybook driver --agent):/statusrefusal passed.desktop-test serve --agent. They were driven over MCP: act, check, capture, progress, and cancellation.Not run:
tree/findhelper commands, with no shared operation contract, authority, or settlement.Pull request checklist
This PR has considered (when applicable):