Repository navigation
Fix desktop CLI errors, shutdown hangs, and run verdicts - #4478
Merged
Jason Morse (JasonVMo) merged 15 commits intoNov 10, 2026
Merged
Conversation
Commander copies exit and output settings only into commands created after they are set, so subcommand usage errors went to an in-memory buffer and exited silently. Apply the reporter output and exit override to the whole command tree. - Route usage errors through the selected reporter: one error record in json and ndjson, a stderr message in plain. - Throw InvalidArgumentError from option parsers so they are reported as usage errors instead of internal. - Resolve the reporter mode once and before any action, so an invalid DESKTOP_REPORTER no longer lets serve start listening. - Write the serve readiness record inside the try so a write failure still closes the server. - Make the NDJSON error record the single terminal record, matching JSON. - Map WebDriver error codes to matching CLI error kinds, including busy for target-busy session failures. - Clip doctor error messages to the same byte limit as other errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Option parsers and input checks threw TypeError, which the reporter classified as internal. Throw InvalidArgumentError so the error is reported as a usage error in every mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Describe the single error record written for usage and configuration failures in json and ndjson modes, and correct the NDJSON terminal record rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Bundle size reportTree-shaken, minified production esbuild bundles with React and React Native runtimes externalized.
The job is advisory: size changes are reported but do not fail the pull request. Bundle or analysis errors still fail. |
Jason Morse (JasonVMo)
enabled auto-merge (squash)
November 10, 2026 12:42
Retried tests no longer pass when a failed attempt is followed only by skipped attempts, selected worker work that reported no tests is incomplete, and presentation errors become warnings that keep the verdict. Incomplete lane runs list their unmet work and throw a typed error, secrets are sanitized once at finalize, run-query errors use a closed code set without filesystem paths, NDJSON milestones never suppress failures, run.json writes are throttled, and IPC producers cannot send collector-only events. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The producer's attempt number is opaque, so retry numbering no longer corrupts workerAttempt, a spec that ran no tests is reported as a warning, and the generated config raises the framework log level so spec import failures show. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Integrations borrow the supervisor scope without closing it; the CLI awaits the integration run outside the scope with a bound, and scope draining is bounded (drainTimeoutMs), so Storybook serve and story tests exit after SIGINT (LIFE-1). storybook-desktop closes only a scope it created. - Stage, suite and storybook-desktop stage commands receive the scope signal; abort stops the POSIX process group as failed, a late spawn is refused, and sync spawn failures leave nothing tracked (LIFE-2, LIFE-9). - Interrupt handlers cover SIGINT/SIGTERM/SIGHUP until cleanup completes; a second signal force-kills owned trees, a third exits (LIFE-3..5). - Close disposes services and resources in one reverse order, isolates observer failures and releases spawn-failure claims (LIFE-10, LIFE-13). - Per-user claim directory, atomic claim replacement and start-time tolerance (LIFE-6, LIFE-8 partial); one short private local-socket helper for process control and reporting transport, with IPC only for services (LIFE-7, REP-11). - Generated WDIO config reads settings from a file; PowerShell reads the AppX name from the environment; suiteTimeoutMs bounds the launcher; NUL is rejected in command args and env (LIFE-11, LIFE-12, LIFE-15, LIFE-17). - Tests: commands, services, signals, localSocket, transport negatives, lifecycle and CLI-interrupt contracts, public-entry consumer, borrowed-scope regression (LIFE-16, REP-16). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Records the fixed idle-CLI-after-interrupt cause, signal escalation, bounded draining, reverse disposal, the per-user claim directory, suiteTimeoutMs and the NUL rule, and the borrowed storybook-desktop scope. Adds changesets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Morse (JasonVMo)
disabled auto-merge
November 10, 2026 12:53
Combines user/jasonvmo/desktop-test-verdicts (REP) with the lifecycle work because the reporting contract asserted the stalled-console verdict REP-4 removes. Resolution against the combined behavior: - reporting contract: a stalled or failing console is a warning, so the run passes; the final-line failure is journaled after run-finished, which the contract now asserts explicitly. - New spec-load-failure WDIO contract: a spec that throws while loading reports no tests yet fails rather than becoming incomplete under REP-2. - reporting.md: SIGHUP and stopped-stage wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the one-second start-time tolerance from workload-claim inspection. The recorded start time comes from the Node.js clock rather than the operating system, and a tolerance would let a reused process ID match a live owner. Inspection compares exactly again; the lifecycle and status references record that `pid-reused` is unproven until the claim records a helper-described start time (LIFE-8 deferred). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Export the desktop-test interrupt policy as `handleDesktopInterrupts` and use it in the storybook-desktop CLI and standalone Storybook channel. The handlers now stay installed until cleanup completes, SIGHUP is handled, a second interrupt force-stops owned commands, and a third exits without cleanup. Previously the one-shot `process.once` listeners were consumed by the first signal, so a second SIGINT or a SIGHUP killed the CLI mid-cleanup. An interrupted one-shot command, such as a stage, now reports `incomplete` instead of passed with exit 0. Tests: interrupted stage under SIGINT/SIGINT and SIGHUP/SIGTERM, standalone channel SIGHUP, and an owned service escalating on a second SIGINT; each fails without the change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Run `taskkill /T /F` synchronously, and only while the command's exit is unrecorded. Windows reuses a process ID only after every handle to the process closes, and Node closes the child handle while processing the exit on the same event loop, so the ID still names the owned leader. A failed taskkill is now judged by the recorded completion rather than by probing a process ID that may have been reused, and the asynchronous force-stop spawn that could race the exit is gone. This is not descendant containment: `/T` follows parent process IDs, so a descendant whose parent already exited still escapes until Job Object containment exists. Emulated-Windows tests cover the exit gate, the synchronous call, and a failed taskkill for a leader that keeps running. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The spec-load-failure contract now also requires the failed work item to be labelled with the spec that threw. The original error text remains in the worker command log only; run.json and junit.xml do not carry it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep overlapping Storybook command handling and desktop-test reporting contracts in a single pull request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> # Conflicts: # packages/agentic/storybook-desktop/src/cli/createDesktopStorybookCommand.ts
The contract runner printed only the outer error stack, so an AggregateError from disposal hid the cleanup failure that caused it. Print the error with `util.inspect` so members and causes are included. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
4 of 10 tasks
Vladimir Morozov (vmoroz)
approved these changes
Nov 10, 2026
Jason Morse (JasonVMo)
deleted the
user/jasonvmo/desktop-driver-cli-errors
branch
November 10, 2026 23:12
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Platforms Impacted
Description of changes
Related to #4435.
This PR combines three groups of fixes from the desktop testing review, because they change the same files (
createDesktopStorybookCommand.tsand the desktop-test contracts):@fluentui-react-native/desktop-driver,@fluentui-react-native/storybook-desktop): usage errors are reported in every reporter mode, and the NDJSON error record is the terminal record.@fluentui-react-native/desktop-test,@fluentui-react-native/storybook-desktop): shutdown hangs, cancellation, and signal handling (LIFE-1 to LIFE-7, LIFE-9 to LIFE-13, LIFE-15 to LIFE-17, and the transport parts of REP-11 and REP-16).@fluentui-react-native/desktop-test): skipped, empty, or incomplete work never counts as passing coverage (REP-1 to REP-8, REP-10, REP-12, and parts of REP-15 and REP-16).The body of commit
5680dc365lists the wrong LIFE IDs. Use the mapping below; the commits were not amended.CLI errors
Subcommand usage errors in
desktop-driverexited 1 with no output in plain, JSON and NDJSON modes.servekept listening after an invalidDESKTOP_REPORTER. NDJSON errors were wrapped in aterminalrecord, unlike JSON.exitOverrideand output settings only to commands created after they are set, but the root call ran after the subcommands existed. The newconfigureDesktopCliCommandTreeinsrc/cli/reporter.tsconfigures the tree recursively, sostories listwithout--urlprintserror: required option '--url <url>' not specifiedto stderr, or ausageerror record in JSON and NDJSON.createDesktopDriverCommand.tsresolves the reporter mode once, before any action, soservewith an invalidDESKTOP_REPORTERexits 1 without listening. Serve writes its readiness line inside the error-handlingtry.{"schemaVersion":1,"type":"error","error":{...}}is the NDJSON terminal record, identical to JSON. desktop-test's CLI uses the same helper, so its NDJSON usage errors change the same way.InvalidArgumentError, so bad ports, depths, and choices areusagerather thaninternal. WebDriverinvalid argumentandinvalid selectormap tousage,unsupported operationtounsupported, andsession not createdwithtarget-busytobusy; unknown codes stayinternal.doctormessages are clipped to 1024 bytes instead of up to 1 MiB.createDesktopDriverCommand: it always setsexitOverride, soparseAsyncrejects with aCommanderErrorinstead of exiting. The only in-repo callers are the CLI and its tests.Lifecycle
serveand story tests never exit.desktop-testtracked the integration run in its service scope withservices.guard(). storybook-desktop closed that borrowed scope from inside the run (driver(),server(), andrunSmokeLifecycle()), andclose()waited for the run, so each waited on the other. This also caused the recorded "idle CLI after Ctrl+C" andserve --agentinterrupt limitation.cli.tsno longer tracks the integration run. It races the run against a stop request, and after a stop or a scope failure it waits up to 60 s for the integration to unwind before closing the scope.ownsServices).DesktopServiceScopedrains tracked work with a bound (drainTimeoutMs, default 60 s). Work still running after that is reported as a cleanup failure and abandoned.runner.run()andrunner.start()take the scope signal. Stage, suite, and storybook-desktopexecutePlancommands now stop when the scope stops, withfailed: true. A command cannot start after the abort.incomplete, not passed, in both the desktop-test CLI and the storybook-desktop CLI. Before this PR an interrupted storybook-desktop stage such asbundlereported[desktop] PASSEDwith exit 0.signals.tskeeps the SIGINT, SIGTERM, and SIGHUP handlers installed until cleanup and the session close finish. A second signal force-stops the owned process trees, and a third exits with status 128 + the signal number. Isolating the helper's process group is in desktop-driver and is not part of this PR.handleDesktopInterrupts. The storybook-desktop CLI (createDesktopStorybookCommand.ts) and the standalone Storybook channel (config/server.cjs) now use it instead ofprocess.oncelisteners. Those listeners were consumed by the first signal, so a second SIGINT, or any SIGHUP, killed the CLI in the middle of cleanup. The handlers are released only after cleanup and the session close finish. The channel can force nothing, so its second signal only warns.ownedService.ts) escalate on a second signal through the same policy.taskkill /T /Fnow runs synchronously (spawnSync, bounded by the shutdown timeout), and only while the command's exit is unrecorded. Windows reuses a process ID only after every handle to the process closes, and Node closes the child handle while it processes the exit on the same event loop, so the ID still names the owned leader when taskkill resolves it. The old asynchronous force-stop spawn could race the exit. A failed taskkill is judged by the recorded completion; the old fallback probedprocess.kill(pid, 0), which could match a reused ID./Tfollows parent process IDs, so a descendant whose parent already exited escapes. Only Job Objects close that gap, and they are deferred. Whether taskkill also checks process creation times before matching a stale parent ID was not verified.localSocket.tsgivesprocessControl.tsandreporting/transport.tsa short socket path in a private owner-only (0700) directory, with a check against the platform path-length limit. It falls back to/tmpwhenTMPDIRis too long. Only service commands get a process-control IPC server.--application-incarnationmode, which verifies a pid against a helper-described start time but cannot describe one. A real fix needs a helper "describe process" mode (native Swift and C++), a helper available when the claim is taken, and a claim schema bump.lifecycle.mdandcurrent-status.mdnow say thatpid-reusedis unproven until then.TMPDIR:~/Library/Caches/com.microsoft.fluentui-react-native.desktop-test/workload%LOCALAPPDATA%\Microsoft\FluentUIReactNative\desktop-test\workload~/.cache/fluentui-react-native/desktop-test/workloadDESKTOP_WDIO_SETTINGS. It no longer splicesJSON.stringify(settings)into code.$env:FURN_APPX_NAME.resource.dispose, which nothing called before, is now how that pass stops the service.createTarget. An integration'screateTargetcan only add the story context (storyOrchestratorandstoryRootTestId). It cannot replace the target's identity or launch modes.lifecyclecovers in-flight cancellation, release of the claim after a spawn failure, and an observer that fails during close.interrupt-clisends SIGINT to the CLI during a stage and expects exit 1 within 10 s, a dead stage process, and anincompleterun.spec-load-failureuses a WDIO spec that throws while loading.resourcesmode now checks the default claim directory.commands.test.tsemulates Windows against a mockedtaskkill(no call after a recorded exit, even while a grandchild holds the output open; a synchronous call for a running leader; and a stop that fails with taskkill's diagnostic when the leader keeps running).ownedService.test.tsescalates an owned service on a second SIGINT. storybook-desktopinterrupts.test.tsinterrupts a stage with SIGINT then SIGINT, and with SIGHUP then SIGTERM.serverIntegration.test.tscloses a standalone channel on SIGHUP. The three signal tests fail without the change.suiteTimeoutMs. By default the WDIO launcher bound grows withtimeoutMs, spec groups, and retries.reporting/transport.ts.Reporting verdicts
failed; otherwise the last attempt's status stands.runQueries.ts, results, JUnit, and the summary all use this rule.incomplete. The WDIO reporter warns about a spec that ran no tests. The Jasmine and Mocha framework loggers log atwarn, so a spec import failure shows up.<run>and<artifacts>in projected text.assertSuccessfulthrows a typedDesktopIncompleteError, which is recorded as the run's verdict.DesktopFailureRegistrysanitizes failures and late-registered secrets when the run is finalized.run.jsonare throttled.flush()and finalize always write it.run-finished.wdio/reporter.ts.Combining the groups
reportingcontract now asserts REP-4's behavior. A slow sink and a late console failure leave the runpassed. The summary sayspassed,junit.xmldoes not list the late failure, and stderr shows[desktop] WARNING: Console output stopped. The warning is journaled afterrun-finished, as{ level: 'warning', message: 'Console output stopped: late terminal fixture' }.incomplete, an empty spec isincomplete, and a spec that throws while loading isfailed. For a spec-load failure,run.jsonhas onefailedwork item labelled with the spec, and an infrastructure failure "selected work did not produce complete executable results". The original error text is only in the worker command log (commands/*.log).run.jsonandjunit.xmldo not carry it; that remains a reporting gap.reporter-clicontract asserts the JSON record), and the delegated success record{"type":"terminal","result":{"runId","delegated":true,"exitCode":0}}is unchanged (asserted by the delegated-output contract).wdio/run.tsstill runsassertSuccessfulbefore it checks the exit code, so an empty spec staysincomplete. Checking the exit code first would report that spec as an infrastructure failure.Documentation and changesets
references/:lifecycle.md,commands.md,configuration.md,reporting.md(a new Verdicts section, plus signals),current-status.md, andnative-qualification.md(the claim paths and Ctrl+C).references/commands.md: integrations borrow the scope, and the signal policy (SIGHUP, escalation, andincompleteinterrupted one-shot commands).references/getting-started.mdandreferences/remaining-work.md: the error record shape and the NDJSON terminal rule.cli-usage-errors.md(desktop-driver and storybook-desktop patch),life-desktop-test-shutdown.mdandrep-desktop-test-verdicts.md(desktop-test patch), andlife-storybook-desktop-scope.md(storybook-desktop patch).Not in this PR
reporter.ts.sbd --reporter json) report the usage message(outputHelp), and its plain-mode configuration errors still print a stack trace.internal:readManifestandstories explainfor a missing story, the SessionManager "does not allow ... sessions" error, and the "is not registered" server error. Theunknown-targetandattach-onlycodes are reserved; nothing produces them yet.run.json.cancelledrun status.Verification
Local runs were on macOS. The native and aggregate runs held the shared native lock. Final PR head
8c202b74a.Build:
yarn buildinpackages/agentic/storybook-desktop(tsc -b, which also builds desktop-driver and desktop-test) passed.packages/agentic/desktop-driver:yarn test src/clipassed: 2 suites, 33 tests (reporter.test.tsandDesktopDriverCli.test.ts).packages/agentic/desktop-test:yarn format,yarn lint, andyarn buildpassed.yarn testpassed: 23 suites, 379 tests passed, 1 skipped (an existing Windows-only test). All 30 contract modes passed, includingreporting,lifecycle,interrupt-cli,spec-load-failure,resources,packageBoundary, thereporter-clierror record, and the delegated success record.desktop-test CLI by hand: a usage error with
--reporter jsonand with--reporter ndjsonprints{"schemaVersion":1,"type":"error","error":{"code":"usage",...}}, exits 1, and leaves stderr empty.packages/agentic/storybook-desktop:yarn format,yarn lint,yarn build, andyarn testpassed: 23 suites, 338 tests passed, 4 skipped (existing PowerShell owned-service tests that run only on Windows). These tests fail on the code without the fix:ownsServicesguard is removed.interrupts.test.tscases and the standalone SIGHUP test fail on the code beforef01039fcb: the CLI is killed by the second SIGINT or by SIGHUP, and the channel exits by signal.Root:
yarn oxfmt --checkon the edited markdown passed,yarn check-agent-instructionspassed, andyarn markdown-link-check -c .github/markdown-link-check-config.json <file>passed for each edited markdown file.Root, under the lock, on
8c202b74a:yarn clean-lage-cache && yarn lage buildci --concurrency 4passed: 178 tasks succeeded, 0 failed (1 min 9 s).Serve and Ctrl+C. In
apps/storybook,yarn desktop-test --target macos servewas started as its own process group, and SIGINT was sent to the group once it was ready.666e32983): the CLI was still running 90 s later and still held its claim.8c202b74a: the CLI was ready in 2 s, exited 1 s after SIGINT with code 0, and reportedPASSED: lifecycle (no tests). No owned process, driver helper, or app was left, and the claim directory was empty.One story, standalone path, on
8c202b74a. The commands wereyarn storybook driver --macos, thenyarn storybook run --macos(exit 0, 18 s), then a wait for the app's Storybook runtime, thenyarn storybook test --macos --test "exposes enabled button semantics".1 selected, 1 executed, 1 passed.runreturned failed withbridge-unavailableand exit 1. That is expected: authored-test navigation does not wait for the bridge.One story, owned-runtime path, on
8c202b74a: not qualified. Two commands were tried:STORYBOOK_SMOKE_STORY=components-button--default yarn desktop-test --target macos test --suite stories, the integration path behind LIFE-1, run twice.STORYBOOK_SMOKE_STORY=components-button--default yarn storybook --verbose smoke --macos --mode stories-and-tests, run once.All three runs exited on their own in 30 to 35 s with code 1 and
FAILED. Each stopped Metro, the channel, and the app, and released its claim. Each failed in the driver's firstnewSession, about 4 s after the app's runtime started, with the macOS helper error "Accessibility returned an incomplete or internally inconsistent window enumeration." The app was then stopped during teardown, so the driver's concurrent bridge-epoch attach also reportedattach-failed.This PR does not change the helper's Swift sources or the attach sequence. The macOS-helper branch (Fix macOS desktop-driver input accounting and application cleanup #4482) separately passed 209 story traversals and 2 authored tests; that does not qualify this branch. A run on this branch's base was not repeated, so whether this is environmental or a base-helper race is unproven.
Combined integration follow-up: passed. The local integration of Harden desktop-driver native artifact cache and build locks #4477 through Check cancellation in more Windows desktop-driver helper waits #4483 merges cleanly at
921a7c9ce. Its earlier headed9353e83passed fresh-cachebuildci(178 tasks); the only later change is contract AggregateError diagnostics, andyarn test contractsin desktop-test passed all 30 cases at921a7c9ce.yarn storybook build-driver --macospassed, building a helper from the combined sources.yarn desktop-driver doctor --platform macos --permissionsreported ready, unlocked, and Accessibility/PostEvent/ScreenCapture granted without prompting or changing permissions.yarn storybook bundle --macos,yarn storybook prep --macos, andyarn storybook build --macoseach passed in the integration checkout; no app from another worktree was borrowed.STORYBOOK_SMOKE_STORY=components-button--default yarn desktop-test --target macos test --suite storiespassed, exit 0 in 291 s: 209 story traversals and 2 authored Button tests, all 211 selected/executed/passed, with zero failed, skipped, incomplete, retried, or flaky work. Run id:76caf360-1703-4cd3-b81c-1af465939dc9.50309ad4...) was the only compatible shared-cache entry. Its use is inferred from that selection state; the run record does not attest the helper path. This successful combined-head run does not isolate the cause of the earlier lifecycle-only attach failures.b16d0349c; the subsequent Windows cancellation follow-up114c2f605is not part of this qualification.STORYBOOK_SMOKE_STORYfilters authored callbacks, not catalog traversal. The three failed owned-runtime attempts stopped before traversal with zero selected work; they provide teardown evidence, not passing story coverage.Hosted CI at
8c202b74a: passed. All checks succeeded, including JS, macOS Fabric/Paper Storybook, iOS/macOS builds, Windows Storybook, and both Win32 Storybook lanes.Not specifically qualified: physical Ctrl+C and real
taskkillon Windows. The dedicated stop regressions use mocked/emulated Windows behavior; passing hosted lanes does not replace that interactive qualification.current-status.mdrecords the remaining Windows gap.desktop-test servewith the Storybook integration was still running 90 s after Ctrl+C and still held the workload claim. A cancelled stage waited for its command and could report passed. A failed test retried into skipped attempts passed, a spec with no tests passed, and a stalled console failed a passing run.incomplete. These runs arefailed,incomplete, andpassedwith a warning.Pull request checklist
This PR has considered (when applicable):