Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/cli-usage-errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@fluentui-react-native/desktop-driver': patch
'@fluentui-react-native/storybook-desktop': patch
---

Report desktop-driver CLI usage errors in every reporter mode. Subcommand usage errors now reach stderr (plain) or a single `error` record (json and ndjson) and exit 1 instead of exiting silently. An invalid `DESKTOP_REPORTER` no longer lets `serve` start. NDJSON failures emit the error record as their terminal record, matching JSON. WebDriver errors map to matching CLI error codes, and `doctor` messages are clipped.
5 changes: 5 additions & 0 deletions .changeset/life-desktop-test-shutdown.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@fluentui-react-native/desktop-test': patch
---

Fix shutdown hangs and cancellation. An integration's run is no longer tracked in the scope it borrows, so a Storybook `serve` or story test exits after SIGINT instead of idling, and scope draining is bounded. Cancellation stops running stage and suite commands, including their POSIX process groups, reports them failed, and refuses late spawns. A second interrupt force-stops owned process trees, SIGHUP is handled, and handlers stay installed until cleanup completes; `handleDesktopInterrupts` exports that policy for integrations. On Windows, `taskkill` runs synchronously and only before the command's exit is recorded, so it cannot address a reused process ID. Close disposes services and resources in one reverse order even when an observer throws. Workload claims move to a per-user directory independent of TMPDIR and are replaced atomically; local sockets use short private paths. Generated WDIO configs read settings from a file, suites accept `suiteTimeoutMs`, and command arguments and environment reject NUL.
5 changes: 5 additions & 0 deletions .changeset/life-storybook-desktop-scope.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@fluentui-react-native/storybook-desktop': patch
---

Leave an injected desktop-test service scope to its owner instead of closing it from `driver`, `server`, and the smoke lifecycle, which made a supervising `desktop-test serve` or story test wait on itself after a stop. Stage commands now receive the scope's cancellation signal. The CLI and the standalone Storybook channel keep their interrupt handlers until cleanup completes, handle SIGHUP, and force-stop owned commands on a second interrupt; an interrupted one-shot command reports `incomplete` instead of passed.
5 changes: 5 additions & 0 deletions .changeset/rep-desktop-test-verdicts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@fluentui-react-native/desktop-test': patch
---

Fix verdicts that passed without coverage. A test takes the status of its decisive attempt, so a failed attempt followed only by skipped or unsupported retries stays failed. Selected worker work that reports no tests is `incomplete`, and the WDIO reporter warns when a spec ran no tests, with the jasmine and mocha framework loggers raised to `warn` so an import failure is visible. WDIO attempts keep their arrival order whatever numbers the runner reports. Console stalls, closed pipes, and queue overflow are warnings that no longer fail a passing run, and the final line comes from the terminal result. Incomplete runs list their unmet work and a JUnit message that names it, and result checks throw a typed `DesktopIncompleteError` that reporters record as the run's verdict instead of an infrastructure failure. Failures and late-registered secrets are sanitized once at finalization, and run-query errors use a closed code set without filesystem paths. NDJSON milestones no longer suppress a later failure after many skips, record IDs are not clipped, a projection failure still emits a summary record, `run.json` snapshots are throttled, and producers can no longer send `run-finished`.
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,10 @@ desktop-driver stories list --platform windows --url http://127.0.0.1:7007 --tar
`--reporter auto|plain|github-actions|json|ndjson` is also configurable with
`DESKTOP_REPORTER`; `--json` aliases `--reporter json`. In `auto`, existing
indented JSON is retained. One-shot commands write one result, while `serve`
emits one readiness result and then waits for shutdown.
emits one readiness result and then waits for shutdown. A usage or
configuration error in `json` or `ndjson` writes one `error` record
(`{"schemaVersion":1,"type":"error","error":{"code","message"}}`) to stdout and
exits 1; `plain` writes the message to stderr.

## Resolve and diagnose

Expand Down
14 changes: 7 additions & 7 deletions packages/agentic/desktop-driver/references/remaining-work.md
Original file line number Diff line number Diff line change
Expand Up @@ -425,12 +425,12 @@ alias for `--reporter json`. Resolve the mode from the flag, then
`DESKTOP_REPORTER`, then configuration, then `auto`. Do not try to detect
whether the caller is an agent.

| Mode | stdout | stderr |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `auto` (default) | `github-actions` when `GITHUB_ACTIONS=true`; otherwise `plain`. A future interactive renderer may be selected only for a TTY. | Current diagnostics |
| `plain`, `github-actions` | Existing human renderers. `reporting.console` remains a compatible configuration source. | Current diagnostics |
| `json` | Exactly one compact document at exit. Long-lived commands (`serve`, Storybook `driver` and `server`) print one readiness document instead and nothing afterward. | Empty, except for failures before mode resolution or loss of stdout |
| `ndjson` | Compact LF-terminated records ending with exactly one terminal record, identical to the `json` document. One-shot commands emit one record. | Same as `json` |
| Mode | stdout | stderr |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `auto` (default) | `github-actions` when `GITHUB_ACTIONS=true`; otherwise `plain`. A future interactive renderer may be selected only for a TTY. | Current diagnostics |
| `plain`, `github-actions` | Existing human renderers. `reporting.console` remains a compatible configuration source. | Current diagnostics |
| `json` | Exactly one compact document at exit. Long-lived commands (`serve`, Storybook `driver` and `server`) print one readiness document instead and nothing afterward. | Empty, except for failures before mode resolution or loss of stdout |
| `ndjson` | Compact LF-terminated records. A run ends with exactly one `terminal` record, identical to the `json` document. A failure before a run starts emits only its `error` record, identical to `json`. | Same as `json` |

`--events <milestones|progress|all>` applies to `ndjson`; the default is
`milestones`:
Expand Down Expand Up @@ -714,7 +714,7 @@ exit-path gaps.
**Complete when:** every reporter mode is tested on every exit path: pass,
test failure, infrastructure failure, usage or configuration error,
interruption, and closed stdout. Structured stdout must parse as exactly one
document or as valid NDJSON ending in one terminal record, and structured
document or as valid NDJSON ending in one terminal or error record, and structured
stderr must be empty on normal paths. Documents and default streams stay within
limits for synthetic runs with at least 10,000 tests, 1,000 failures, and
1 MiB messages. Secrets remain redacted, and delegated runs render once.
Expand Down
106 changes: 105 additions & 1 deletion packages/agentic/desktop-driver/src/cli/DesktopDriverCli.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import path from 'node:path';
import { NativeDriverError } from '../native/NativeDriverError';
import type { NativeDriverArtifact } from '../native/types';
import type { DesktopStoryManifest } from '../storybook.js';
import { createDesktopDriverCommand } from './createDesktopDriverCommand';
import { createDesktopDriverCommand, runDesktopDriverCli } from './createDesktopDriverCommand';

jest.setTimeout(30_000);

Expand Down Expand Up @@ -134,6 +134,84 @@ describe('desktop-driver CLI', () => {
}
});

test('reports subcommand usage errors on stderr in plain mode', async () => {
const result = await runInProcessCli(['stories', 'list']);

expect(result.stdout).toBe('');
expect(result.stderr).toContain("required option '--url <url>' not specified");
expect(result.exitCode).toBe(1);
});

test('reports subcommand usage errors as one structured record in JSON and NDJSON modes', async () => {
const json = await runInProcessCli(['--reporter', 'json', 'stories', 'list']);
const ndjson = await runInProcessCli(['--reporter', 'ndjson', 'stories', 'list']);

for (const result of [json, ndjson]) {
expect(result.stderr).toBe('');
expect(result.exitCode).toBe(1);
expect(JSON.parse(result.stdout)).toMatchObject({
schemaVersion: 1,
type: 'error',
error: { code: 'usage', message: expect.stringContaining("required option '--url <url>' not specified") },
});
}
expect(ndjson.stdout).toBe(json.stdout);
});

test('reports bad choices and invalid option values as usage errors', async () => {
const choice = await runInProcessCli([
'--reporter',
'ndjson',
'stories',
'list',
'--platform',
'linux',
'--url',
'https://127.0.0.1:1',
'--target',
'target',
]);
const port = await runInProcessCli(['--reporter', 'json', 'serve', '--port', 'abc']);

expect(JSON.parse(choice.stdout)).toMatchObject({
type: 'error',
error: { code: 'usage', message: expect.stringContaining('Allowed choices are macos, windows') },
});
expect(JSON.parse(port.stdout)).toMatchObject({
type: 'error',
error: { code: 'usage', message: expect.stringContaining('Port must be an integer between 0 and 65535') },
});
});

test('does not start serve when DESKTOP_REPORTER is invalid', async () => {
const result = await runInProcessCli(['serve', '--port', '0'], 'xml');

expect(result.stdout).toBe('');
expect(result.stderr).toContain('Reporter must be one of auto, plain, github-actions, json, ndjson. Received "xml".');
expect(result.exitCode).toBe(1);
});

test('clips long doctor failure messages', async () => {
const output: string[] = [];
const program = createDesktopDriverCommand({
resolveDriver: jest.fn(async () => {
throw new Error('native helper failure '.repeat(200));
}),
stdout: {
write(value) {
output.push(String(value));
return true;
},
},
});

await program.parseAsync(['node', 'test', 'doctor', '--platform', 'macos']);

const message = JSON.parse(output.join('')).error.message as string;
expect(Buffer.byteLength(message)).toBeLessThanOrEqual(1024);
expect(message.endsWith('…')).toBe(true);
});

test('does not crash when the consumer closes stdout before a structured error is written', async () => {
const response = await new Promise<{ code: number | null; stderr: string }>((resolve, reject) => {
const child = spawnCli(['--reporter', 'json', 'unknown']);
Expand Down Expand Up @@ -417,6 +495,32 @@ describe('desktop-driver CLI', () => {
}
});

async function runInProcessCli(argv: readonly string[], reporterEnvironment?: string) {
const previousReporter = process.env.DESKTOP_REPORTER;
const stdout: string[] = [];
const stderr: string[] = [];
const writeOut = jest.spyOn(process.stdout, 'write').mockImplementation((chunk: string | Uint8Array) => {
stdout.push(String(chunk));
return true;
});
const writeErr = jest.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array) => {
stderr.push(String(chunk));
return true;
});
try {
if (reporterEnvironment === undefined) delete process.env.DESKTOP_REPORTER;
else process.env.DESKTOP_REPORTER = reporterEnvironment;
process.exitCode = undefined;
await runDesktopDriverCli(['node', 'desktop-driver', ...argv]);
return { stdout: stdout.join(''), stderr: stderr.join(''), exitCode: process.exitCode };
} finally {
writeOut.mockRestore();
writeErr.mockRestore();
if (previousReporter === undefined) delete process.env.DESKTOP_REPORTER;
else process.env.DESKTOP_REPORTER = previousReporter;
}
}

function spawnCli(args: readonly string[]) {
return spawn(process.execPath, [path.resolve(__dirname, '../../config/cli.cjs'), ...args], {
stdio: ['ignore', 'pipe', 'pipe'],
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs';

import { Command, Option } from 'commander';
import { Command, InvalidArgumentError, Option } from 'commander';

import { connectDesktopAgent } from '../agent/DesktopAgent.js';
import { validateDesktopStoryTests } from '../authoring/storyTests.js';
import { FakeDesktopHost } from '../hosts/fake/FakeDesktopHost.js';
import { buildNativeDesktopDriver, resolveNativeDesktopDriver } from '../native/nativeDriver.js';
import {
configureDesktopCliCommandTree,
describeDesktopCliError,
desktopReporterModes,
formatDesktopReporterError,
formatDesktopReporterValue,
Expand Down Expand Up @@ -66,18 +68,22 @@ export function createDesktopDriverCommand(options: CreateDesktopDriverCommandOp
.name('desktop-driver')
.description('Drive React Native desktop applications through W3C WebDriver without Appium.')
.addOption(new Option('--reporter <mode>', 'output format').choices([...desktopReporterModes]))
.option('--json', 'alias for --reporter json')
.configureOutput({
writeErr: (value) => stderr.write(value),
writeOut: (value) => stdout.write(value),
});
.option('--json', 'alias for --reporter json');
let resolvedReporterMode: ResolvedDesktopReporterMode | undefined;
const reporterMode = (): ResolvedDesktopReporterMode => {
const flags = program.opts<{ reporter?: string; json?: boolean }>();
return resolveDesktopReporterMode({
flag: flags.json ? 'json' : flags.reporter,
environment: process.env.DESKTOP_REPORTER,
});
if (!resolvedReporterMode) {
const flags = program.opts<{ reporter?: string; json?: boolean }>();
resolvedReporterMode = resolveDesktopReporterMode({
flag: flags.json ? 'json' : flags.reporter,
environment: process.env.DESKTOP_REPORTER,
});
}
return resolvedReporterMode;
};
// Resolving before any action keeps an invalid DESKTOP_REPORTER from running a command such as serve.
program.hook('preAction', () => {
reporterMode();
});
const writeValue = (value: unknown) => stdout.write(formatDesktopReporterValue(reporterMode(), value));

addNativeBuildCommand(program, buildDriver, writeValue);
Expand Down Expand Up @@ -117,8 +123,8 @@ export function createDesktopDriverCommand(options: CreateDesktopDriverCommandOp
},
],
});
writeValue({ server: server.url, targetId: flags.target });
try {
writeValue({ server: server.url, targetId: flags.target });
await waitForSignal();
} finally {
await server.close();
Expand Down Expand Up @@ -234,6 +240,10 @@ export function createDesktopDriverCommand(options: CreateDesktopDriverCommandOp
}),
);

configureDesktopCliCommandTree(program, {
writeErr: (value) => stderr.write(value),
writeOut: (value) => stdout.write(value),
});
return program;
}

Expand Down Expand Up @@ -306,7 +316,7 @@ function addNativeDoctorCommand(
writeValue({
error: {
code: error instanceof Error && 'code' in error ? String(error.code) : 'unknown',
message: error instanceof Error ? error.message : String(error),
message: describeDesktopCliError(error),
},
ready: false,
});
Expand All @@ -330,7 +340,7 @@ export async function runDesktopDriverCli(argv: readonly string[] = process.argv
reporterMode = 'plain';
}
}
const program = createDesktopDriverCommand({ stderr: { write: (text) => ((diagnostics += text), true) } }).exitOverride();
const program = createDesktopDriverCommand({ stderr: { write: (text) => ((diagnostics += text), true) } });
try {
await program.parseAsync([...argv]);
} catch (error) {
Expand Down Expand Up @@ -403,7 +413,7 @@ function toResolveOptions(flags: NativeResolveFlags): NativeDriverResolveOptions

function parseChoice<T extends string>(value: string, choices: readonly T[], name: string): T {
if (!choices.includes(value as T)) {
throw new TypeError(`${name} must be one of ${choices.join(', ')}. Received "${value}".`);
throw new InvalidArgumentError(`${name} must be one of ${choices.join(', ')}. Received "${value}".`);
}
return value as T;
}
Expand Down Expand Up @@ -457,31 +467,31 @@ function waitForSignal(): Promise<void> {
function parsePort(value: string): number {
const port = Number(value);
if (!Number.isInteger(port) || port < 0 || port > 65_535) {
throw new TypeError(`Port must be an integer between 0 and 65535. Received "${value}".`);
throw new InvalidArgumentError(`Port must be an integer between 0 and 65535. Received "${value}".`);
}

return port;
}

function requireOption(value: string | undefined, name: string): string {
if (!value) {
throw new TypeError(`${name} requires a non-empty value.`);
throw new InvalidArgumentError(`${name} requires a non-empty value.`);
}
return value;
}

function parseNonNegativeInteger(value: string): number {
const parsed = Number(value);
if (!Number.isInteger(parsed) || parsed < 0) {
throw new TypeError(`Expected a non-negative integer. Received "${value}".`);
throw new InvalidArgumentError(`Expected a non-negative integer. Received "${value}".`);
}
return parsed;
}

function parsePositiveInteger(value: string): number {
const parsed = Number(value);
if (!Number.isInteger(parsed) || parsed < 1) {
throw new TypeError(`Expected a positive integer. Received "${value}".`);
throw new InvalidArgumentError(`Expected a positive integer. Received "${value}".`);
}
return parsed;
}
Expand Down
Loading
Loading