Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/desktop-test-operation-logging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@fluentui-react-native/desktop-test': patch
---

Report each command and step as a one-line outcome header with its duration and summary. Command output stays in its log and shows only when the step fails, in a verbose run, or inside a collapsed GitHub Actions group, and `runDesktopOperation` exposes the same headers to other producers. Neutralize workflow commands per line instead of wrapping every message in `::stop-commands::` fences, drop the per-message command identifiers, and remove the unused `@rnx-kit/reporter` dependency.
5 changes: 5 additions & 0 deletions .changeset/storybook-desktop-smoke-summary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@fluentui-react-native/storybook-desktop': patch
---

Run story rendering and story tests as reported operations, so a smoke run shows one header with counts such as `209 stories rendered` and `101 of 101 story tests executed: 101 passed, 0 failed, 0 skipped`, and says `no story tests executed` when none ran. `--verbose` streams command output live; other runs keep it in the logs and show the end of a failing step's log.
2 changes: 1 addition & 1 deletion packages/agentic/desktop-test/jest.config.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ const config = require('@fluentui-react-native/scripts/jest-config');

module.exports = {
...config,
transformIgnorePatterns: ['node_modules[/\\\\](?!.*(?:\\.mjs$|@rnx-kit(?:[/\\\\]reporter|-reporter-npm-)))'],
transformIgnorePatterns: ['node_modules[/\\\\](?!.*\\.mjs$)'],
moduleNameMapper: {
...config.moduleNameMapper,
'^(\\.{1,2}/.*)\\.js$': '$1',
Expand Down
1 change: 0 additions & 1 deletion packages/agentic/desktop-test/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,6 @@
},
"dependencies": {
"@fluentui-react-native/desktop-driver": "workspace:*",
"@rnx-kit/reporter": "catalog:",
"@wdio/reporter": "catalog:",
"commander": "^14.0.2",
"cross-spawn": "7.0.6",
Expand Down
5 changes: 5 additions & 0 deletions packages/agentic/desktop-test/references/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,11 @@ and `--json` (an alias for `--reporter json`). Mode precedence is CLI flag,
`DESKTOP_REPORTER`, config, then `auto`. `auto` selects GitHub Actions rendering
when `GITHUB_ACTIONS=true`, otherwise plain output.

Plain and GitHub Actions output reports each command as one header with its
outcome and duration, and shows the end of its log only when it fails. Pass
`--verbose` to stream every command's output live, with start markers and log
paths. See [console output](reporting.md#console-output).

One-shot JSON commands emit a bounded summary; `--json` no longer prints the
complete `run.json`. NDJSON starts with run metadata, emits milestone records
by default, then ends with the same summary. Use `--events progress|all` for
Expand Down
56 changes: 52 additions & 4 deletions packages/agentic/desktop-test/references/reporting.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,64 @@ are flags, not credentials, and redacting them would rewrite machine-readable
output. Register a shorter credential explicitly with the reporter's
`addSecret`.

Stderr and test progress are live without verbose mode. Failed commands replay
the collected output. GitHub Actions grouping and annotations are rendered by
the supervisor; child output cannot issue workflow commands. Upload the full
run directory in an always-run artifact step.
Command output goes to these logs, not to the console; see
[console output](#console-output) for what the console shows and when. Upload
the full run directory in an always-run artifact step.

Statuses preserve failures, retries, flaky tests, skips, unsupported
capabilities, and unfinished work. A zero worker exit cannot override failed
assertions or incomplete work. Optional suites can be skipped without being
labelled passed.

## Console output

A command, or a step such as a story test run, is an operation. When it ends the
reporter prints one header with its outcome and duration, and a summary when the
operation has one:

```text
✔ pod install (42s)
✔ Render stories (1m 08s): 209 stories rendered
✔ Story tests (3m 12s): 101 of 101 story tests executed: 101 passed, 0 failed, 0 skipped
✖ rnx-cli build (4m 03s): exit code 65
```

The operation's output stays in its log, and the console shows it only when it
helps:

- **Default.** A passing operation prints its header on stdout. A failing one
prints its header on stderr, the last 200 lines of its log indented beneath
it, and `Full log: <path>`.
- **GitHub Actions.** A passing operation that has a log prints a collapsed
`::group::` titled with the header, holding the last 1000 lines of the log and
`Full log: <path>`. A failing operation prints as it does by default, so an
error is never folded away, and each failure also gets an `::error`
annotation.
- **`--verbose`.** The operation's output streams live as `[label] stdout: text`
and `[label] stderr: text`, after a `▶ label` start line and an `artifact:`
line naming its log. The header follows, and a failing operation adds
`Full log: <path>`.

Without `--verbose` a passing, skipped, or unsupported test appears only in the
final counts, so a run that executes nothing shows as `0 executed` instead of
disappearing into a long list. Failed tests, retries, and diagnostics always
print, and the root supervisor's stderr is always live. Every run finishes with a
`[desktop] PASSED: …` or `[desktop] FAILED: …` line carrying the counts.

Output is untrusted text. The reporter neutralizes each line it prints that it
did not build as a workflow command, so a command cannot start, stop, or forge
one: `::` at the start of a line (after any white space) becomes `: :`, and `##[`
becomes `# #[`. Only text that would have been a command changes, and the logs
keep the original. The console has no `::stop-commands::` fence, per-message
token lines, or command identifiers; agents that need complete output read the
logs and `run.json`.

Delegated producers and the Storybook CLI report operations the same way through
`runDesktopOperation` from `@fluentui-react-native/desktop-test/reporting`. Its
`summarize` sets the header's summary. Its `log` records a line of detail that is
live only with `--verbose`; it is not kept in a log, so it is not replayed when
the operation fails.

## CLI reporters

Select `--reporter auto|plain|github-actions|json|ndjson`; `--json` aliases
Expand Down
2 changes: 1 addition & 1 deletion packages/agentic/desktop-test/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ export async function runDesktopTestCli(argv = process.argv): Promise<void> {
.option('--json', 'alias for --reporter json')
.addOption(new Option('--events <level>', 'NDJSON event detail').choices([...desktopEventLevels]))
.option('--paper', 'select macOS Paper (run prep after switching renderers)')
.option('--verbose', 'show informational service output');
.option('--verbose', 'stream command output live, with start markers and log paths, instead of only the outcome of each step');
let parseDiagnostics = '';
program.exitOverride().configureOutput({
writeErr: (message) => {
Expand Down
46 changes: 31 additions & 15 deletions packages/agentic/desktop-test/src/commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,7 @@ export class NodeDesktopCommandRunner implements DesktopCommandRunner {
try {
const code = await running.completed;
if (code !== 0)
throw (
running.failure ?? new Error(`Command "${command.label ?? command.command}" exited with code ${code}. Log: ${running.logPath}`)
);
throw running.failure ?? new Error(`Command "${commandLabel(command)}" exited with code ${code}. Log: ${running.logPath}`);
} catch (error) {
failure = error;
}
Expand Down Expand Up @@ -96,6 +94,7 @@ export class NodeDesktopCommandRunner implements DesktopCommandRunner {
reporter.addSecret(value);
}
const id = `command-${randomUUID()}`;
const label = commandLabel(command);
const emit = reporter.source(id);
const relativeLog = `commands/${id}.log`;
const logPath = reporter.artifactPath(relativeLog);
Expand All @@ -104,12 +103,8 @@ export class NodeDesktopCommandRunner implements DesktopCommandRunner {
flag: 'wx',
mode: 0o600,
});
emit(
{ kind: 'scope-start', payload: { label: command.label ?? command.command, category: command.category ?? 'service' } },
id,
command.context,
);
emit({ kind: 'artifact', payload: { path: relativeLog, description: command.label ?? command.command } }, id, command.context);
emit({ kind: 'scope-start', payload: { label, category: command.category ?? 'service' } }, id, command.context);
emit({ kind: 'artifact', payload: { path: relativeLog, description: label } }, id, command.context);
let failed: unknown;
const processControl = createDesktopProcessControl((error) => {
failed ??= error;
Expand All @@ -132,6 +127,7 @@ export class NodeDesktopCommandRunner implements DesktopCommandRunner {
let stopped = false;
let closed = false;
let exitFailure: Error | undefined;
let exitSummary: string | undefined;
let stopPromise: Promise<void> | undefined;
let closeDeadline: ReturnType<typeof setTimeout> | undefined;
let logBytes = fs.statSync(logPath).size;
Expand Down Expand Up @@ -190,18 +186,18 @@ export class NodeDesktopCommandRunner implements DesktopCommandRunner {
child.once('exit', () => {
processControl.stop();
closeDeadline = setTimeout(() => {
failed ??= new Error(
`Child exited but inherited output streams did not close within the drain deadline: ${command.label ?? command.command}`,
);
failed ??= new Error(`Child exited but inherited output streams did not close within the drain deadline: ${label}`);
child.stdout.destroy();
child.stderr.destroy();
}, this.options.shutdownTimeoutMs ?? 10_000);
});
child.once('close', (code, signal) => {
clearTimeout(closeDeadline);
closed = true;
if (code !== 0 && !stopped)
exitFailure = new Error(`Command "${command.label ?? command.command}" exited with code ${code ?? signal}. Log: ${logPath}`);
if (code !== 0 && !stopped) {
exitSummary = code === null ? `terminated by ${signal}` : `exit code ${code}`;
exitFailure = new Error(`Command "${label}" exited with code ${code ?? signal}. Log: ${logPath}`);
}
if (failed !== undefined) reject(failed);
else resolve(code ?? (signal ? 1 : 0));
});
Expand Down Expand Up @@ -264,7 +260,12 @@ export class NodeDesktopCommandRunner implements DesktopCommandRunner {
}
const status =
options.failed || failed !== undefined || cleanupError !== undefined || (!wasRunning && code !== 0) ? 'failed' : 'passed';
emit({ kind: 'scope-end', payload: { status, logPath: relativeLog } }, id, command.context);
const reason = status === 'failed' ? (describeFailure(failed ?? cleanupError) ?? exitSummary) : undefined;
emit(
{ kind: 'scope-end', payload: { status, logPath: relativeLog, ...(reason ? { summary: reason } : {}) } },
id,
command.context,
);
emit({ kind: 'producer-finished', payload: { status } }, id, command.context);
this.active.delete(running);
await processControl.close();
Expand All @@ -283,6 +284,21 @@ export class NodeDesktopCommandRunner implements DesktopCommandRunner {
}
}

// A first argument such as `install` in `pod install` or `lage` in `yarn lage` names the work. Flags, paths and other values do not.
const subcommandPattern = /^[a-z][a-z0-9-]{0,31}$/i;

function commandLabel(command: PreparedDesktopCommand): string {
if (command.label) return command.label;
const name = path.basename(command.command);
const [first] = command.args;
return first !== undefined && subcommandPattern.test(first) ? `${name} ${first}` : name;
}

function describeFailure(error: unknown): string | undefined {
if (error === undefined) return undefined;
return (error instanceof Error ? error.message : String(error)).split('\n', 1)[0];
}

function needsWindowsCommandShim(command: string): boolean {
return (
process.platform === 'win32' &&
Expand Down
59 changes: 46 additions & 13 deletions packages/agentic/desktop-test/src/contracts.contract.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,17 @@ async function main() {
const journal = fs.readFileSync(path.join(run.root, 'events.jsonl'), 'utf8');
assert(!journal.includes('secret-value'));
assert(journal.includes('root cause'));
assert(err.text().includes('::stop-commands::'));
assert(
!err.text().includes('::stop-commands::') && !out.text().includes('::stop-commands::'),
'text is defused in place, not fenced',
);
assert(err.text().includes(': :error::untrusted [REDACTED]'), 'root stderr is shown with its workflow command defused');
assert(
!err
.text()
.split('\n')
.some((line) => line.trimStart().startsWith('::error::untrusted')),
);
assert.equal((err.text().match(/::error title=/g) || []).length, 1);
const junit = fs.readFileSync(path.join(run.root, 'junit.xml'), 'utf8');
assert(junit.includes('attempt'));
Expand Down Expand Up @@ -523,16 +533,37 @@ async function main() {
error: { code: 'unknown-target', message: 'Unknown target "missing".' },
});
} else if (mode === 'commands') {
const run = reporter({ kind: 'lifecycle', console: 'github-actions' });
const liveOut = capture();
const liveErr = capture();
const live = reporter({ kind: 'lifecycle', verbose: true, output: liveOut, errorOutput: liveErr });
const liveRunner = new NodeDesktopCommandRunner({ reporter: live });
const liveCommand = liveRunner.start(
command(
"process.stdout.write('ready\\n'); setTimeout(()=>process.stderr.write('live diagnostic\\n'),30); setInterval(()=>{},1000);",
),
);
try {
await waitFor(() => liveErr.text().includes('live diagnostic'));
assert(!live.resultSnapshot, 'diagnostic must arrive before shutdown');
} finally {
await liveCommand.stop();
}
assert((await live.finish()).status === 'passed');
assert(liveOut.text().includes('[fixture] stdout: ready'), 'verbose child stdout is live and stays on stdout');
assert(liveErr.text().includes('[fixture] stderr: live diagnostic'));
assert(!(liveOut.text() + liveErr.text()).includes('[command-'), 'source identifiers do not leak into the display');

const quietOut = capture();
const quietErr = capture();
const run = reporter({ kind: 'lifecycle', console: 'github-actions', output: quietOut, errorOutput: quietErr });
const runner = new NodeDesktopCommandRunner({ reporter: run });
const running = runner.start(
command(
"process.stdout.write('ready\\n'); setTimeout(()=>process.stderr.write('live diagnostic\\n'),30); setInterval(()=>{},1000);",
"process.stdout.write('ready\\n'); setTimeout(()=>process.stderr.write('quiet diagnostic\\n'),30); setInterval(()=>{},1000);",
),
);
try {
await waitFor(() => err.text().includes('live diagnostic'));
assert(!run.resultSnapshot, 'diagnostic must arrive before shutdown');
await waitFor(() => fs.readFileSync(running.logPath, 'utf8').includes('[stderr] quiet diagnostic'));
} finally {
await running.stop();
}
Expand All @@ -554,14 +585,16 @@ async function main() {
.map((file) => fs.readFileSync(path.join(run.root, 'commands', file), 'utf8'))
.join('');
assert(logs.includes('\u20ac'));
assert(logs.includes('[stderr] live diagnostic'));
assert.equal((out.text().match(/::group::/g) || []).length, 2);
assert.equal((out.text().match(/::endgroup::/g) || []).length, 2);
const groupOne = out
.text()
.slice(out.text().indexOf('::group::one'), out.text().indexOf('::endgroup::', out.text().indexOf('::group::one')));
assert(groupOne.includes('::stop-commands::'));
assert(logs.includes('[stderr] quiet diagnostic'));
assert(!quietErr.text().includes('quiet diagnostic'), 'child output is not live without verbose');
const quiet = quietOut.text();
assert.equal((quiet.match(/::group::/g) || []).length, 3, 'each passed operation is one collapsed group');
assert.equal((quiet.match(/::endgroup::/g) || []).length, 3);
assert(!quiet.includes('::stop-commands::'));
const groupOne = quiet.slice(quiet.indexOf('::group::✔ one '), quiet.indexOf('::endgroup::', quiet.indexOf('::group::✔ one ')));
assert(groupOne.includes('[stdout] ::warning::one'));
assert(!groupOne.includes('::warning::two'));
assert(!quiet.split('\n').some((line) => line.startsWith('::warning::')), 'child output cannot start a workflow command');
} else if (mode === 'command-secrets') {
const run = reporter({ kind: 'lifecycle' });
const runner = new NodeDesktopCommandRunner({ reporter: run });
Expand Down Expand Up @@ -756,7 +789,7 @@ async function main() {
);
} else if (mode === 'owned-service') {
const { startReportTransport } = await import(lib('reporting/transport.js'));
const run = reporter({ kind: 'lifecycle' });
const run = reporter({ kind: 'lifecycle', verbose: true });
const transport = await startReportTransport(run);
const runner = new NodeDesktopCommandRunner({ reporter: run, environment: transport.environment, shutdownTimeoutMs: 10_000 });
const descriptor = path.join(root, 'service.json');
Expand Down
Loading
Loading