Skip to content

Infrastructure: Qualify and require native Storybook CI coverage #4349

Open

Description

@JasonVMo

Current status - 2026-11-16

Authoritative native coverage and reliability promotion remain open. Hosted Storybook lanes exist for macOS, Windows Fabric and Win32 Paper; no physical-input lane is required or has a reliability window yet.

Landed (PR links):

Remaining (this issue's gates):

Green semantic lanes, one-off hosted passes and aggregate checks alone do not satisfy these. Per-run evidence is recorded in pull request descriptions and issue comments; each package's references/current-status.md keeps a status overview, stable topic sections, and only the latest result per native endpoint with a link.

Goal

Qualify and promote the native Storybook pipelines delivered by #4256 into reliable required desktop coverage. Parent: #4275.

Status - 2026-10-01

Open; all three pipelines exist and run native apps. macOS, Windows Fabric, and Win32 Paper explicitly build/verify native helpers, execute the owned stories-and-tests lifecycle, and retain diagnostics.

All three Storybook jobs passed in run 36927750317. The macOS log and retained Windows/Win32 reports each record 29 authored tests: 7 passed, 22 skipped. Current main catalogs contain 168 macOS, 158 Windows, and 152 Win32 stories. Hosted Windows/Win32 disable keyboard, physical click, and wheel; hosted macOS disables physical click.

Current main rules require only the PR status. The aggregate's needs excludes all three Storybook jobs and bundle-size; a green aggregate therefore does not make those checks required.

Remaining scope and acceptance criteria

  • Provision authoritative, owned interactive Windows/Win32 and managed macOS lanes with the capabilities their required tests actually need. Locked sessions, denied authority, helper failures, app crashes, and assertion failures have distinguishable results.
  • Publish an auditable executed cohort, not just discovered tests or render counts; integrate the regression-parity and accessibility follow-ups.
  • Reconcile endpoint-specific catalog identities and the actual WDIO story/test/source digests. Full catalogs intentionally differ. portablePlanDigest currently hashes only legacy static plans (none in the component catalog), so matching that digest does not prove callback equivalence.
  • Retain versioned machine-readable run, JUnit, event, process-ownership, helper identity, native log, tree, and available screenshot evidence for the active WDIO path. Failure upload must not mask the original error.
  • Define bounded retry policy and expiring quarantine records with owner, reason/issue, and expiry. Quarantined/skipped cases do not count toward exit coverage.
  • Name runner-capacity, escalation, artifact-retention/redaction/access, and fork-PR safety owners and document their operating policy.
  • Define one consistent helper build and verification policy across the macOS, Windows and Win32 lanes (source-verified cache versus prebuilt-only) and document it in ci-integration.md.
  • Make the timing-sensitive boundedCommand and privateStorage driver tests deterministic or isolate them, so a plain parallel local yarn lage buildci can serve as acceptance evidence alongside hosted CI.
  • Measure at least 100 representative executions per endpoint over 14 days, with at least 99% infrastructure success, zero leaked app/helper processes, zero unreleased input, and actionable metadata for every infrastructure failure.
  • Prevent all-skipped selected cohorts from being reported as successful completed coverage; preserve truthful mixed pass/skip reports.
  • Promote only qualified capabilities through the real required-status path, and prove a relevant failing native job blocks merging.
  • Preserve authenticated readiness, exact app/helper leases, loopback-only listeners, source verification, serial physical input, and cleanup/cancellation contracts.
  • Document adding/filtering/reproducing native tests and diagnosing failures. Pixel differences remain advisory, not an implicit merge gate.

Evidence and dependencies

Activity

  1. JasonVMo commented on Nov 16, 2026

    @JasonVMo
    ContributorAuthor

    Status update. #4500 added an advisory macOS Storybook PR (physical) job (Fabric, --input physical, no disabled input features), outside the aggregate pr needs. Hosted run 43344510349 passed 343 checks with 3 declared unsupported and 0 failures. That is one run: managed runners, required-status promotion and the 100-run/14-day window remain open here. The existing macOS lanes keep their physicalClick mask. #4498's hosted Windows/Win32 jobs passed on attempt 1. Added gaps: one consistent helper build/verification policy across lanes, the hosted Windows ownership consumer opt-in (0 of 11 ran), and the timing-sensitive boundedCommand/privateStorage tests that fail in a plain parallel local aggregate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions