Skip to content

ci(release): advisory Playwright web-UI sweep on tag builds (QA gate T2) - #1030

Merged
github-actions[bot] merged 5 commits into
mainfrom
ci/release-qa-gate-playwright-t2
Aug 24, 2026
Merged

github-actions[bot] merged 5 commits into
mainfrom
ci/release-qa-gate-playwright-t2

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 23, 2026

Copy link
Copy Markdown
Member

Spec 081 T2 (roadmap release-qa-gate-playwright): the release QA gate now runs the Playwright Web UI sweep on every tag build — release.yml / prerelease.yml already call the reusable gate — against the Web UI served by the candidate binary (embedded frontend, not a dev server) with a live mcpfixture stdio upstream.

Advisory, enforced twice (it cannot block a release):

  • continue-on-error: true on the web-ui-sweep job — a failing job inside a reusable workflow would otherwise become the workflow_call conclusion the publishers' needs: gate on;
  • non-blocking manifest entry advisory/web-ui-sweep (renamed from reserved/web-ui-sweep), whose failures are recorded as advisory-fail and listed in advisory_failures via a new release-gate run-suite --advisory flag.

Promotion to blocking (FR-016 end state) is Blocking: true + dropping continue-on-error, after 3 consecutive clean tags.

Reuse over copy-paste: the job calls scripts/run-web-smoke.sh, the same launcher a maintainer runs by hand. That script was dead — it ran web-smoke.spec.ts from the gitignored .playwright-mcp/ — so this PR points it at a committed sweep (e2e/web-ui-sweep: servers list, server detail + security tab, tools page + search, activity log, settings; uncaught page exceptions fail it) and adds the optional fixture upstream.

Artifacts: Playwright HTML report + server log upload as web-ui-sweep-playwright-report (14 days).

Also: verdict now needs: the sweep job so its fragment is merged, and docs (docs/development/release-gate.md, docs/development/web-ui-verification.md) + roadmap.yaml/ROADMAP.md are updated.

Spec 081 T2. The release QA gate now runs the Playwright Web UI sweep on
every tag build (release.yml / prerelease.yml already call the reusable
gate), against the Web UI served by the candidate binary with a live
mcpfixture stdio upstream.

Advisory by design, enforced twice: `continue-on-error: true` on the job
(a failing job inside a reusable workflow would otherwise become the
workflow_call conclusion the publishers gate on), and a non-blocking
manifest entry `advisory/web-ui-sweep` (renamed from
`reserved/web-ui-sweep`) whose failures land in `advisory_failures` via a
new `release-gate run-suite --advisory` flag. Promotion to blocking is
Blocking:true + dropping continue-on-error.

The sweep itself is now committed (e2e/web-ui-sweep: servers list, server
detail + security tab, tools page + search, activity log, settings; fails
on uncaught page exceptions) and scripts/run-web-smoke.sh — previously
dead, it ran a spec file that lives only in a gitignored dir — is the
single launcher for both the manual run and the CI job.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 23, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4d35f00
Status: ✅  Deploy successful!
Preview URL: https://07fbdd76.mcpproxy-docs.pages.dev
Branch Preview URL: https://ci-release-qa-gate-playwrigh.mcpproxy-docs.pages.dev

View logs

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 57.14286% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
cmd/release-gate/main.go 0.00% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Aug 23, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: ci/release-qa-gate-playwright-t2

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (25 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (20 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 32687351114 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

Algis Dumbris and others added 4 commits August 23, 2026 22:00
…launcher

Cross-model review follow-ups on the T2 Playwright sweep.

The gate runs as a reusable workflow, so its artifacts land in the
publisher's own run — and release.yml/prerelease.yml build release
assets by globbing every *.tar.gz/*.zip from ALL run artifacts. A
Playwright HTML report keeps its traces as playwright-report/data/
<sha>.zip, retained on failure (and on a flaky-then-green retry), so a
red advisory sweep on a tag would have attached unnamed trace zips to
the public release. Both publishers now exclude the sweep artifact from
that glob, guarded by TestPublishersDoNotShipSweepArtifacts.

scripts/run-web-smoke.sh:
- generate a throwaway API key instead of honouring MCPPROXY_API_KEY,
  which the sweep writes into every navigation URL and Playwright then
  stores in the HTML report and failure traces; pin it into the server's
  environment too, since the env var outranks the config file and an
  ambient key otherwise left the server demanding one key while the
  probe used another (46s of 401s, then a readiness timeout)
- fail fast when the candidate exits before becoming ready instead of
  burning the full timeout, and give the readiness curl connect/max
  timeouts so a port held by a non-responding process cannot block the
  loop forever (occupied-port repro: 119s -> 5s)
- preserve the server log from the cleanup trap so an early failure is
  diagnosable from the uploaded artifact, not just the CI console
- reject fixture paths containing a quote/backslash/newline rather than
  silently emitting a malformed config heredoc
- npm ci against a committed package-lock.json instead of resolving the
  mutable ^1.49.0 range during release qualification
Round-2 review follow-up. `npm ci` ran only when the playwright binary
was missing, so a tree installed by an older revision of this script
(unlocked `npm install`) kept serving an unreviewed version forever —
the lockfile was enforced on fresh CI runners but not in existing
worktrees.

Compare the installed @playwright/test version against the lockfile
instead of merely checking for presence: a mismatch reinstalls, a match
still skips. That closes the hole without making every hand-run pay a
registry round-trip, which running `npm ci` unconditionally would.
Round-3 review follow-up, closing two low-severity residuals in the
install guard.

Comparing only @playwright/test's version still missed a lockfile that
moved a transitive dependency (playwright-core) without touching the
top-level version. Hash the whole lockfile instead and stamp it into
node_modules after a successful npm ci; any drift reinstalls. Verified
against a transitive-only lockfile bump, which the version check would
have skipped. The stamp is written only on success, so a failed npm ci
leaves the previous, still-accurate stamp in place.

Also pass the path through process.argv rather than interpolating it
into the node -e source: a checkout under a path containing a quote
produced invalid JavaScript and a silently empty result.
Cross-model review (opencode/gpt-5.6-sol) findings on the advisory Web UI
sweep, both verified against the running sweep before fixing:

* The throwaway instance's API key was `web-sweep-<epoch>-<pid>`. It guards
  the full REST admin API, which can register a stdio upstream — i.e. run an
  arbitrary command as whoever launched the sweep — and 127.0.0.1 is reachable
  by every local account, so a clock+PID key is brute-forceable for the minute
  the instance is up. Now 24 bytes from crypto.randomBytes.

* The tools-search assertion (`count <= before` plus first-row-matches) passed
  vacuously when the search box did nothing: the unfiltered set is trivially
  <= itself and the fixture's first tool is already `echo`. It now requires
  every surviving row to match, which is also race-free — a strict `<` would
  have raced the background indexer, since `before` can be sampled mid-render.
  Verified non-vacuous by mutation: with the query emptied the check fails,
  where the old one passed.

A third finding (readiness loop could latch onto a foreign server already on
the port) was rejected as a false positive: the probe carries this run's unique
key, and a foreign mcpproxy answers 401 — measured — while a second instance on
a taken port exits 2 and is caught by the loop's liveness check.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved (Model B): Paperclip review verdicts = ACCEPT and qa-gate green at this head SHA. Arming auto-merge; GitHub merges when all required checks pass.

@github-actions
github-actions Bot merged commit a52e90a into main Aug 24, 2026
39 checks passed
github-actions Bot pushed a commit that referenced this pull request Aug 25, 2026
…specs 101/102 tracked (#1041)

- release-qa-gate-playwright (T2): in_review -> done, pr #1030 (advisory
  Playwright web-UI sweep merged into the release gate).
- mcp-2026-upgrade: blocked -> in_progress. The mcp-go gate cleared with
  v1.0.0-beta.1 (2026-08-12); spec 058 revision in review as #1033. Note
  rewritten: the remaining blocker is the Spec 057 cross-spec conflict,
  not the dependency.
- tpa-db: todo -> in_review with spec: specs/101-tpa-db and pr #1028
  (spec stage only; all three child tasks stay todo).
- New epic schema-deferred (P1, spec 102, in_progress): deferred-schema
  serialization for the direct tools/list surface, spec merged in #1035
  (issue #971), plan in progress.

Reconnect-backoff work (#1014/#1039/#1040) is deliberately not added —
the roadmap never tracked it; git history does.

ROADMAP.md regenerated; gen-roadmap.py --check passes.
Dumbris added a commit that referenced this pull request Oct 17, 2026
…1 T4) (#1655)

Spec 081 T4 (US5, FR-018). The release gate now blocks a tag when the
CLI, MCP or Web UI shows a server state different from what the core
reports.

## What it does
New **blocking** gate entry `invariant/surface-consistency`, run by a
new job `surface-consistency`. It replaces the reserved T4 slot.

1. Boots the same five-type server matrix (stdio/http/sse/docker/oauth).
Those cells are still reported by `matrix-invariants`.
2. Adds four servers so every admin state is on screen: live (healthy),
quarantined, disabled, and broken (the endpoint refuses connections, so
it is unhealthy).
3. Compares each surface with the core's `GET /api/v1/servers`:

| Surface | Read path | Fields |
|---|---|---|
| CLI | `mcpproxy upstream list -o json --config …` | enabled,
quarantined, connected, connection state,
health.level/admin_state/status, tool_count |
| MCP | `upstream_servers` `operation=list` (its own storage + manager
projection) | enabled, quarantined, connection state,
health.level/admin_state/status |
| Web UI | rendered cards on `/ui/servers`, read by
`e2e/web-ui-sweep/surface-snapshot.mjs` | enabled, quarantined, tool
count, status chip colour and text. The expected values come from the
core row using `ServerCard.vue`'s rules. |
| Tray | — | recorded as `not_compared` (T3) |

It runs in three phases: **settled**, **live** (a server is disabled
through REST while the page stays open, which tests the SSE/poll update
path) and **reload**. Each phase retries until the surfaces converge. An
attempt counts only if the core state did not change while the surfaces
were being read. A divergence that remains after the retries fails the
gate with a per-field diff, for example `webui gate-cons-quarantined
webui.chip_class: core=badge-secondary webui=badge-success`.

**Negative path:** `--seed-mismatch surface:server:field=value`, or the
dry-run-only `seed_consistency_mismatch` workflow_dispatch input. A
tag's `workflow_call` never reads it, and an audit test enforces that.

**Local:** `./scripts/test-surface-consistency.sh` runs next to
`scripts/test-api-e2e.sh`.

No new dependencies. The reader reuses the sweep's locked
`@playwright/test`.

## Review
Cross-reviewed with opencode `github-copilot/gpt-6.1-sol` (zcode login
was down at round 1): round 1 → 10 findings, all fixed in 54adbf1; round
2 → 9/10 confirmed fixed + 1 new, both fixed in 03ad256; round 3 → NO
NEW FINDINGS.

## Testing
- Written test-first: comparator tests cover agreement, a stale CLI
level, an MCP connection state, a Web UI chip and tool count, missing
and extra servers, a capture error, and the seed path. A workflow audit
checks that the job blocks, compares the Web UI, is in the verdict job's
`needs` list, and only reads the seed on dispatch.
- Local live run, all five cells (stdio/http/sse/docker/oauth) + the 4
mixed servers, isolated instance (free port, scratch data dir): all
three phases pass. The captured rows show the disable reaching the CLI,
MCP and the open Web UI page.
- Local seeded run
(`webui:gate-cons-quarantined:webui.chip_class=badge-success`): fails in
every phase with the diff above.
- golangci-lint v2 (bare and `--build-tags server`): 0 issues.
- Gate dry runs on this branch: clean
https://github.com/smart-mcp-proxy/mcpproxy-go/actions/runs/39038591871
· seeded
https://github.com/smart-mcp-proxy/mcpproxy-go/actions/runs/39038596517

Related #819 #1030
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants