Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 101 additions & 4 deletions .github/workflows/release-qa-gate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,13 @@ name: Release QA Gate
# entry):
# advisory/web-ui-sweep — T2 Playwright sweep against the candidate
#
# T3/T4 extension slots (reserved manifest entries, recorded as
# not-run/"not-implemented-yet" until their stage lands):
# T4 landed as a BLOCKING entry:
# invariant/surface-consistency — CLI / MCP / Web UI agree with the core's
# /api/v1/servers on every server's state
#
# T3 extension slot (reserved manifest entry, recorded as
# not-run/"not-implemented-yet" until its stage lands):
# reserved/macos-smoke — T3 macOS tray smoke (advisory, FR-019/020)
# reserved/surface-consistency — T4 REST/CLI/Web-UI/tray state agreement
# See docs/development/release-gate.md.

on:
Expand All @@ -40,6 +43,10 @@ on:
description: "Ref to qualify (dry run — publishes nothing, FR-001a)"
required: false
type: string
seed_consistency_mismatch:
description: "Dry-run only: prove T4 fails — override surface:server:field=value (e.g. cli:gate-cons-live:health.level=degraded)"
required: false
type: string

permissions:
contents: read
Expand Down Expand Up @@ -409,6 +416,96 @@ jobs:
retention-days: 1
if-no-files-found: warn

# ---------------------------------------------------------------------------
# T4 (US5, FR-018): surface-state consistency. Boots the same five-type
# server matrix (its matrix fragments go to a scratch dir — the
# matrix-invariants job reports those cells), then `release-gate
# consistency` adds quarantined/disabled/unhealthy/live servers and asserts
# that the CLI (`upstream list -o json`), the MCP `upstream_servers list` and
# the RENDERED Web UI (one long-lived Chromium page) agree with the core's
# /api/v1/servers on enabled, quarantined, connection state, health.level,
# health.admin_state, health.status and tool counts — settled, after a live
# disable with no page reload, and after a reload. BLOCKING: a post-retry
# divergence fails the gate with a per-field diff. The macOS tray is T3.
#
# Negative proof: dispatch this workflow with seed_consistency_mismatch set;
# the knob is read only on workflow_dispatch, never on a tag's workflow_call.
# ---------------------------------------------------------------------------
surface-consistency:
name: Surface consistency (CLI / MCP / Web UI vs core)
needs: build-candidate
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.ref || github.sha }}

- name: Set up Go # build-fixture-image.sh builds the static fixture binary
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version: ${{ env.GO_VERSION }}
cache: true

- name: Set up Node.js # the Web UI reader drives Chromium via @playwright/test
uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with:
node-version: ${{ env.NODE_VERSION }}

- name: Download candidate binaries
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: gate-candidate
path: dist-bin

- name: Stage binaries
run: chmod +x dist-bin/*

- name: Install the locked Playwright + Chromium
run: |
cd e2e/web-ui-sweep
npm ci
npx playwright install --with-deps chromium

- name: Build fixture Docker image (docker cell)
run: scripts/gate/build-fixture-image.sh

- name: Run surface consistency check
env:
# Read only on a manual dry run; empty on every tag (workflow_call).
SEED_MISMATCH: ${{ github.event_name == 'workflow_dispatch' && inputs.seed_consistency_mismatch || '' }}
run: |
set -uo pipefail
WORK="${RUNNER_TEMP}/consistency-work"
mkdir -p "$WORK" "${GATE_REPORT_DIR}"
./dist-bin/release-gate matrix \
--binary ./dist-bin/mcpproxy \
--fixture ./dist-bin/mcpfixture \
--oauth-server ./dist-bin/oauthserver \
--report-dir "${WORK}/matrix-report" \
--state-file "${WORK}/gate-state.json" \
--work-dir "$WORK"
echo "matrix exit=$? (cells are reported by matrix-invariants)"
SEED_ARGS=()
if [ "${{ github.event_name }}" = "workflow_dispatch" ] && [ -n "${SEED_MISMATCH}" ]; then
SEED_ARGS=(--seed-mismatch "${SEED_MISMATCH}")
fi
./dist-bin/release-gate consistency \
--state-file "${WORK}/gate-state.json" \
--report-dir "${GATE_REPORT_DIR}" \
--webui-script "${GITHUB_WORKSPACE}/e2e/web-ui-sweep/surface-snapshot.mjs" \
${SEED_ARGS[@]+"${SEED_ARGS[@]}"}

- name: Upload report fragment
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gate-fragment-surface-consistency
path: ${{ env.GATE_REPORT_DIR }}/
retention-days: 1
if-no-files-found: warn

# ---------------------------------------------------------------------------
# T2 (US3, FR-016): Playwright Web UI sweep against the Web UI served by the
# CANDIDATE binary (embedded frontend, not a dev server), with a live stdio
Expand Down Expand Up @@ -488,7 +585,7 @@ jobs:
# ---------------------------------------------------------------------------
verdict:
name: Gate verdict
needs: [build-candidate, suite-api-e2e, suite-race, suite-scan-eval, matrix-invariants, web-ui-sweep]
needs: [build-candidate, suite-api-e2e, suite-race, suite-scan-eval, matrix-invariants, surface-consistency, web-ui-sweep]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 10
Expand Down
10 changes: 6 additions & 4 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,17 +158,19 @@ graph LR
release_qa_gate_matrix --> release_qa_gate_consistency

classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff;
classDef in_review fill:#9a6700,stroke:#5c3d00,color:#ffffff;
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
class release_qa_gate_matrix,release_qa_gate_playwright done;
class release_qa_gate_macos,release_qa_gate_consistency todo;
class release_qa_gate_consistency in_review;
class release_qa_gate_macos todo;
```

| Task | Status | Refs |
| --- | --- | --- |
| T1: tag-blocking release-gate workflow: server-type matrix (stdio/http/sse/docker/oauth) + invariants (activity-log/request-id, token+telemetry counters, quarantine flow, reconnect, upgrade-in-place), publish jobs gated on the verdict, scan-eval unconditional on tags | 🟢 Done | #819 |
| T2: wire the Playwright Web UI sweep into the gate (currently manual-trigger only) | 🟢 Done | #1030 |
| T3: macOS app smoke on a macos runner, advisory until 3 consecutive passes (today zero CI automation for the tray app) | ⚪ Todo | — |
| T4: surface-state consistency check (tray/Web UI/CLI agree with core on server states) | ⚪ Todo | — |
| T4: surface-state consistency check (tray/Web UI/CLI agree with core on server states) | 🟡 In review | — |

</details>

Expand Down Expand Up @@ -884,7 +886,7 @@ graph LR
| Epic | Status | Priority | Progress | Spec | PR |
| --- | --- | --- | --- | --- | --- |
| Web UI + macOS app UX audit | In progress | P0 | — | | |
| Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | — | [081-release-qa-gate](./specs/081-release-qa-gate/) | |
| Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | 3/4 (75%) | [081-release-qa-gate](./specs/081-release-qa-gate/) | |
| Action log / transparency — info at a glance | In progress | P1 | — | | |
| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | |
| Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | |
Expand Down Expand Up @@ -1008,7 +1010,7 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—`
| [078-connect-trust-preview](./specs/078-connect-trust-preview/) | — | — |
| [079-upgrade-nudge](./specs/079-upgrade-nudge/) | — | — |
| [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | — | — |
| [081-release-qa-gate](./specs/081-release-qa-gate/) | — | — |
| [081-release-qa-gate](./specs/081-release-qa-gate/) | `in-flight` | 3/4 (75%) |
| [082-work-sessions](./specs/082-work-sessions/) | — | — |
| [083-discovery-profiler](./specs/083-discovery-profiler/) | `in-flight` | 4/41 (10%) |
| [084-toon-output](./specs/084-toon-output/) | `in-flight` | 40/43 (93%) |
Expand Down
Loading
Loading