Add - Themegrill QA Setup for E2E Testing (#1415)

* Add claudegrill QA setup and a Playwright suite

Sets this plugin up for claudegrill and ports the gate tier of the existing
Robot Framework suite (themegrill/UR-Automation) to Playwright.

QA configuration
- .themegrill-qa/suite.json describing the suite, with area_paths so CI can
  narrow a run to the areas a diff touches
- .themegrill-qa/knowledge.md, reviewed: product surfaces, persistence, the
  unauthenticated AJAX surface, capability boundaries, both upgrade ladders,
  and fragile areas evidenced from git history
- .themegrill-qa/docs/ and testcases/, ingested from the docs site and from
  UR-Automation, as porting references
- .github/workflows/qa-suite.yml

Playwright suite (16 tests, 8 files)
- admin surfaces, registration, my account, security settings, login methods,
  ajax login, membership plan creation, registration emails
- Specs assert on stored state rather than on success notices, because the
  fix history here is full of notices printed over writes that did not happen
- Every spec that creates a user deletes it again; membership subscription
  rows outlive their users, so leaked accounts eventually make plans
  undeletable

Supporting changes
- @playwright/test pinned to 1.57.0, the version @wordpress/scripts already
  resolved, so this adds a direct dependency rather than a new download
- .gitignore: /tests/ excluded the whole directory, which would have kept the
  suite out of the repo and left CI running nothing. Narrowed to /tests/*
  with !/tests/e2e/, and added Playwright artifacts and the doc scrape cache

Credentials in the ingested UR-Automation test cases (a MaxMind license key and
an account id, hardcoded in the Robot source) are redacted here. They are still
live in that repository and should be rotated and moved to secrets there.

* Keep the QA suite out of the release zip and source exports

The suite is source that CI runs, not something end users install, so it
belongs in the repo but in neither distributed artifact.

.gitattributes (governs `git archive` / GitHub source exports)
- tests, playwright.config.ts: previously exported. `/.*` already covered
  .themegrill-qa, and /tests/ used to be gitignored entirely so it never
  existed to export; now that the suite is committed, both need excluding.
  Verified: git archive now emits 0 entries for all three.

.distignore (governs the release zip, and the WordPress.org deploy action
which reads this file directly rather than through Grunt)
- playwright.config.ts: the Gruntfile's compress task already excludes
  tests/** and dotfiles, but nothing excluded the Playwright config.

* Add test:e2e scripts and make the config self-sufficient

There was no way to run the suite by hand: only run-suite.mjs could invoke it,
and that is the CI path, not the one a developer wants while writing a spec.

Scripts follow the zakra/magazine-blocks shape (config at the plugin root,
plain `playwright test`) rather than the colormag-pro one (config under
tests/e2e with an explicit --config), because that is what this checkout's
layout already is and what claudegrill's manifest inference resolves natively.

  test:e2e          test:e2e:fresh    test:e2e:demo
  test:e2e:ui       test:e2e:report   test:e2e:install

No script uses shell quoting or a lookahead --grep pattern. colormag-pro's
test:e2e:pro wraps '(?=.*@fresh)(?=.*@Pro)' in single quotes, which bash
strips but cmd.exe does not treat as quoting at all, and ( ) are cmd
metacharacters — so that form is shell-dependent on Windows. This plugin is
free-only and needs no such tier, so the question does not arise here.

playwright.config.ts now reads .themegrill-qa/.env.local itself, so the
scripts work without sourcing it first. Variables already present in the
environment take precedence, leaving run-suite.mjs authoritative under CI.
Parsing strips CR, so a .env.local written on Windows works unchanged.

* Make the @fresh tier actually run on a fresh install

The QA suite failed on PR #1415 with three @login-forms failures that pass on
test-urm.local. Both causes are the specs reading a configured site as if it
were product behaviour; neither is a bug in the plugin.

A clean install has no registration form and no /my-account/ page.
UR_Install::install() creates options, tables and roles and records the
first-run flag, but calls neither create_form() nor create_pages() — those
belong to the setup wizard. On a developer's site the wizard ran months ago; on
a disposable CI site it never has, so the Registrations list is legitimately
empty and firstFormId() waited the full action timeout for a row that was never
coming, reporting a 20s TimeoutError on a locator that was fine. ensureFirstRun()
now provisions the site the way the product does — an authenticated GET to
/wp-json/user-registration/v1/getting-started runs install_initial_pages() and
ensure_default_form() — and bails out when a form already exists, because that
bootstrap also writes users_can_register and must not edit a real site.

Playground's --login makes browser.newContext() an admin session, not a
visitor. Every new context is authenticated as admin on its first request, so
the "visitor" that registers an account then lands on the My Account dashboard
and never sees the login form. newVisitor() lets the auto-login happen once and
drops the wordpress* cookies while keeping Playground's own marker, which is
what stops the next request logging it back in. This also fixes a false pass:
"a subscriber keeps the admin bar" was being satisfied by an admin session.
loginAsAdmin() checks /wp-admin/ before driving wp-login.php, which is the race
that failed one CI attempt on #wpadminbar and passed on retry.

Two latent failures of the same class, in areas that run only when a PR touches
them, are fixed too: the Edit Profile spec needs a Country field the default
four-field form does not have, so it is @demo now (in its own describe — grep
sees the parent title); and mailAvailable() reports false under
TGQA_ENV=playground, since Playground sends no mail at all and a machine running
Local answers on :10000 anyway.

Verified on a reset Playground with CI's own ceilings (--timeout 45000,
--max-failures 3): 14 passed, 1 skipped, 0 failed, slowest test 37.7s. Was
0 passed, 3 failed, 7 skipped. Still 15/15 green against test-urm.local, where
the bootstrap correctly does nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Give the @fresh tier the headroom CI actually needs

The previous commit fixed the two root causes but left every spec running at
the edge of CI's 45s per-test ceiling, and the runner is slower than any
developer machine. Run 33855568937: the three login specs passed only on retry
and the ajax spec timed out at 45s, while both change-password specs failed with
net::ERR_ABORTED at ~42s. Nothing was wrong with them except time.

Most of the overshoot was self-inflicted. restNonce() costs a full
options-general.php load and four helpers call it, so each test paid for three
redundant ones; it is now cached per browser context. Not run-wide, which is the
part that matters: a nonce embeds the session token, every test gets a fresh
context and therefore a fresh login, and a run-wide cache would hand out stale
nonces. ensureFirstRun() and the form id are cached run-wide instead, because
those are site state and do not expire with a session.

Two more cuts: setAjaxLogin() returns the previous value so the heavy Login
Forms builder loads twice per test instead of four times, and its two fixed
3s sleeps are gone in favour of waiting for the save response — which is also
more correct, since three seconds was not always enough and navigating away
mid-flight aborted the write. Restoring the setting now skips when the old value
was never read, rather than guessing false and switching it off on a site that
had it on.

gotoChangePassword() waits for domcontentloaded rather than load: the plugin
redirects inside the account area while assets are still loading, which
supersedes the navigation and is what ERR_ABORTED was.

Together these take the slowest spec from 37.7s to 25.6s locally. That is still
not margin enough on a slower runner for a suite of full user journeys, so
qa-suite.yml sets test_timeout_ms to 90000 — about 3x the measured worst case,
so a genuinely hung spec is still capped, and max_failures (3) x 90s stays far
inside the 30-minute run ceiling.

Verified on a reset Playground at the old 45s ceiling: 14 passed, 1 skipped,
0 failed, no retries, slowest 25.6s. And 16/16 against test-urm.local.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Stop the run's first test paying a retry for a cold start

Run 33857705477 went green — 8 passed, 0 failed — but the ajax spec was still
flaky: attempt 0 failed in loginAsAdmin with "#wpadminbar element(s) not found"
and the retry passed. It was the first test of the run, and Playground reports
settled while WASM PHP is still cold, so 15s (the expect default) is not enough
for the very first admin page load of a run. Nothing was wrong; the retry just
found a warm site.

That assertion now gets 45s. A cold start is not a regression and should not
cost a ~40s retry to discover, nor teach anyone that red-then-green is normal
here.

The run also gave real CI durations, which are now in knowledge.md: login specs
35-37s, change-password 42s and 47s, registration 36s. The 47s confirms the old
45s ceiling was genuinely too tight and not merely unlucky.

Verified on a reset Playground: 14 passed, 1 skipped, 0 failed, no retries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: iamprazol

.distignore

@@ -16,3 +16,4 @@ package.json
 phpcs.xml
 /tests/
 phpunit.watcher.yml.dist
+playwright.config.ts

.gitattributes

@@ -11,3 +11,5 @@ webpack.config.js export-ignore
 phpcs.xml.dist export-ignore
 phpunit.xml.dist export-ignore
 README.md export-ignore
+tests export-ignore
+playwright.config.ts export-ignore

.github/workflows/qa-suite.yml

@@ -0,0 +1,55 @@
+# Written by `/claudegrill:setup`. Safe to edit — it is a plain caller.
+#
+# THIS IS THE ONLY PR CHECK MOST PRODUCTS NEED. No ANTHROPIC_API_KEY, no agent,
+# runner minutes only.
+#
+# It works without AI because the developer already wrote the spec: they ran
+# /claudegrill:verify-fix locally, write-spec added a @fresh spec to their
+# branch, and they committed it with the fix. By the time CI sees the PR the
+# regression is already a deterministic assertion.
+#
+# ONE job, on pull requests only. Nothing runs on a schedule: the team's model is
+# a local /claudegrill:verify-fix while the fix is being written, and the e2e
+# suite on the PR. No cron, no nightly, no WordPress-core watcher.
+#
+# `scope: changed` narrows a run to the areas the diff maps to. That is safe by
+# construction — a changed file matching no `area_paths` pattern in
+# .themegrill-qa/suite.json falls back to the full tier, so an unmapped change
+# costs runner time, never coverage.
+#
+# Worth knowing, since nothing else runs the suite now: a spec whose area is
+# never touched by any PR will not execute. Set `scope: full` below if you would
+# rather every PR run the whole @fresh tier.
+
+name: QA suite
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review]
+  workflow_dispatch:
+
+jobs:
+  # The only check. Scoped, fast, runs on every PR including drafts.
+  pr:
+    uses: ThemeGrill/claudegrill/.github/workflows/suite.yml@main
+    with:
+      product_slug: user-registration
+      product_type: plugin
+      scope: changed
+      # The shared default is 45s, and it is too tight for THIS product.
+      #
+      # Nearly every spec here is a whole user journey — provision a form,
+      # publish a page, register an account on the front end, log in as that
+      # account, then delete it — which is a dozen page loads against
+      # Playground's WASM PHP. Measured on a reset Playground locally the
+      # slowest is ~25s, and the GitHub runner is comfortably slower than a
+      # developer's machine: at 45s the three login specs passed only on retry
+      # and the ajax spec timed out outright, all of them for want of headroom
+      # rather than because anything was wrong.
+      #
+      # 90s is roughly 3x the measured worst case, so a genuinely hung spec is
+      # still capped. It stays well inside the ceilings above it, which the
+      # reusable workflow asks for: max_failures (3) x 90s is 4.5 minutes
+      # against a 30-minute timeout_ms and a 40-minute job limit.
+      test_timeout_ms: 90000
+

.gitignore

@@ -31,7 +31,12 @@ Thumbs.db
 /vendor/
 
 # PHP Unit Tests
-/tests/
+# Directory contents are excluded, but the entry itself must stay listable
+# ('/tests/' would exclude the directory, and git cannot re-include a path
+# whose parent directory is excluded).
+/tests/*
+# The Playwright QA suite is source, not a build artifact — CI runs it.
+!/tests/e2e/
 
 # act local secrets (do not commit)
 .act/.secrets
@@ -44,3 +49,15 @@ phpunit-watcher.yml
 
 #Release
 /release/
+
+# claudegrill — never track QA credentials
+.themegrill-qa/.env.local
+
+# Playwright run artifacts
+/test-results/
+/playwright-report/
+/blob-report/
+/.cache/
+
+# claudegrill: raw doc scrape cache, regenerated by ingest-docs.mjs
+.themegrill-qa/.docs-cache/