Repository navigation
Build/Test Tools: Run E2E specs in parallel and serial projects. - #14897
adimoldovan wants to merge 10 commits into
Conversation
The E2E suite runs on one worker. Split it into two Playwright projects: - `parallel` runs 15 spec files on several workers. - `serial` runs the 6 specs that break the site for other tests, on one worker. Each project has its own npm script. CI runs each project in a separate matrix job, so a failure in one project does not hide the results of the other.
Test using WordPress PlaygroundThe changes in this pull request can previewed and tested using a WordPress Playground instance. WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser. Some things to be aware of
For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation. |
The base config from `@wordpress/scripts` sets `workers: 1` for the whole run, so the `parallel` project ran on one worker. Set `workers: '50%'` at the top level. This is the Playwright default. Move the 7 specs that delete all posts or media to `serial`. On several workers, they delete content that tests on other workers use.
… project. Add an `all` project that runs every spec on one worker. It is the only default project, so `npm run test:e2e` runs the whole suite serially, as before. Set `default: false` on `parallel` and `serial`, which needs Playwright 1.64. CI selects them with `--project`, so the two npm scripts are no longer needed. Move `themes.test.js` to `serial`, because it changes the active theme for the whole site.
Release branches call the `trunk` copy of the reusable E2E workflow, and their Playwright configs have no `parallel` or `serial` project. Add a `test-group` input to the reusable workflow. When it is empty, the workflow runs `npm run test:e2e` as before. `end-to-end-tests.yml` sets it from its own matrix. Document `--shuffle` in the E2E README as a check for new specs.
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. Props Bot commandsYou can trigger Props Bot actions by commenting on this PR:
|
…n page. See #66392.
There was a problem hiding this comment.
🔵 Needs a closer look
Concurrent specs share mutable WordPress state, so isolation and repeated CI results need human validation.
No open issues 🎉
What changed in this PR
This PR splits WordPress E2E tests into parallel and serial Playwright projects to reduce CI runtime while preserving the existing single-worker default command.
Changes:
- Adds
parallel,serial, and defaultallprojects, separating specs that modify shared state. - Runs parallel and serial groups in separate CI matrix jobs with distinct artifact names.
- Documents project selection, spec classification, and shuffled test runs.
| File | Description |
|---|---|
tests/e2e/README.md |
Documents projects and guidance for validating spec isolation. |
tests/e2e/playwright.config.js |
Defines project partitions and worker limits. |
.github/workflows/reusable-end-to-end-tests.yml |
Accepts a project input and distinguishes job and artifact names. |
.github/workflows/end-to-end-tests.yml |
Adds parallel and serial groups to the CI matrix. |
Review effort: Balanced
lancewillett
left a comment
There was a problem hiding this comment.
Reviewed the project split and test isolation. All 101 tests are assigned exactly once: 29 parallel and 72 serial. CI passed both groups with SCRIPT_DEBUG enabled and disabled, without retried tests. Local actionlint, zizmor, and ESLint passed.
A shuffled local two-worker run passed 27 tests with no retries. The two user-management cases need the single-site UI; this local site is multisite, and the user-creation failure also reproduced with one worker. CI covered both cases successfully. No blocking issues found.
AI review · gpt-6
|
@lucatume Could you also review the approach, please? |
lucatume
left a comment
There was a problem hiding this comment.
In parallel, requestUtils.login() at tests/e2e/specs/media-upload.test.js:13 invalidates the worker's REST nonce, so deleteMedia() at line 108 retries through setupRest(), which rewrites artifacts/storage-states/admin.json in place while other workers read it for new contexts ("Error reading storage state ... Unexpected end of JSON input"; reproduced by widening the write window in rest.js, 2 and 6 taxonomies failures over 2 runs). Global setup already writes the storage state, so dropping those two lines fixes it; all 29 parallel tests pass without them:
--- a/tests/e2e/specs/media-upload.test.js
+++ b/tests/e2e/specs/media-upload.test.js
@@ -11,5 +11,3 @@
test( 'Test dismissing failed upload works correctly', async ({ page, admin, requestUtils }) => {
- // Log in before visiting admin page.
- await requestUtils.login();
await admin.visitAdminPage( '/media-new.php' );Alternatively, add media-upload.test.js to SERIAL_SPECS.
…test. The extra login invalidates the worker's REST nonce. The next REST call then rewrites the shared storage state file while other workers read it. See #66392.
Trac Ticket MissingThis pull request is missing a link to a Trac ticket. For a contribution to be considered, there must be a corresponding ticket in Trac. To attach a pull request to a Trac ticket, please include the ticket's full URL in your pull request description. More information about contributing to WordPress on GitHub can be found in the Core Handbook. |
lancewillett
left a comment
There was a problem hiding this comment.
Re-reviewed a410f30: it removes exactly the redundant login Luca identified. Global setup already establishes authentication; avoiding that extra login prevents the stale nonce from triggering a shared storage-state rewrite during media cleanup.
Local discovery confirms 101 tests are assigned exactly once: 29 parallel and 72 serial. ESLint and actionlint passed. The four current CI jobs passed all 202 test executions with SCRIPT_DEBUG enabled and disabled, with no retries reported. I did not repeat the browser runs locally during this re-review.
No remaining blocker found. @lucatume, your requested fix is now in place for a final look.
AI review · gpt-6
Split the Playwright suite into `parallel` and `serial` projects, running each in a separate CI job for both `SCRIPT_DEBUG` settings. Keep the default `all` project on one worker for local runs, and give each CI job a distinct artifact name. Remove the redundant media-upload login to avoid concurrent writes to shared authentication state. Document how to classify specs and check their execution order. Developed in: #14897 Props adrianmoldovanwp, lucatume. See #66392. git-svn-id: https://develop.svn.wordpress.org/trunk@65049 602fd350-edb4-49c9-b593-d223f7449a82
|
PR #14897 landed in https://core.trac.wordpress.org/changeset/65049 |
Split the Playwright suite into `parallel` and `serial` projects, running each in a separate CI job for both `SCRIPT_DEBUG` settings. Keep the default `all` project on one worker for local runs, and give each CI job a distinct artifact name. Remove the redundant media-upload login to avoid concurrent writes to shared authentication state. Document how to classify specs and check their execution order. Developed in: WordPress/wordpress-develop#14897 Props adrianmoldovanwp, lucatume. See #66392. Built from https://develop.svn.wordpress.org/trunk@65049 git-svn-id: http://core.svn.wordpress.org/trunk@64199 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Adds two Playwright projects so that CI can run part of the E2E suite in parallel. Today the whole suite runs on one worker, because
@wordpress/scriptssetsworkers: 1.parallelruns 10 spec files on several workers, one for every two CPU cores. Tests inside one file still run in order.serialruns the other 14 spec files on one worker. These specs break the site for other tests or change content that other tests use. For example,install.test.jsdrops the database tables, andedit-posts.test.jsdeletes all posts.A third project,
all, is set as default and runs every spec on one worker as before. It is added for convenience, to keep the existingnpm run test:e2ecommand running the whole suite.The config sets
workers: '50%', the Playwright default, to replace theworkers: 1from@wordpress/scripts.In CI,
parallelandserialrun in separate matrix jobs.Testing
npm cinpm exec --no -- playwright install chromiumnpm run env:start && npm run env:installnpm run test:e2e -- --listlists 101 tests in 24 files, all in theallproject.npm run test:e2e -- --project=serial --listlists 72 tests in 14 files.npm run test:e2e -- --project=parallel --listlists 29 tests in the other 10 files.npm run test:e2e -- --project=parallel, thennpm run test:e2e -- --project=serialTrac ticket: #66392
Use of AI Tools
AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Opus 5.5
Used for: Config, scripts, workflow change, test runs and this description. I reviewed the result.
This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.