Skip to content

Build/Test Tools: Run E2E specs in parallel and serial projects. - #14897

Closed
adimoldovan wants to merge 10 commits into
WordPress:trunkfrom
adimoldovan:e2e/split-parallel-serial-projects
Closed

adimoldovan wants to merge 10 commits into
WordPress:trunkfrom
adimoldovan:e2e/split-parallel-serial-projects

Conversation

@adimoldovan

@adimoldovan adimoldovan commented Oct 26, 2026 •

Copy link
Copy Markdown

Adds two Playwright projects so that CI can run part of the E2E suite in parallel. Today the whole suite runs on one worker, because @wordpress/scripts sets workers: 1.

  • parallel runs 10 spec files on several workers, one for every two CPU cores. Tests inside one file still run in order.
  • serial runs the other 14 spec files on one worker. These specs break the site for other tests or change content that other tests use. For example, install.test.js drops the database tables, and edit-posts.test.js deletes all posts.

A third project, all, is set as default and runs every spec on one worker as before. It is added for convenience, to keep the existing npm run test:e2e command running the whole suite.

The config sets workers: '50%', the Playwright default, to replace the workers: 1 from @wordpress/scripts.

In CI, parallel and serial run in separate matrix jobs.

Testing

  1. npm ci
  2. npm exec --no -- playwright install chromium
  3. npm run env:start && npm run env:install
  4. npm run test:e2e -- --list lists 101 tests in 24 files, all in the all project.
  5. npm run test:e2e -- --project=serial --list lists 72 tests in 14 files.
  6. npm run test:e2e -- --project=parallel --list lists 29 tests in the other 10 files.
  7. npm run test:e2e -- --project=parallel, then npm run test:e2e -- --project=serial

Trac ticket: #66392

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Opus 5.5
Used for: Config, scripts, workflow change, test runs and this description. I reviewed the result.


This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

The E2E suite runs on one worker. Split it into two Playwright projects:

- `parallel` runs 15 spec files on several workers.
- `serial` runs the 6 specs that break the site for other tests, on one worker.

Each project has its own npm script. CI runs each project in a separate matrix job, so a failure in one project does not hide the results of the other.
@adimoldovan adimoldovan self-assigned this Oct 26, 2026
@github-actions

Copy link
Copy Markdown

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

  • All changes will be lost when closing a tab with a Playground instance.
  • All changes will be lost when refreshing the page.
  • A fresh instance is created each time the link below is clicked.
  • Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
    it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

The base config from `@wordpress/scripts` sets `workers: 1` for the whole run, so the `parallel` project ran on one worker. Set `workers: '50%'` at the top level. This is the Playwright default.

Move the 7 specs that delete all posts or media to `serial`. On several workers, they delete content that tests on other workers use.
… project.

Add an `all` project that runs every spec on one worker. It is the only default project, so `npm run test:e2e` runs the whole suite serially, as before.

Set `default: false` on `parallel` and `serial`, which needs Playwright 1.64. CI selects them with `--project`, so the two npm scripts are no longer needed.

Move `themes.test.js` to `serial`, because it changes the active theme for the whole site.
Release branches call the `trunk` copy of the reusable E2E workflow, and their Playwright configs have no `parallel` or `serial` project. Add a `test-group` input to the reusable workflow. When it is empty, the workflow runs `npm run test:e2e` as before. `end-to-end-tests.yml` sets it from its own matrix.

Document `--shuffle` in the E2E README as a check for new specs.
@adimoldovan
adimoldovan marked this pull request as ready for review October 27, 2026 14:32
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs.

Core Committers: Use this line as a base for the props when committing in SVN:

Props adrianmoldovanwp, lancewillett, lucatume.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.


Props Bot commands

You can trigger Props Bot actions by commenting on this PR:

  • @propsbot add <username> to give props to a GitHub user. Requires write access.
  • @propsbot remove <username> to leave a user out of the props. Requires write access.
  • @propsbot refresh to refresh this list. Adding the props-bot label works too.

@adimoldovan
adimoldovan requested a review from a team October 27, 2026 14:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Concurrent specs share mutable WordPress state, so isolation and repeated CI results need human validation.

No open issues 🎉

What changed in this PR

This PR splits WordPress E2E tests into parallel and serial Playwright projects to reduce CI runtime while preserving the existing single-worker default command.

Changes:

  • Adds parallel, serial, and default all projects, separating specs that modify shared state.
  • Runs parallel and serial groups in separate CI matrix jobs with distinct artifact names.
  • Documents project selection, spec classification, and shuffled test runs.
File Description
tests/​e2e/​README.md Documents projects and guidance for validating spec isolation.
tests/​e2e/​playwright.config.js Defines project partitions and worker limits.
.github/​workflows/​reusable-end-to-end-tests.yml Accepts a project input and distinguishes job and artifact names.
.github/​workflows/​end-to-end-tests.yml Adds parallel and serial groups to the CI matrix.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Parallel tests share mutable WordPress state, so test isolation needs confirmation through repeated CI runs and final human review.

No open issues 🎉

Review effort: Balanced

@lancewillett lancewillett left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the project split and test isolation. All 101 tests are assigned exactly once: 29 parallel and 72 serial. CI passed both groups with SCRIPT_DEBUG enabled and disabled, without retried tests. Local actionlint, zizmor, and ESLint passed.

A shuffled local two-worker run passed 27 tests with no retries. The two user-management cases need the single-site UI; this local site is multisite, and the user-creation failure also reproduced with one worker. CI covered both cases successfully. No blocking issues found.

AI review · gpt-6

@lancewillett
lancewillett requested a review from lucatume October 27, 2026 18:50
@lancewillett

Copy link
Copy Markdown
Member

@lucatume Could you also review the approach, please?

@lucatume lucatume left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In parallel, requestUtils.login() at tests/e2e/specs/media-upload.test.js:13 invalidates the worker's REST nonce, so deleteMedia() at line 108 retries through setupRest(), which rewrites artifacts/storage-states/admin.json in place while other workers read it for new contexts ("Error reading storage state ... Unexpected end of JSON input"; reproduced by widening the write window in rest.js, 2 and 6 taxonomies failures over 2 runs). Global setup already writes the storage state, so dropping those two lines fixes it; all 29 parallel tests pass without them:

--- a/tests/e2e/specs/media-upload.test.js
+++ b/tests/e2e/specs/media-upload.test.js
@@ -11,5 +11,3 @@
 test( 'Test dismissing failed upload works correctly', async ({ page, admin, requestUtils }) => {
-	// Log in before visiting admin page.
-	await requestUtils.login();
 	await admin.visitAdminPage( '/media-new.php' );

Alternatively, add media-upload.test.js to SERIAL_SPECS.

…test.

The extra login invalidates the worker's REST nonce. The next REST call then rewrites the shared storage state file while other workers read it.

See #66392.
@github-actions

Copy link
Copy Markdown

Trac Ticket Missing

This pull request is missing a link to a Trac ticket. For a contribution to be considered, there must be a corresponding ticket in Trac.

To attach a pull request to a Trac ticket, please include the ticket's full URL in your pull request description. More information about contributing to WordPress on GitHub can be found in the Core Handbook.

@lancewillett
lancewillett requested a balanced review from Copilot October 29, 2026 14:57

@lancewillett lancewillett left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed a410f30: it removes exactly the redundant login Luca identified. Global setup already establishes authentication; avoiding that extra login prevents the stale nonce from triggering a shared storage-state rewrite during media cleanup.

Local discovery confirms 101 tests are assigned exactly once: 29 parallel and 72 serial. ESLint and actionlint passed. The four current CI jobs passed all 202 test executions with SCRIPT_DEBUG enabled and disabled, with no retries reported. I did not repeat the browser runs locally during this re-review.

No remaining blocker found. @lucatume, your requested fix is now in place for a final look.

AI review · gpt-6

@adimoldovan
adimoldovan requested a review from lucatume October 29, 2026 15:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the E2E suite’s parallel/serial split.

No open issues 🎉

Review effort: Balanced

wporg-sync pushed a commit that referenced this pull request Oct 30, 2026
Split the Playwright suite into `parallel` and `serial` projects, running each in a separate CI job for both `SCRIPT_DEBUG` settings. Keep the default `all` project on one worker for local runs, and give each CI job a distinct artifact name.

Remove the redundant media-upload login to avoid concurrent writes to shared authentication state. Document how to classify specs and check their execution order.

Developed in: #14897

Props adrianmoldovanwp, lucatume.
See #66392.


git-svn-id: https://develop.svn.wordpress.org/trunk@65049 602fd350-edb4-49c9-b593-d223f7449a82
@lancewillett

Copy link
Copy Markdown
Member

PR #14897 landed in https://core.trac.wordpress.org/changeset/65049

wporg-sync pushed a commit to WordPress/WordPress that referenced this pull request Oct 30, 2026
Split the Playwright suite into `parallel` and `serial` projects, running each in a separate CI job for both `SCRIPT_DEBUG` settings. Keep the default `all` project on one worker for local runs, and give each CI job a distinct artifact name.

Remove the redundant media-upload login to avoid concurrent writes to shared authentication state. Document how to classify specs and check their execution order.

Developed in: WordPress/wordpress-develop#14897

Props adrianmoldovanwp, lucatume.
See #66392.

Built from https://develop.svn.wordpress.org/trunk@65049


git-svn-id: http://core.svn.wordpress.org/trunk@64199 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

4 participants