diff --git a/tests/e2e/README.md b/tests/e2e/README.md index ff5584dd547c5..d8c6697064e0b 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -1 +1 @@ -# E2E Tests End-To-End (E2E) tests for WordPress. ## Running the tests The e2e tests require a production-like environment to run. By default, they will assume an environment is available at `http://localhost:8889`, with username `admin` and password `password`. If you don't already have an environment ready, you can set one up by following [these instructions](https://github.com/WordPress/wordpress-develop/blob/master/README.md). Then you can launch the tests by running: ``` npm run test:e2e ``` which will run the test suite using a headless browser. If your environment has a different url, username or password to the default, you can provide the base URL, username and password like this: ``` WP_BASE_URL=http://mycustomurl WP_USERNAME=username WP_PASSWORD=password npm run test:e2e ``` **DO NOT run these tests in an actual production environment, as they will delete all your content.** For debugging purposes, you might want to follow the test visually. You can do so by running the tests in an interactive mode: ``` npm run test:e2e -- --ui ``` [UI Mode](https://playwright.dev/docs/test-ui-mode) let's you explore, run and debug tests with a time travel experience complete with watch mode. All test files are loaded into the testing sidebar where you can expand each file and describe block to individually run, view, watch and debug each test. You can also run a single test file separately: ``` npm run test:e2e tests/e2e/specs/hello.test.js ``` ## Projects The suite has three Playwright projects, defined in `playwright.config.js`: * `all`: every spec, one at a time on one worker. `npm run test:e2e` runs this project. * `parallel`: specs that can share the site with other tests. Spec files run at the same time on several workers, one for every two CPU cores. Tests inside one file still run in order. Run it with `npm run test:e2e -- --project=parallel`. * `serial`: specs that break the site for other tests. They run one at a time on one worker. Run it with `npm run test:e2e -- --project=serial`. CI runs `parallel` and `serial` in separate jobs. New specs go in `parallel` by default. Add a spec to `SERIAL_SPECS` only if it makes the site unusable for other tests, or if it changes content or settings that other tests use, for example with `requestUtils.deleteAllPosts()`. Before you open a pull request with a new spec, run its project a few times with `--shuffle`. Playwright then runs the spec files in a random order, so a spec that needs another spec to run first, or that breaks a spec that runs after it, fails: ``` npm run test:e2e -- --project=parallel --shuffle ``` Playwright prints the seed of each run, for example `shuffle seed 317753389`. To run the same order again, give the seed: `--shuffle 317753389`. ## Logged-out pages Most specs run logged in as the admin user. There are two ways to run a test logged out: * To make the `page` of a whole `describe` group logged out, add `test.use( { storageState: { cookies: [], origins: [] } } )` to the group. See `specs/dashboard.test.js`. * To use a logged-out page next to the logged-in `page` in one test, import `test` and `expect` from `../fixtures` and use the `loggedOutPage` fixture. Its browser context has no cookies, and it closes when the test ends. Unlike `page`, it resolves a URL that starts with `/` against the host only, not the full `WP_BASE_URL`, so it does not work with a site in a subdirectory. See `specs/cache-control-headers-directives.test.js`. ## Fixtures To change a setting, an option, or a must-use plugin, import `test` and `expect` from `../fixtures`, and use one of the fixtures in `fixtures/site.js`. Each fixture restores the state when the test ends, also if the test fails. If a restore fails, the test fails. * `siteSettings`: reads and changes the settings of `/wp/v2/settings` through REST, with `get()` and `set( patch )`. It restores only the keys passed to `set`. See `specs/privacy.test.js`. * `siteOptions`: reads and changes options through WP-CLI, with `get( names )` and `set( values )`. It reads and restores the stored `gmt_offset`, not the value calculated for a named timezone. It restores only the keys passed to `set`. See `specs/general-settings-invalid-timezone.test.js`. * `muPlugin`: writes a must-use plugin with `write( fileName, php )`, and deletes the file when the test ends. Call `write` before the test visits a page. See `specs/fatal-error-handler.test.js`. If the test changes a setting or an option through a form, first call `set` with the current value. The fixture then knows that it must restore the key. ## Documentation * Block Editor Handbook end to end testing overview: https://developer.wordpress.org/block-editor/contributors/code/testing-overview/#end-to-end-testing * Gutenberg e2e-test-utils-playwright package API docs: https://github.com/WordPress/gutenberg/tree/trunk/packages/e2e-test-utils-playwright * Playwright API docs: https://playwright.dev/docs (the version we are using is indicated in the `@wordpress/scripts` package: https://github.com/WordPress/gutenberg/blob/trunk/packages/scripts/package.json) \ No newline at end of file +# E2E Tests End-To-End (E2E) tests for WordPress. ## Running the tests The e2e tests require a production-like environment to run. By default, they will assume an environment is available at `http://localhost:8889`, with username `admin` and password `password`. If you don't already have an environment ready, you can set one up by following [these instructions](https://github.com/WordPress/wordpress-develop/blob/master/README.md). Then you can launch the tests by running: ``` npm run test:e2e ``` which will run the test suite using a headless browser. If your environment has a different url, username or password to the default, you can provide the base URL, username and password like this: ``` WP_BASE_URL=http://mycustomurl WP_USERNAME=username WP_PASSWORD=password npm run test:e2e ``` **DO NOT run these tests in an actual production environment, as they will delete all your content.** For debugging purposes, you might want to follow the test visually. You can do so by running the tests in an interactive mode: ``` npm run test:e2e -- --ui ``` [UI Mode](https://playwright.dev/docs/test-ui-mode) let's you explore, run and debug tests with a time travel experience complete with watch mode. All test files are loaded into the testing sidebar where you can expand each file and describe block to individually run, view, watch and debug each test. You can also run a single test file separately: ``` npm run test:e2e tests/e2e/specs/hello.test.js ``` ## Projects The suite has three Playwright projects, defined in `playwright.config.js`: * `all`: every spec, one at a time on one worker. `npm run test:e2e` runs this project. * `parallel`: specs that can share the site with other tests. Spec files run at the same time on several workers, one for every two CPU cores. Tests inside one file still run in order. Run it with `npm run test:e2e -- --project=parallel`. * `serial`: specs that break the site for other tests. They run one at a time on one worker. Run it with `npm run test:e2e -- --project=serial`. CI runs `parallel` and `serial` in separate jobs. New specs go in `parallel` by default. Add a spec to `SERIAL_SPECS` if it makes the site unusable for other tests. If it changes content or settings that other tests use, for example with `requestUtils.deleteAllPosts()`, follow [Parallel safety](#parallel-safety). Before you open a pull request with a new spec, run its project a few times with `--shuffle`. Playwright then runs the spec files in a random order, so a spec that needs another spec to run first, or that breaks a spec that runs after it, fails: ``` npm run test:e2e -- --project=parallel --shuffle ``` Playwright prints the seed of each run, for example `shuffle seed 317753389`. To run the same order again, give the seed: `--shuffle 317753389`. ## Parallel safety Specs in `parallel` share one site. A test that changes shared data can break a test in another file that runs at the same time. To make a spec safe, use the first of these that works: 1. Scope the data. Let the test create its own data, for example a post with a unique title, and read only that data. 2. If the test must change shared data, such as a site setting, give it a lock. Do this only if every spec that writes the same data holds the same lock. 3. If you cannot lock every spec that writes the data, add the spec to `SERIAL_SPECS`. A lock is a name. Playwright never runs two tests that hold the same lock name at the same time. A lock only schedules tests. It does not isolate data: a test without the lock can still change the data while a locked test runs. The lock names are in `utils/locks.js`. Import the constant, so that all specs use the same name. Playwright gives no error for a lock name that no other test uses, or for an import name that is not in `utils/locks.js`: ``` import { GENERAL_SETTINGS } from '../utils/locks'; test.describe( 'Settings -> General', { lock: GENERAL_SETTINGS }, () => {} ); ``` A form save writes every field on the page, not only the field that the test changed. A spec that saves the General Settings form (`options-general.php`) must hold `GENERAL_SETTINGS`. A spec in `parallel` that changes a General Settings option in a different way, for example with `siteSettings` or `siteOptions`, must also hold it. ## Logged-out pages Most specs run logged in as the admin user. There are two ways to run a test logged out: * To make the `page` of a whole `describe` group logged out, add `test.use( { storageState: { cookies: [], origins: [] } } )` to the group. See `specs/dashboard.test.js`. * To use a logged-out page next to the logged-in `page` in one test, import `test` and `expect` from `../fixtures` and use the `loggedOutPage` fixture. Its browser context has no cookies, and it closes when the test ends. Unlike `page`, it resolves a URL that starts with `/` against the host only, not the full `WP_BASE_URL`, so it does not work with a site in a subdirectory. See `specs/cache-control-headers-directives.test.js`. ## Fixtures To change a setting, an option, or a must-use plugin, import `test` and `expect` from `../fixtures`, and use one of the fixtures in `fixtures/site.js`. Each fixture restores the state when the test ends, also if the test fails. If a restore fails, the test fails. * `siteSettings`: reads and changes the settings of `/wp/v2/settings` through REST, with `get()` and `set( patch )`. It restores only the keys passed to `set`. See `specs/privacy.test.js`. * `siteOptions`: reads and changes options through WP-CLI, with `get( names )` and `set( values )`. It reads and restores the stored `gmt_offset`, not the value calculated for a named timezone. It restores only the keys passed to `set`. See `specs/general-settings-invalid-timezone.test.js`. * `muPlugin`: writes a must-use plugin with `write( fileName, php )`, and deletes the file when the test ends. Call `write` before the test visits a page. See `specs/fatal-error-handler.test.js`. If the test changes a setting or an option through a form, first call `set` with the current value. The fixture then knows that it must restore the key. ## Documentation * Block Editor Handbook end to end testing overview: https://developer.wordpress.org/block-editor/contributors/code/testing-overview/#end-to-end-testing * Gutenberg e2e-test-utils-playwright package API docs: https://github.com/WordPress/gutenberg/tree/trunk/packages/e2e-test-utils-playwright * Playwright API docs: https://playwright.dev/docs (the version we are using is indicated in the `@wordpress/scripts` package: https://github.com/WordPress/gutenberg/blob/trunk/packages/scripts/package.json) \ No newline at end of file diff --git a/tests/e2e/specs/general-settings-invalid-timezone.test.js b/tests/e2e/specs/general-settings-invalid-timezone.test.js index 35cb231b96db2..597219f0259ab 100644 --- a/tests/e2e/specs/general-settings-invalid-timezone.test.js +++ b/tests/e2e/specs/general-settings-invalid-timezone.test.js @@ -1,6 +1,7 @@ import { test, expect } from '../fixtures'; +import { GENERAL_SETTINGS } from '../utils/locks'; -test.describe( 'Settings -> General', () => { +test.describe( 'Settings -> General', { lock: GENERAL_SETTINGS }, () => { const invalidTimezones = [ '', '0', 'Barry/Gary' ]; test.beforeEach( async ( { admin } ) => { diff --git a/tests/e2e/specs/general-settings.test.js b/tests/e2e/specs/general-settings.test.js index 6401bf7110e10..59171605d4f21 100644 --- a/tests/e2e/specs/general-settings.test.js +++ b/tests/e2e/specs/general-settings.test.js @@ -1,6 +1,7 @@ import { test, expect } from '@wordpress/e2e-test-utils-playwright'; +import { GENERAL_SETTINGS } from '../utils/locks'; -test.describe( 'Settings -> General', () => { +test.describe( 'Settings -> General', { lock: GENERAL_SETTINGS }, () => { test( 'Saving General Settings preserves the active theme options', async ( { admin, page, diff --git a/tests/e2e/utils/locks.js b/tests/e2e/utils/locks.js new file mode 100644 index 0000000000000..6c2afafe7223c --- /dev/null +++ b/tests/e2e/utils/locks.js @@ -0,0 +1 @@ +export const GENERAL_SETTINGS = 'general-settings';