Skip to content

Commit 525e71d

Browse files
raju-opticlaude
andauthored
[FSSDK-13413] Run browser tests with Vitest 5 (#1186)
## Summary Vitest browser mode is not stable in version 3. It fails frequently in CI, is very slow and also requires hacky modification to files in node_modules. This PR runs the tests using vitest@5 in a separate subproject. The root project stays on Vitest 3 so the Node tests keep running on Node 18. - Add `tests/browser`, a standalone Vitest 5 project that runs the same browser and UMD spec files from `lib/` in real browsers, locally or on BrowserStack. It has its own `package.json`/lockfile, configs and runner (`run.js`). - Point CI's `browser_tests` job at it. The job now uses Node 22 (Vitest 5 needs 22.12+), installs root deps, runs `npm ci` in `tests/browser`, then `npm run test-browser`. Matrix and secrets are unchanged. - Run the UMD tests in CI again (`npm run test-umd` in `tests/browser`, as a step in the same job). It builds the UMD bundle and tests it in the same browsers, and runs even if the browser tests fail. The old UMD job was commented out. - Make the `backoff_retry_runner` cancel test deterministic. It depended on how many microtask ticks an `await` takes, which differs on Safari 15. It now uses `ConstantBackoff(100)` and stops the last iteration 10ms short of the retry, so the cancel lands while the retry is waiting on a fake timer. The expected count is still 6. Changes needed for the same specs to pass on Vitest 5 (details in `tests/browser/README.md` and comments in `vitest.shared.mts`): - **`isolate: true`.** With shared iframes, a module loaded unmocked by one file stays unmocked for later files under Vitest 5. - **`browser.importMapMocks: false`.** Always mock by request interception, as Vitest 3 does and as old browsers require. - **`clearMocks: false` + `vitest3-compat.setup.ts`.** Vitest 5 defaults `clearMocks` to true, and `vi.restoreAllMocks()` no longer resets `vi.fn()` mocks. The specs rely on the Vitest 3 behavior of both. - **Polyfills for `Array.prototype.at`, `findLast(Index)` and `Object.hasOwn`**, injected into both the orchestrator page (`browser.orchestratorScripts`) and the test iframe. Vitest 5's browser runtime calls them, and Edge 89/90 and Firefox 91 lack them. Without the polyfills, those runs hang until the connect timeout. `lib/` doesn't use these functions, so the polyfills can't hide an SDK issue today. - **Oxc target set to the oldest tested browsers instead of `es2015`.** Vite 8 replaces esbuild with Oxc, and Vitest 5's mock modules use top-level `await` and string export names, which can't be compiled down to ES2015. Dependencies are still compiled to ES2015 via Rolldown. - **`lightningcss` is replaced by an empty local stub** (`tests/browser/stubs/lightningcss`, via `overrides`). It is MPL-2.0, which license policy doesn't allow. Vite 8 depends on it but only loads it lazily for `css.transformer: 'lightningcss'` or CSS minification in production builds, neither of which happens under Vitest. If anything ever loads it, the stub throws a clear error. This also removes its 11 platform binaries. - **No viewport patch.** Vitest 5 falls back to classic WebDriver's `setWindowSize`. - **UMD plugin and console capture use `transformIndexHtml`.** Patching the response doesn't reach the tester page under Vitest 5. - **Tunnels start without `force: true`, with a unique `localIdentifier`.** Runs can go in parallel, one per browser. - **Small runner fixes.** - Tunnel startup errors now fail the run with exit 1. Previously they were swallowed and the run exited 0 with no tests run. - `USE_LOCAL_BROWSER` is passed to Vitest when credentials are missing. - Extra arguments are passed through to Vitest (e.g. `--reporter=verbose <spec>`). - Set `WDIO_LOG_LEVEL` to see webdriverio's commands. Type tests (`*.test-d.ts`) are not run in this project. They run `tsc` in Node, and the root `npm test` already covers them. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 95422f5 commit 525e71d

15 files changed

Lines changed: 5232 additions & 10 deletions

‎.github/workflows/javascript.yml‎

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -107,17 +107,33 @@ jobs:
107107
TEST_BROWSER: ${{ matrix.browser }}
108108
steps:
109109
- uses: actions/checkout@v3
110+
# Runs the browser tests with Vitest 5 from tests/browser (see its README).
111+
# Needs Node 22.23.4+ (Vitest 5 / webdriverio); the root project stays on Vitest 3 for Node 18.
110112
- name: Set up Node
111113
uses: actions/setup-node@v3
112114
with:
113-
node-version: 20
115+
node-version: 22
114116
cache: 'npm'
115-
cache-dependency-path: ./package-lock.json
116-
- name: Browser tests - ${{ matrix.browser }}
117+
cache-dependency-path: |
118+
./package-lock.json
119+
./tests/browser/package-lock.json
120+
- name: Install SDK dependencies
117121
working-directory: .
118-
run: |
119-
npm install
120-
npm run test-browser
122+
run: npm install
123+
- name: Install browser test dependencies
124+
id: browser-deps
125+
working-directory: ./tests/browser
126+
run: npm ci
127+
- name: Browser tests - ${{ matrix.browser }}
128+
working-directory: ./tests/browser
129+
run: npm run test-browser
130+
# Builds the UMD bundle (root build-browser-umd) and tests it in the same browsers.
131+
# Runs even if the browser tests failed, so both results are reported, but not
132+
# if installing dependencies failed (browser-deps is skipped if the SDK install fails).
133+
- name: UMD tests - ${{ matrix.browser }}
134+
if: ${{ !cancelled() && steps.browser-deps.outcome == 'success' }}
135+
working-directory: ./tests/browser
136+
run: npm run test-umd
121137

122138
# crossbrowser_and_umd_unit_tests:
123139
# runs-on: ubuntu-latest

‎lib/utils/executor/backoff_retry_runner.spec.ts‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import { vi, describe, it, expect, beforeEach, afterEach } from 'vitest';
22
import { runWithRetry } from './backoff_retry_runner';
33
import { advanceTimersByTime } from '../../tests/testUtils';
4+
import { ConstantBackoff } from '../repeater/repeater';
45

56
const exhaustMicrotasks = async (loop = 100) => {
67
for(let i = 0; i < loop; i++) {
@@ -116,18 +117,19 @@ describe('runWithRetry', () => {
116117
throw new Error('error');
117118
};
118119

119-
const { result, cancelRetry } = runWithRetry(task, undefined, 100);
120+
const { result, cancelRetry } = runWithRetry(task, new ConstantBackoff(100), 100);
120121

121122
for(let i = 0; i < 5; i++) {
122123
await exhaustMicrotasks();
123-
await advanceTimersByTime(0);
124+
// the last iteration stops 10ms short, so the next retry is still waiting on its backoff
125+
await advanceTimersByTime(i < 4 ? 100 : 90);
124126
}
125-
127+
126128
cancelRetry();
127129

128130
for(let i = 0; i < 100; i++) {
129131
await exhaustMicrotasks();
130-
await advanceTimersByTime(0);
132+
await advanceTimersByTime(100);
131133
}
132134

133135
try {

‎tests/browser/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Browser and UMD tests on Vitest 5
2+
3+
Runs the same browser and UMD spec files as the root setup (`npm run test-browser` / `npm run test-umd`), but with Vitest 5 instead of Vitest 3. It's a separate project because the root has to stay on Vitest 3 for Node 18. The root setup is unchanged and still works.
4+
5+
Requires Node 22.23.4+, 24.22.1+ or 26.12.1+ (the minimums of the locked webdriverio packages; Vitest 5 itself needs 22.12+).
6+
7+
```sh
8+
cd tests/browser
9+
npm install
10+
11+
npm run test-browser-local # Chrome on this machine
12+
TEST_BROWSER=firefox npm run test-browser-local # a different local browser
13+
npm run test-umd-local # builds the UMD bundle in the repo root first
14+
15+
npm run test-browser # BrowserStack if credentials are set, local otherwise
16+
npm run test-umd
17+
```
18+
19+
Same environment variables as the root setup: `TEST_BROWSER`, `USE_LOCAL_BROWSER`, `BROWSERSTACK_USERNAME` / `BROWSERSTACK_ACCESS_KEY`, `VITEST_REQUEST_LOGGER`, `VITEST_CONSOLE_CAPTURE`. Same browser matrix as the root setup (see `BROWSER_CONFIGS` in `run.js`).
20+
21+
Runs can go in parallel, for example one terminal per browser: each run starts its own BrowserStack Local tunnel with a unique identifier. (The root scripts start the tunnel with `force: true`, which kills other tunnels, so they can't run alongside each other or this project.)
22+
23+
Extra arguments go to Vitest, for example `node run.js browser --reporter=verbose ../../lib/core/bucketer/index.spec.ts`.
24+
25+
## Differences from the root (Vitest 3) setup
26+
27+
- No viewport patch. Vitest 5 falls back to classic WebDriver when BiDi is off.
28+
- Each spec file runs in its own iframe (`isolate: true`). Shared iframes break `vi.mock` under Vitest 5.
29+
- Modules are always mocked by request interception (`browser.importMapMocks: false`), the same way Vitest 3 does it and the way old browsers must.
30+
- `clearMocks: false` plus `vitest3-compat.setup.ts` keep the Vitest 3 mock-reset behavior the specs rely on.
31+
- Source is compiled for the oldest tested browsers, not ES2015, because Vitest 5's mock modules can't be compiled down to ES2015. Dependencies are still compiled to ES2015.
32+
- `legacy-polyfills-plugin.ts` adds `Array.prototype.at`, `findLast`/`findLastIndex` and `Object.hasOwn` to both pages Vitest loads (the orchestrator page and the test iframe). Vitest 5's browser runtime calls them, and Edge 89/90 and Firefox 91 don't have them; without the polyfills the run hangs until it times out. The SDK code under test sees the polyfills too (`lib/` doesn't use these functions today).
33+
- `lightningcss` is replaced by an empty stub (`stubs/lightningcss`, wired in through `overrides` in `package.json`). It's MPL-2.0, which our license policy doesn't allow. Vite 8 depends on it but only loads it for `css.transformer: 'lightningcss'` or CSS minification in production builds, neither of which happens here. If something ever loads it, the stub throws an error saying so.
34+
- Type tests (`*.test-d.ts`) are skipped. The root `npm test` already runs them.
35+
36+
See the comments in `vitest.shared.mts` for details.
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
/**
2+
* Copyright 2026, Optimizely
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
import type { Plugin } from 'vite';
17+
import { consoleCapturePlugin as rootConsoleCapturePlugin } from '../../vitest/console-capture-plugin';
18+
19+
// Vitest 5 version of the root vitest/console-capture-plugin.ts. Reuses its
20+
// /__vitest_console__ endpoint, but injects vitest/public/console-capture.js
21+
// through transformIndexHtml, because the root plugin's response patching
22+
// doesn't reach the tester page under Vitest 5. (The root plugin skips pages
23+
// that already include the script, so it is never added twice.)
24+
export function consoleCapturePlugin(): Plugin {
25+
return {
26+
...(rootConsoleCapturePlugin() as unknown as Plugin),
27+
transformIndexHtml() {
28+
return [
29+
{
30+
tag: 'script',
31+
attrs: { src: '/console-capture.js' },
32+
injectTo: 'head-prepend',
33+
},
34+
];
35+
},
36+
};
37+
}
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
/**
2+
* Copyright 2026, Optimizely
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
import type { Plugin } from 'vite';
17+
18+
// Vitest 5's in-browser runtime (the tester page and the pre-bundled vitest
19+
// runtime) calls a few built-ins that the oldest browsers we test don't have:
20+
//
21+
// Array/String/TypedArray.prototype.at Chromium 92, Firefox 90, Safari 15.4
22+
// Object.hasOwn Chromium 93, Firefox 92, Safari 15.4
23+
// Array.prototype.findLast(Index) Chromium 97, Firefox 104, Safari 15.4
24+
//
25+
// Without them the runtime throws before it can report back, and the run hangs
26+
// until it times out (seen on Edge 89 and 90). Vitest 3 didn't depend on these, so the
27+
// root setup doesn't need this.
28+
//
29+
// The polyfills go into both pages Vitest loads: the orchestrator page (via
30+
// `browser.orchestratorScripts`, see legacyPolyfillsOrchestratorScript) and the
31+
// test iframe (via this plugin's transformIndexHtml). They run before any other
32+
// script, so the SDK code under test also sees them. The SDK source (lib/) doesn't use any of
33+
// these today. If it starts to, tests on these old browsers would no longer
34+
// catch it.
35+
const POLYFILLS = `
36+
(function () {
37+
function define(target, name, value) {
38+
if (target && !target[name]) {
39+
Object.defineProperty(target, name, { value: value, writable: true, configurable: true });
40+
}
41+
}
42+
43+
function at(index) {
44+
var length = this.length;
45+
var i = Math.trunc(index) || 0;
46+
if (i < 0) i += length;
47+
return i < 0 || i >= length ? undefined : this[i];
48+
}
49+
50+
function findLast(predicate, thisArg) {
51+
for (var i = this.length - 1; i >= 0; i--) {
52+
if (predicate.call(thisArg, this[i], i, this)) return this[i];
53+
}
54+
return undefined;
55+
}
56+
57+
function findLastIndex(predicate, thisArg) {
58+
for (var i = this.length - 1; i >= 0; i--) {
59+
if (predicate.call(thisArg, this[i], i, this)) return i;
60+
}
61+
return -1;
62+
}
63+
64+
var typedArrayPrototype = Object.getPrototypeOf(Int8Array.prototype);
65+
66+
define(Array.prototype, 'at', at);
67+
define(String.prototype, 'at', at);
68+
define(typedArrayPrototype, 'at', at);
69+
define(Array.prototype, 'findLast', findLast);
70+
define(Array.prototype, 'findLastIndex', findLastIndex);
71+
define(typedArrayPrototype, 'findLast', findLast);
72+
define(typedArrayPrototype, 'findLastIndex', findLastIndex);
73+
define(Object, 'hasOwn', function (object, key) {
74+
return Object.prototype.hasOwnProperty.call(Object(object), key);
75+
});
76+
})();
77+
`;
78+
79+
// For `browser.orchestratorScripts`. The orchestrator page doesn't go through
80+
// transformIndexHtml. A classic script (not a module) runs while the page is
81+
// parsed, before Vitest's module scripts.
82+
export const legacyPolyfillsOrchestratorScript = {
83+
id: 'legacy-polyfills.js',
84+
content: POLYFILLS,
85+
type: 'text/javascript',
86+
};
87+
88+
// Adds the polyfills to the test iframe page
89+
export function legacyPolyfillsPlugin(): Plugin {
90+
return {
91+
name: 'legacy-polyfills',
92+
transformIndexHtml() {
93+
return [
94+
{
95+
tag: 'script',
96+
children: POLYFILLS,
97+
injectTo: 'head-prepend',
98+
},
99+
];
100+
},
101+
};
102+
}

0 commit comments

Comments
 (0)