Speed up yarn test-flows: shard CI, skip marketing, pause UI renderer - #1830
Speed up yarn test-flows: shard CI, skip marketing, pause UI renderer#1830pablo-mayrgundter wants to merge 8 commits into
Conversation
CI playwright-run has grown to ~18m of tests (188 tests, 34m serial, 2 workers on 8GB). Split the Conway suite across two shards so four Chromiums run without putting four wasm heaps on one machine; keep the required check name `playwright-run` as an aggregator. Playwright builds skip the Next.js marketing overlay (no flow spec loads those routes). Dialog/panel specs freeze the WebGL loop after the model is ready. GoogleDriveConnect no longer double-loads index.ifc.
✅ Deploy Preview for bldrs-share-dev ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
✅ Deploy Preview for bldrs-share-prod ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
pauseRenderer holds the last painted canvas frame. Notes/Comment goldens include the scene, so CI failed at ~9% pixel diff (threshold 2%). Keep the freeze on dialog/panel specs that do not screenshot the canvas.
|
@codex review — second commit drops pauseRenderer on full-page screenshot specs (Notes/Comment goldens include the scene; first run failed at ~9% pixel diff). |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Share is public, so standard ubuntu-24.04 is 4 vCPU / 16 GB and unlimited-free. The paid 8 GB larger runner was the RAM ceiling (2 workers green, 3+ starved) and billed even on a public repo. Keep 2 workers per box — four SwiftShader Chromiums on four cores contend — and split --shard=N/4 so eight Chromiums run across four free machines instead of packing one box. Screenshots still use the Playwright noble container.
|
@codex review — moved shards onto free public ubuntu-24.04 (16 GB) and split --shard=N/4, still 2 workers per box. Paid 8 GB larger runner is out of this workflow. |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Four shards each ran yarn clean && build. Esbuild is seconds, but that is four copies of the tree the suite claims to have tested. playwright-build uploads docs/ (maps stripped); shards set PLAYWRIGHT_SERVE_ONLY and only http-server it. Shards still yarn install — specs import app modules — so this is the one-build guarantee, not a skip of the install floor.
|
@codex review — playwright-build produces docs/ once; shards PLAYWRIGHT_SERVE_ONLY and http-server it. Yarn install remains per-shard (specs import app modules). |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Shards only need @playwright/test (browsers already in the image) and a handful of dep-free constant modules. npm install that one package, serve docs with a zero-dep static server that still returns 404.html for SPA routes. CutPlane.spec imported waitForModelReady via tsconfig baseUrl; switch it to a relative path so the runner does not need the Share node_modules tree to resolve it.
|
@codex review — shards skip Share yarn install: npm install @playwright/test@1.56.1 only, serve docs with tools/playwright/serveDocs.mjs. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be8613e7d4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD is set on the image. Do not | ||
| # `yarn install` the Share tree — that was ~50s per shard. | ||
| - name: Install Playwright test runner | ||
| run: npm install --no-save --no-package-lock @playwright/test@1.56.1 |
There was a problem hiding this comment.
In a clean playwright-shard checkout, this command runs from the repository root, so npm processes the root package.json and installs the Share dependency graph in addition to Playwright; --no-save only prevents updating the manifest. The npm install documentation confirms that dependencies listed in package.json are installed by default. This repeats the large install on all four shards, defeating the intended speed/disk optimization and potentially consuming most of the new 15-minute timeout; install into an isolated empty prefix/directory and expose that runner to the repo instead.
Useful? React with 👍 / 👎.
npm install at the repo root still resolved ts-jest/babel-jest peers and failed ERESOLVE. Put the runner in .pw-runner and symlink node_modules so specs can import @playwright/test.
Drop the custom serveDocs.mjs and the map-stripping step. Shards install @playwright/test@1.56.1 plus http-server in .pw-runner and serve docs/ with npx http-server, same as yarn test-flows-serve. The artifact is what yarn test-flows-build left, so the suite tests that tree rather than a rewritten one.
Playwright's webServer cwd is the config directory (tools/), so `npx http-server docs` would serve tools/docs. Serve-only sets cwd to the built artifact and runs `npx http-server` there, with PORT from env.
1 participant
Why
playwright-runon main was ~23m wall (18m of tests). Latest green run on main: 188 tests, 34m serial, 2 workers on the paid 8 GB larger runner. A third Chromium there starves the wasm heap. Share is public, so standardubuntu-24.04is 4 vCPU / 16 GB and unlimited-free; larger runners are billed even on public repos.Coverage is unchanged. Shards serve the built
docs/asyarn test-flows-buildleft it —npx http-serverinside that directory, no custom server, no map-stripping.What
--shard=N/4, each--workers=2, on freeubuntu-24.04. Eight Chromiums across four machines, 2-per-box (the density that stayed green). Packing 4 workers onto 4 vCPU would contend on SwiftShader; more shards buy cores instead.playwright-buildjob producesdocs/(SKIP_MARKETING=true). Shards download that artifact and setPLAYWRIGHT_SERVE_ONLY=true.yarn install. They npm-install@playwright/test@1.56.1plushttp-serverinto a throwaway.pw-runnerprefix (a rootnpm installstill reads Sharepackage.jsonand dies on ts-jest/babel-jest peers) and runnpx http-serverwith cwddocs/(Playwright's webServer cwd is the config dir,tools/).pauseRenderer: true). Full-page screenshot specs still paint.index.ifc.CI runtime (this PR)
docs/+ per-shard yarnneeds:serializes yarndocs/+ skip yarn,npx http-serverindocs/Per-shard (this run): 4m52s / 4m46s / 4m51s / 5m09s. Slowest shard: container 25s, npm install 3s, tests 4m28s.
playwright-build1m40s.playwright-webifc-run3m10s.build6m37s. $0.Verified locally
tools/marketing/build.test.js(skip path)