Skip to content

Commit f6d4282

Browse files
test(mosaic): run screenshot compare and update in the pinned image locally and in CI
1 parent a1b2d01 commit f6d4282

3 files changed

Lines changed: 57 additions & 50 deletions

File tree

‎packages/mosaic/docs/testing.md‎

Lines changed: 25 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -119,23 +119,27 @@ The `visual` project runs `*.visual.test.tsx` in Chromium, using the feature
119119
project's Clerk and MSW setup. Tests and their fixtures live in Mosaic, beside
120120
the feature they cover. They do not import Swingset stories or fixtures.
121121

122-
Baselines come from CI, not from your machine. Fonts and Chromium draw slightly
123-
differently on each operating system, so every baseline is captured in one
124-
digest-pinned Playwright image. The `Mosaic Visual Tests` job builds the
125-
workspace on the runner, then runs the `visual` project inside that image.
126-
127-
To add or change a baseline, push the change and wait for the job. When
128-
screenshots differ, it uploads
129-
`mosaic-visual-failure-artifacts-<run-id>-<run-attempt>` with actual, expected,
130-
and diff images under `.vitest/attachments/`. Review them, then copy the actual
131-
images into place and commit them:
122+
Screenshots are generated and compared only inside one digest-pinned Playwright
123+
image, locally and in CI, because fonts and Chromium draw slightly differently
124+
on each operating system.
132125

133126
```bash
134-
pnpm --filter @clerk/mosaic test:visual:pull <run-id>
127+
pnpm --filter @clerk/mosaic test:visual
128+
pnpm --filter @clerk/mosaic test:visual:update
135129
```
136130

137-
`test:visual` runs the comparison itself. It needs a Linux x64 install of the
138-
workspace, so it only runs in CI.
131+
Both commands require Docker. Locally, they mount your checkout into the image
132+
and install Linux dependencies into Docker volumes, so your own `node_modules`
133+
is untouched and later runs reuse the install. The first command compares
134+
screenshots without changing baselines. The second deletes the Linux baselines,
135+
lets Vitest write every screenshot again, and restores the old ones if the run
136+
fails, so new screenshots are created and ones no test produces are removed.
137+
Review the changed images and commit them under each test's `__screenshots__/`
138+
directory. Failed comparisons write actual, expected, and diff images under
139+
`packages/mosaic/.vitest/attachments/`.
140+
141+
Use these package commands for screenshots. Calling Vitest directly bypasses
142+
the pinned image.
139143

140144
Add a screenshot only when it covers a distinct layout or visual state that the
141145
existing ones miss. Capture the element under test (a dialog, a popup, or the
@@ -150,10 +154,14 @@ fixture data and avatar fallbacks, and wait for content before capturing the
150154
page so portaled menus and overlays are included.
151155

152156
Baselines include the browser and operating system in their filenames. The
153-
job's comparison is advisory: screenshot differences appear as a warning and in
154-
the job summary without blocking the rest of CI. A failed build or a comparison
155-
that crashes before producing images still fails the job. Rerun the job after
156-
pulling baselines to confirm it reports no differences.
157+
`Mosaic Visual Tests` job builds the workspace on the runner, then compares in
158+
the same pinned image. Its comparison is advisory: screenshot differences
159+
appear as a warning and in the job summary without blocking the rest of CI. A
160+
failed build, or a comparison that crashes before writing images, still fails
161+
the job. On a difference, download
162+
`mosaic-visual-failure-artifacts-<run-id>-<run-attempt>` from the run and review
163+
the images in `.vitest/attachments/`, then run `test:visual:update` locally and
164+
commit the approved baselines.
157165

158166
## Feature tests
159167

‎packages/mosaic/package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@
6363
"test:ci": "vitest run --project primitives --project mosaic --maxWorkers=70%",
6464
"test:feature": "vitest run --project feature",
6565
"test:visual": "bash ./scripts/test-visual.sh compare",
66-
"test:visual:pull": "bash ./scripts/test-visual.sh pull",
66+
"test:visual:update": "bash ./scripts/test-visual.sh update",
6767
"test:webkit": "vitest run --project webkit",
6868
"typecheck": "tsc --noEmit"
6969
},

‎packages/mosaic/scripts/test-visual.sh‎

Lines changed: 31 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -3,41 +3,40 @@ set -euo pipefail
33

44
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)
55
image=mcr.microsoft.com/playwright:v1.62.1-noble@sha256:dcc5531e97840b9b5e794f2814476b21571c5124a3fca2267d73041f56e7580e
6+
vitest='node /repo/node_modules/vitest/vitest.mjs run --project visual'
67

78
case "${1:-}" in
8-
compare)
9-
if [[ "$(uname -s)-$(uname -m)" != Linux-x86_64 ]]; then
10-
printf 'compare needs a Linux x64 install of the workspace; it runs in CI.\n' >&2
11-
exit 2
12-
fi
13-
rm -rf "$root/packages/mosaic/.vitest/attachments"
14-
docker run --rm --init --ipc=host --user "$(id -u):$(id -g)" --env HOME=/tmp \
15-
--volume "$root:/repo" --workdir /repo/packages/mosaic \
16-
"$image" node /repo/node_modules/vitest/vitest.mjs run --project visual --update none
17-
;;
18-
pull)
19-
run="${2:?Usage: $0 pull <run-id>}"
20-
download=$(mktemp -d)
21-
trap 'rm -rf "$download"' EXIT
22-
gh run download "$run" --pattern 'mosaic-visual-failure-artifacts-*' --dir "$download"
23-
count=0
24-
while IFS= read -r actual; do
25-
relative=${actual#*/.vitest/attachments/}
26-
test_file=$(dirname "$relative")
27-
name=$(basename "$relative")
28-
baseline="$root/packages/mosaic/$(dirname "$test_file")/__screenshots__/$(basename "$test_file")/${name/-actual-/-}"
29-
mkdir -p "$(dirname "$baseline")"
30-
cp "$actual" "$baseline"
31-
printf 'updated %s\n' "${baseline#"$root"/}"
32-
count=$((count + 1))
33-
done < <(find "$download" -path '*/.vitest/attachments/*' -name '*-actual-*.png')
34-
if [[ "$count" -eq 0 ]]; then
35-
printf 'Run %s has no changed screenshots.\n' "$run" >&2
36-
exit 1
37-
fi
38-
;;
9+
compare) update=none ;;
10+
update) update=all ;;
3911
*)
40-
printf 'Usage: %s compare | pull <run-id>\n' "$0" >&2
12+
printf 'Usage: %s compare|update\n' "$0" >&2
4113
exit 2
4214
;;
4315
esac
16+
17+
run=(docker run --rm --init --ipc=host --platform linux/amd64 --volume "$root:/repo" --workdir /repo/packages/mosaic)
18+
19+
if [[ "${CI:-}" == true ]]; then
20+
run+=(--user "$(id -u):$(id -g)" --env HOME=/tmp "$image" sh -c "$vitest --update $update")
21+
else
22+
run+=(--volume clerk-mosaic-visual-store:/pnpm-store --volume clerk-mosaic-visual-modules:/repo/node_modules)
23+
for package in "$root"/packages/*/ "$root"/packages/electron-passkeys/npm/*/; do
24+
relative=${package#"$root"/}
25+
run+=(--volume "clerk-mosaic-visual-${relative//\//-}modules:/repo/${relative}node_modules")
26+
done
27+
run+=("$image" sh -c "corepack enable && pnpm install --frozen-lockfile --store-dir /pnpm-store && pnpm turbo build --filter=@clerk/mosaic... && $vitest --update $update")
28+
fi
29+
30+
baselines=$(find "$root/packages/mosaic/src" -path '*/__screenshots__/*-chromium-linux.png')
31+
rm -rf "$root/packages/mosaic/.vitest/attachments"
32+
if [[ "$update" == all && -n "$baselines" ]]; then
33+
printf '%s\n' "$baselines" | xargs rm
34+
fi
35+
36+
status=0
37+
"${run[@]}" || status=$?
38+
39+
if [[ "$update" == all && "$status" -ne 0 ]]; then
40+
git -C "$root" checkout -- ':(glob)packages/mosaic/src/**/__screenshots__/**' 2>/dev/null || true
41+
fi
42+
exit "$status"

0 commit comments

Comments
 (0)