Commit 7809756
authored
fix: give three tests distinct screenshot names, so a failure keeps its evidence (#290)
`index-without-img-cropped` was asserted by THREE tests, not the two #284
reported: the `color_distance_limit` case (:117), `crop accepts css selector`
(:129), and the `skip_area: "input"` case (:200).
A differing screenshot is written to its baseline path by design, so when one of
them fails the next re-captures the same name, matches, and leaves the file
clean. The failure's own evidence is gone before artifacts upload.
Measured on the real failing run 32740082766, artifact
`screenshots-selenium_headless-chunky_png-diffs`: the job reported a **26.06%
difference** and its uploaded `.base.png`, `.png` and the committed baseline were
all `5fccb56f4a47d41b3a8c8cdf38531da6`. Anyone opening that artifact to debug the
flake finds two identical PNGs and concludes it was spurious. That is why #285
has never been diagnosed.
All three visit `/index-without-img.html` and crop `"form"`, so they share one
rendering -- which is exactly why the collision went unnoticed for so long. Each
renamed test therefore gets a copy of that same baseline in all six
os/driver directories; nothing is re-recorded and no comparison changes.
The `color_distance_limit` test keeps the original name so its baseline history
stays continuous.
NOT verified locally, and deliberately not claimed to be: this file reports
`0 verified -- NOTHING WAS VERIFIED` on this machine, and does so on unmodified
master too, so a local green here is not evidence. `Test Drivers` is where these
baselines actually compare; this PR carries `full-ci` for that reason.1 parent 9987b43 commit 7809756
13 files changed
Lines changed: 2 additions & 2 deletions
File tree
- test
- fixtures/app/doc/screenshots
- linux
- cuprite
- selenium_chrome_headless
- selenium_headless
- macos
- cuprite
- selenium_chrome_headless
- selenium_headless
- integration
0 commit comments