Skip to content

chore(ci): Run UI tests on local emulator instead of SauceLabs - #5639

Closed
romtsn wants to merge 3 commits into
mainfrom
rz/chore/ignore-ui-test-timeout
Closed

romtsn wants to merge 3 commits into
mainfrom
rz/chore/ignore-ui-test-timeout

Conversation

@romtsn

@romtsn romtsn commented Jun 25, 2026 •

Copy link
Copy Markdown
Contributor

The Ui tests workflow now runs sentry-uitest-android on a local API 35 x86_64 emulator on the GitHub runner, instead of on SauceLabs. The tests run through Gradle connectedReleaseAndroidTest (with -Denvironment=github), and the AVD snapshot is cached between runs to make emulator boot fast.

This removes the SauceLabs infrastructure failures (for example "Suite timed out" and "Process crashed") that blocked PRs, and also removes the Verify Test Results log-grep workaround. The job now fails only when a test fails. It also runs on fork PRs, because it does not need SAUCE_USERNAME/SAUCE_ACCESS_KEY anymore.

Review focus: this also removes the step that uploads replay snapshots to Sentry (sentry-cli build snapshots), because that step read the PNGs from the SauceLabs artifacts. If we still need the snapshot upload, we must add it again from the emulator output in a follow-up.

🤖 Generated with Claude Code

@sentry

sentry Bot commented Jun 25, 2026 •

Copy link
Copy Markdown

📲 Install Builds

Android

🔗 App Name App ID Version Configuration
SDK Size io.sentry.tests.size 8.63.0 (1) release

⚙️ sentry-android Build Distribution Settings

@runningcode

Copy link
Copy Markdown
Contributor

If these don't block merging, wouldn't this just hide the fact that these are happening more often?

Otherwise a PR might go green because this failure was ignored rather than rerun because it could have been a real failure.

@github-actions

github-actions Bot commented Jun 25, 2026 •

Copy link
Copy Markdown
Contributor

Performance metrics 🚀

  Plain With Sentry Diff
Startup time 354.34 ms 416.46 ms 62.12 ms
Size 0 B 0 B 0 B

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
18c0bc2 306.73 ms 349.77 ms 43.03 ms
0eaac1e 316.82 ms 357.34 ms 40.52 ms
d15471f 303.49 ms 439.08 ms 135.59 ms
fc5ccaf 276.52 ms 370.46 ms 93.93 ms
e2dce0b 308.96 ms 360.10 ms 51.14 ms
5b1a06b 352.27 ms 413.70 ms 61.43 ms
37ec571 366.04 ms 424.28 ms 58.23 ms
9fbb112 361.43 ms 427.57 ms 66.14 ms
bbc35bb 324.88 ms 425.73 ms 100.85 ms
ff8eea4 313.42 ms 337.08 ms 23.66 ms

App size

Revision Plain With Sentry Diff
18c0bc2 1.58 MiB 2.13 MiB 557.33 KiB
0eaac1e 1.58 MiB 2.19 MiB 619.17 KiB
d15471f 1.58 MiB 2.13 MiB 559.54 KiB
fc5ccaf 1.58 MiB 2.13 MiB 557.54 KiB
e2dce0b 0 B 0 B 0 B
5b1a06b 0 B 0 B 0 B
37ec571 0 B 0 B 0 B
9fbb112 1.58 MiB 2.11 MiB 539.18 KiB
bbc35bb 1.58 MiB 2.12 MiB 553.01 KiB
ff8eea4 1.58 MiB 2.28 MiB 718.64 KiB

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@romtsn romtsn changed the title chore(ci): ignore UI test suite timeout failures chore(ci): Run UI tests on local emulator instead of SauceLabs Nov 6, 2026
romtsn and others added 2 commits November 6, 2026 09:25
SauceLabs suite timeouts are infrastructure-level failures, not test
failures. Treat them the same as "Process crashed" — exit 0 so they
don't block PRs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SauceLabs introduces flaky timeouts and infrastructure failures.
Switch to GH Actions emulator (same setup as integration-tests-ui-critical.yml).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@romtsn
romtsn force-pushed the rz/chore/ignore-ui-test-timeout branch from ffef220 to b6648bd Compare November 6, 2026 08:26
@romtsn
romtsn requested a review from a team as a code owner November 6, 2026 08:26

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b6648bd. Configure here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants