Repository navigation
Commit 6a1c02c
ci(macrobenchmark): Run the startup benchmark on Sauce Labs and recover its results (#5909)
* ci(macrobenchmark): Run startup benchmark on Sauce Labs (POC)
The sentry-uitest-android-macrobenchmark module currently only runs on a
locally connected device. This wires it to Sauce Labs so we can evaluate
whether the cold-start timeToInitialDisplay benchmark can run on the
real-device cloud already used by our other benchmarks.
This is a proof-of-concept, gated behind workflow_dispatch and kept off the
per-PR path. Device-guard errors are intentionally not suppressed: on
non-rooted, unlocked-clock cloud devices the guards (UNLOCKED, DEBUGGABLE,
...) are expected to fire, and seeing which ones fire is the point of the
POC. It also tells us whether timeToInitialDisplay can be retrieved from
Sauce artifacts (benchmark JSON) or must be parsed from the device log.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci(macrobenchmark): Call gradle directly instead of a Makefile target
Drop the assembleMacrobenchmark Makefile target and invoke the two gradle
assemble tasks directly from the workflow. Only one workflow uses them, so
the extra indirection isn't worth it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci(macrobenchmark): Pull benchmark results via Real Device Access API
saucectl cannot retrieve Macrobenchmark's output. Its artifacts.download.match
filters a hardcoded list of assets Sauce hosts for a job -- device.log,
junit.xml, video.mp4, network.har, crash.json, screenshots.zip -- and never
reads the device filesystem, so benchmarkData.json and the per-iteration
perfetto traces stay stranded on the device. The POC's "*-benchmarkData.json"
match line could never have matched anything; drop it.
Drive the run through the Real Device Access API instead, which exposes adb
shell and pullFile: reserve a device, install both APKs, run the
instrumentation, then pull the results off. This also recovers the perfetto
traces, the only way to resolve sub-millisecond SDK changes that
timeToInitialDisplay cannot see.
The API is an open-beta add-on, so the workflow probes entitlement before
spending a Gradle build on it, and defaults to a one-iteration smoke run.
* ci(macrobenchmark): Fix app upload status and drop the iterations override
App Storage upload answers 201, not 200, so the strict status check rejected
every upload before a device was even reserved.
Also drop --iterations: androidx.benchmark.iterations only feeds the
microbenchmark path (BenchmarkStateLegacy, MicrobenchmarkPhase).
Macrobenchmark reads its iteration count from the test source and ignores the
argument, so skipping AOT compilation is the only way to shorten a run that
still writes results -- dryRunMode would shorten it but forces outputEnable to
false, producing no benchmarkData.json to retrieve.
Track installations by installationId rather than by the app reference, which
Sauce may echo back normalised.
* ci(macrobenchmark): Resolve the device from the catalog before opening a session
The first run reached the API but matched zero devices: Real Device Access
descriptors are not the ids used in .sauce/*.yml, which carry an OS version and
a region suffix (Google_Pixel_9_Pro_XL_15_real_sjc1).
Fetch the catalog unfiltered and match client-side against both descriptor id
and name, then open the session with the resolved concrete id rather than a
pattern. On no match, print every available Android device so a naming mismatch
is diagnosable from a single run -- the endpoint's own deviceId filter cannot
distinguish "no devices on this account" from "your pattern matched nothing".
Also run the no-match check before --probe-only returns. It previously came
after, so the probe reported success and the real run failed on it minutes
later.
* ci(macrobenchmark): Probe every region when the device catalog is empty
An empty catalog answers 200 exactly like a populated one, so the first probe
reported the API as reachable while there was in fact nothing to run on. Report
device availability and the other regions' catalogs too, so an unentitled
account is distinguishable from a misconfigured region.
* ci(macrobenchmark): Resolve devices from /devices/status, not /devices
On a public-cloud account /devices answers 200 with an empty list, which made
the probe look like a missing entitlement. /devices/status returns the whole
fleet -- 392 devices, all public, 368 available -- and its descriptors are
exactly the ids already used in .sauce/*.yml, so the original
Google_Pixel_9_Pro_XL_15_real_sjc1 is back as the default.
Prefer AVAILABLE devices when a pattern matches several, and report each
match's state so an IN_USE device explains itself.
* ci(macrobenchmark): Recover results through logcat, not the device API
The Real Device Access API can pull files off a device, but it cannot reserve
one on a public-cloud account: POST /sessions rejects every request with
deviceClasses=[PRIVATE_DEVICE], and the request body has no way to ask for a
public device. Everything up to reservation worked -- App Storage upload, device
resolution from /devices/status, the session request itself -- so this is a
commercial limit, not a technical one. Revert to saucectl and drop the script;
git history has it if we ever lease private devices, since that route would also
return the per-iteration perfetto traces.
Instead have the benchmark echo its own benchmarkData.json into logcat in
chunks, which reaches CI as device.log, the one asset Sauce does hand back.
scripts/parse-macrobenchmark-log.py reassembles it, validates the JSON, and
writes a timeToInitialDisplay table to the job summary.
* ci(macrobenchmark): Decode Sauce's JSON-lines device log
The recovery worked on the device but not in the parser. Sauce returns
device.log as JSON lines -- {"tag", "message", "level", ...} -- so the payload
arrives with its quotes escaped and cannot be regexed straight out of the raw
line. Decode each entry and match against its message, falling back to the raw
line so the same parser still handles `adb logcat` output from a local run.
Enable pipefail in the workflow step too. Piping into tee meant the step exited
on tee's status, so this very failure reported success while producing no
results.
Verified against the device.log from run 31117974318: timeToInitialDisplayMs
min 443.5 / median 477.3 / max 571.2 over 12 iterations on a Pixel 9 Pro XL.
* docs(macrobenchmark): Record how Sauce results are retrieved
Document the three routes that do not work -- artifacts.download.match,
Macrobenchmark's own reporting channels, and the Real Device Access API on a
public-cloud account -- so the next person does not re-derive them.
* refactor(macrobenchmark): Collect log chunks per file
Chunks were merged across every *.log found, so two devices each emitting the
same number of chunks would overwrite each other's indices and reassemble into
one bogus document. Collect per file and report when more than one log carries
results, which also drops the cross-file total comparison that was standing in
for this check.
Build the summary table and per-iteration details in one pass while here.
* ci(macrobenchmark): Drop unused submodule checkout and encryption key env
The repo has no submodules -- no .gitmodules, no gitlink entries -- so
submodules: 'recursive' does nothing; sentry-native-ndk is a Maven dependency
pinned in gradle/libs.versions.toml. GRADLE_ENCRYPTION_KEY is never read as
env.* or $GRADLE_ENCRYPTION_KEY either; setup-gradle takes the secret directly
via cache-encryption-key, which stays.
* ci(macrobenchmark): Record why the benchmark runs on a high-end device
Older hardware would resolve SDK-init regressions better, but
Google_Pixel_4_13_real_us could not be allocated in three attempts -- each
queued ~61 minutes before Sauce gave up with a concurrency retry-timeout,
leaving an empty device log and tests="0". The Pixel 9 Pro XL allocates
reliably and is the device this pipeline is proven green on.
* ref(macrobenchmark): Narrow the deprecation suppression to the deprecated call
@Suppress("DEPRECATION") covered the whole candidateDirs expression, including
externalCacheDir, which is not deprecated -- so a future deprecation there would
have been silently swallowed. Scope it to getExternalMediaDirs and record why it
cannot be migrated: it is deprecated in favour of MediaStore, which returns
content URIs rather than the filesystem path androidx.benchmark writes to.
* ref(macrobenchmark): Trim comments and drop the cache encryption key
The retrieval rationale these comments carried is documented in the module
README, so removing the duplicated prose loses nothing.
Also drops cache-encryption-key from the Setup Gradle step, leaving this
workflow without the Gradle configuration-cache encryption the other workflows
still configure.
* ci(macrobenchmark): Remove the temporary push trigger (JAVA-679)
The push trigger was scaffolding: workflow_dispatch cannot target a
workflow that is absent from the default branch, so the workflow had to
fire on pushes to the development branch to be exercised at all. Once
this merges that constraint is gone, and leaving the trigger in place
would start a Sauce Labs run on every push to a same-named branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci(macrobenchmark): Skip result steps on cancellation (JAVA-679)
always() keeps a step running even when the job is cancelled, so a run
cancelled by the concurrency group would still grind through result
recovery and artifact upload instead of stopping promptly.
!cancelled() keeps the intended behaviour, running on both success and
failure so a failed benchmark still reports what it measured, while
letting a cancellation take effect. This matches integration-tests-ui,
which already gates its post-Sauce steps the same way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>1 parent 9a425e5 commit 6a1c02c
5 files changed
Lines changed: 326 additions & 7 deletions
File tree
- .github/workflows
- .sauce
- scripts
- sentry-android-integration-tests/sentry-uitest-android-macrobenchmark
- src/main/java/io/sentry/uitest/android/macrobenchmark
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
37 | 65 | | |
38 | 66 | | |
39 | 67 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
10 | 14 | | |
11 | 15 | | |
12 | 16 | | |
| |||
23 | 27 | | |
24 | 28 | | |
25 | 29 | | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
51 | | - | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
52 | 53 | | |
53 | 54 | | |
54 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
55 | 104 | | |
56 | 105 | | |
0 commit comments