Skip to content

Commit 18cc528

Browse files
committed
docs(macrobenchmark): Record how Sauce results are retrieved
Document the three routes that do not work -- artifacts.download.match, Macrobenchmark's own reporting channels, and the Real Device Access API on a public-cloud account -- so the next person does not re-derive them.
1 parent e5a6795 commit 18cc528

1 file changed

Lines changed: 28 additions & 0 deletions

File tree

  • sentry-android-integration-tests/sentry-uitest-android-macrobenchmark

‎sentry-android-integration-tests/sentry-uitest-android-macrobenchmark/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,34 @@ Connect a device, then:
3131
Results print to the console and are written to
3232
`build/outputs/connected_android_test_additional_output/.../*-benchmarkData.json`.
3333

34+
## Running on Sauce Labs
35+
36+
The `Integration Tests - Macrobenchmark` workflow (manual trigger) runs the same benchmark on a
37+
Sauce Labs real device. It reports numbers only — it is not a PR gate, because cloud devices run
38+
with unlocked CPU clocks and the run-to-run spread swamps most SDK-init changes.
39+
40+
Getting the numbers *back off* the device is the awkward part, so if you are changing this, know
41+
what has already been ruled out:
42+
43+
- **`artifacts.download.match` in `.sauce/*.yml` cannot reach the device.** It filters a
44+
hardcoded list of assets Sauce hosts for the job (`device.log`, `junit.xml`, `video.mp4`,
45+
`network.har`, `crash.json`, `screenshots.zip`); nothing enumerates device storage. A
46+
`*-benchmarkData.json` pattern there matches nothing.
47+
- **Macrobenchmark's own reporting channels don't survive.** The readable summary goes into the
48+
instrumentation status bundle, which only Studio and AGP consume, and `benchmarkData.json` is
49+
written to the app's external media dir, which Sauce never pulls.
50+
- **The Real Device Access API can pull device files, but not on our account.** It offers
51+
`pullFile` and `executeShellCommand`, and `GET /rdc/v2/devices/status` even lists the whole
52+
public fleet — but `POST /sessions` answers `deviceClasses=[PRIVATE_DEVICE]` and there is no
53+
parameter to request a public device. It would need leased private devices. That route would
54+
also return the per-iteration perfetto traces, so it is worth revisiting if we ever get them.
55+
56+
So `SentryStartupBenchmark` echoes its own `benchmarkData.json` into logcat in chunks, which
57+
reaches CI inside `device.log`, and `scripts/parse-macrobenchmark-log.py` reassembles it and
58+
writes a `timeToInitialDisplay` table to the job summary. Note this recovers the metrics only —
59+
the perfetto traces are megabytes each and cannot go through logcat, so sub-millisecond work
60+
still needs a local device.
61+
3462
### Device hygiene (do this for trustworthy numbers)
3563

3664
- **Wake and unlock the device first** — the launch check fails with "Unable to confirm activity

0 commit comments

Comments
 (0)