Skip to content

Commit 4f26d59

Browse files
committed
ci(macrobenchmark): Recover results through logcat, not the device API
The Real Device Access API can pull files off a device, but it cannot reserve one on a public-cloud account: POST /sessions rejects every request with deviceClasses=[PRIVATE_DEVICE], and the request body has no way to ask for a public device. Everything up to reservation worked -- App Storage upload, device resolution from /devices/status, the session request itself -- so this is a commercial limit, not a technical one. Revert to saucectl and drop the script; git history has it if we ever lease private devices, since that route would also return the per-iteration perfetto traces. Instead have the benchmark echo its own benchmarkData.json into logcat in chunks, which reaches CI as device.log, the one asset Sauce does hand back. scripts/parse-macrobenchmark-log.py reassembles it, validates the JSON, and writes a timeToInitialDisplay table to the job summary.
1 parent 6285430 commit 4f26d59

4 files changed

Lines changed: 197 additions & 447 deletions

File tree

‎.github/workflows/integration-tests-macrobenchmark.yml‎

Lines changed: 24 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,18 @@
11
name: 'Integration Tests - Macrobenchmark'
22
# Runs the sentry-uitest-android-macrobenchmark cold-start benchmark on a Sauce Labs real
3-
# device via the Real Device Access API, and pulls benchmarkData.json plus the per-iteration
4-
# perfetto traces off the device. saucectl cannot do the pulling: it only downloads assets
5-
# Sauce hosts for a job, never files from the device. See scripts/macrobenchmark-sauce.py.
3+
# device and recovers timeToInitialDisplay from the device log.
4+
#
5+
# Getting the numbers back is the awkward part. saucectl only downloads assets Sauce hosts
6+
# for a job (device.log, junit.xml, video.mp4, ...) and never reads the device filesystem,
7+
# so Macrobenchmark's benchmarkData.json cannot be fetched directly. The Real Device Access
8+
# API can pull device files, but POST /sessions accepts only private devices
9+
# (deviceClasses=[PRIVATE_DEVICE]) and we run on the public cloud. So the benchmark echoes
10+
# its own results into logcat, and scripts/parse-macrobenchmark-log.py reassembles them.
611
#
712
# Manual trigger only -- this reports numbers, it does not gate PRs. Cloud devices have
813
# unlocked CPU clocks, so run-to-run spread is far wider than most SDK-init changes.
914
on:
1015
workflow_dispatch:
11-
inputs:
12-
mode:
13-
description: 'smoke = skip AOT compilation (faster, validates the plumbing); full = the benchmark as declared'
14-
type: choice
15-
options:
16-
- smoke
17-
- full
18-
default: smoke
19-
device-name:
20-
description: 'Sauce device id or regex'
21-
required: false
22-
default: 'Google_Pixel_9_Pro_XL_15_real_sjc1'
2316
# Temporary scaffolding: workflow_dispatch cannot target a workflow that does not exist on
2417
# the default branch yet, so trigger on pushes to this branch while we validate the flow.
2518
# Remove before merge.
@@ -39,27 +32,14 @@ jobs:
3932
# we copy the secret to the env variable in order to access it in the workflow
4033
env:
4134
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
42-
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
4335
GRADLE_ENCRYPTION_KEY: ${{ secrets.GRADLE_ENCRYPTION_KEY }}
44-
# Defaults apply to the push trigger too, where `inputs` is empty.
45-
MODE: ${{ inputs.mode || 'smoke' }}
46-
DEVICE_NAME: ${{ inputs.device-name || 'Google_Pixel_9_Pro_XL_15_real_sjc1' }}
4736

4837
steps:
4938
- name: Git checkout
5039
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
5140
with:
5241
submodules: 'recursive'
5342

54-
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
55-
with:
56-
python-version: '3.10.5'
57-
58-
- name: Install Python dependencies
59-
run: |
60-
python3 -m pip install --upgrade pip
61-
python3 -m pip install -r requirements.txt
62-
6343
- name: 'Set up Java: 17'
6444
uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5
6545
with:
@@ -71,32 +51,29 @@ jobs:
7151
with:
7252
cache-encryption-key: ${{ secrets.GRADLE_ENCRYPTION_KEY }}
7353

74-
# Probe first: the Real Device Access API is an open-beta add-on, so a missing
75-
# entitlement should fail in seconds rather than after a full Gradle build.
76-
- name: Check Real Device Access API entitlement
77-
if: env.SAUCE_USERNAME != null
78-
run: python3 scripts/macrobenchmark-sauce.py --probe-only --device-name "$DEVICE_NAME"
79-
8054
- name: Assemble target app and Macrobenchmark apk
8155
if: env.SAUCE_USERNAME != null
8256
run: ./gradlew :sentry-samples:sentry-samples-android:assembleRelease :sentry-android-integration-tests:sentry-uitest-android-macrobenchmark:assembleBenchmark
8357

84-
- name: Run Macrobenchmark on Sauce Labs
58+
- name: Run Macrobenchmark in SauceLab
59+
uses: saucelabs/saucectl-run-action@283660aa934c02723c497efa151d582a3acc5801 # pin@v3
8560
if: env.SAUCE_USERNAME != null
61+
env:
62+
GITHUB_TOKEN: ${{ github.token }}
63+
with:
64+
sauce-username: ${{ secrets.SAUCE_USERNAME }}
65+
sauce-access-key: ${{ secrets.SAUCE_ACCESS_KEY }}
66+
config-file: .sauce/sentry-uitest-android-macrobenchmark.yml
67+
68+
# Runs even when the suite fails: a failed benchmark still logs whatever it managed to
69+
# measure, and the parser's own error explains what was missing.
70+
- name: Recover benchmark results from the device log
71+
if: always() && env.SAUCE_USERNAME != null
8672
run: |
87-
if [ "$MODE" = "smoke" ]; then
88-
mode_args=(--skip-compilation)
89-
else
90-
mode_args=()
91-
fi
92-
python3 scripts/macrobenchmark-sauce.py \
93-
--app sentry-samples/sentry-samples-android/build/outputs/apk/release/sentry-samples-android-release.apk \
94-
--test-app sentry-android-integration-tests/sentry-uitest-android-macrobenchmark/build/outputs/apk/benchmark/sentry-uitest-android-macrobenchmark-benchmark.apk \
95-
--device-name "$DEVICE_NAME" \
96-
--out-dir ./artifacts/macrobenchmark \
97-
"${mode_args[@]}"
73+
python3 scripts/parse-macrobenchmark-log.py ./artifacts \
74+
--json-out ./artifacts/benchmarkData.json | tee -a "$GITHUB_STEP_SUMMARY"
9875
99-
- name: Upload benchmark results
76+
- name: Upload Sauce artifacts
10077
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
10178
if: always()
10279
with:

0 commit comments

Comments
 (0)