Skip to content

Commit 1664d46

Browse files
runningcodeclaude
andauthored
test(android): Add SentryAndroid.init timing to macrobenchmark (JAVA-685) (#5903)
Read the SentryAndroid.init android.os.Trace section via a TraceSectionMetric so the cold-start macrobenchmark reports SDK-init duration on its own, separate from timeToInitialDisplay. This resolves init-cost changes that the whole-start metric would bury in cold-start noise. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 79a3290 commit 1664d46

2 files changed

Lines changed: 27 additions & 15 deletions

File tree

‎sentry-android-integration-tests/sentry-uitest-android-macrobenchmark/README.md‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,17 @@ performance changes on a real device in a **stable, reproducible** way. Not run
55

66
## What it measures
77

8-
`SentryStartupBenchmark` runs a cold start and reports **`timeToInitialDisplay`**
9-
(`StartupTimingMetric`) per iteration — the whole app cold start, taken from framework trace
10-
events. No trace markers are required in the SDK or the app.
11-
12-
The flip side of marker-free measurement: an SDK change has to be large enough (roughly tens of
13-
milliseconds) to show above cold-start noise. Sub-millisecond changes are not resolvable with
14-
`timeToInitialDisplay` alone; for those, capture a perfetto trace and inspect the relevant slices
15-
directly (each iteration's trace is saved under
8+
`SentryStartupBenchmark` runs a cold start and reports two metrics per iteration:
9+
10+
- **`timeToInitialDisplay`** (`StartupTimingMetric`) — the whole app cold start, taken from
11+
framework trace events. Because it captures the entire start, an SDK change has to be large enough
12+
(roughly tens of milliseconds) to show above cold-start noise.
13+
- **`SentryAndroid.init`** (`TraceSectionMetric`) — the duration of the `SentryAndroid.init`
14+
`android.os.Trace` section the SDK emits, which isolates SDK-init cost from the rest of the start
15+
and resolves changes that `timeToInitialDisplay` would lose in the noise.
16+
17+
For even finer detail (sub-millisecond changes, or cost inside init), capture a perfetto trace and
18+
inspect the relevant slices directly (each iteration's trace is saved under
1619
`build/outputs/connected_android_test_additional_output/`).
1720

1821
`CompilationMode.Full()` pins ART AOT so dexopt state can't drift between runs. `StartupMode.COLD`
@@ -49,5 +52,6 @@ Results print to the console and are written to
4952
Macrobenchmark measures one build per run, so compare separate runs — but **interleave them**:
5053
running all of variant A followed by all of variant B lets thermal drift systematically penalize
5154
whichever variant runs second. Instead, alternate A/B rounds (build variant A, run, build variant
52-
B, run, repeat 2–3 times), keep each round's `*-benchmarkData.json`, and compare the
53-
`timeToInitialDisplay` values pooled per variant.
55+
B, run, repeat 2–3 times), keep each round's `*-benchmarkData.json`, and compare the values pooled
56+
per variant. Prefer the `SentryAndroid.init` metric for SDK-init changes — it isolates init cost, so
57+
it moves on changes that `timeToInitialDisplay` would bury in cold-start noise.

‎sentry-android-integration-tests/sentry-uitest-android-macrobenchmark/src/main/java/io/sentry/uitest/android/macrobenchmark/SentryStartupBenchmark.kt‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
package io.sentry.uitest.android.macrobenchmark
22

33
import androidx.benchmark.macro.CompilationMode
4+
import androidx.benchmark.macro.ExperimentalMetricApi
45
import androidx.benchmark.macro.StartupMode
56
import androidx.benchmark.macro.StartupTimingMetric
7+
import androidx.benchmark.macro.TraceSectionMetric
68
import androidx.benchmark.macro.junit4.MacrobenchmarkRule
79
import androidx.test.ext.junit.runners.AndroidJUnit4
810
import org.junit.Rule
@@ -13,17 +15,20 @@ import org.junit.runner.RunWith
1315
* Cold-start benchmark for the sentry-samples-android app, used to evaluate SDK-init changes on a
1416
* real device in a stable, repeatable way.
1517
*
16-
* Reports timeToInitialDisplay ([StartupTimingMetric]) per iteration. This measures the whole app
17-
* cold start from framework trace events, with no trace markers in the SDK or the app — which also
18-
* means SDK changes need to be large enough (roughly tens of milliseconds) to show above cold-start
19-
* noise.
18+
* Reports two metrics per iteration:
19+
* - timeToInitialDisplay ([StartupTimingMetric]) — the whole app cold start from framework trace
20+
* events. Because it captures the entire start, an SDK change has to be large enough (roughly
21+
* tens of milliseconds) to show above cold-start noise.
22+
* - SentryAndroid.init ([TraceSectionMetric]) — the duration of the `SentryAndroid.init`
23+
* [android.os.Trace] section the SDK emits, isolating SDK-init cost from the rest of the start.
2024
*
2125
* [CompilationMode.Full] pins ART AOT compilation so dexopt state does not drift between runs.
2226
* Iterations are capped at 12: on an unthrottled Pixel 3, back-to-back cold starts hit thermal
2327
* throttling after ~14 iterations, which inflates the tail of longer runs. This is NOT a CI test;
2428
* it requires a connected device. To A/B an SDK change, see README.md (build the app twice, once
2529
* per SDK variant, in interleaved rounds).
2630
*/
31+
@OptIn(ExperimentalMetricApi::class)
2732
@RunWith(AndroidJUnit4::class)
2833
class SentryStartupBenchmark {
2934

@@ -33,7 +38,7 @@ class SentryStartupBenchmark {
3338
fun startupFullCompilation() =
3439
benchmarkRule.measureRepeated(
3540
packageName = TARGET_PACKAGE,
36-
metrics = listOf(StartupTimingMetric()),
41+
metrics = listOf(StartupTimingMetric(), TraceSectionMetric(INIT_TRACE_SECTION)),
3742
compilationMode = CompilationMode.Full(),
3843
startupMode = StartupMode.COLD,
3944
iterations = 12,
@@ -44,5 +49,8 @@ class SentryStartupBenchmark {
4449

4550
private companion object {
4651
const val TARGET_PACKAGE = "io.sentry.samples.android"
52+
53+
// Matches the android.os.Trace section name in SentryAndroid.init.
54+
const val INIT_TRACE_SECTION = "SentryAndroid.init"
4755
}
4856
}

0 commit comments

Comments
 (0)