Skip to content

Commit 4d15f0e

Browse files
markushimarkushi
authored
docs: Add develop-docs with Perfetto profiling documentation (#5750)
1 parent d08313e commit 4d15f0e

3 files changed

Lines changed: 502 additions & 0 deletions

File tree

‎develop-docs/README.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Develop Docs
2+
3+
This folder holds internal developer documentation for the Sentry Java/Android SDK:
4+
architecture notes, feature deep-dives, design decisions, and cross-module concepts
5+
that don't belong in the public [Sentry docs](https://docs.sentry.io) or in inline
6+
code comments.
7+
8+
If you are documenting **how** or **why** something works for the people who maintain
9+
this SDK, it goes here. If you are documenting **how to use** the SDK for end users,
10+
it belongs in the public docs instead.
11+
12+
## Rules
13+
14+
These rules keep the docs consistent, easy to navigate, and easy to grep.
15+
16+
### Directory structure
17+
18+
Documents live in **subdirectories**, one level per level of grouping. Directories are
19+
cheap: reach for a new one as soon as a topic has more than one document, or as soon as you
20+
can name the group.
21+
22+
Every document sits under one of these top-level categories:
23+
24+
- `general/` — cross-cutting topics (e.g. `general/architecture.md`, `general/pipeline.md`)
25+
- `feature/` — a specific SDK feature (e.g. `feature/errors/`, `feature/profiling/`)
26+
- `integration/` — a specific integration or module (e.g. `integration/opentelemetry/`, `integration/spring/`)
27+
- `platform/` — platform-specific concerns (e.g. `platform/android/`, `platform/jvm/`)
28+
- `process/` — team processes and workflows (e.g. `process/release.md`)
29+
30+
Add a new category only when an existing one clearly does not fit, and keep the list above
31+
up to date.
32+
33+
Below the category, nest by topic and then by sub-topic. A fully grown feature might look
34+
like this:
35+
36+
```text
37+
develop-docs/
38+
README.md
39+
general/
40+
pipeline.md
41+
feature/
42+
profiling/
43+
overview.md
44+
perfetto.md
45+
anr.md
46+
symbolication/
47+
deobfuscation.md
48+
```
49+
50+
- Give a directory an `overview.md` once it holds several documents, and link to its
51+
siblings from there.
52+
- Do not create a directory that will only ever hold one document — put the document
53+
directly in the category (`general/pipeline.md`, not `general/pipeline/pipeline.md`).
54+
55+
### File naming
56+
57+
- File names are **lowercase**, except for this `README.md`, which GitHub renders as the
58+
folder's landing page.
59+
- Use **dashes** (`-`) as separators, never underscores or spaces. For example, use
60+
`session-replay.md`, not `session_replay.md` or `Session Replay.md`.
61+
- Use the `.md` extension for all text documents.
62+
- **Do not repeat the path in the file name.** The directories carry the namespace, so the
63+
file name only needs the part that distinguishes it from its siblings:
64+
`feature/profiling/perfetto.md`, not `feature/profiling/perfetto-profiling.md`.
65+
- Choose short, descriptive names (`feature/replay/masking.md`, not
66+
`feature/replay/how-masking-works.md`).
67+
68+
### Images and other assets
69+
70+
- When a document embeds images (or other binary assets), store them in an **`assets/`
71+
folder next to the document**. Documents in the same directory share it:
72+
73+
```text
74+
develop-docs/
75+
feature/
76+
profiling/
77+
perfetto.md
78+
assets/
79+
pipeline.png
80+
overview.svg
81+
```
82+
83+
- Reference assets with **relative paths**: `![Profiling pipeline](assets/pipeline.png)`.
84+
- Asset file names follow the same rules as documents: lowercase, dashes, descriptive.
85+
- Prefer **vector formats** (SVG) for diagrams and screenshots where practical
86+
- Prefer **Mermaid** over a static image whenever a diagram can be expressed as one
87+
(see below) — it lives in the document, is versioned as text, and is easy to update.
88+
89+
### Writing style
90+
91+
- Write in the **present tense** and the **active voice**. Describe how the system
92+
behaves now ("The transport retries failed envelopes"), not how it will or did behave.
93+
This way there's no need to update the docs once a feature ships.
94+
- Keep one **top-level `# ` heading** per document (the title), and nest sections with
95+
`##`, `###`, etc. Do not skip heading levels.
96+
- Keep documents focused on a **single topic**. Split large topics into several documents
97+
in a shared directory and link between them rather than growing one giant file.
98+
- Use fenced **code blocks with a language identifier** (```kotlin `,
99+
` ```bash `) so syntax highlighting works.
100+
- Prefer Kotlin snippets over Java.
101+
- When referencing code, link to the file with a **relative path** (e.g.
102+
`../../../sentry/src/main/java/io/sentry/Sentry.java`) rather than pasting large excerpts
103+
that fall out of date. Count the `../` from the document's own directory.
104+
- Avoid pinning content to a specific SDK version or date unless it is genuinely
105+
version-specific; keep docs evergreen.
106+
- Cross-link related documents with relative links (e.g.
107+
`[the ingestion pipeline](../../general/pipeline.md)`).
108+
109+
### Structuring a feature document
110+
111+
Most feature documents answer the same four questions, and following that order makes them
112+
easier to compare and to keep current:
113+
114+
1. **Surface area** — where and when the SDK collects the data.
115+
2. **Collection** — how the SDK collects it.
116+
3. **Format** — what the collected data looks like on the wire.
117+
4. **Pipeline** — how the backend ingests, stores, and serves it.
118+
119+
Do not restate (4) in every document. Describe the shared path once in
120+
[general/pipeline.md](general/pipeline.md) and cover only the deviations a feature
121+
introduces. Omit any of the four that a feature does not have, and keep each as high-level
122+
as the topic allows so the document stays true for longer.
123+
124+
### Diagrams with Mermaid
125+
126+
- Prefer [Mermaid](https://mermaid.js.org/) for diagrams. It renders directly on GitHub
127+
and lives in the document as text, so it versions and reviews like code.
128+
- Embed a Mermaid diagram in a fenced block tagged `mermaid`:
129+
130+
````markdown
131+
```mermaid
132+
flowchart LR
133+
Event[SentryEvent] --> Processor[EventProcessors]
134+
Processor --> Transport
135+
Transport --> Sentry[(Sentry)]
136+
```
137+
````
138+
139+
- For complex diagrams, include a link to the [Mermaid Live Editor](https://mermaid.live/)
140+
so reviewers can iterate quickly.
141+
- Fall back to static images (stored per the asset rules above) if mermaid is not practicable.
142+
143+
## Adding a new document
144+
145+
1. Pick the right top-level category (or introduce a new one and document it above).
146+
2. Pick or create the topic directory below it.
147+
3. Create the document, naming it for what distinguishes it from its siblings.
148+
4. If the directory now holds several documents, add or update its `overview.md`.
149+
5. If the document embeds assets, put them in an `assets/` folder next to it.

‎develop-docs/feature/profiling/perfetto.md‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
# Perfetto profiling on Android
2+
3+
This document describes how continuous profiling works on Android when the SDK
4+
captures traces through the OS-level [`android.os.ProfilingManager`](https://developer.android.com/reference/android/os/ProfilingManager)
5+
API (available on API 35+), and how a captured **profile chunk** flows all the way
6+
from the device to a downloadable profile in Sentry.
7+
8+
## What Perfetto is
9+
10+
[Perfetto](https://perfetto.dev/) is Google's tracing framework for Android and Linux, and
11+
the tooling Android itself is instrumented with. Its
12+
[callstack sampler](https://perfetto.dev/docs/getting-started/cpu-profiling) interrupts the
13+
app at a fixed frequency, records the native and Java call stacks of the running threads,
14+
and writes them to a binary `.pftrace` file (a serialized
15+
[Perfetto protobuf](https://perfetto.dev/docs/reference/trace-packet-proto)).
16+
Starting with Android 15, apps can request such traces at
17+
runtime via `ProfilingManager` without root or `adb`, which is what makes on-device
18+
continuous profiling possible.
19+
20+
Useful Perfetto references:
21+
22+
- Perfetto docs: https://perfetto.dev/docs/
23+
- CPU profiling with Perfetto: https://perfetto.dev/docs/getting-started/cpu-profiling
24+
- Trace format (`TracePacket` proto): https://perfetto.dev/docs/reference/trace-packet-proto
25+
- Perfetto UI (to open a downloaded `.pftrace`): https://ui.perfetto.dev/
26+
27+
## Pipeline overview
28+
29+
Profile chunks travel the standard ingestion path described in
30+
[general/pipeline.md](../../general/pipeline.md) — SDK envelope,
31+
[Relay](https://develop.sentry.dev/ingestion/relay/) (Sentry's ingestion proxy), Kafka, a
32+
monolith processing task, then storage and a read API. Read that first; the rest of this
33+
document covers only where Perfetto deviates from it.
34+
35+
The deviations are:
36+
37+
- The envelope item carries **JSON and raw binary in one payload**, subdivided by a
38+
`meta_length` header rather than base64-encoding the trace ([details](#envelope-format-and-the-meta_length-header)).
39+
- Relay **converts** the Perfetto trace into the existing Sample v2 profile format, and
40+
additionally **keeps the raw `.pftrace`** in the object store so it can be downloaded
41+
later ([details](#relay-getsentryrelay)).
42+
43+
```mermaid
44+
flowchart TD
45+
subgraph device["Android device — sentry-java"]
46+
PM[android.os.ProfilingManager]
47+
PP[PerfettoProfiler]
48+
PCP[PerfettoContinuousProfiler]
49+
PC[ProfileChunk]
50+
ENV["Envelope item<br/>[JSON metadata][raw .pftrace]<br/>header: meta_length"]
51+
PM --> PP --> PCP --> PC --> ENV
52+
end
53+
54+
subgraph relay["Relay (processing mode)"]
55+
SPLIT[Split payload at meta_length]
56+
CONV[Convert Perfetto → Sample v2]
57+
OS1[Upload raw .pftrace to object store]
58+
KAFKA[["Kafka topic: profiles<br/>ProfileChunkKafkaMessage<br/>(Sample v2 + attachment stored_id)"]]
59+
SPLIT --> CONV --> KAFKA
60+
SPLIT --> OS1
61+
end
62+
63+
subgraph monolith["Monolith — getsentry/sentry"]
64+
TASK[process_profile_task]
65+
SYM[Symbolicate / deobfuscate]
66+
VR[vroomrs: parse + normalize]
67+
OS2[(Object store)]
68+
SNUBA[(Snuba: function metrics)]
69+
DB[(ProfileChunkAttachment row)]
70+
TASK --> SYM --> VR
71+
VR --> OS2
72+
VR --> SNUBA
73+
TASK --> DB
74+
end
75+
76+
ENV -->|envelope| relay
77+
KAFKA --> TASK
78+
OS1 -.stored_id.-> DB
79+
VROOM[getsentry/vroom<br/>serve + merge flamegraphs]
80+
OS2 --> VROOM
81+
SNUBA --> VROOM
82+
```
83+
84+
## SDK (getsentry/sentry-java)
85+
86+
On API 35+, [`AndroidOptionsInitializer`](../../../sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java)
87+
wires up `PerfettoContinuousProfiler` automatically. On older devices the SDK falls back
88+
to the legacy `Debug`-based [`AndroidContinuousProfiler`](../../../sentry-android-core/src/main/java/io/sentry/android/core/AndroidContinuousProfiler.java),
89+
gated by the `enableLegacyProfiling` option (manifest key
90+
`io.sentry.profiling.enable-legacy-profiling`, defaults to `true`). Only **continuous
91+
profiling** is supported on the Perfetto path — transaction-based and app-start profiling
92+
are not.
93+
94+
### Capturing chunks
95+
96+
Continuous profiling emits a stream of independent [`ProfileChunk`](../../../sentry/src/main/java/io/sentry/ProfileChunk.java)s
97+
rather than one profile per transaction. `PerfettoContinuousProfiler` drives a chained
98+
loop: each chunk runs for `MAX_CHUNK_DURATION_MILLIS` (60s) via `PerfettoProfiler`, which
99+
calls `ProfilingManager.requestProfiling(PROFILING_TYPE_STACK_SAMPLING, …)` at
100+
`PROFILING_FREQUENCY_HZ` (101 Hz). When a chunk's trace file is ready, a new chunk starts,
101+
so profiling runs continuously.
102+
103+
A chunk keeps a stable `profilerId` across the session and a per-chunk `chunkId`. When the
104+
OS produces the trace file, the profiler builds a `ProfileChunk` tagged with the Perfetto
105+
content type:
106+
107+
```kotlin
108+
ProfileChunk.Builder(profilerId, chunkId, measurements, traceFile, timestamp, ProfileChunk.PLATFORM_ANDROID)
109+
.setContentType(ProfileChunk.CONTENT_TYPE_PERFETTO) // "application/x-perfetto-trace"
110+
.build()
111+
```
112+
113+
The chunk is captured via `scopes.captureProfileChunk(...)` and sent as its own envelope
114+
with item type [`SentryItemType.ProfileChunk`](../../../sentry/src/main/java/io/sentry/SentryItemType.java)
115+
(wire name `profile_chunk`).
116+
117+
### Envelope format and the `meta_length` header
118+
119+
A legacy chunk base64-encodes its trace into the `ProfileChunk` JSON. A Perfetto chunk is
120+
much larger, so [`SentryClient`](../../../sentry/src/main/java/io/sentry/SentryClient.java) instead
121+
routes it through the new `SentryEnvelopeItem.fromPerfettoProfileChunk(...)` factory, which
122+
avoids base64 by sending the raw binary alongside the JSON.
123+
124+
The trick is a single envelope **item** whose payload concatenates the JSON metadata and
125+
the raw `.pftrace` bytes with **no delimiter**:
126+
127+
```text
128+
[ProfileChunk JSON bytes][raw .pftrace binary bytes]
129+
```
130+
131+
A new `meta_length` property on the [envelope item header](../../../sentry/src/main/java/io/sentry/SentryEnvelopeItemHeader.java)
132+
tells the server where the JSON ends and the binary begins. The standard envelope item
133+
structure (header line + newline + payload) is unchanged; `meta_length` simply subdivides
134+
the payload:
135+
136+
```text
137+
{"type":"profile_chunk","content_type":"application/x-perfetto-trace","filename":"…","length":<total>,"meta_length":<json bytes>}
138+
<ProfileChunk JSON><raw perfetto binary>
139+
```
140+
141+
- `length` — total payload size (JSON + binary), as for any envelope item.
142+
- `meta_length` — byte length of the JSON prefix. It is only known after the payload is
143+
serialized, so the header computes it lazily (via a `Callable<Integer>`) and omits the
144+
field entirely for non-Perfetto items, keeping the change backward compatible.
145+
146+
## Relay (getsentry/relay)
147+
148+
In processing mode Relay:
149+
150+
1. **Splits** the compound item payload at `meta_length` into `(metadata JSON, raw profile)`
151+
and reads `content_type: "perfetto"` from the metadata.
152+
2. **Converts** the binary Perfetto trace into the existing **Sample v2** profile JSON
153+
format (`relay_profiling::expand_perfetto(...)`, backed by a checked-in subset of the
154+
Perfetto protobuf definitions).
155+
3. **Uploads** the raw `.pftrace` blob to object store (usecase `profiles`, keyed per
156+
org/project, with an attachment-retention TTL).
157+
4. **Produces** a `ProfileChunkKafkaMessage` to the `profiles` Kafka topic. The message
158+
carries the expanded Sample v2 JSON as `payload` plus an `attachments` array, where each
159+
attachment records:
160+
- `name` (e.g. `profile.perfetto`),
161+
- `content_type` (e.g. `application/x-perfetto-trace`),
162+
- `stored_id` — the object store key of the uploaded raw blob.
163+
164+
```json
165+
{
166+
"organization_id": 1,
167+
"project_id": 42,
168+
"received": 1720000000,
169+
"retention_days": 30,
170+
"payload": "<expanded Sample v2 profile JSON>",
171+
"attachments": [
172+
{
173+
"name": "profile.perfetto",
174+
"content_type": "application/x-perfetto-trace",
175+
"stored_id": "<object store key of the raw .pftrace blob>"
176+
}
177+
]
178+
}
179+
```
180+
181+
The monolith later uses `stored_id` to fetch the raw trace back.
182+
183+
## Monolith (getsentry/sentry)
184+
185+
`process_profile_task` (in `src/sentry/profiles/task.py`) consumes the `profiles` topic.
186+
Because Relay already converted the trace to Sample v2, the task treats a Perfetto chunk
187+
like any other: deobfuscate, hand it to `vroomrs` to parse and normalize
188+
(`vroomrs.profile_chunk_from_json_str(...)`), compress and store it, and emit function
189+
metrics to Snuba.
190+
191+
The Perfetto-specific step is the last one: for each attachment on the message the task
192+
persists a lightweight **`ProfileChunkAttachment`** row — `project_id`, `profiler_id`,
193+
`chunk_id`, `name`, `content_type`, and the `stored_id` object store key. The row exists so
194+
the raw trace can be downloaded by ID without exposing the `stored_id`.
195+
196+
Flamegraphs themselves are served by `getsentry/vroom`, which reads the stored chunks and
197+
the Snuba-indexed metadata and merges several chunks into one flamegraph. The endpoint
198+
lives in the monolith and passes the request through.
199+
200+
### Perfetto format dispatch (vroom / vroomrs)
201+
202+
Older Android SDKs emit the legacy Android trace format tagged as a "faulty" `version=2`,
203+
and the pipeline historically keyed off the platform rather than the version. To
204+
distinguish legacy from Sample v2 chunks, `ProfileChunk` carries a dedicated `version`
205+
field, and both `vroom` and `vroomrs` now dispatch on it instead of the platform:
206+
207+
- Version `""` or `2.android-trace` → legacy Android trace format.
208+
- Any other version → Sample v2.
209+
210+
## Downloading a Perfetto profile
211+
212+
The monolith exposes two feature-gated endpoints:
213+
214+
- **List attachments** — `GET /organizations/{org}/profiling/chunk-attachments/`
215+
(`sentry-api-0-organization-profiling-chunk-attachments`). Requires a `project` and
216+
`profiler_id`; resolves the visible `chunk_id`s (same logic as the flamegraph) and returns
217+
the matching `ProfileChunkAttachment` metadata.
218+
- **Download** — `GET /projects/{org}/{project}/profiling/chunks/{profiler_id}/{chunk_id}/attachments/{attachment_id}/?download`
219+
(`sentry-api-0-project-profiling-chunk-attachment`). The `?download` param is required; it
220+
streams the raw blob back from object store via the stored `stored_id`. Access requires
221+
the org's configured attachments role, analogous to generic event attachments.
222+
223+
In the flamegraph UI, a toolbar button (added for continuous profiles when the feature is
224+
enabled and at least one attachment exists) lists and provides a way to download these traces.
225+
226+
## References
227+
228+
- SDK: [sentry-java#5251](https://github.com/getsentry/sentry-java/pull/5251) — Android `ProfilingManager` (Perfetto) support
229+
- Relay: [#5659](https://github.com/getsentry/relay/pull/5659), [#5932](https://github.com/getsentry/relay/pull/5932), [#6099](https://github.com/getsentry/relay/pull/6099), [#6102](https://github.com/getsentry/relay/pull/6102) — Perfetto parsing, pipeline, and object-store routing
230+
- vroom: [#672](https://github.com/getsentry/vroom/pull/672) — version dispatch for Android trace profiles
231+
- vroomrs: [#93](https://github.com/getsentry/vroomrs/pull/93) — accept Android profiles in Sample v2 format
232+
- Monolith: [sentry#118029](https://github.com/getsentry/sentry/pull/118029) (chunk attachments + endpoints), [sentry#118071](https://github.com/getsentry/sentry/pull/118071) (flamegraph download button)

0 commit comments

Comments
 (0)