Skip to content

Commit 00a30a7

Browse files
jpnurmiromtsn
andauthored
feat(android): Add experimental out-of-process crash reporting (#6347)
* feat(android): Add service-managed NDK crash daemon Run the NDK crash daemon from an isolated Android service process so it can release the crashing app, merge the resulting tombstone, and send the enriched envelope before shutdown. * Update CHANGELOG.md * fix(android): Narrow exception handling in the NDK crash service Replace the broad `Throwable` catches in `SentryNdkCrashService` with the exceptions the guarded code can throw, and restore the interrupt flag when the service bind is interrupted. * fix(android): Stop the NDK crash service when its daemon exits The service was started with `startService` but never stopped, so its process kept running after `Sentry.close()` or after the app process died. Call `stopSelf` with the start ID that was current when the worker started, so a newer start from a relaunched app is not stopped. * fix(android): Match the NDK crash service stop to its own start A new service process receives `onBind` before `onStartCommand`, so a worker started from the binder configuration could capture a stale start ID and never stop the service. Tag each start with its shared memory path and stop only when the latest start belongs to the exiting worker. * fix(android): Write the NDK tombstone marker after a successful send The marker was written as soon as the merged envelope existed. If the send then failed with a network error, the daemon discarded the envelope and the next launch skipped the tombstone as already reported, so the crash was lost. Write the marker only once the send succeeds. * fix(android): Bind the NDK crash service for the app's lifetime `startService` throws when the app starts in the background, and the started service stopped about a minute after the app left the foreground, so a later crash found no daemon. Native init then fell back to fork/exec, which cannot exec `libsentry-crash.so` from inside the APK with default packaging, and stalled startup for 10 s. Bind the service with `BIND_AUTO_CREATE` and keep the binding until `SentryNdk.close()`, so the service process keeps the app's importance and is not frozen. Let crash processing continue after `onDestroy`, which runs as soon as a crashed app's binding goes away. Fail NDK init instead of falling back to fork/exec when the service is unavailable: below API 30, without a context, or when binding or configuring fails. Move the inproc `SentryNdkTest` to inproc-only test sources, since outproc init now requires a context. * fix(android): Match NDK tombstones by the app's pid The crash service picked the native crash exit info closest to the event's timestamp and ignored its pid. In a crash loop, the first poll can run before the current crash has an exit info, and the previous crash's tombstone was merged instead. Skip exit infos of other pids. The worker captures the pid when it starts, so a later configure call cannot change it. * fix(android): Start an NDK crash daemon for each relaunch When the app relaunched while the previous daemon was still sending its crash, or before it noticed that its process was gone, the new config was stored but no worker started, and the old worker's cleanup then cleared it. The relaunched app's next crash waited 10 s in the handler and was lost, and its descriptors leaked. Keep the per-app daemon state in a config owned by its worker, and start a worker for each config right away. The old daemon is past crash processing or only waiting for its process to go away, so the two share only process-wide native state such as the logger. * ci(temp): Publish sentry-native-ndk-outproc to Maven Local `sentry-android-ndk-outproc` depends on `io.sentry:sentry-native-ndk-outproc`, which is not on Maven Central yet. Build it from the sentry-native feature branch and publish it to Maven Local so the Build job can resolve it. TODO: Revert. This is temporary until sentry-native publishes the artifact. * fix(android): Wait for the NDK tombstone outside the send budget The tombstone wait and the send shared `shutdownTimeoutMillis`, so a slow tombstone left the send no time at all. Wait for the app's exit info instead. The app exits right after debuggerd writes its tombstone, so the wait ends when the tombstone is there, capped at the 30 s match tolerance. Stop early when the app did not crash natively or the tombstone does not parse. Give the send the full `shutdownTimeoutMillis`, and write the marker for a merged envelope that was sent or kept for the next launch. * fix(android): Remove stale NDK crash IPC files Name the shared-memory file after the app pid and delete leftovers whose process no longer runs when the crash service starts. A setup that fails before the daemon attaches would otherwise leave the file behind, while other processes of a multi-process app keep theirs. * fix(android): Fall back to the 1-argument NDK init With a context, NdkIntegration only looked up `SentryNdk.init(Context, SentryAndroidOptions)`. An older sentry-android-ndk without it disabled NDK. Fall back to `SentryNdk.init(SentryAndroidOptions)` and log a warning. * ref(android): Stop passing the tombstone flag to NdkOptions sentry-native-ndk-outproc no longer reads it, because it always sends native crash envelopes without a minidump. The crash service still gets the tombstone flag through its configure call. * fix(android): Skip SDK init in the NDK crash service process The crash service runs in its own `.sentry_ndk_crash` process, which runs the app's `Application.onCreate`. Apps that call `SentryAndroid.init` there started a second SDK on every launch, with the tombstone, ANR and outbox senders working on the same cache directory as the app process. Return early from `SentryAndroid.init` in that process. * ref(android): Remove the NDK crash service process check SentryAndroid.init now returns early in the `.sentry_ndk_crash` process, so SentryNdk.init is no longer called there. * fix(android): Keep a newer NDK tombstone marker Crash service workers for processes that crash close together can finish out of order. A worker that wrote an older tombstone timestamp let `TombstoneIntegration` report the newer crash again on the next launch. Write the marker only when it is newer than the current one, and serialize the workers. * ci: Stop the sentry-native Gradle daemon after publishing The publish step left its 4 GB Gradle daemon running alongside the 8 GB daemon used by `make preMerge`. The job was killed about 12 minutes into the tests, most likely because the runner ran out of memory. * fix(android): Bind the NDK crash service as important Without `BIND_IMPORTANT`, a foreground app raises its bound crash service process only to the visible oom adj (100), so lmkd would kill the service before the app under memory pressure. The flag gives the service the app's own adj (0) while the app is on top, without lowering the app's priority the way `BIND_ABOVE_CLIENT` would. * Apply batched suggestions from code review Co-authored-by: Roman Zavarnitsyn <rom4ek93@gmail.com> * chore(android): Update the outproc API dump The outproc module shares NdkScopeObserver with the inproc module, which gained setEnvironment on main (#5772). * fix(android): Reattach the NDK crash daemon after a service restart When the crash service process died while the app was bound, Android restarted it and called `onServiceConnected` again, but the connection had already closed the descriptors it configured with, so the app ran without a crash daemon. Configure the restarted service with new pipes and the app's shared memory file, then reattach the native crash IPC. Bind again when the binding dies, and log a lost connection at ERROR. * ref(android): Build the NDK outproc package on top of sentry-android-ndk sentry-android-ndk-outproc compiled sentry-android-ndk's sources through extra source folders, shared its namespace and borrowed its ProGuard rules, so the same files belonged to both modules, and the deprecated `BaseExtension` wired it up. Depend on sentry-android-ndk instead, without its `sentry-native-ndk`, and keep only the crash service and the `SentryNdkOutproc` entry point in the `io.sentry.android.ndk.outproc` namespace. `NdkIntegration` now loads that entry point first and falls back to `SentryNdk`. * fix(android): Write the NDK tombstone marker before the send The native side keeps the envelope for the next launch until it is sent, so the crash is delivered whatever the send result. Write the marker before the send, so that a send that a frozen service finishes late no longer races TombstoneIntegration on the next launch. * fix(android): Close rejected NDK crash service fds `configureDaemon` left the file descriptors of a binder transaction open until garbage collection when it rejected the configuration. Close them before returning. * fix(android): Time the NDK tombstone poll with a monotonic clock The crash service polled for a matching tombstone until a wall clock deadline, so a clock step backward delayed the crash send by the size of the step. Use `SystemClock.uptimeMillis()`, which also drives `Thread.sleep()`. * fix(android): Narrow the NDK outproc library load catch The library loader thread swallowed every `Throwable`, including fatal errors. Catch only the `UnsatisfiedLinkError` of a missing or broken library, which `init()` reports later. `ExceptionUtils.rethrowIfFatal` does not fit, because it rethrows `LinkageError` and would turn a missing library into an app crash. * fix(android): Load the NDK outproc libraries before binding the service A missing or broken native library surfaced only in `initCrashDaemon`, after the crash service was bound and the IPC fds were detached. The `UnsatisfiedLinkError` left the service bound with `BIND_IMPORTANT` and the fds open for the app's lifetime. Load the libraries first, so the failure leaves nothing behind. * fix(android): Write the NDK tombstone marker when the merge fails When the crash service matched the app's native crash exit but could not merge its tombstone, because it stayed unavailable until the deadline or the merge threw, it sent the native event without writing the marker. On the next launch, `TombstoneIntegration` then reported the same crash again as a standalone tombstone event. Write the marker for a matched tombstone, merged or not. * docs(android): Remove the Gradle plugin from the NDK outproc README The `ndkBackend` option of the Gradle plugin is parked for now, so the outproc package is selected through its dependency only. * fix(android): Hand the NDK crash memfd to the crash service The NDK now backs the crash IPC with a memfd instead of a file, because the sparse file could raise `SIGBUS` on a full disk. Create the memfd before configuring the service and send it over Binder, which reattaching already does. This removes the file path from the protocol, and the cleanup of stale files. * docs(changelog): Move the outproc entry back to Unreleased The merge of `origin/main` placed it under `8.63.0`, which was released without out-of-process crash reporting. * build(deps): Bump sentry-native-ndk-outproc to 1.0.0-beta.1 sentry-native now publishes `sentry-native-ndk-outproc` to Maven Central, so CI no longer builds it from the sentry-native branch into Maven Local. * meta(android): Register the NDK outproc module for release Craft's `registry` target fails the release when a published module has no entry. Also list the module in the README packages table and the Android issue template. * ref(android): Pass the thread id as the NDK crash daemon's appTid `appTid` received `System.nanoTime()`, but the daemon expects the parent thread id. On Android it only names the shared memory file that the daemon unlinks, and none exists because the memory is a memfd. * ref(android): Time the NDK tombstone poll with a Deadline New code reads clocks through `io.sentry.time` instead of `SystemClock`. The poll deadline now comes from the options' monotonic ticker, which also counts time the device spends suspended. * ref(android): Document the NDK crash service's broad catches The exception rules in `AGENTS.md` allow a broad catch only at a boundary, with a comment on why and what happens on failure. The two `Exception` catches also call `ExceptionUtils.rethrowIfFatal`, which restores the interrupt flag of a swallowed `InterruptedException`. * fix(android): Unbind the NDK crash service on any native init failure `SentryNdkOutproc.init` only unbound the crash service when native init threw `IllegalStateException`. Any other throwable, such as an `UnsatisfiedLinkError`, left the service bound with `BIND_IMPORTANT`, and the disabled NDK integration never closed it. * docs(changelog): Explain how to enable Android outproc crash reporting --------- Co-authored-by: Roman Zavarnitsyn <rom4ek93@gmail.com>
1 parent 0fb4971 commit 00a30a7

29 files changed

Lines changed: 1895 additions & 9 deletions

File tree

‎.craft.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,11 @@ targets:
124124
packageUrl: https://central.sonatype.com/artifact/io.sentry/sentry-android-ndk
125125
mainDocsUrl: https://docs.sentry.io/platforms/android/
126126
apiDocsUrl: https://javadoc.io/doc/io.sentry
127+
maven:io.sentry:sentry-android-ndk-outproc:
128+
name: io.sentry:sentry-android-ndk-outproc
129+
packageUrl: https://central.sonatype.com/artifact/io.sentry/sentry-android-ndk-outproc
130+
mainDocsUrl: https://docs.sentry.io/platforms/android/configuration/using-ndk/
131+
apiDocsUrl: https://javadoc.io/doc/io.sentry
127132
maven:io.sentry:sentry-android-timber:
128133
name: io.sentry:sentry-android-timber
129134
packageUrl: https://central.sonatype.com/artifact/io.sentry/sentry-android-timber

‎.github/ISSUE_TEMPLATE/bug_report_android.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ body:
1010
options:
1111
- sentry-android
1212
- sentry-android-ndk
13+
- sentry-android-ndk-outproc
1314
- sentry-android-timber
1415
- sentry-android-fragment
1516
- sentry-android-navigation3

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@
2222
- Quartz `SentryJobListener` can send a monitor config derived from the trigger with check-ins, so Sentry creates or updates the monitor from code. Set the `sentry-upsert-monitor-config` job or trigger data key to `"true"` to send the schedule ([#6216](https://github.com/getsentry/sentry-java/pull/6216))
2323
- Send a monitor config derived from Spring `@Scheduled` with `@SentryCheckIn` check-ins, so Sentry creates or updates the monitor from code. Set `@SentryCheckIn(upsertMonitorConfig = true)` to send the schedule ([#6215](https://github.com/getsentry/sentry-java/pull/6215))
2424
- Sentry OpenTelemetry integrations now honor `OTEL_BSP_SCHEDULE_DELAY` (`otel.bsp.schedule.delay`) for how often spans are sent to Sentry, instead of always waiting 5 seconds ([#6388](https://github.com/getsentry/sentry-java/pull/6388))
25+
- Add experimental, opt-in out-of-process crash reporting on Android ([#6347](https://github.com/getsentry/sentry-java/pull/6347))
26+
- To enable it, add the `sentry-android-ndk-outproc` dependency and exclude `sentry-native-ndk`. Requires Android 11 (API level 30) or later.
27+
- See the [NDK docs](https://docs.sentry.io/platforms/android/configuration/using-ndk/#out-of-process-crash-handling) for details.
2528

2629
### Behavioral Changes
2730

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ Sentry SDK for Java and Android
2222
| sentry-android-core | [![Maven Central Version](https://img.shields.io/maven-central/v/io.sentry/sentry-android-core?style=for-the-badge&logo=sentry&color=green)](https://central.sonatype.com/artifact/io.sentry/sentry-android-core) | 21 |
2323
| sentry-android-distribution | [![Maven Central Version](https://img.shields.io/maven-central/v/io.sentry/sentry-android-distribution?style=for-the-badge&logo=sentry&color=green)](https://central.sonatype.com/artifact/io.sentry/sentry-android-distribution) | 21 |
2424
| sentry-android-ndk | [![Maven Central Version](https://img.shields.io/maven-central/v/io.sentry/sentry-android-ndk?style=for-the-badge&logo=sentry&color=green)](https://central.sonatype.com/artifact/io.sentry/sentry-android-ndk) | 21 |
25+
| sentry-android-ndk-outproc | [![Maven Central Version](https://img.shields.io/maven-central/v/io.sentry/sentry-android-ndk-outproc?style=for-the-badge&logo=sentry&color=green)](https://central.sonatype.com/artifact/io.sentry/sentry-android-ndk-outproc) | 21 |
2526
| sentry-android-timber | [![Maven Central Version](https://img.shields.io/maven-central/v/io.sentry/sentry-android-timber?style=for-the-badge&logo=sentry&color=green)](https://central.sonatype.com/artifact/io.sentry/sentry-android-timber) | 21 |
2627
| sentry-android-fragment | [![Maven Central Version](https://img.shields.io/maven-central/v/io.sentry/sentry-android-fragment?style=for-the-badge&logo=sentry&color=green)](https://central.sonatype.com/artifact/io.sentry/sentry-android-fragment) | 21 |
2728
| sentry-android-navigation | [![Maven Central Version](https://img.shields.io/maven-central/v/io.sentry/sentry-android-navigation?style=for-the-badge&logo=sentry&color=green)](https://central.sonatype.com/artifact/io.sentry/sentry-android-navigation) | 21 |

‎build-logic/src/main/kotlin/io/sentry/gradle/AarTypes.kt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ private val androidLibs =
2525
setOf(
2626
"sentry-android-core",
2727
"sentry-android-ndk",
28+
"sentry-android-ndk-outproc",
2829
"sentry-android-fragment",
2930
"sentry-android-navigation",
3031
"sentry-android-navigation3",

‎gradle/libs.versions.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,7 @@ reactor-core = { module = "io.projectreactor:reactor-core", version = "3.5.3" }
194194
retrofit = { module = "com.squareup.retrofit2:retrofit", version.ref = "retrofit" }
195195
retrofit-gson = { module = "com.squareup.retrofit2:converter-gson", version.ref = "retrofit" }
196196
sentry-native-ndk = { module = "io.sentry:sentry-native-ndk", version = "0.17.6" }
197+
sentry-native-ndk-outproc = { module = "io.sentry:sentry-native-ndk-outproc", version = "1.0.0-beta.1" }
197198
servlet-api = { module = "javax.servlet:javax.servlet-api", version = "3.1.0" }
198199
servlet-jakarta-api = { module = "jakarta.servlet:jakarta.servlet-api", version = "6.1.0" }
199200
slf4j-api = { module = "org.slf4j:slf4j-api", version.ref = "slf4j" }

‎scripts/update-sentry-native-ndk.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ set-version)
1616

1717
echo "Setting sentry-native-ndk version to '$version'"
1818

19-
PATTERN='(module = "io\.sentry:sentry-native-ndk", version = ")[0-9.]+(")'
19+
PATTERN='(module = "io\.sentry:sentry-native-ndk(?:-outproc)?", version = ")[0-9.]+(")'
2020
perl -pi -e "s/$PATTERN/\${1}$version\${2}/" "$GRADLE_NDK_FILEPATH"
2121
;;
2222
*)

‎sentry-android-core/api/sentry-android-core.api‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -376,6 +376,7 @@ public final class io/sentry/android/core/NdkHandlerStrategy : java/lang/Enum {
376376

377377
public final class io/sentry/android/core/NdkIntegration : io/sentry/Integration, java/io/Closeable {
378378
public static final field SENTRY_NDK_CLASS_NAME Ljava/lang/String;
379+
public static final field SENTRY_NDK_OUTPROC_CLASS_NAME Ljava/lang/String;
379380
public fun <init> (Ljava/lang/Class;)V
380381
public fun close ()V
381382
public final fun register (Lio/sentry/IScopes;Lio/sentry/SentryOptions;)V
@@ -680,6 +681,7 @@ public final class io/sentry/android/core/TombstoneIntegration$TombstonePolicy :
680681
public fun getLastReportedTimestamp ()Ljava/lang/Long;
681682
public fun markReported (J)V
682683
public fun matches (Landroid/app/ApplicationExitInfo;)Z
684+
public static fun mergeNativeCrashes (Lio/sentry/SentryEvent;Lio/sentry/SentryEvent;)V
683685
public fun shouldReportHistorical ()Z
684686
}
685687

‎sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package io.sentry.android.core;
22

33
import static io.sentry.android.core.NdkIntegration.SENTRY_NDK_CLASS_NAME;
4+
import static io.sentry.android.core.NdkIntegration.SENTRY_NDK_OUTPROC_CLASS_NAME;
45
import static io.sentry.util.IntegrationUtils.addIntegrationToSdkVersion;
56

67
import android.annotation.SuppressLint;
@@ -438,8 +439,12 @@ static void installDefaultIntegrations(
438439

439440
// Integrations are registered in the same order. NDK before adding Watch outbox,
440441
// because sentry-native move files around and we don't want to watch that.
441-
final Class<?> sentryNdkClass = loadClass.loadClass(SENTRY_NDK_CLASS_NAME, options.getLogger());
442-
options.addIntegration(new NdkIntegration(sentryNdkClass));
442+
// the outproc backend builds on the inproc one, so it goes first, and quietly
443+
Class<?> sentryNdkClass = loadClass.loadClass(SENTRY_NDK_OUTPROC_CLASS_NAME, null);
444+
if (sentryNdkClass == null) {
445+
sentryNdkClass = loadClass.loadClass(SENTRY_NDK_CLASS_NAME, options.getLogger());
446+
}
447+
options.addIntegration(new NdkIntegration(context, sentryNdkClass));
443448

444449
if (buildInfoProvider.getSdkInfoVersion() >= Build.VERSION_CODES.S) {
445450
options.addIntegration(new TombstoneIntegration(context));

‎sentry-android-core/src/main/java/io/sentry/android/core/NdkIntegration.java‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import static io.sentry.util.IntegrationUtils.addIntegrationToSdkVersion;
44

5+
import android.content.Context;
56
import io.sentry.IScopes;
67
import io.sentry.Integration;
78
import io.sentry.SentryLevel;
@@ -19,11 +20,20 @@ public final class NdkIntegration implements Integration, Closeable {
1920

2021
public static final String SENTRY_NDK_CLASS_NAME = "io.sentry.android.ndk.SentryNdk";
2122

23+
public static final String SENTRY_NDK_OUTPROC_CLASS_NAME =
24+
"io.sentry.android.ndk.outproc.SentryNdkOutproc";
25+
26+
private final @Nullable Context context;
2227
private final @Nullable Class<?> sentryNdkClass;
2328

2429
private @Nullable SentryAndroidOptions options;
2530

2631
public NdkIntegration(final @Nullable Class<?> sentryNdkClass) {
32+
this(null, sentryNdkClass);
33+
}
34+
35+
NdkIntegration(final @Nullable Context context, final @Nullable Class<?> sentryNdkClass) {
36+
this.context = context != null ? ContextUtils.getApplicationContext(context) : null;
2737
this.sentryNdkClass = sentryNdkClass;
2838
}
2939

@@ -50,9 +60,25 @@ public final void register(final @NotNull IScopes scopes, final @NotNull SentryO
5060
}
5161

5262
try {
53-
final Method method = sentryNdkClass.getMethod("init", SentryAndroidOptions.class);
54-
final Object[] args = new Object[1];
55-
args[0] = this.options;
63+
Method method = null;
64+
Object[] args = null;
65+
if (context != null) {
66+
try {
67+
method = sentryNdkClass.getMethod("init", Context.class, SentryAndroidOptions.class);
68+
args = new Object[] {context, this.options};
69+
} catch (NoSuchMethodException e) {
70+
// an older sentry-android-ndk
71+
this.options
72+
.getLogger()
73+
.log(
74+
SentryLevel.WARNING,
75+
"SentryNdk.init(Context, SentryAndroidOptions) not found, using SentryNdk.init(SentryAndroidOptions).");
76+
}
77+
}
78+
if (method == null) {
79+
method = sentryNdkClass.getMethod("init", SentryAndroidOptions.class);
80+
args = new Object[] {this.options};
81+
}
5682
method.invoke(null, args);
5783

5884
this.options.getLogger().log(SentryLevel.DEBUG, "NdkIntegration installed.");

0 commit comments

Comments
 (0)