Skip to content

Commit 41d7c51

Browse files
runningcodeclaude
andauthored
perf(core): Batch and coalesce scope-persistence disk writes (#5791)
* perf(core): Batch and coalesce scope-persistence disk writes (JAVA-628) Scope persistence wrote to disk on every scope mutation, which dominated SDK cost during startup: each breadcrumb triggered a synchronous fsync'd QueueFile append, and every other scope field (contexts, trace, user, tags, ...) rewrote its whole file on each change even though only the latest value matters. Coalesce mutations instead of writing eagerly. Each field keeps only its latest pending value and is flushed once per debounce window; breadcrumbs are buffered and appended together behind a single fsync (QueueFile gains an opt-in buffered-write mode plus sync()). This trades a small data-loss window (~100ms before the process dies) for far fewer writes and fsyncs. Persistence exists to enrich crash/ANR events on the next launch, and was already asynchronous, so the widened loss window is acceptable. * Add changelog entry for #5791 * perf(core): Coalesce scope writes on submit instead of a timer (JAVA-628) Scope-persistence flushes were debounced 100ms behind a scheduled task. The debounce was unnecessary: the Sentry executor is single-threaded, so a submitted flush task already sits in the queue long enough for mutations arriving behind it to be folded into the same write. That is exactly the window that matters, since the queue is deepest during startup. Submit the flush instead of scheduling it. Coalescing now tracks executor load rather than a fixed delay, which closes the data-loss window the debounce introduced. Guard against the executor rejecting the task without throwing once its queue is full, which would otherwise leave the pending flag set and stop scope persistence for the rest of the process. Also record why resetCache() deliberately leaves pending mutations alone: they only ever hold values from the current process, so dropping them would lose scope state set during init rather than clearing the previous run's data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(core): Keep breadcrumbs added during a scope flush (JAVA-628) Batching split one ordered stream of breadcrumb operations into two independent pieces of state: a queue of pending adds and a separate clear flag. Independent state cannot preserve an ordering that the old FIFO executor queue gave for free. flushPending consumed the clear flag with a CAS at the top, then spent the rest of the method serializing and writing each breadcrumb. A clear plus a subsequent add landing in that window had its clear applied after the new breadcrumb was already on disk, so the follow-up flush wiped a breadcrumb that was added after the clear. Breadcrumbs are added from arbitrary threads while the flush runs on the executor thread, so this needs no unusual timing. Enqueue the clear into the breadcrumb queue as a sentinel instead, mirroring the DELETE_MARKER pattern already used for pendingWrites. Ordering becomes intrinsic to the queue rather than something the flush has to reconstruct. This also stops setBreadcrumbs(emptyList()) from calling clear() on the shared queue, which could discard a breadcrumb offered concurrently by another thread even with no flush in flight. * docs(core): Clarify scope-persistence comments (JAVA-628) "Durable" overstated what the buffered-write mode guarantees. sync() issues an fsync, but durability also depends on the storage stack, so describe what the code does — write to disk — rather than promise an outcome it cannot ensure on its own. Also name the pendingBreadcrumbs queue for what it holds: requests to add and clear, not breadcrumbs already buffered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(core): Fold the scope flush into one method (JAVA-628) flushPending existed only so the @testonly flush() could do the disk write without the latch bookkeeping that belongs to the queued task. Nothing in production called flush(), and its only test asserted the behaviour of flush() itself, so the second entry point was buying two near-identical methods and a way to corrupt hasPendingFlush. Drop flush() and merge the pair: flush() is now the queued task and writePending() the write it performs. Clear hasPendingFlush in a finally so an unexpected throw cannot leave the flag set, which would stop scope persistence for the rest of the process. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(core): Drop redundant buffered-write QueueFile test (JAVA-628) bufferedWritesSurviveReopenAfterSync was testAddOneElement with a second element and a sync() call: same write, close, reopen, read-it-back shape, and testAddAndRemoveElements already covers round-tripping far more thoroughly. The sync() it appeared to exercise was not actually load-bearing. close() flushes through to the OS whichever mode the file was opened in, so the assertions would hold even if sync() did nothing — the name promised durability semantics the test could not verify in-process. Verifying that would take a killed process, not a reopen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(core): Record that the vendored Tape fork has diverged (JAVA-628) Square archived Tape on 2024-10-25, so the upstream link in each file header points at a snapshot that will never receive fixes. Our copy has also drifted from it: corruption recovery, a bounded ring size, and optional buffered writes. Say both things where a reader will look — the file headers, the class javadoc, and THIRD_PARTY_NOTICES.md — so nobody diffs against upstream expecting it to explain our behaviour, or files a bug there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(core): Iterate pendingWrites directly when flushing (JAVA-628) writePending() copied the key set into an ArrayList before draining it, which allocates on every flush for no benefit: ConcurrentHashMap's iterator is weakly consistent, so walking it while removing entries is already safe. Removal still goes through the map, not the iterator, because that is an atomic get-and-remove — we never drop a value stored by a mutation racing with the loop. Keys added after iteration starts can be missed, which is fine since flush() re-checks and queues another write. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 0ab689f commit 41d7c51

9 files changed

Lines changed: 372 additions & 93 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@
1111

1212
- Remove an unused lock from `SentryPerformanceProvider`, which was allocated on every cold start in `ContentProvider.onCreate` without ever being acquired ([#5871](https://github.com/getsentry/sentry-java/pull/5871))
1313
- Parse the app start profiling config with only the deserializer it needs instead of building a full `JsonSerializer` and `SentryOptions`, cutting 188 of 221 allocations on the main thread before `Application.onCreate` ([#5867](https://github.com/getsentry/sentry-java/pull/5867))
14+
- Batch and coalesce scope-persistence disk writes to reduce startup cost ([#5791](https://github.com/getsentry/sentry-java/pull/5791))
15+
- Scope mutations are now coalesced (latest value per field) and breadcrumbs are appended in batches behind a single fsync, instead of one synchronous disk write per mutation.
1416

1517
## 8.51.0
1618

‎THIRD_PARTY_NOTICES.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,14 +136,16 @@ limitations under the License.
136136

137137
## Square — Tape (Apache 2.0)
138138

139-
**Source:** https://github.com/square/tape (Commit: 445cd3fd0a7b3ec48c9ea3e0e86663fe6d3735d8)<br>
139+
**Source:** https://github.com/square/tape (Commit: 445cd3fd0a7b3ec48c9ea3e0e86663fe6d3735d8, archived 2024-10-25)<br>
140140
**License:** Apache License 2.0<br>
141141
**Copyright:** Copyright (C) 2010 Square, Inc.
142142

143143
### Scope
144144

145145
The Sentry Java SDK includes an adapted version of Square's Tape library, a file-based FIFO queue implementation used for reliable event storage. The code resides in the `io.sentry.cache.tape` package and includes `QueueFile`, `FileObjectQueue`, and `ObjectQueue`.
146146

147+
Upstream was archived on 2024-10-25 and is no longer maintained. This copy is maintained in-tree and has diverged from the linked commit: it recovers from file corruption by recreating the file, bounds the queue to a maximum number of elements, and supports optional buffered writes flushed by an explicit `sync()`.
148+
147149
```
148150
Copyright (C) 2010 Square, Inc.
149151

‎sentry/api/sentry.api‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4934,6 +4934,7 @@ public abstract class io/sentry/cache/tape/ObjectQueue : java/io/Closeable, java
49344934
public fun remove ()V
49354935
public abstract fun remove (I)V
49364936
public abstract fun size ()I
4937+
public fun sync ()V
49374938
}
49384939

49394940
public abstract interface class io/sentry/cache/tape/ObjectQueue$Converter {
@@ -4954,13 +4955,15 @@ public final class io/sentry/cache/tape/QueueFile : java/io/Closeable, java/lang
49544955
public fun remove ()V
49554956
public fun remove (I)V
49564957
public fun size ()I
4958+
public fun sync ()V
49574959
public fun toString ()Ljava/lang/String;
49584960
}
49594961

49604962
public final class io/sentry/cache/tape/QueueFile$Builder {
49614963
public fun <init> (Ljava/io/File;)V
49624964
public fun build ()Lio/sentry/cache/tape/QueueFile;
49634965
public fun size (I)Lio/sentry/cache/tape/QueueFile$Builder;
4966+
public fun synchronousWrites (Z)Lio/sentry/cache/tape/QueueFile$Builder;
49644967
public fun zero (Z)Lio/sentry/cache/tape/QueueFile$Builder;
49654968
}
49664969

‎sentry/src/main/java/io/sentry/cache/PersistingScopeObserver.java‎

Lines changed: 150 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@
77
import io.sentry.Breadcrumb;
88
import io.sentry.IScope;
99
import io.sentry.ScopeObserverAdapter;
10-
import io.sentry.SentryExecutorService;
1110
import io.sentry.SentryLevel;
1211
import io.sentry.SentryOptions;
1312
import io.sentry.SpanContext;
@@ -30,14 +29,26 @@
3029
import java.io.Writer;
3130
import java.nio.charset.Charset;
3231
import java.util.Collection;
32+
import java.util.Iterator;
3333
import java.util.Map;
34+
import java.util.Queue;
35+
import java.util.concurrent.ConcurrentHashMap;
36+
import java.util.concurrent.ConcurrentLinkedQueue;
37+
import java.util.concurrent.Future;
38+
import java.util.concurrent.atomic.AtomicBoolean;
3439
import org.jetbrains.annotations.NotNull;
3540
import org.jetbrains.annotations.Nullable;
3641

3742
public final class PersistingScopeObserver extends ScopeObserverAdapter {
3843

3944
private static final Charset UTF_8 = Charset.forName("UTF-8");
4045

46+
/** Sentinel value marking a file that should be deleted rather than written on the next flush. */
47+
private static final Object DELETE_MARKER = new Object();
48+
49+
/** Sentinel value marking a breadcrumb clear in the pending breadcrumb operations. */
50+
private static final Object CLEAR_MARKER = new Object();
51+
4152
public static final String SCOPE_CACHE = ".scope-cache";
4253
public static final String USER_FILENAME = "user.json";
4354
public static final String BREADCRUMBS_FILENAME = "breadcrumbs.json";
@@ -65,7 +76,11 @@ public final class PersistingScopeObserver extends ScopeObserverAdapter {
6576
final File file = new File(cacheDir, BREADCRUMBS_FILENAME);
6677
try {
6778
try {
68-
queueFile = new QueueFile.Builder(file).size(options.getMaxBreadcrumbs()).build();
79+
queueFile =
80+
new QueueFile.Builder(file)
81+
.size(options.getMaxBreadcrumbs())
82+
.synchronousWrites(false)
83+
.build();
6984
} catch (IOException e) {
7085
// if file is corrupted we simply delete it and try to create it again. We accept
7186
// the trade
@@ -74,7 +89,11 @@ public final class PersistingScopeObserver extends ScopeObserverAdapter {
7489
// update where the new format was introduced
7590
file.delete();
7691

77-
queueFile = new QueueFile.Builder(file).size(options.getMaxBreadcrumbs()).build();
92+
queueFile =
93+
new QueueFile.Builder(file)
94+
.size(options.getMaxBreadcrumbs())
95+
.synchronousWrites(false)
96+
.build();
7897
}
7998
} catch (IOException e) {
8099
options.getLogger().log(ERROR, "Failed to create breadcrumbs queue", e);
@@ -106,140 +125,190 @@ public void toStream(Breadcrumb value, OutputStream sink) throws IOException {
106125
});
107126
});
108127

128+
// Latest pending value per file (or DELETE_MARKER), coalesced until the next flush.
129+
private final @NotNull Map<String, Object> pendingWrites = new ConcurrentHashMap<>();
130+
// Tracks requests to add and clear breadcrumb since the last flush, applied together behind a
131+
// single fsync. Adds and clears share one queue so their relative order survives batching:
132+
// a clear must not wipe a breadcrumb added after it.
133+
private final @NotNull Queue<Object> pendingBreadcrumbs = new ConcurrentLinkedQueue<>();
134+
private final @NotNull AtomicBoolean hasPendingFlush = new AtomicBoolean(false);
135+
109136
public PersistingScopeObserver(final @NotNull SentryOptions options) {
110137
this.options = options;
111138
}
112139

113140
@Override
114141
public void setUser(final @Nullable User user) {
115-
serializeToDisk(
116-
() -> {
117-
if (user == null) {
118-
delete(USER_FILENAME);
119-
} else {
120-
store(user, USER_FILENAME);
121-
}
122-
});
142+
enqueue(USER_FILENAME, user);
123143
}
124144

125145
@Override
126146
public void addBreadcrumb(@NotNull Breadcrumb crumb) {
127-
serializeToDisk(
128-
() -> {
129-
try {
130-
breadcrumbsQueue.getValue().add(crumb);
131-
} catch (IOException e) {
132-
options.getLogger().log(ERROR, "Failed to add breadcrumb to file queue", e);
133-
}
134-
});
147+
if (!options.isEnableScopePersistence()) {
148+
return;
149+
}
150+
pendingBreadcrumbs.offer(crumb);
151+
requestFlush();
135152
}
136153

137154
@Override
138155
public void setBreadcrumbs(@NotNull Collection<Breadcrumb> breadcrumbs) {
139156
if (breadcrumbs.isEmpty()) {
140157
// we only clear the queue if the new collection is empty (someone called clearBreadcrumbs)
141158
// If it's not empty, we'd add breadcrumbs one-by-one in the method above
142-
serializeToDisk(
143-
() -> {
144-
try {
145-
breadcrumbsQueue.getValue().clear();
146-
} catch (IOException e) {
147-
options.getLogger().log(ERROR, "Failed to clear breadcrumbs from file queue", e);
148-
}
149-
});
159+
if (!options.isEnableScopePersistence()) {
160+
return;
161+
}
162+
pendingBreadcrumbs.offer(CLEAR_MARKER);
163+
requestFlush();
150164
}
151165
}
152166

153167
@Override
154168
public void setTags(@NotNull Map<String, @NotNull String> tags) {
155-
serializeToDisk(() -> store(tags, TAGS_FILENAME));
169+
enqueue(TAGS_FILENAME, tags);
156170
}
157171

158172
@Override
159173
public void setExtras(@NotNull Map<String, @NotNull Object> extras) {
160-
serializeToDisk(() -> store(extras, EXTRAS_FILENAME));
174+
enqueue(EXTRAS_FILENAME, extras);
161175
}
162176

163177
@Override
164178
public void setRequest(@Nullable Request request) {
165-
serializeToDisk(
166-
() -> {
167-
if (request == null) {
168-
delete(REQUEST_FILENAME);
169-
} else {
170-
store(request, REQUEST_FILENAME);
171-
}
172-
});
179+
enqueue(REQUEST_FILENAME, request);
173180
}
174181

175182
@Override
176183
public void setFingerprint(@NotNull Collection<String> fingerprint) {
177-
serializeToDisk(() -> store(fingerprint, FINGERPRINT_FILENAME));
184+
enqueue(FINGERPRINT_FILENAME, fingerprint);
178185
}
179186

180187
@Override
181188
public void setLevel(@Nullable SentryLevel level) {
182-
serializeToDisk(
183-
() -> {
184-
if (level == null) {
185-
delete(LEVEL_FILENAME);
186-
} else {
187-
store(level, LEVEL_FILENAME);
188-
}
189-
});
189+
enqueue(LEVEL_FILENAME, level);
190190
}
191191

192192
@Override
193193
public void setTransaction(@Nullable String transaction) {
194-
serializeToDisk(
195-
() -> {
196-
if (transaction == null) {
197-
delete(TRANSACTION_FILENAME);
198-
} else {
199-
store(transaction, TRANSACTION_FILENAME);
200-
}
201-
});
194+
enqueue(TRANSACTION_FILENAME, transaction);
202195
}
203196

204197
@Override
205198
public void setTrace(@Nullable SpanContext spanContext, @NotNull IScope scope) {
206-
serializeToDisk(
207-
() -> {
208-
if (spanContext == null) {
209-
// we always need a trace_id to properly link with traces/replays, so we fallback to
210-
// propagation context values and create a fake SpanContext
211-
store(scope.getPropagationContext().toSpanContext(), TRACE_FILENAME);
212-
} else {
213-
store(spanContext, TRACE_FILENAME);
214-
}
215-
});
199+
// we always need a trace_id to properly link with traces/replays, so we fallback to
200+
// propagation context values and create a fake SpanContext
201+
enqueue(
202+
TRACE_FILENAME,
203+
spanContext == null ? scope.getPropagationContext().toSpanContext() : spanContext);
216204
}
217205

218206
@Override
219207
public void setContexts(@NotNull Contexts contexts) {
220-
serializeToDisk(() -> store(contexts, CONTEXTS_FILENAME));
208+
enqueue(CONTEXTS_FILENAME, contexts);
221209
}
222210

223211
@Override
224212
public void setReplayId(@NotNull SentryId replayId) {
225-
serializeToDisk(() -> store(replayId, REPLAY_FILENAME));
213+
enqueue(REPLAY_FILENAME, replayId);
226214
}
227215

228-
@SuppressWarnings("FutureReturnValueIgnored")
229-
private void serializeToDisk(final @NotNull Runnable task) {
216+
private void enqueue(final @NotNull String fileName, final @Nullable Object entity) {
230217
if (!options.isEnableScopePersistence()) {
231218
return;
232219
}
233-
if (SentryExecutorService.isSentryExecutorThread()) {
234-
// we're already on the sentry executor thread, so we can just execute it directly
235-
runSafely(task);
220+
// latest value wins; a null entity means the file should be deleted on the next flush
221+
pendingWrites.put(fileName, entity == null ? DELETE_MARKER : entity);
222+
requestFlush();
223+
}
224+
225+
/**
226+
* Queues a flush unless one is already queued. Coalescing comes from the flush task sitting in
227+
* the executor queue: every mutation that arrives before it runs is folded into the same write.
228+
* The executor is single-threaded, so during startup — when mutations are frequent and the queue
229+
* is deep — that window covers many mutations.
230+
*/
231+
private void requestFlush() {
232+
if (!hasPendingFlush.compareAndSet(false, true)) {
233+
// a flush is already queued; it will pick up the latest pending state
236234
return;
237235
}
238-
239236
try {
240-
options.getExecutorService().submit(() -> runSafely(task));
237+
final @NotNull Future<?> future = options.getExecutorService().submit(this::flush);
238+
if (future.isCancelled()) {
239+
// the executor rejects tasks without throwing once its queue is full, so clear the flag or
240+
// no later mutation would ever be able to queue a flush again
241+
hasPendingFlush.set(false);
242+
}
241243
} catch (Throwable e) {
242-
options.getLogger().log(ERROR, "Serialization task could not be scheduled", e);
244+
hasPendingFlush.set(false);
245+
options.getLogger().log(ERROR, "Scope persistence flush could not be submitted", e);
246+
}
247+
}
248+
249+
/**
250+
* Writes all coalesced scope state to disk. Runs as the queued flush task on the Sentry executor,
251+
* which is also the only place it may run: it does I/O, and it owns clearing {@link
252+
* #hasPendingFlush} once the write is done.
253+
*/
254+
private void flush() {
255+
try {
256+
runSafely(this::writePending);
257+
} finally {
258+
// clear the flag before re-checking, otherwise a mutation landing between the drain and the
259+
// clear would see a flush still queued and be left with nobody to write it. In a finally so
260+
// an unexpected throw can't leave the flag set and stop persistence for the whole process.
261+
hasPendingFlush.set(false);
262+
if (!pendingWrites.isEmpty() || !pendingBreadcrumbs.isEmpty()) {
263+
requestFlush();
264+
}
265+
}
266+
}
267+
268+
private void writePending() {
269+
// ConcurrentHashMap's iterator is weakly consistent, so removing while iterating is safe. Keys
270+
// added after iteration starts may be missed, but flush() re-checks and queues another write.
271+
// We remove through the map rather than the iterator: that is an atomic get-and-remove, so we
272+
// never drop a value written by a mutation racing with this loop.
273+
final @NotNull Iterator<String> fileNames = pendingWrites.keySet().iterator();
274+
while (fileNames.hasNext()) {
275+
final @NotNull String fileName = fileNames.next();
276+
final @Nullable Object entity = pendingWrites.remove(fileName);
277+
if (entity == null) {
278+
// removed by a concurrent flush
279+
continue;
280+
}
281+
if (entity == DELETE_MARKER) {
282+
delete(fileName);
283+
} else {
284+
store(entity, fileName);
285+
}
286+
}
287+
288+
boolean breadcrumbsChanged = false;
289+
final @NotNull ObjectQueue<Breadcrumb> queue = breadcrumbsQueue.getValue();
290+
291+
Object operation;
292+
while ((operation = pendingBreadcrumbs.poll()) != null) {
293+
try {
294+
if (operation == CLEAR_MARKER) {
295+
queue.clear();
296+
} else {
297+
queue.add((Breadcrumb) operation);
298+
}
299+
breadcrumbsChanged = true;
300+
} catch (IOException e) {
301+
options.getLogger().log(ERROR, "Failed to apply breadcrumb change to file queue", e);
302+
}
303+
}
304+
305+
if (breadcrumbsChanged) {
306+
try {
307+
// single fsync for the whole batch instead of one per breadcrumb
308+
queue.sync();
309+
} catch (IOException e) {
310+
options.getLogger().log(ERROR, "Failed to sync breadcrumbs file queue", e);
311+
}
243312
}
244313
}
245314

@@ -286,9 +355,15 @@ public static <T> void store(
286355
* I/O and should be called from a background thread.
287356
*/
288357
public void resetCache() {
358+
// NOTE: pending mutations are deliberately left alone. They only ever hold values from the
359+
// current process, which is exactly what this reset is clearing the way for; dropping them
360+
// would lose scope state set during init.
289361
// since it keeps a reference to the file and we cannot delete it, breadcrumbs we just clear
290362
try {
291-
breadcrumbsQueue.getValue().clear();
363+
final @NotNull ObjectQueue<Breadcrumb> queue = breadcrumbsQueue.getValue();
364+
queue.clear();
365+
// breadcrumbs use buffered writes, so make the clear durable explicitly
366+
queue.sync();
292367
} catch (IOException e) {
293368
options.getLogger().log(ERROR, "Failed to clear breadcrumbs from file queue", e);
294369
}

0 commit comments

Comments
 (0)