Repository navigation
fix: trim the persisted queue in one pass - #225
Open
marian-bitstack wants to merge 1 commit into
Open
marian-bitstack wants to merge 1 commit into
marian-bitstack wants to merge 1 commit into
Conversation
When uploads keep failing, every queue change re-encoded the whole queue and, past 512 KB, _trimQueueToTargetMap re-encoded it again after each removed event, which is quadratic and runs on the UI isolate. encodeQueue encodes events newest first, each at most once, and stops once the limit is passed, so a write costs at most ~512 KB of encoding regardless of queue length. Same 512 KB / 475 KB thresholds as before. Fixes segmentio#199
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #199.
Problem
When uploads keep failing (offline, captive portal, a VPN or DNS filter blocking the API host), the event queue grows past the 512 KB file cap added in #188. From then on, every queue change runs on the UI isolate:
_writeFileencodes the whole queue to check its size._trimQueueToTargetMapencodes it again, then removes the oldest event and re-encodes the entire remaining queue after each removal until it fits under 475 KB. That is quadratic in the number of events dropped.Only the copy written to disk is trimmed, so the in-memory queue stays above the cap and this repeats on every write until a flush succeeds. In a production app, a profile on an iPhone 17 Pro Max under Network Link Conditioner showed
_writeFile/setPersisted/_trimQueueToTargetMapat ~35 % / ~34 % / ~29 % of main-isolate samples, with visible freezes of 0.5–1.3 s.Fix
io.dart).encodeQueuewalks the queue newest first, encodes each event at most once, tracks the byte size, and stops as soon as the 512 KB limit is passed. If the queue fits, it is written unchanged; otherwise the newest events that fit in 475 KB are kept, exactly as before. The output is assembled from the already-encoded events, so there is no extra full encode. A write now costs at most ~512 KB of encoding regardless of how long the in-memory queue is. The now-unreachable "still too big, write an empty queue" fallback is removed; an event larger than the limit on its own still results in an empty queue.QueueState.toJsonand 2–3 ms inencodeQueue. Capping the in-memory queue would drop events that could still be uploaded if the network recovers before the app is killed, so it is left for a separate discussion.Write coalescing is left out:
PersistedStatealready coalesces updates while a write is in flight, and changing write scheduling would grow this PR.Tests
test/utils/store/io_test.dart: an under-limit queue is encoded unchanged; an over-limit queue is trimmed to ≤ 475 KB, keeps a contiguous run of the newest events, and one more event would not have fit; with 20,000 events, each event'stoJsonis called at most once and fewer than 600 are encoded at all (the previous implementation encoded each surviving event dozens of times); a single oversized event yields an empty queue.flutter analyze(no new issues) andflutter testpass inpackages/core.