Repository navigation
[Core] Size FluxUtil.toFluxByteBuffer buffers to the data read - #51105
Merged
Isabelle (ibrandes) merged 4 commits intoNov 10, 2026
Merged
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 33 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
6 tasks done
Isabelle (ibrandes)
marked this pull request as ready for review
November 5, 2026 20:03
Isabelle (ibrandes)
requested review from
a team,
Jonathan Giles (JonathanGiles),
Alan Zimmer (alzimmermsft) and
vcolin7
as code owners
November 5, 2026 20:03
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the memory-retention fix in FluxUtil.toFluxByteBuffer.
No open issues 🎉
What changed in this PR
This PR fixes excessive memory retention in Azure Core’s FluxUtil.toFluxByteBuffer, addressing storage upload failures while preserving early emission of partial reads.
Changes:
- Reuses one read array per subscription and copies emitted data into exact-size buffers.
- Adds tests for partial-read sizing, mixed read sizes, and independent backing arrays.
- Documents the memory fix in the changelog.
| File | Description |
|---|---|
sdk/core/azure-core/src/test/java/com/azure/core/util/FluxUtilTest.java |
Adds buffer-sizing and data-independence regression tests. |
sdk/core/azure-core/src/main/java/com/azure/core/util/FluxUtil.java |
Reuses the read array and emits exact-size copies. |
sdk/core/azure-core/CHANGELOG.md |
Records the memory-retention and upload failure fix. |
Review effort: Balanced
Alan Zimmer (alzimmermsft)
approved these changes
Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
#50837 (shipped in azure-core 1.61.0) changed
FluxUtil.toFluxByteBuffer(InputStream, int)to emit a buffer as soon as a read returns andavailable()reports nothing more, instead of waiting to fillchunkSize. Each emitted buffer still wrapped a newly allocatedchunkSizearray, so any consumer that holds onto emitted buffers keeps a fullchunkSizearray alive for each partial read.Storage is one such consumer. When an
InputStreamis uploaded without a length (e.g.BlobClient.uploadWithResponse(new BlobParallelUploadOptions(stream), ...)), storage converts it with a 64 MiBchunkSizeand buffers the result up tomaxSingleUploadSizeto decide between a single-shot and a chunked upload. For a stream returning 1 MiB reads, every 1 MiB of data kept a 64 MiB array alive, a 64× amplification. This has caused theOutOfMemoryErrorfailures in the storage live test pipeline since storage moved to core 1.61.0 (#51010). A heap dump from one of those runs showed a single storage upload buffer object (PayloadSizeGate) holding 7.78 GB (99% of the 8 GB heap) for about 116 MiB of actual data, duringLargeBlobTests.uploadLargeInputSyncNoLengthGiven.Changes
FluxUtil.toFluxByteBuffer(InputStream, int): each subscription allocates onechunkSizearray (insideFlux.defer) and reuses it for every read. Each emission is an exact-size copy of the bytes read (Arrays.copyOf(buffer, offset)).chunkSizearray per subscription. Before this change, every read allocated a newchunkSizearray.chunkSizearray on every read.FileInputStreammemory-mapping path is unchanged.FluxUtilTestcoverage:Validation
mvn -f sdk/core/azure-core/pom.xml test checkstyle:check "-Dtest=FluxUtilTest" "-Djacoco.skip" "-Drevapi.skip" "-Dspotbugs.skip" "-Dcodesnippet.skip" "-Denforcer.skip": passed. 49 tests run, 0 failures, 1 skipped; 0 Checkstyle violations.End-to-end storage repro. A standalone harness reproduces
LargeBlobTests.uploadLargeInputSyncNoLengthGivenagainst Azurite: a 2,501 MiBSequenceInputStreamof 1 MiB parts, uploaded with no length and a 2,500 MiB block size. It uses storage built from source with-Xmx8g, the same heap CI uses. Live heap is the peak after GC, taken from GC logs.OutOfMemoryErrorafter about 4 sAbout 2.5 GB of live heap is expected, since the upload must hold the whole 2,500 MiB block. The extra upload time compared with 1.60.0 comes from [azure-core] Fix partial InputStream response buffering #50837 emitting 2,501 buffers of 1 MiB instead of 40 of 64 MiB. It isn't caused by this change: see the next bullet. The 1.60.0 run had to use netty 1.16.8, so its timing isn't strictly comparable.
toFluxByteBufferconversion only. Converting the same 2.5 GB stream with a 64 MiBchunkSizetook about 260 ms on 1.60.0 (40 buffers), about 5,000 ms on 1.61.0 (2,501 buffers), and about 260 ms with this PR (2,501 buffers).