Skip to content

[Storage] STG105 Photon on by Default - #50477

Merged
browndav-msft merged 24 commits into
Azure:feature/storage/stg105basefrom
browndav-msft:stg105/photonOnByDefault
Oct 28, 2026
Merged

browndav-msft merged 24 commits into
Azure:feature/storage/stg105basefrom
browndav-msft:stg105/photonOnByDefault

Conversation

@browndav-msft

Copy link
Copy Markdown
Member

Summary

  • Changes the default Blob listing response format from XML to Apache Arrow.
  • Treats unset and AUTO formats as Arrow while preserving explicit XML.
  • Centralizes ListBlobsOptions copying and default application across sync/async flat and hierarchical listings.
  • Preserves options when pagination overrides the page size.
  • Adds request-level tests verifying the generated Accept header for all listing paths (null, unset, ARROW, AUTO, and XML`)
  • Removes the obsolete error-response-body deserialization test.

Why

Blob listing should use Apache Arrow by default.

Testing

  • Added 16 cases covering null, unset, ARROW, AUTO, and XML` across sync/async flat and hierarchical listings.
  • Ran targeted tests and Checkstyle successfully.

@github-actions github-actions Bot added the Storage Storage Service (Queues, Blobs, Files) label Sep 15, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
34 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@browndav-msft

Copy link
Copy Markdown
Member Author

Isabelle (@ibrandes) tests that are failing in CI right now aren't for the arrow enablement. Those have to do with issues with the service not supporting the features. So I don't know why this is happening. But this should be good now

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
34 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@browndav-msft

Copy link
Copy Markdown
Member Author

/azp run java - storage - ci

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@ibrandes

Copy link
Copy Markdown
Member

browndav-msft could you look into these recording failures? these don't seem similar to the key based auth error / smart access tier error that we've normally been getting in CI:

CPKNAsyncTests.containerListBlobsFlat expectation "assertNext" failed (expected: onNext(); actual: onError(com.azure.storage.blob.models.BlobStorageException: Status code 409, "<?xml version="1.0" encoding="utf-8"?><Error><Code>EncryptionScopeNotAvailable</Code><Message>The given encryption scope is not available. (link)

CPKNAsyncTests.containerListBlobsHierarchical expectation "assertNext" failed (expected: onNext(); actual: onError(com.azure.storage.blob.models.BlobStorageException: Status code 409, "<?xml version="1.0" encoding="utf-8"?><Error><Code>EncryptionScopeNotAvailable</Code><Message>The given encryption scope is not available. (link)

ServiceAsyncApiTests.restoreContainerIntoOtherContainer expectation "assertNext" failed (expected: onNext(); actual: onError(java.lang.NullPointerException: The argument must not be null or an empty string. Argument name: deletedContainerVersion.)) (link)

VersioningAsyncTests.listBlobsWithVersion » Runtime Test proxy exception: Unable to find a record for the request PUT https://redacted/cb1326cc2listblobswithversiondbc697453214e98957084?restype=container (link)

^there were a couple of other tests in VersioningAsyncTests and VersioningTests that had the same error too

@ibrandes
Isabelle (ibrandes) requested a balanced review from Copilot October 1, 2026 01:05

@ibrandes Isabelle (ibrandes) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the changes themselves look good to me, but please look into the failing tests i linked in an earlier comment

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The default-format change needs an entry in the package's unreleased CHANGELOG before approval.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
What changed in this PR

Changes blob listings to prefer Apache Arrow while retaining explicit XML support.

Changes:

  • Defaults unset and AUTO formats to Arrow.
  • Centralizes option copying/default resolution across listing paths.
  • Adds sync/async request-header coverage and updates recordings.
File Description
ContainerAsyncApiTests.java Updates async default-format test naming.
ContainerApiTests.java Updates sync default-format test naming.
BlobErrorDeserializationTests.java Removes obsolete error deserialization test.
BlobContainerClientTests.java Adds sync Accept header tests.
BlobContainerAsyncClientTests.java Adds async Accept header tests.
ModelHelper.java Defaults to Arrow and centralizes option handling.
BlobContainerClient.java Applies resolved options to sync listings.
BlobContainerAsyncClient.java Applies resolved options to async listings.
assets.json Updates test recording assets.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Remove explicit XML overrides added during recording restoration. Preserve the restored assets, container deletion fix, and hierarchy request parameters. Existing XML recordings are intentionally not updated; this isolated experiment is intended for live CI and can be reverted afterward.
@browndav-msft

Copy link
Copy Markdown
Member Author

/azp run java - storage - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Default synchronous listings can exceed caller-supplied timeouts because streamed response bodies are consumed outside the timed operation.

0 open findings

1 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Medium severity Consume Arrow listing bodies within the request timeout

sdk/​storage/​azure-storage-blob/​src/​main/​java/​com/​azure/​storage/​blob/​implementation/​util/​ModelHelper.java:96

Making Arrow the default exposes a timeout regression for synchronous listBlobs and listBlobsByHierarchy calls with unset or AUTO formats. Both Arrow branches in BlobContainerClient obtain an InputStream inside the timed StorageImplUtils.sendRequest call, but consume it afterward (lines 1057–1081 and 1252–1274). With OkHttp, that stream reads directly from the response body, so prompt headers followed by a slow body can exceed the caller's timeout—even when the service returns XML. The previous XML path deserialized the body inside the timed operation. Move body consumption, parsing, and stream closure into the timed per-page operation in both branches, and add delayed-body timeout tests for flat and hierarchical listings, including XML fallback.

🧠 Review effort: Balanced

Recover the 16 FileApiTests and FileAsyncApiTests listAllRanges recordings from the original feature-branch assets tag 0f234e8e09. Publish them alongside existing File Share fixtures as tag 74b3ffac97 without changing test code or existing recordings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Compatibility of the new default for tier-sensitive listings still needs successful-response validation.

0 open findings

Previously missed (2)

In code that hasn't changed since last review

Low severity Add unset/AUTO listing tests for tier properties

sdk/​storage/​azure-storage-blob/​src/​main/​java/​com/​azure/​storage/​blob/​implementation/​util/​ModelHelper.java:96

The SMART-tier, SMART-rehydration, and premium page-blob tier listing tests now explicitly select XML, so they no longer exercise the default used by existing callers. The new request tests return HTTP 500 and cannot verify returned properties. Please retain the XML cases and add successful unset/AUTO cases in the sync and async tests, asserting the expected access tier, smart access tier, and archive status. Otherwise, regressions in these properties on the new default path can go undetected.

Low severity Test endBefore preservation with overridden page size

sdk/​storage/​azure-storage-blob/​src/​main/​java/​com/​azure/​storage/​blob/​implementation/​util/​ModelHelper.java:705

The existing endBefore tests only check rejection with explicit XML, and the new header tests leave it unset. A regression that drops this bound during a page-size override would therefore pass those tests. Please add request-level cases for unset/AUTO formats across sync/async flat and hierarchical listings. Assert that the request preserves endBefore, uses the overridden page size, and leaves the caller's options unchanged.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new default exposes synchronous listings to streaming body reads outside the caller’s timeout.

1 open finding
Previously missed (2)

In code that hasn't changed since last review

Medium severity Add pagination override tests for filters, details, and option immutability

sdk/​storage/​azure-storage-blob/​src/​main/​java/​com/​azure/​storage/​blob/​implementation/​util/​ModelHelper.java:706

The new request tests populate only the serialization format and assert Accept. They would still pass if this copier dropped prefix, startFrom, endBefore, or listing details when pagination overrides the page size. Add sync/async flat and hierarchical request cases with populated filters and an options page size different from the override. Assert the outgoing query parameters and that the caller's options remain unchanged, including a positive endBefore case rather than only XML-validation failures.

Medium severity Test SMART-tier properties on default and Arrow response paths

sdk/​storage/​azure-storage-blob/​src/​test/​java/​com/​azure/​storage/​blob/​specialized/​BlobBaseApiTests.java:897

Both SMART-tier listing suites now force XML, so their assertions no longer verify getSmartAccessTier() or REHYDRATE_PENDING_TO_SMART through the new default response path. The new header tests return 500 before deserialization, and the Arrow decoder tests do not assert these properties. Keep the XML cases and add successful default/Arrow response cases for both sync and async clients that assert these properties, so enabling Arrow by default does not leave this behavior untested.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

@browndav-msft

Copy link
Copy Markdown
Member Author

/azp run java - storage - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@ibrandes Isabelle (ibrandes) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good implementation wise - just some comments about the tests.

@browndav-msft

Copy link
Copy Markdown
Member Author

/azp run java - storage - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@ibrandes Isabelle (ibrandes) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me! let's wait to merge this in until after the API view meeting though, in case they request any changes.

@ibrandes Isabelle (ibrandes) changed the title Stg105/photon on by default [Storage] STG105 Photon on by Default Oct 22, 2026
@browndav-msft
browndav-msft merged commit 6cd984d into Azure:feature/storage/stg105base Oct 28, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Storage Storage Service (Queues, Blobs, Files)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants