Skip to content

[Storage] STG105 Base Branch - #50998

Draft
Isabelle (ibrandes) wants to merge 21 commits into
mainfrom
feature/storage/stg105base
Draft

Isabelle (ibrandes) wants to merge 21 commits into
mainfrom
feature/storage/stg105base

Conversation

@ibrandes

Copy link
Copy Markdown
Member

No description provided.

browndav-msft and others added 14 commits July 9, 2026 20:26
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
* Storage - STG105 NFS List V2 Files and Directories

Restores the NFS List V2 Files and Directories feature (#49246), which was
reverted from stg104 in #49770, rebased onto stg105base.

Handwritten files match #49246 exactly. Generated code was produced by
running autorest against the swagger link from Azure/azure-rest-api-specs#42720
already configured in the swagger README.

The only handwritten deviations from #49246 are those forced by this base:
- DirectoryListingDeserializationTests uses V2027_03_07; V2026_12_06 does
  not exist in ShareServiceVersion on stg105base.
- Dropped the ArrayList import in the two ShareDirectory clients; it is
  unused here and fails checkstyle.
- ModelHelper omits toShareFileRangeItems and its imports. That helper is
  STG103 paginated Get Range work, which is not on this base and is
  unrelated to this feature.

Recordings are not included. assets.json stays at the stg105base tag until
the feature is enabled in canary and the tests can be re-recorded.

* revert revert of nfsv2

* add recordings

* fix javadoc tag

---------

Co-authored-by: browndav <browndav@microsoft.com>
…50775)

* Revert "Storage - STG105 NFS List V2 Files and Directories (#50036)"

This reverts commit 83dc6e2.

* add recordings for prev features that were missing
… Shares (#50652)

* autorest

* finish feature, push assets

* Fix share creation time in list responses and add coverage

* Restore missing listAllRanges recordings in file share assets

* spotless apply

* changelog entry and shareproperties serialization / deserialization schema name update

* Revert "Storage - STG105 NFS List V2 Files and Directories (#50036)"

This reverts commit 83dc6e2.

* add recordings for prev features that were missing

* Update File Share Swagger spec reference

---------

Co-authored-by: Isabelle <ibrandes@microsoft.com>
* Restore missing listAllRanges recordings in file share assets

* autorest

* wire up clients, options, and responses

* spotless apply

* change primitives to wrapped reference types

* fix spelling issues

* remove model tests, add live tests, push assets

* change all docs to say 2026-06-06 for change feed

* spotless apply

* changelog update

* adding more documentation to retention getters and setters

* Revert "Storage - STG105 NFS List V2 Files and Directories (#50036)"

This reverts commit 83dc6e2.

* add recordings for prev features that were missing

---------

Co-authored-by: Isabelle <ibrandes@microsoft.com>
* Regenerate azure-storage-file-share for STG105 Files by ID

* first pass with harness

* Regenerate File Share property-by-ID APIs from updated Swagger

* Keep Files-by-ID timestamp handling consistent with existing SDK overrides

Extend the Files-by-ID response timestamp directive to Get Hard Links and regenerate its headers. ISO-8601 SMB creation, last-write, and change timestamps cannot be parsed by the RFC 1123 deserializer selected by the shared Swagger.

Research: the pre-existing Java swagger/README.md already overrides these response headers to date-time for directory create/get/set/rename and file create/get/download/set/rename/link operations. Generated FilesGetPropertiesHeaders and DirectoriesCreateHeaders use OffsetDateTime.parse, as does the hand-written FileSmbProperties header mapping. Python's azure-storage-file-share/swagger/README.md instead overrides all three response headers to strings to preserve timestamp precision.

Keep the shared Microsoft.FileStorage/stable/2027-03-07/file.json unchanged: SDKs already apply language-specific handling, and changing existing shared definitions would broaden this feature's compatibility impact. Scope the new Java directives to file/directory properties by ID and Get Hard Links. Preserve previous operations and the RFC 1123 handling of HTTP Date, Last-Modified, and copy-completion timestamps. Generated Java was produced by AutoRest, not manually edited.

The local Files-by-ID mock regressions cover fractional-second timestamps and sync/async file and directory property calls. The hard-link cases exposed the same parser mismatch and now pass with the regenerated model.

* Use generated APIs for file and directory properties by ID

Replace the hand-written FileIdOperations REST client with generated
operations in both synchronous and asynchronous clients.

Centralize file-ID validation and addressing-mode guards in ShareErrors.
Reuse property response mapping across path- and ID-addressed requests
while preserving existing path-based behavior.

Add regression coverage for timestamp and metadata mapping, service
errors, and addressing-mode restrictions.

* remove directory by id through swagger update

* change time changes from loop to hardcode in swagger doc

* Remove directory-by-ID support from File Share clients

* Regenerate File Share with explicit timestamp overrides

* Reuse file property headers for ID-based requests

* remove cosmetic change in ModelHelper

* remove change for utf8 encoding for quotes

* spotless apply

* add live tests

* add live tests and recordings

* add getDirectoryByFileId, add customization to avoid file-by-id variants for getproperties

* add tests with recordings

* add recordings for prev features that were missing

* move check to validatePathOperation to encapsulate teh policy

* spotless apply

* restrict more paths when using file id

* move unsupported operations tests into a separate file

* remove redundant cases

* update javadocs to reflect smb vs file operations

* Revert "Storage - STG105 NFS List V2 Files and Directories (#50036)"

This reverts commit 83dc6e2.

* add recordings for prev features that were missing

* remove public apis for sharefileasync  client and sync client

* Update File Share Swagger input

* fix sharefileasyncclient convenience method

* update filesimpl to reflect that hardlinks are smb only

* add changelog entry

* add error logging for ShareError in othe rplaces in code

* url encode fileId

* change null and empty checks to use code from common package

* spotless apply

* add liveonly tests

* block creation of lease clients with file-id based clients

* add remote swagger link

* preserveConfiguredFileID

---------

Co-authored-by: Isabelle <ibrandes@microsoft.com>
* Bifrostbase/create session (#50157)

* Feature/storage/bifrost create session (#49448)

* generate base files based on swagger docs

* create live tests for createSession

- downgrade blobserviceversion to 2026_04_06
- change AZURE_LIVE_TEST_SERVICE_VERSION to V2026_04_06 in ci.system.properties in azure-storage-common
- create both sync and async

* add recordings

* create new files based on swagger update

* add two params to BlobContainerClient#createSessionWithResponse

* add sanitizers for SessionToken and SessionKey to BlobTestBase

* add recording for createSessionReturnsTokenAndKey

* create StorageSessionCredential with isExpired

* create BlobSessionClient so that BlobSessionProvider takes it as a dep instead of ContainersImpl

* create BlobSEssionClient with tests

* add recorings for BlobSessionClient

* fix BlobContainerAsyncClient to match new swagger, add new recording

* add SessionProvider and SessionProviderTest

* add accountName to BlobSessionClient

* add accountName to StorageSessionCredential and SesionTestHelper

* wip

* change sessionprovider to SEssionTokenCredentialPolicy

* wip

* move session tests from containerapi to blobsessionclienttests

* fix blobsessiontests and add place holder for end-to-end tests in containerapitests

* add recordings for blobsessionclient

* linting

* refactor cache into separate class so it follows BearerTokenAuthenticationPolicy + AccessTokenCache pattern

* add 503 fallback

* add tests for udsas, but disabled for now

* refactor createContext to use hardcoded endpoint

* add SessionMode and tests for SessionMode

* add sessionOptions to buildPipeline, add null to builders not using sessions

* move SessionTokenCredentialPolicy ahead of StorageBearerTokenChallengeAuthorizationPolicy

* fix linting issues

* add session to BlobServiceClients and BlobServiceClientBuildeer

* change expiration so that it defaults to 5 minutes, if there is no expiration

* move SessionOptions so that it is public

* remove old SessionOptions

* remove unnecessary references to containerName and serviceVersion

* add BlobContainerSessionInfo, add other Copilot recommendations

* delete BlobContainerSessionInfo, restore return CreateSessionResponse

* create createSession end-to-end test with recordings

* only allow get requests for getblob

* wrap tests in try-with-resources

* make createSession package private

* fixes based on copilot suggestions

* Update sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/blob/implementation/util/BlobSessionClient.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* add containerName to SessionOptions

* move accountName to SessionOptions

* refactor: SessionTokenCredentialPolicy accepts bearer policy as constructor dependency

SessionTokenCredentialPolicy now takes StorageBearerTokenChallengeAuthorizationPolicy
as a constructor dependency instead of relying on pipeline ordering to coordinate
with the bearer policy. Key changes:

- SessionTokenCredentialPolicy delegates to bearerPolicy.process() for non-session
  requests (non-GetBlob, NONE mode, AUTO first request) and fallback (503).
- Added getBearerPolicy() accessor for use in per-container pipeline construction.
- BuilderHelper updated to pass bearer policy through to SessionTokenCredentialPolicy
  constructor in both addSessionPolicyIfEnabled and wrapWithSessionPolicy.
- Tests updated to mock bearerPolicy and verify delegation behavior.

This is step 1 of the session auth refactor to align with the .NET SDK's
SessionAuthenticationPolicy pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor: introduce AuthStrategy enum and consolidate analyzeRequest

Consolidate isGetBlobRequest() and shouldUseSession() into a single
analyzeRequest() method that returns an AuthStrategy enum
(USE_BEARER_TOKEN or USE_SESSION_TOKEN), following the .NET
SessionAuthenticationPolicy pattern.

Also extract response handling into handleSessionResponse() and
handleSessionResponseSync() methods for cleaner process()/processSync()
flow. The process methods now have a clear structure:
1. analyzeRequest() -> decide strategy
2. USE_BEARER_TOKEN -> delegate to bearer policy
3. USE_SESSION_TOKEN -> acquire session, sign, send, handle response

No behavioral changes - all existing tests pass unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor: remove redundant restype check from analyzeRequest

The restype query parameter check was redundant because requests with
restype=container are container-level operations that have no blob name.
They are already filtered by the blob name presence check. This aligns
more closely with the .NET SessionAuthenticationPolicy which relies on
URL structure rather than query parameter checks.

The comp check is retained as a safety measure to exclude sub-operations
like GetBlobMetadata (comp=metadata) that have a blob name but are not
download operations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* change sessionmode from always to singlespecifciedcontainer, add resolve method

* wrap bearer token in sessioncredentialpolicy

* fix NPE for SessionOptions, sessionoptions always non null

* add tests for sessiontokencredpolicy and storagesessioncred

* add logic to avoid wrapping Bearertoken, if session is not needed

* add overloaded oauth in blobtestbase to be able to add sessionoptions

* add overloaded getOAuthServiceAsyncClient to be able to pass session options

* add custom buildStringToSign to remove `0` from get requests

* readd versions

* readd ci.system.properties

* change session options check for null in BuilderHelper which affected other tests

* add recordings for create sessions, change time to testResource time

* add requestInspectionPolicy and overloaded getoauth client in base test

* fix null sessionsoptions issue

* add fix in storagesessioncredntial for query params

* add SessionTokenCredPolicy to checks for anonymousAccess

* remove constructor for BlobSessionClients that uses parse url

* linting issues, remove SessionOptions from service methods

* add comments to policyrefreshNearExpiry test

* fix linting issues

* add check for container name

* fix return javadoc for SessionMode

* add LOGGER and appropriate error throwing in BlobSessionClient

* changes based on feedback from isabelle

* add single retry for all 401 errors

* remove unused imports

* add suppression for SessionTokenPolicy for linting

* fix ubuntu tests hanging by removing local dns bypass

* add session for blob client with tests

* create unified http transport between data requests and session request, add ResourceLock for live tests

* test multiple concurrency

* add branching if for bearer policy

* refactor validateSessionMode to builderhelper

* add revert to bearertoken for 400 errors

* remove unused imports

* readd comment to bearertokechallenge

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix bug in storage session credential with get content length 0

* patch to enable session with environement variables

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* add code to ensure container name only accepted when session enabled

* fix formatting and javadocs alignment

* add forceRefreshSessionInBackground() to SessionTokenCredentialPolicy

* remove .env initialization for sessions

* change SessionCredentialCache to AutoRefreshingCache<T>

* rename params for SessionTokenCrednetialPolicy

* add autorefreshcasche specific tests for blob session credential

* wip

* Session provider lifecycle refactor: expand public contract, move caching into BlobSessionClient

- Expand SessionProvider with default invalidateSession/refreshSession lifecycle methods
- Move AutoRefreshingCache per-container caching from deleted SessionCredentialManager into BlobSessionClient
- Extract account cooldown into narrow SessionAcquisitionCooldown helper
- SessionTokenCredentialPolicy depends directly on SessionProvider; no manager wrapper
- BYO custom providers are used directly without SDK-owned cache wrapping
- Remove SessionOptions.orDefault (internal helper made public accidentally)
- Update CHANGELOG and SessionOptions JavaDoc to reflect new BYO provider ownership model
- Add BlobSessionClientCacheTest, SessionAcquisitionCooldownTest, update SessionTokenCredentialPolicyTest

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* remove sessionacquisitioncooldown, remove log entry

* remove SessionRequestSigner, rename SessionClient to SessionProvider

* rename SessionProviderSeamTests to SessionProviderTests

* create helper method for session provider inside BuilderHelper

* refactor beginAccountCooldown

* inline error codes, make async code easier to read

* rename BlobSessionProvider to TokenCredentialSessionProvider

* consolidate testing resources in session package in common, improve session tests

* rework SessionTokenCredentialPolicyTest to remove warnings

* remove normal and trims from TokenCredentialSessionProviderCacheTests

* remove unnecessary tests in BuilderHelper

* add recordings for tests

* fix linting

* consolidate session mocks to scriptedHttpClient, remove createSessionMockHttpClient

* refactor builderhelper test to test customer provided SessionProvider

* clean up javadoc for sessionprovider, remove unneeded returns for ScriptedHttpClient

* add support for dfs hns

* create live tests for datalake

* add public accessors to session methods

* fix failing compilations, update swagger to use 105

* merge in recordings from create session

* remove container level createSession calls from public api, fix tests for Autorefcache

* undo spacing from builder helper

* fix builder helper so that diff shows appropriately for createBearerPipeline

* fix bytebuddy testing dependency in pom

* Share post-authentication pipeline policies with the CreateSession pipeline

Session credentials are bound to the network context of the call that created
them, so CreateSession must take the same network path as the data requests its
credential signs. Hoist the per-retry, after-retry, response validation, logging
and ScrubEtag policies into a postAuthenticationPolicies list shared by the data
pipeline and the bearer-only CreateSession pipeline. Main pipeline ordering is
unchanged.

* Cool down accounts that repeatedly reject session credentials

Invalidating a rejected session makes the next request create a brand new one,
so an environment where sessions can never be used would mint and lose a session
on every request indefinitely. Count consecutive 401s per account and, after
three, place the account into the existing five minute cooldown so requests fall
straight through to bearer. Any non-401 response clears the count.

* Sanitize session secrets and mark DataLake session tests live-only

Add SessionToken and SessionKey body sanitizers so session secrets never reach
recordings, matching the pair already present in BlobTestBase.
readFileOverSessionAuth cannot be recorded at all: session credentials are bound
to the network context of the CreateSession call and the TLS-terminating test
proxy invalidates them, so mark it @liveonly instead of @disabled.

* De-duplicate pipeline construction in BuilderHelper

The HttpPipelineBuilder configuration was written out twice, and the bearer-only
CreateSession pipeline was threaded through an extra single-caller method that
re-resolved a service version its caller had already resolved. Extract
createPipeline for the shared transport, client options and tracer setup, and
fold buildBearerPipeline into createDefaultSessionProvider. No behavior change.

* moved helps from SessionTestHelper to BlobTestBase

* Throttle background refresh retries after a failed session creation

A failed background refresh left nextRefreshTime in the past, so isRefreshDue stayed true and the next getValidValue call fired another attempt. Because the session policy resolves a credential per request, a persistently failing CreateSession produced one attempt per request for the credential's entire remaining life. The existing 401 cooldown in SessionTokenCredentialPolicy does not cover this: it counts rejections of a credential, not failures to mint one.

Arm a thirty second backoff on failure, cleared on any successful creation, and check it in refreshValueInBackground. That is the single choke point, so it also gates forceRefreshValueInBackground. This is deliberate: a failing service still returns the session_expiring hint on every response, so a force path that bypassed the backoff would reproduce the same storm.

The foreground path stays unthrottled. Once the cached value expires, callers still create unconditionally and receive either a value or a real error, never a silently throttled stall.

* Pin the reentrant forced-refresh behavior of AutoRefreshingCache

setActiveValue clears the in-flight creation marker eagerly rather than relying on doFinally, because a downstream onNext handler reacting to the session_expiring hint runs before the creation Mono terminates and must be able to start a fresh creation. That line had no test covering it.

Add a characterization test that forces a refresh from inside a downstream onNext and asserts a second creation is started. Verified to fail if the eager clear is removed.

* Fix reentrant forced-refresh behavior for joiners

* Remove redundant session credential shadow-copy from ContainerSessionCache

* Refine session test HTTP transports

* Localize the CreateSession test transport

* Observe session auth requests with the wire tap client

* Align session auth javadocs with the cooldown implementation

The session docs described the account back-off and bearer fallback
rules in ways the policy does not implement:

- SessionProvider named only a 400/403/5xx acquisition failure as a
  trigger, omitting three consecutive HTTP 401 rejections, and framed
  bearer fallback as applying to ineligible requests only.
- SessionMode claimed session failures fall back to bearer for any
  reason; the policy retries with bearer on 400 and 401 only, and its
  idle-session behavior belongs to the built-in provider.
- Neither noted that back-off state lives on the client pipeline, so
  clients sharing a provider instance do not share it.

Replace the "account-level acquisition cooldown" phrasing with a plain
description of the behavior under a new Division of responsibility
heading, and document handleSessionAcquisitionFailure to match its
sibling handleSessionRejection.

* add session options to all clients

* Remove unused session options field

* Remove Mockito dependency from storage common tests

* add tests for full expiration test

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com>

* run autorest

* create autorefreshingcache branch

* change ValueProvider to Supplier

* create autorefreshingcache branch

* change ValueProvider to Supplier

* fix compilation error

* adjust autorefreshingcache logic to match dotnet

* change generic names in autorefreshing cache

* add 30 second timeout for refreshes

* Simplify autofresh cache readability

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* change names in autorefreshcache to make it easier to read funcitons and vars

* remove unnecessary tests in AutoRefreshingCacheTests

* Simplify cache test suppliers and share per-test setup

* Remove duplicate import introduced by cache test merge

* shorten javadocs, change syncronized to locks

* shorten javadocs, change sync to implement async + block()

* rename SessionProvider implementation and session auth policy files

* refactor so Session Provider handles eligibility for sessions

* remove containerName from options

* make accountName optional, create isServiceEndpoint parser in common package

* update error codes to match onenote

* change account cooldowns to container cooldowns

* remove x-ms-date header on fallback to hearer

* make sessions off by default

* fix linting issues

* fix linting issues

* fix linting issues

* change session tests to test for sessions off by default

* move SessionAuthPolicy from implementation to new policy package

* fix linting issues

* update recordings to match change test names

* remove unsused containerName var from SessionOptions

* update datalake to check for sessions off by default, add sessions to sessions-enabled tests

* make sessions off by default, update documentation and tests

* nest SessionMode enum in sessionOptions

* docs: correct session authentication comments and JavaDoc

* update SessionAuthPolicyTest to update test names and comments

* address github copilot comments

* change sessionOptions in policy to snapshot sessionMode and accountName

* spotless apply

* Restore missing File Share range-listing recordings

Update the File Share assets tag to 74b3ffac97, which restores the 16 FileApiTests and FileAsyncApiTests listAllRanges recordings from the original feature branch. Preserve all existing recordings and test code.

* clear authentication policy containerCooldowns hashmap after 1 minute

* remove retry marker

* create custom logging policy for session to prevent creds from printing in logs

* remove live tokens from logging during live runs

* Explore package-private session authentication policy

Move SessionAuthenticationPolicy and its tests into implementation.util,
make the class and constructors package-private, and remove the public
policy package export. Delegate the sync/async builders' type checks to
BuilderHelper so the concrete policy is not exposed outside its package.

Customers configure sessions through SessionOptions and SessionProvider;
constructing the authentication policy is not an intended customer scenario.

Blob and Data Lake compile and 84 targeted tests pass, but this experiment
intentionally records the following unresolved Checkstyle failures at
SessionAuthenticationPolicy.java:57:1 (messages reproduced verbatim):

Class 'SessionAuthenticationPolicy' implementing 'HttpPipelinePolicy' and should be a public class
Class 'SessionAuthenticationPolicy' implementing 'HttpPipelinePolicy' and should not be a implementation package or sub-package of it

Both failures come from:
io.clientcore.linting.extensions.checkstyle.checks.HttpPipelinePolicyCheck

Reproduce with:
mvn -f sdk\storage\pom.xml -pl com.azure:azure-storage-blob,com.azure:azure-storage-file-datalake -am checkstyle:check

The rule is declared at eng/lintingconfigs/checkstyle/track2/checkstyle.xml:363-366.
No Checkstyle configuration or suppressions have been changed. Whether to
allow an exception for this internal policy is an architecture-board
decision. Revert this experiment in a follow-up commit to retain the
proposal and its exact blocker in history without leaving the PR failing.

PR: #50575

* Revert package-private session policy pending architecture-board decision

This reverts commit 0f37737.

Restore the public policy package so PR #50575 continues to comply with
HttpPipelinePolicyCheck. The experimental commit preserves the internal
policy design and both exact Checkstyle failures for architecture review.

The policy is builder-managed, but allowing it to remain package-private
under implementation requires an architecture-board decision on an
exception to the policy-placement rule. No lint rules were overridden.

* update to always refresh signing

* remove isExpired from the public api

* change sessionprovider to enable public creation

* clear stale cache atomic

* move SessionAuthPolicy to implementation

* change isValueMissingOrExpired in cache to be inclusive of expiration time

* change accessor pattern iplementation for session provider

* spotless apply

* spotless apply

* add creds to cspell

* fix spelling

* Move SessionOptions to options package

* Remove unused imports from session provider snippet

* Revert "Storage - STG105 NFS List V2 Files and Directories (#50036)"

This reverts commit 83dc6e2.

* add recordings for prev features that were missing

* revert add creds to cspell.json

* create SessionOptions constructor to create SessionProvider internally

* Restore content validation upload recordings

* Rename container session client to provider

* Rename custom session provider factory

* sanitize logs in dfs for auth headers

* refactor session creation so that sessions shared between clients that use session options

* spotless apply

* wip session policy and cache split

* rename methods and variables to clarity

* make sessionrequestcontext immutable and package private

* change sessionoptions to sessioncontext

* remove extra parse on string to get blob parts

* consolidate accountName logic

* move sessioncontext from options package to parent package

* consolidate logic for checking SessionMode

* small change to ternary for clearer reading

* add changelog entry

* remove tokens from logs

* address O(n2) time for clearing caches

* remove stale caches logic, keep only expired tokens logic

* consolidate session tests, add context data to maintain sessions across clients

* restore logic for custom loggers

* create live tests for session sharing

* fix tests so tokens arent in logs

* clean up tests

* refactor blob/dfs url logic into common package

* add javadocs for sessioncontext, add changelog entry for datalake

* fix sessioncontext javadocs

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com>
Co-authored-by: Isabelle <ibrandes@microsoft.com>
@github-actions github-actions Bot added the Storage Storage Service (Queues, Blobs, Files) label Oct 27, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
35 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

browndav-msft and others added 7 commits October 28, 2026 11:15
* change default serialization format to arrow

* consolidate setters for photon, photon on by default

* add ARROW enum check

* change default tests to check for arrow and not xml

* rerecord failing tests so arrow returned and not xml in recordings

* recorded more failing tests for arrow

* change previous tests to check for xml explicitly, restore previous recordings

* restore recordings for failing tests, add xml explitly

* spotless apply

* Exercise default Arrow listing in restored tests for live CI

Remove explicit XML overrides added during recording restoration. Preserve the restored assets, container deletion fix, and hierarchy request parameters. Existing XML recordings are intentionally not updated; this isolated experiment is intended for live CI and can be reverted afterward.

* Revert "Exercise default Arrow listing in restored tests for live CI"

This reverts commit 9ae3cfc.

* Use new Arrow recordings for encryption-scope and version listings

Remove explicit XML overrides for the six cases with new Arrow responses. Keep XML for tests still using their existing XML recordings.

Replaced recordings:
- CPKNTests.containerListBlobsFlat.json
- CPKNTests.containerListBlobsHierarchical.json
- CPKNAsyncTests.containerListBlobsFlat.json
- CPKNAsyncTests.containerListBlobsHierarchical.json
- VersioningTests.listBlobsWithoutVersion.json
- VersioningAsyncTests.listBlobsWithoutVersion.json

Update blob assets tag from 6bde428684 to b0417a0e0b.

* Restore missing File Share range-listing recordings

Recover the 16 FileApiTests and FileAsyncApiTests listAllRanges recordings from the original feature-branch assets tag 0f234e8e09. Publish them alongside existing File Share fixtures as tag 74b3ffac97 without changing test code or existing recordings.

* spotless apply

* push updated recordings for failing tests

* retain xml header request for single failing test, setAttributesTier

* change hard asserts for HOT to what was reqeusted

* move tests from BlobContainerAsyncCLientTests to ContainerApiTests and sync variant

* add entry to changelog

* file share test recordings

* blob assets

* updating recordings

---------

Co-authored-by: Isabelle <ibrandes@microsoft.com>
* Feature/storage/datalocality private drop (#50136)

* Feature/storage/bifrost create session (#49448)

* generate base files based on swagger docs

* create live tests for createSession

- downgrade blobserviceversion to 2026_04_06
- change AZURE_LIVE_TEST_SERVICE_VERSION to V2026_04_06 in ci.system.properties in azure-storage-common
- create both sync and async

* add recordings

* create new files based on swagger update

* add two params to BlobContainerClient#createSessionWithResponse

* add sanitizers for SessionToken and SessionKey to BlobTestBase

* add recording for createSessionReturnsTokenAndKey

* create StorageSessionCredential with isExpired

* create BlobSessionClient so that BlobSessionProvider takes it as a dep instead of ContainersImpl

* create BlobSEssionClient with tests

* add recorings for BlobSessionClient

* fix BlobContainerAsyncClient to match new swagger, add new recording

* add SessionProvider and SessionProviderTest

* add accountName to BlobSessionClient

* add accountName to StorageSessionCredential and SesionTestHelper

* wip

* change sessionprovider to SEssionTokenCredentialPolicy

* wip

* move session tests from containerapi to blobsessionclienttests

* fix blobsessiontests and add place holder for end-to-end tests in containerapitests

* add recordings for blobsessionclient

* linting

* refactor cache into separate class so it follows BearerTokenAuthenticationPolicy + AccessTokenCache pattern

* add 503 fallback

* add tests for udsas, but disabled for now

* refactor createContext to use hardcoded endpoint

* add SessionMode and tests for SessionMode

* add sessionOptions to buildPipeline, add null to builders not using sessions

* move SessionTokenCredentialPolicy ahead of StorageBearerTokenChallengeAuthorizationPolicy

* fix linting issues

* add session to BlobServiceClients and BlobServiceClientBuildeer

* change expiration so that it defaults to 5 minutes, if there is no expiration

* move SessionOptions so that it is public

* remove old SessionOptions

* remove unnecessary references to containerName and serviceVersion

* add BlobContainerSessionInfo, add other Copilot recommendations

* delete BlobContainerSessionInfo, restore return CreateSessionResponse

* create createSession end-to-end test with recordings

* only allow get requests for getblob

* wrap tests in try-with-resources

* make createSession package private

* fixes based on copilot suggestions

* Update sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/blob/implementation/util/BlobSessionClient.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* add containerName to SessionOptions

* move accountName to SessionOptions

* refactor: SessionTokenCredentialPolicy accepts bearer policy as constructor dependency

SessionTokenCredentialPolicy now takes StorageBearerTokenChallengeAuthorizationPolicy
as a constructor dependency instead of relying on pipeline ordering to coordinate
with the bearer policy. Key changes:

- SessionTokenCredentialPolicy delegates to bearerPolicy.process() for non-session
  requests (non-GetBlob, NONE mode, AUTO first request) and fallback (503).
- Added getBearerPolicy() accessor for use in per-container pipeline construction.
- BuilderHelper updated to pass bearer policy through to SessionTokenCredentialPolicy
  constructor in both addSessionPolicyIfEnabled and wrapWithSessionPolicy.
- Tests updated to mock bearerPolicy and verify delegation behavior.

This is step 1 of the session auth refactor to align with the .NET SDK's
SessionAuthenticationPolicy pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor: introduce AuthStrategy enum and consolidate analyzeRequest

Consolidate isGetBlobRequest() and shouldUseSession() into a single
analyzeRequest() method that returns an AuthStrategy enum
(USE_BEARER_TOKEN or USE_SESSION_TOKEN), following the .NET
SessionAuthenticationPolicy pattern.

Also extract response handling into handleSessionResponse() and
handleSessionResponseSync() methods for cleaner process()/processSync()
flow. The process methods now have a clear structure:
1. analyzeRequest() -> decide strategy
2. USE_BEARER_TOKEN -> delegate to bearer policy
3. USE_SESSION_TOKEN -> acquire session, sign, send, handle response

No behavioral changes - all existing tests pass unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor: remove redundant restype check from analyzeRequest

The restype query parameter check was redundant because requests with
restype=container are container-level operations that have no blob name.
They are already filtered by the blob name presence check. This aligns
more closely with the .NET SessionAuthenticationPolicy which relies on
URL structure rather than query parameter checks.

The comp check is retained as a safety measure to exclude sub-operations
like GetBlobMetadata (comp=metadata) that have a blob name but are not
download operations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* change sessionmode from always to singlespecifciedcontainer, add resolve method

* wrap bearer token in sessioncredentialpolicy

* fix NPE for SessionOptions, sessionoptions always non null

* add tests for sessiontokencredpolicy and storagesessioncred

* add logic to avoid wrapping Bearertoken, if session is not needed

* add overloaded oauth in blobtestbase to be able to add sessionoptions

* add overloaded getOAuthServiceAsyncClient to be able to pass session options

* add custom buildStringToSign to remove `0` from get requests

* readd versions

* readd ci.system.properties

* change session options check for null in BuilderHelper which affected other tests

* add recordings for create sessions, change time to testResource time

* add requestInspectionPolicy and overloaded getoauth client in base test

* fix null sessionsoptions issue

* add fix in storagesessioncredntial for query params

* add SessionTokenCredPolicy to checks for anonymousAccess

* remove constructor for BlobSessionClients that uses parse url

* linting issues, remove SessionOptions from service methods

* add comments to policyrefreshNearExpiry test

* fix linting issues

* add check for container name

* fix return javadoc for SessionMode

* add LOGGER and appropriate error throwing in BlobSessionClient

* changes based on feedback from isabelle

* add single retry for all 401 errors

* remove unused imports

* add suppression for SessionTokenPolicy for linting

* fix ubuntu tests hanging by removing local dns bypass

* add session for blob client with tests

* create unified http transport between data requests and session request, add ResourceLock for live tests

* test multiple concurrency

* add branching if for bearer policy

* refactor validateSessionMode to builderhelper

* add revert to bearertoken for 400 errors

* remove unused imports

* readd comment to bearertokechallenge

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* add forceRefreshSessionInBackground() to SessionTokenCredentialPolicy

* remove .env initialization for sessions

* change SessionCredentialCache to AutoRefreshingCache<T>

* Regenerate azure-storage-blob from updated swagger spec for STG105 Data Locality

Points swagger/README.md at the Data Locality fork spec (nickliu-msft/azure-rest-api-specs@5c678e4, 2026-10-06/blob.json) matching .NET PR Azure/azure-sdk-for-net#57554, and runs autorest to regenerate the implementation layer: new GetLayout REST operation (comp=layout), BlobLayout/BlobLayoutRanges/BlobLayoutEndpoints models, BlobsGetLayoutHeaders, and the new x-ms-download-hint header on BlobsDownloadHeaders.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Implement Data Locality getLayout API for STG105 in azure-storage-blob

Adds the paged getLayout / getLayoutAsync API for BlobClient/BlobAsyncClient (via BlobClientBase/BlobAsyncClientBase), returning BlobLayoutInfo with the blob's layout ranges/endpoints plus standard properties. New public types: BlobLayoutInfo, BlobLayoutRange, BlobGetLayoutOptions, DownloadHint. Wires the new x-ms-download-hint response header into BlobDownloadHeaders. ModelHelper.transformBlobLayoutInfo bridges the generated BlobLayout/BlobsGetLayoutHeaders response into the public BlobLayoutInfo model, reusing the existing getObjectReplicationDestinationPolicyId/getObjectReplicationSourcePolicies helpers for parity with other blob property transforms.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add tests for getLayout API in azure-storage-blob (STG105)

Sync (BlobClientBaseGetLayoutApiTests) and async (BlobClientBaseGetLayoutAsyncApiTests) coverage for BlobClientBase/BlobAsyncClientBase.getLayout: basic success, empty blob, ranged request, page-size limiting, continuation token, request-condition success/failure (mirroring PageBlobApiTests listPageRanges AC/ACFail pattern via BlobTestBase allConditionsSupplier/allConditionsFailSupplier), and a not-found error case. All tests gated with RequiredServiceVersion min 2027-03-07. No recordings yet -- fail as expected in playback with recording does not exist, which is normal at this stage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add implementation notes for STG105 Data Locality getLayout API

Comprehensive explanation of what was built (getLayout paged API and DownloadHint signal, scoped to azure-storage-blob only), how it works (generated-layer wire protocol, hand-written model/client bridge, test coverage), and why key decisions were made (eager endpoint resolution in BlobLayoutRange, matching the existing listPageRanges pattern, deferring the data-locality routing infrastructure and DataLake module as follow-on work, using the unmerged fork spec to match the still-open .NET PR). Lists outstanding items: missing test recordings due to no credentials in this environment, the unmerged spec dependency, deferred routing/DataLake work, and a pending CHANGELOG entry.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Fix BlobLayoutInfo missing blob-content fields and async test bug

The generated BlobsGetLayoutHeaders exposes both generic (Content-Length, Content-Type, etc from the layout response body) and blob-specific (x-ms-blob-content-length, x-ms-blob-content-type, etc reflecting the blobs actual properties) header pairs, matching the .NET model split between BlobLayoutInfo.ContentLength and BlobLayoutInfo.BlobContentLength/BlobContentType/BlobContentEncoding/BlobContentMD5/BlobCreatedOn. The initial Java BlobLayoutInfo only mapped the generic set, so getContentLength() returned null against a real service response. Adds the missing blobContentLength/blobContentType/blobContentEncoding/blobContentMd5/blobCreatedOn fields end to end (BlobLayoutInfo, ModelHelper.transformBlobLayoutInfo) and updates tests to assert on the correct getBlobContentLength(). Also fixes a StepVerifier logic bug in getLayoutAC (async): a successful conditional request legitimately emits one item, so the assertion needs response.then() before verifyComplete() to only assert on completion, not zero emissions -- matching the pattern already used by getLayoutContinuationToken/getLayoutRange in the same file. Found and fixed by actually recording against a real (preprod) storage account with AZURE_LIVE_TEST_SERVICE_VERSION=V2026_02_06.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Fix BlobTestBase afterTest cleanup client ignoring AZURE_LIVE_TEST_SERVICE_VERSION

afterTest built its cleanup BlobServiceClient directly from BlobServiceClientBuilder (httpClient/credential/endpoint only), bypassing the existing getServiceClient(TestAccount) helper that beforeTest already uses for every other client in this class. That helper routes through getServiceClientBuilder -> StorageCommonTestUtils.instrument, which is the only place that reads the AZURE_LIVE_TEST_SERVICE_VERSION environment variable and applies it via builder.serviceVersion(...). As a result, the cleanup listBlobContainers call always used the SDK's default/latest service version regardless of any environment override, causing every test to fail during teardown when running against an account that only supports an older pinned version (discovered while recording STG105 getLayout tests against a preprod account limited to V2026_02_06 -- the actual test bodies passed, only cleanup failed). Fix: reuse getServiceClient(ENVIRONMENT.getPrimaryAccount()) like the rest of the class. Pre-existing bug, unrelated to the STG105 feature itself, but blocking for anyone using AZURE_LIVE_TEST_SERVICE_VERSION in this module.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add DataLocalityPolicy to azure-storage-common (STG105)

Java port of .NET's DataLocalityPolicy (Azure.Storage.Common/src/Shared/DataLocalityPolicy.cs): a no-op HttpPipelinePolicy unless a caller opts a request into locality-aware routing by setting DataLocalityPolicy.LAYOUT_ENDPOINT_KEY on the per-call Context to a non-empty endpoint URL. When set, rewrites the outgoing request's host/port to that endpoint (for routing to an alternate storage node returned by GetLayout) while preserving the original host as the Host header, so server-side virtual-hosting/TLS-SNI routing still works. Runs PER_RETRY (mirroring .NET's use on the raw HttpMessage before each send/retry). Malformed endpoint values are logged and skipped rather than failing the request.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add tests for DataLocalityPolicy (STG105)

Real HttpPipeline-based tests (no live service dependency) using a captured NoOpHttpClient and MockHttpResponse: no-op when context data is absent, rewrites host/port and preserves the original Host header when a layout endpoint is set, and no-op for an empty-string endpoint value.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add enableDataLocality option to BlobDownloadToFileOptions and BlobInputStreamOptions

Adds the EnableDataLocality-equivalent toggle (mirroring .NET's
BlobDownloadToOptions/BlobOpenReadOptions) so callers can opt in to
layout-aware routing for parallel file downloads and streamed reads.
Wiring of the flag into the download/read paths is tracked separately.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add BlobLayoutCacheValue and BlobLayoutRangeResolver for data locality routing

Mirrors .NET's BlobLayoutSegmentCacheValue (cache wrapper distinguishing
populated/empty/failed layout state) and the binary-search
GetLayoutEndpoint helper from PR Azure/azure-sdk-for-net#57554, adapted
to reuse the existing public BlobLayoutRange model (endpoint indices are
already resolved by ModelHelper.transformBlobLayoutInfo) and the
AutoRefreshingCache<T> added to azure-storage-common.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add tests for BlobLayoutCacheValue and BlobLayoutRangeResolver

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Wire DataLocalityPolicy and layout cache into Blob download paths

Registers DataLocalityPolicy in the Blob HTTP pipeline (previously
built but unregistered) and wires locality-aware routing into the two
download entry points, opt-in via BlobDownloadToFileOptions /
BlobInputStreamOptions.isEnableDataLocality():

- BlobAsyncClientBase.downloadToFileImpl: after the first chunk
  resolves, if enabled and the initial response's DownloadHint is
  LAYOUT, builds one AutoRefreshingCache<BlobLayoutCacheValue> per
  download (etag-locked, scoped to the remaining un-downloaded range)
  and resolves an endpoint per chunk via BlobLayoutRangeResolver,
  routing only chunks 1+ (chunk 0 already completed before the cache
  exists, matching the .NET PartitionedDownloader design).
- BlobClientBase.openInputStream / BlobInputStream.dispatchRead: same
  pattern, scoped to the full requested stream range since reads can
  seek to arbitrary offsets.
- Added a context-propagating internal getLayout(options, context)
  overload and fetchLayoutCacheValueAsync helper on
  BlobAsyncClientBase; soft-fails (caches a null/no-routing value) on
  BlobStorageException rather than failing the download.

Mirrors Azure/azure-sdk-for-net#57554's PartitionedDownloader.cs and
BlobBaseClient.DownloadStreamingInternal wiring. Default
(enableDataLocality=false) behavior is unchanged; verified via
BlobApiTests/BlobAsyncApiTests (738 tests) and
BlobClientBaseGetLayout*ApiTests (38 tests), all passing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add getLayout and data locality wiring for DataLake file client

Mirrors Azure/azure-sdk-for-net#57554's DataLakeFileClient.GetLayout
and locality-aware routing, taking advantage of DataLakeFileClient
already wrapping a BlockBlobClient internally:

- New DataLakeFileLayoutInfo/DataLakeFileLayoutRange public models and
  DataLakeFileGetLayoutOptions options bag (DataLake-domain shapes,
  field-mapping conventions matched to Transforms.toPathProperties).
- DataLakeFileClient/DataLakeFileAsyncClient.getLayout(...) proxies the
  already-implemented BlockBlobClient/BlockBlobAsyncClient getLayout
  API and maps BlobLayoutInfo -> DataLakeFileLayoutInfo via new
  Transforms helpers. Documented as an explicit proxy since DataLake
  does not yet have its own generated layout REST operation; revisit
  if/when one is added.
- Added enableDataLocality to ReadToFileOptions/
  DataLakeFileInputStreamOptions, threaded through to the underlying
  BlobDownloadToFileOptions/BlobInputStreamOptions. Since
  readToFileWithResponse/openInputStream already delegate to
  BlockBlobClient (which now has full locality routing wired in), this
  makes DataLake downloads/streams locality-aware with no additional
  chunking/routing logic needed on the DataLake side.

Default (enableDataLocality=false) behavior is unchanged. Verified via
FileApiTest/FileAsyncApiTests/FileInputStreamTests (827 tests), all
passing; compile/checkstyle/spotbugs clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add end-to-end tests for data locality download/stream wiring

Adds BlobDataLocalityDownloadApiTests covering enableDataLocality on
BlobDownloadToFileOptions/BlobInputStreamOptions: single-chunk and
multi-chunk downloadToFileWithResponse, openInputStream (full and
partial range), the enableDataLocality=false regression path, and
that BlobProperties are still returned correctly.

Recorded against a real (preprod) storage account with
AZURE_LIVE_TEST_SERVICE_VERSION=V2026_02_06 (via the standard
temporary TestEnvironment.java preprod patch, applied and reverted,
not included in this commit). Finding: this account does not set
x-ms-download-hint: Layout on plain download responses, so these
recordings validate the wiring's regression-safety (the opt-in flag
never changes downloaded bytes) end-to-end against a real service;
the routing-selection logic itself (DownloadHint gating, per-chunk
endpoint resolution, DataLocalityPolicy URL rewrite) remains covered
by the existing unit tests for BlobLayoutRangeResolver,
BlobLayoutCacheValue, AutoRefreshingCache, and DataLocalityPolicy.
Verified passing in both RECORD and PLAYBACK (6/6).

Recordings pushed to Azure/azure-sdk-assets as
java/storage/azure-storage-blob_dfdc202e2d.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* Add CHANGELOG entries for STG105 data locality feature

Documents the new AutoRefreshingCache/DataLocalityPolicy internal
infrastructure (azure-storage-common), getLayout + enableDataLocality
download/stream routing (azure-storage-blob), and the proxied
getLayout + enableDataLocality pass-through (azure-storage-file-datalake),
plus the 2027-03-07 service version support for the latter two.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* remove implementation notes

* update swagger generatation removes session implementation

* remove sessions implementation files

* remove sessions implementation files

* fix swagger for 105 generation

* remove more sessions from testing

* bring back autorefreshingcachetests, rename public functions with generic names

* update autorefreshcache to match sessions implementation, update references

* Revert "update autorefreshcache to match sessions implementation, update references"

This reverts commit f64688c9d85d86c0fd766eafa9bbb2fce5e49a24.

* Revert "fix swagger for 105 generation"

This reverts commit dd35386021cdf48f4b6dc26bfa42676735c57dac.

* remove unneeded code from DataLocalityPolicy

* remove unneeded Datalocality policy test

* remove enableDataLocality, make datalocality on by default

* move autorefreshing cache tests

* undo all changes that were just for spacing or ternary changes

* remove unused imports, params, and unneeded no-op changes

* add caching for blobLayoutcachevalue

* create and move cache factory to implmeentation

* remove unused imports

* fix linting

* Add explicit data locality endpoints to Blob downloads

* Refactor Blob layout cache retrieval

* Linting fixes: add missing JavaDoc and format test method calls

* Add data locality endpoint support to DataLake read APIs

- Add dataLocalityEndpoint property to DataLakeFileInputStreamOptions and ReadToFileOptions
- Propagate endpoint through Transforms.addDataLocalityEndpoint helper
- Update DataLakeFileAsyncClient and DataLakeFileClient to pass endpoint to Blob client
- Add DataLocalityPolicyTests for policy rewrite validation
- Add DataLakeDataLocalityOptionsTests for option propagation

* Fix linting: make valueProvider final in AutoRefreshingCache

* Fix DataLake layout access via blob accessor

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix blob layout range resolution gaps

* Align blob layout fetch failure handling with data locality spec

* move Blob client failure tests to the base tests files

* Send RFC 9110 conformant If-Match on layout continuation pages

ScrubEtagPolicy strips the quotes from response ETag headers, so the ETag
the SDK reads off the first getLayout page is a bare value. Sending that
value straight back as If-Match produces a header that violates RFC 9110,
which makes the surrounding quotes part of the entity-tag itself
(section 8.8.3) and restricts If-Match to "*" or a list of entity-tags
(section 13.1.1).

Add StorageImplUtils.toETagHeaderValue, an idempotent helper mirroring the
.NET Azure.Core ETag "H" (header) format: null, empty, "*", already quoted,
and W/"..." values pass through untouched; everything else is wrapped in
quotes. Apply it to the layout ETag in
BlobAsyncClientBase.copyRequestConditionsWithIfMatch so getLayout
continuation pages match the wire format the .NET client already sends.
DataLake inherits the fix because its getLayout proxies through the blob
client.

The caller-supplied If-Match is deliberately left untouched. The first
layout page is issued with the caller's request conditions as given, so
normalizing only on continuation pages would make page one and page two
send different values for the same request.

Azure Storage accepts both the quoted and unquoted forms, so this is not a
behavior change against the service. The same gap exists in the GA download
and client-side encryption paths and is tracked separately.

* add recordings for blob tests

* Add Java BlobLayout customization

* Regenerate BlobLayout internal model

* Expose public BlobLayout model

* shorten bloblayout import

* remove RequestInspectionPolicy and unnecessary helpers from BuilderHelperTests

* ETag lock getLayout continuation pages in the public blob layout APIs

getLayout requires If-Match on the original response ETag when following a continuation marker; without it the service rejects the follow-up page request. The internal layout-cache path already did this, but the public BlobClientBase.getLayout and BlobAsyncClientBase.getLayoutWithResponse APIs re-sent the caller's original request conditions on every page, so any layout spanning more than one page failed after the first.

Record the page-one ETag and send it as If-Match on subsequent pages, preserving the caller's other conditions. The async paths use PagedFlux.create so each subscription gets its own lock. ETags are re-quoted through StorageImplUtils.toETagHeaderValue because ScrubEtagPolicy strips the quotes RFC 9110 requires on the wire.

DataLakeFileClient.getLayout and DataLakeFileAsyncClient.getLayout inherit the fix by delegation.

* Fix two defects that silently disabled data locality routing

UrlBuilder.getPort() returns Integer, but UrlBuilder only exposes setPort(String) and setPort(int). DataLocalityPolicy called setPort(endpointUrlBuilder.getPort()), which auto-unboxed and threw NullPointerException for any layout endpoint given without an explicit port. The surrounding catch (RuntimeException) downgraded that to a log warning and skipped the rewrite, so data locality silently did nothing for the endpoint format the service actually returns. The existing test only passed because it hardcoded :443.

Separately, DataLocalityPolicy was never registered in the Data Lake pipeline, so every Data Lake data locality path was a no-op regardless. Register it in the same pipeline position the blob builder uses, immediately after AddHeadersFromContextPolicy and before the credential policy. Host rewriting does not affect SharedKey signing because the canonicalized resource is derived from the URL path and the credential account name, not the host.

Cover both a port-less and an explicit non-default port endpoint so neither regresses.

* Let Data Lake one-shot reads take a data locality endpoint

ReadToFile and OpenRead could already be given a layout endpoint, but ReadStreaming could not: readWithResponse takes positional parameters with no options bag, so there was nowhere to supply one. The blob equivalents, downloadStreamWithResponse and downloadContentWithResponse, have had setDataLocalityEndpoint since the feature landed.

Add FileReadOptions, mirroring BlobDownloadStreamOptions, plus readWithResponse overloads on the sync and async file clients that accept it. The existing positional overloads are untouched, so nothing breaks.

Tests assert the endpoint reaches the wire: the outbound request host is rewritten to the layout endpoint while the Host header retains the original account authority, and no rewrite happens when no endpoint is supplied.

Also drops the now-dead lossy toDataLakeFileLayoutInfo(BlobLayout) overload, which built its result from 26 nulls, and replaces a no-op 'assert options != null' in readToFileWithResponse with StorageImplUtils.assertNotNull, matching uploadWithResponse in the same file. Assertions are disabled at runtime by default, so the assert protected nothing.

* Remove unreachable blob layout paging helpers

getLayoutPage and toPublicLayoutPagedResponse were never called. getPublicLayoutPage became unreachable when the public paging APIs moved to getPublicLayoutPageWithLockedETag for the continuation ETag lock.

* Document data locality in the blob and Data Lake changelogs

The Data Lake changelog had no data locality entries at all, and the blob changelog covered only getLayout, omitting the locality-aware routing of buffered downloads and the setDataLocalityEndpoint options on the one-shot download paths.

* Refresh the layout cache proactively so downloads never stall on it

The layout is only an optimization: the ETag on each range GET guarantees correctness no matter which endpoint serves it. So an expired or unavailable layout must degrade to the client's configured endpoint, never block or fail a download.

Three problems prevented that. Jitter was dead code, because BlobLayoutCacheValue.getRefreshOn returned a fixed expiry-minus-30s that overrode the cache's jittered calculation, so every cache in the process refreshed at the same instant and stampeded getLayout. Refresh was only triggered by access, so an idle openInputStream would find the value expired on its next read and block. And a failed refresh left the next attempt already due, so each subsequent chunk launched another request at a service that was already failing.

Refresh now happens on a Reactor timer at a jittered point 90 to 30 seconds before expiry, failures back off 30 seconds as the SDK spec requires, and an expired value reports no ranges so callers fall back instead of stalling. The five minute TTL is unchanged.

Because a cache belongs to a single download, the timer needs a lifetime: the cache is now Closeable, disposed when the download terminates and when BlobInputStream is closed, and it stops rescheduling while nothing is reading it so an abandoned stream cannot refresh forever.

* Confine the data locality endpoint pass-in to one-shot reads

A buffered download needs a different endpoint per chunk, so a single caller-supplied endpoint has no coherent meaning there. ReadToFileOptions and DataLakeFileInputStreamOptions accepted one anyway: it applied to the initial request and was then silently overwritten as soon as the implicit layout resolved an endpoint for a chunk, with no error and no signal to the caller.

Remove it from those two buffered options, matching Blob, where BlobDownloadToFileOptions and BlobInputStreamOptions never had it. Both options are unreleased, so nothing breaks. FileReadOptions keeps setDataLocalityEndpoint because readWithResponse is one-shot.

Also scopes the sync getLayout continuation ETag per enumeration. It was allocated once per getLayout call, so two concurrent enumerations of the same PagedIterable could overwrite each other's locked ETag and take a spurious 412. Building the PagedIterable from PagedFlux.create gives each subscription its own reference, mirroring the async path; the retriever still runs the blocking sendRequest on the caller's thread.

* Distinguish a caller's bad endpoint from the service's

DataLocalityPolicy previously wrapped the whole rewrite in catch (RuntimeException) and only logged a warning. That is how a real NullPointerException in setPort went unnoticed, and it also meant a caller who explicitly supplied a bad endpoint for a one-shot download got no signal that their routing was ignored.

The policy now validates up front and throws IllegalArgumentException naming the offending value. A bare host[:port], which is the documented service format, and a full URL both remain valid; an absent or empty endpoint is still a silent no-op.

That is right for caller-supplied endpoints but wrong for endpoints the SDK resolved from a layout, where failing the download would contradict the layout being only an optimization. So BlobLayoutRangeResolver now validates service-supplied endpoints and returns null for anything unusable, which routes that chunk to the original endpoint. The ETag on each range GET keeps the data correct either way.

* Gate the layout tests on a service version that exists

Every data locality test required a minimum service version of 2027-03-07. No service accepts that value: a signed GET ?comp=layout sending x-ms-version: 2027-03-07 comes back 400 InvalidHeaderValue, meaning the version is not recognised rather than gated. ci.system.properties runs live tests at V2026_10_06, so RequiredServiceVersion silently skipped all of these in CI. That is why the recorded sessions contain no comp=layout requests at all.

getLayout is in fact already available at 2026-10-06 on an ordinary account. Verified directly: the same request at that version returns 200 with a populated BlobLayout, and range GETs on large enough blobs come back with x-ms-download-hint: layout. Retargeting the gate lets these tests actually run; the full set passes live against a real account.

* Prove chunk downloads really route to the layout endpoint

The download tests used a 16 KB blob. The service only returns x-ms-download-hint: layout above a threshold between 4 MB and 16 MB, so the layout path never engaged and the tests silently asserted nothing but the fallback. The class javadoc even described the wiring as a documented no-op.

Live runs now use a 16 MiB blob with a 4 MiB block size, which does receive the hint. The new live-only test attaches a pipeline policy that records the host and Host header of every outbound request, then asserts that at least one chunk was sent to a host other than the account and that those requests still carry the original account authority in the Host header. Observed in a live run: requests to blob.hkg24prdstf03f.store.core.windows.net carrying Host: vd7xscnhkg24ptf03fx.blob.core.windows.net.

The assertion was verified to be non-vacuous by inverting it and confirming it fails. Playback keeps the smaller blob so existing recordings stay valid, and the routing test is LiveOnly so it skips rather than fails without a recording.

* add new swagger doc with arrow customizations

* Point the blob swagger at the spec that actually has the STG105 operations

The generation config pointed at a 2026-10-06 spec that predates several operations already committed here, so anyone who ran autorest silently deleted them: ContainersImpl lost the Apache Arrow list-blob operations, and BlockBlobsUploadHeaders and BlockBlobsPutBlobFromUrlHeaders lost x-ms-content-crc64. Regeneration was effectively a destructive operation.

Repoint to the 2027-03-07 spec, which carries getLayout, x-ms-download-hint, Apache Arrow and content-crc64 together. Note this only changes what is generated, not what goes on the wire: AzureBlobStorageImpl takes its version from BlobServiceVersion at runtime, which stays at 2026-10-06.

Generating from that spec failed with 'no common parent found for client models []' because Container_ListBlobHierarchySegment_ApacheArrow is marked x-ms-pageable but returns a binary Arrow stream with no item model. Drop the extension for both Arrow list operations, matching the existing directives that do the same for the other list and page-range operations.

An earlier attempt at this repoint was reverted because a stray 'git status' had been pasted onto the license-header line, corrupting the header of every generated file. The swagger choice itself was correct.

The generated output is now taken verbatim. Previous commits had stripped trailing whitespace from generated javadoc, which the rest of the repo keeps, so every regeneration produced spurious churn. BlobsDownloadHeaders picks up the access-tier response headers from the newer spec; the change is purely additive.

* clean up ModelHelper to be more readable

* remove closeable from autorefreshingcache implementation

* remove BlobLayoutCacheFactory in favor of a functional ValueProvider

BlobLayoutCacheFactory only adapted a Supplier<Mono<BlobLayoutCacheValue>> to AutoRefreshingCache.ValueProvider, implementing createSync() as createAsync().block(). Make ValueProvider a @FunctionalInterface with that same blocking default so both layout cache call sites construct AutoRefreshingCache directly with a lambda.

Blocking is the right default here: azure-core's sync stack already blocks internally. Both HttpClient.sendSync and NettyAsyncHttpClient.sendSync are send(request, context).block(), so a hand-written synchronous layout fetch would duplicate the paging, ETag pinning and error policy of the async path only to avoid a block that the adjacent download call performs anyway.

Also reorder the BlobInputStream constructor so layoutCache precedes context, matching its call site in openInputStream.

* converge AutoRefreshingCache on the session implementation

Adopt the AutoRefreshingCache design from the Bifrost session-credential work so both features share one cache. The class is no longer bound to an ExpiringValue interface; callers supply a Function<T, OffsetDateTime> that extracts the expiration. SessionCredential is a public model and ExpiringValue lives in an implementation package, so the bound would have forced a public-to-implementation dependency plus a duplicate getExpiration()/getExpiresAt() pair. BlobLayoutCacheValue drops the interface and passes getExpiresOn as a method reference.

This also removes a real defect: creation no longer runs while holding a monitor. Ownership is claimed by compare-and-set on a Sinks.One latch, so the provider call happens outside any lock and sync and async callers join a single in-flight creation instead of serializing across a network round trip. Failed creations now arm a retry backoff, and the refresh point uses proportional rather than absolute jitter so it degrades sensibly across both the five-minute layout TTL and longer session credential lifetimes.

ValueProvider keeps the functional-interface shape with a blocking default createSync(), which lets the layout call sites stay lambdas while the session provider overrides it with its generated synchronous call.

Refresh is now opportunistic on access rather than scheduled on a timer, so there is nothing to cancel: AutoRefreshingCache.close(), the BlobInputStream.close() override, and the doFinally cleanup in downloadToFile are all removed. An expired layout is now re-fetched instead of being returned and degraded to a null range list; the failure sentinel still carries a full TTL, so a known-bad layout is not retried.

* refactor addDatalocalityEndpoint method to StorageImplUtils

* refactor getLayout to mirror the async variant

* use the synchronous getLayout stack for the sync layout cache

BlobInputStream.dispatchRead calls AutoRefreshingCache.getValidValueSync, which
fell through to the default createSync implementation of createAsync().block().
That drove the reactive HTTP stack from a purely synchronous API.

Add BlobClientBase.fetchLayoutCacheValueSync, built on the existing synchronous
getLayout paging, and supply it through an anonymous ValueProvider that overrides
createSync. The explicit type argument on the anonymous class is required because
the library still targets Java 8.

Extract the layout fetch error policy into isFatalLayoutFetchStatus and
createLayoutFallbackValue so both the sync and async paths share one definition
of which statuses fail a download and which fall back to the original endpoint.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* route seekable byte channel reads through data locality

openSeekableByteChannelRead is a chunked random access read path structurally
identical to BlobInputStream, but it never consulted the blob layout and passed
a null Context to every range GET, so its chunks always went to the client's
configured endpoint.

Build the layout cache from the download hint on the initial response and hand it
to the read behavior, which resolves an endpoint per chunk. The layout is
requested for the whole blob rather than the remaining bytes, and there is no
positive remaining count gate, because a channel can seek arbitrarily backward.

The lookup deliberately sits outside the existing try/catch. That block maps
errors onto -1 to signal end of file, so a layout failure raised inside it would
silently truncate the caller's data.

The five argument constructor is kept as a delegating overload so existing tests
are untouched. The class is package private, so this is not an API change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* do not apply data locality to client side encrypted clients

EncryptedBlobClientBuilder never installs DataLocalityPolicy, but encrypted
clients delegate to the base download paths, which fetched a layout and placed a
resolved endpoint in the request context that nothing could act on.

Beyond the wasted round trip, the combination is unsound. The decryption policy
treats any GET carrying a body as a download to decrypt, and a layout response is
XML. The encrypted range type rewrites plaintext ranges into ciphertext ranges.
Layout offsets are ciphertext as stored by the service while endpoint resolution
uses the caller's plaintext offset, so routing would be wrong at region
boundaries under client side encryption v2.

The Azure SDK for .NET does not support the combination either; its client side
encryption tests explicitly opt out with LayoutAwareRouting.Disabled. Whether it
should ever be supported is an open design question.

Gate all three layout sites on the pipeline actually containing a
DataLocalityPolicy. This needs no new public or protected API, keeps the
suppression automatic rather than opt out, self corrects for any other builder
that omits the policy, and is trivially reversible.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* add tests for the sync layout cache and channel routing

Mock transport tests rather than recordings, because the recorded sessions
contain no comp=layout traffic and cannot exercise this feature.

fetchLayoutCacheValueSync is covered for single page, multi page, and empty
layouts, and for the error policy: 403, 404, 409 and 412 propagate while 429 and
500 fall back to a null range list. A wiring test asserts the layout request goes
through the synchronous pipeline by recording whether processSync or process was
invoked, which is what actually proves the createSync override is used.

Seekable channel tests cover layout caching across reads, routing distinct
offsets to distinct endpoints, backward seeks, absence of the download hint,
soft layout failure falling back to the original endpoint, caller context
survival, and fatal layout errors propagating rather than being converted to end
of file.

The clients are built with retries disabled so request count assertions are not
coupled to the retry policy.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* let callers turn off locality aware routing

Data locality routing was always on whenever the service advertised a layout,
with no way for a caller to opt out. The Azure SDK for .NET exposes this choice
as LayoutAwareRouting on its download options, so add the Java equivalent.

Introduce LayoutAwareRouting with AUTO, DISABLED and ENABLED. AUTO is the
default and resolves to enabled today, but is deliberately distinct from ENABLED
so the library can change what the default means without a breaking change.
Routing only affects which endpoint serves a chunk, never the bytes returned.

The setting is available on the three chunked download paths that route:
downloadToFile, openInputStream and openSeekableByteChannelRead. Data Lake gets
its own mirrored enum and a Transforms conversion, matching how
ConsistentReadControl is already handled there rather than reusing the blob type
the way .NET does.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* add tests for the locality aware routing setting

Mock transport rather than recordings, since the recorded sessions deliberately
use a blob too small for the service to advertise a layout.

Cover DISABLED suppressing the layout lookup on all three chunked download
paths, AUTO and null still routing so the default stays on, ENABLED routing, and
the Data Lake to blob enum conversion including null.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* prove data lake chunked reads route to the layout endpoint

The existing data lake locality tests were all negative: they asserted that a
layout is not fetched when routing is disabled. Those pass just as happily if
routing were never wired up at all, so they said nothing about whether data lake
actually inherits the behavior from the blob client.

Add the positive direction for openInputStream, readToFile and its async
counterpart. Each asserts the layout is fetched exactly once rather than once
per chunk, that chunk requests are sent to the endpoint the layout advertises,
and that the original account authority survives in the Host header. Sync
readToFile had no chunked coverage at all before this.

Also stop getDataRequestRecords from returning the layout request, which would
otherwise be counted as a data request by these assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix the byte buddy version tag group id in storage common

The x-version-update tags for byte-buddy and byte-buddy-agent named the group
net.byte-buddy, but the group is net.bytebuddy, which is how external
dependencies records it and how every other storage pom spells it. The pom
version scanner could not match the tags to the declared dependencies and failed
the Analyze stage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* reorganize blob data locality tests

* pin resumed layout enumerations to their first observed page ETag

A caller that resumes getLayout from a marker it obtained earlier never
observes the layout's first page, so there was no ETag to pin its
continuation requests to and every later page went out unconditioned.

Set the pinned ETag from whichever page the enumeration sees first,
rather than only from a marker-less first page. The resumed page itself
still cannot be conditioned, since its marker came from an enumeration
whose ETag this client never saw.

* restore setter ordering in ReadToFileOptions

setUserPrincipalName and setLayoutAwareRouting had their javadoc and
bodies interleaved, leaving each setter documented as the other. Put
each method back with its own documentation.

* cover blob getLayout against a live service

getLayout was proven only by recordings of a seven byte blob and by mock
HttpClients, so nothing showed that a real layout parses, covers the
blob, or names a backend endpoint distinct from the account.

Add live coverage for a small blob, for a sixteen megabyte blob (the
size at which the service begins sending x-ms-download-hint), and for a
range scoped request. Assertions check that the returned ranges are
contiguous and cover the requested window rather than checking a range
count, because the service is free to paginate a layout and today
returns a single range for blobs of every size tested, up to 5.66 GB.

Also assert the untested half of a service requirement: a continuation
must repeat the original range header, or omit it when the original
request had none. The omit case rests on BlobRange.toHeaderValue
returning null for a default range, which nothing exercised.

Ranged enumeration is capped at its first page. Following the marker
trips a known service defect, recorded in a disabled test that asserts
the behavior expected once the defect is fixed.

* cover data locality download routing against a live service

The download side had no live proof that the layout fetch is actually
gated on x-ms-download-hint, nor that the endpoint the download routes
to is the one getLayout reports.

Add a live test over a four megabyte blob, which is below the size at
which the service sends the hint, asserting that no comp=layout request
is issued, that every request stays on the account host, and that no
Host header is rewritten. Its block size is smaller than the blob so the
download is genuinely chunked; sized otherwise the layout would be
skipped because no bytes remain after the initial chunk, and the test
would pass without exercising the hint check at all.

Strengthen the existing sixteen megabyte routing test to assert that the
layout is fetched exactly once, since it is cached for the duration of
the download, and that every routed chunk host appears among the
endpoints getLayout reports. Enumerations are counted rather than
requests, because continuation pages are also comp=layout calls.

Requests now record their URL so layout calls can be identified and so
a live failure prints what was actually sent.

* Flatten Data Lake layout pagination

* wire in bloblayout through ModelHelper

* remove blobranges since bloblayout was laready included

* remove enumerated values for properties, replace with properties models

* remove pageSize, change BlobRequestConditions to a shallow copy on set

* getLayout now pages directly over blobLayoutRange, change from bloblayout containing list of ranges

* feat(storage): type data locality endpoints

Replace raw String endpoint values across storage-common, Blob, and Data Lake with a validated DataLocalityEndpoint value type. This prevents malformed endpoint strings from reaching the request pipeline while preserving fallback for invalid service layout data.

* Revert "feat(storage): type data locality endpoints"

This reverts commit b9c7679596902e8fd1b9d187d3fd9261589308d7.

* add description for to getters for `getEndpoint`

* Fix Data Lake file client Javadoc link

* Reduce Data Lake timeout test flakiness

* Fix storage layout Javadoc links

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7

* fix linting issues

* create autorefreshingcache branch

* change ValueProvider to Supplier

* remove unused imports

* adjust autorefreshingcache logic to match dotnet

* change generic names in autorefreshing cache

* Simplify autofresh cache readability

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* change names in autorefreshcache to make it easier to read funcitons and vars

* remove unnecessary tests in AutoRefreshingCacheTests

* Simplify cache test suppliers and share per-test setup

* update autorefreshingcache call sites

* shorten javadocs, change syncronized to locks

* fix linting issues

* update call sites for AutoRefreshingCache

* update inputstream test to use async for sync call with .block()

* change downloadToFileImp to include contentValidation

* change isValueMissingOrExpired in cache to be inclusive of expiration time

* change getLayoutWithResponse to getLayout

* spotless apply

* add missing recordings

* Revert "Storage - STG105 NFS List V2 Files and Directories (#50036)"

This reverts commit 83dc6e28852ace9e902975682ab5510671ca56e2.

* add recordings for prev features that were missing

* move LayoutAwareRouting into common

* change DataLocalityPol to extend HttpPipelineSyncPolicy

* change string to uri for bloblayout endpoint

* fix String to URI change for datalake

* change dataawarerouting to disabled by default

* add error handing for fetchLayoutCacheValueSync/Async so that only real errors crash

* disable data locality for encrypted clients

* add warning for download with data locality enabled for encrypted, add test for encrypted

* move datalocality policy before retries so retries do not overwrite the secondary url

* add documentation about jdk configuration required for data locality

* trigger storage CI rerun

* fix malformed StorageImplUtilsTests file

* revert assets tag to tag with recordings

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Isabelle <ibrandes@microsoft.com>
Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
* [Storage] Add Expect: 100-continue support

Applies the HTTP header Expect: 100-continue to requests that carry a
body, so that a body is not uploaded just to be rejected while the
service is under load. The service can reject the request before a
single byte of body is sent, which cuts the bandwidth wasted retrying
large uploads against a throttling service.

- Adds Request100ContinueOptions and Request100ContinueMode (AUTO,
  ALWAYS, NEVER) to azure-storage-common. AUTO, the default, applies the
  header only for a window after the service responds 429, 500, or 503,
  so the extra round trip is only paid when it earns its keep.
- The policies are added below the retry policy so a throttling response
  takes effect on the very next attempt rather than only on a later
  call, and above the credential policies since headers may affect the
  string to sign.
- Wired into the blob, data lake, file share, and queue pipelines, and
  exposed on the blob client builders via request100ContinueOptions.
- Can be turned off without a code change by setting the system property
  or environment variable AZURE_STORAGE_DISABLE_EXPECT_CONTINUE_HEADER.

Ported from azure-sdk-for-net #40611 and #52438.

* [Storage] Address PR feedback on Expect: 100-continue

- Rename Request100ContinueOptions/Request100ContinueMode to
  ExpectContinueOptions/ExpectContinueMode, the mode values to
  APPLY_ON_THROTTLE/ON/OFF, autoInterval to throttleInterval, and the
  builder method to expectContinueBehavior, matching the .NET names.
- Make ExpectContinueOnThrottlePolicy extend HttpPipelineSyncPolicy
  rather than ExpectContinuePolicy. Both policies are now final and
  share their logic through a package private helper, so neither
  advertises an extension point that callers outside the package cannot
  actually use.
- Allow the Expect header in the blob default log options, so it is
  visible when debugging whether a request negotiated the handshake.
- Replace the unrelated timeouts link in the options javadoc with RFC
  9110.
- Add ExpectContinueTransportTests, which drives each HTTP client against
  a real socket and asserts on what reached the wire and when.

The transport tests show that setting the header does not by itself make
a client wait. Only okhttp performs the handshake. netty and vertx send
the header and stream the body immediately, and jdk-httpclient drops the
header entirely since Expect is restricted by java.net.http.HttpClient.
That is recorded in the test matrix and documented on ExpectContinueOptions
and in the changelog rather than left implicit.

* [Storage] Test the allowed Expect log header, trim changelog

Cover the default log options change with an assertion, and keep the
changelog to the feature itself. The transport caveat stays on
ExpectContinueOptions, where the behavior it describes lives.

* [Storage] Match .NET exactly on enum names and log the Expect header everywhere

Use the .NET casing for the mode values, ApplyOnThrottle / On / Off,
rather than Java's upper snake case. Checkstyle accepts it.

Allow the Expect header in the default log options of the queue, file
share and data lake builders too, not only blob, as the review comment
was a general one.

* [Storage] Mirror the .NET expect-continue tests and cover the async path

Restructure ExpectContinueTests so the first section mirrors
ExpectContinueTests.cs test for test, with the same names and the same
case values, making the two suites directly comparable. Java specific
coverage, including the per retry behaviour that .NET has no test for,
moves to a clearly separated second section.

ThrottlePolicyRevertsAfterBackoff is disabled in .NET
(Azure/azure-sdk-for-net#41368); the Java mirror is enabled, since the
window is a monotonic nanoTime deadline and oversleeping it can only
close the window.

Extend the transport test to transport x sync/async, as those are
separate code paths in every client. All four transports behave the same
on both paths, so the earlier finding is unchanged.

* [Storage] Scope expect-continue to blob only and trim comments

Revert the queue, file share and data lake packages entirely, so the
header is applied only where a customer can configure it.

Drop the explanatory comments that were not carrying their weight,
including the cross references to the .NET implementation.

* [Storage] Remove an unrelated file that was committed by mistake

sdk/storage/TYPESPEC_MIGRATION_ESTIMATE.md is unrelated to this change
and was picked up accidentally.

* [Storage] Move the wire level tests into ExpectContinueTests

Keep them in the class the review comment was on rather than a separate
file. Each one opens and closes its own server, so the socket fixture
does not run for the tests that do not need it.

* [Storage] Address review feedback on Expect: 100-continue

- Drop the azure-storage-common changelog entry; the changes there are
  implementation level and the customer facing option is on the blob
  builders.
- Remove the okhttp, vertx and jdk test dependencies. They already come
  transitively from azure-core-test, so they were redundant.
- Move ExpectContinuePolicy and ExpectContinueOnThrottlePolicy into
  com.azure.storage.common.policy alongside the other storage policies,
  and rename the shared helper to ExpectContinuePolicyHelper.
- Use upper snake case for the mode constants.
- Reject a negative content length threshold or throttle interval rather
  than silently changing behaviour, with boundary tests for both.

* [Storage] Do not apply Expect: 100-continue unless it is configured

The handshake needs the HTTP client to withhold the request body until
the service responds, and the default transport does not do that, so a
default client would pay for the header and save nothing. The policy is
now added only when expectContinueBehavior is set.

* adding support for expect continue 100 to azure core netty

* Avoid the deprecated URL constructor in the netty observation test

The core modules compile with -Werror, and URL(String) is deprecated as
of Java 20, so the test failed to compile on the JDK 25 build agents and
took the whole build down with it.

* Expect the JDK and Vert.x clients to defer the request body

Records the behaviour those transports have once the azure-core fixes in
#50311 and #50312 ship. Until Storage picks up those versions the four
cases fail, which is the intended signal rather than a silent mismatch.

Netty stays as it is: the change there observes the interim response and
does not defer the body.

* Let the JDK and Vert.x expectations follow the azure-core version

Those two clients honour the handshake from azure-core 1.2.0-beta.1
onwards, so the behaviour depends on whether the build resolves a
released artifact or the repository source. Asserting either one fails
half the pipeline, so assert what holds either way: the exchange
completes, the body arrives in full, and a withheld body implies the
header reached the wire.

Once Storage consumes a release carrying the fix these become
DEFERS_BODY. Netty and okhttp keep their exact expectations.

* Revert "Let the JDK and Vert.x expectations follow the azure-core version"

This reverts commit 644c196.

* Revert "Storage - STG105 NFS List V2 Files and Directories (#50036)"

This reverts commit 83dc6e2.

* add recordings for prev features that were missing

* addressing comments and fixing problems left by base merge

* addressing comment

* preventing expect continute jdk tests from running on java 11 or lower

* moving to internal.policy

* addressing comments

* removing dupe import

* fixing flaky test

* assets tag update

---------

Co-authored-by: Isabelle <ibrandes@microsoft.com>
Co-authored-by: browndav <browndav@microsoft.com>
@ibrandes

Copy link
Copy Markdown
Member Author

/azp run java - storage - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Storage Storage Service (Queues, Blobs, Files)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants