Repository navigation
[Storage] STG105 Datalocality - #50512
Merged
browndav-msft merged 45 commits intoOct 28, 2026
Merged
browndav-msft merged 45 commits into
browndav-msft merged 45 commits into
Conversation
* Feature/storage/bifrost create session (Azure#49448) * generate base files based on swagger docs * create live tests for createSession - downgrade blobserviceversion to 2026_04_06 - change AZURE_LIVE_TEST_SERVICE_VERSION to V2026_04_06 in ci.system.properties in azure-storage-common - create both sync and async * add recordings * create new files based on swagger update * add two params to BlobContainerClient#createSessionWithResponse * add sanitizers for SessionToken and SessionKey to BlobTestBase * add recording for createSessionReturnsTokenAndKey * create StorageSessionCredential with isExpired * create BlobSessionClient so that BlobSessionProvider takes it as a dep instead of ContainersImpl * create BlobSEssionClient with tests * add recorings for BlobSessionClient * fix BlobContainerAsyncClient to match new swagger, add new recording * add SessionProvider and SessionProviderTest * add accountName to BlobSessionClient * add accountName to StorageSessionCredential and SesionTestHelper * wip * change sessionprovider to SEssionTokenCredentialPolicy * wip * move session tests from containerapi to blobsessionclienttests * fix blobsessiontests and add place holder for end-to-end tests in containerapitests * add recordings for blobsessionclient * linting * refactor cache into separate class so it follows BearerTokenAuthenticationPolicy + AccessTokenCache pattern * add 503 fallback * add tests for udsas, but disabled for now * refactor createContext to use hardcoded endpoint * add SessionMode and tests for SessionMode * add sessionOptions to buildPipeline, add null to builders not using sessions * move SessionTokenCredentialPolicy ahead of StorageBearerTokenChallengeAuthorizationPolicy * fix linting issues * add session to BlobServiceClients and BlobServiceClientBuildeer * change expiration so that it defaults to 5 minutes, if there is no expiration * move SessionOptions so that it is public * remove old SessionOptions * remove unnecessary references to containerName and serviceVersion * add BlobContainerSessionInfo, add other Copilot recommendations * delete BlobContainerSessionInfo, restore return CreateSessionResponse * create createSession end-to-end test with recordings * only allow get requests for getblob * wrap tests in try-with-resources * make createSession package private * fixes based on copilot suggestions * Update sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/blob/implementation/util/BlobSessionClient.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add containerName to SessionOptions * move accountName to SessionOptions * refactor: SessionTokenCredentialPolicy accepts bearer policy as constructor dependency SessionTokenCredentialPolicy now takes StorageBearerTokenChallengeAuthorizationPolicy as a constructor dependency instead of relying on pipeline ordering to coordinate with the bearer policy. Key changes: - SessionTokenCredentialPolicy delegates to bearerPolicy.process() for non-session requests (non-GetBlob, NONE mode, AUTO first request) and fallback (503). - Added getBearerPolicy() accessor for use in per-container pipeline construction. - BuilderHelper updated to pass bearer policy through to SessionTokenCredentialPolicy constructor in both addSessionPolicyIfEnabled and wrapWithSessionPolicy. - Tests updated to mock bearerPolicy and verify delegation behavior. This is step 1 of the session auth refactor to align with the .NET SDK's SessionAuthenticationPolicy pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: introduce AuthStrategy enum and consolidate analyzeRequest Consolidate isGetBlobRequest() and shouldUseSession() into a single analyzeRequest() method that returns an AuthStrategy enum (USE_BEARER_TOKEN or USE_SESSION_TOKEN), following the .NET SessionAuthenticationPolicy pattern. Also extract response handling into handleSessionResponse() and handleSessionResponseSync() methods for cleaner process()/processSync() flow. The process methods now have a clear structure: 1. analyzeRequest() -> decide strategy 2. USE_BEARER_TOKEN -> delegate to bearer policy 3. USE_SESSION_TOKEN -> acquire session, sign, send, handle response No behavioral changes - all existing tests pass unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: remove redundant restype check from analyzeRequest The restype query parameter check was redundant because requests with restype=container are container-level operations that have no blob name. They are already filtered by the blob name presence check. This aligns more closely with the .NET SessionAuthenticationPolicy which relies on URL structure rather than query parameter checks. The comp check is retained as a safety measure to exclude sub-operations like GetBlobMetadata (comp=metadata) that have a blob name but are not download operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * change sessionmode from always to singlespecifciedcontainer, add resolve method * wrap bearer token in sessioncredentialpolicy * fix NPE for SessionOptions, sessionoptions always non null * add tests for sessiontokencredpolicy and storagesessioncred * add logic to avoid wrapping Bearertoken, if session is not needed * add overloaded oauth in blobtestbase to be able to add sessionoptions * add overloaded getOAuthServiceAsyncClient to be able to pass session options * add custom buildStringToSign to remove `0` from get requests * readd versions * readd ci.system.properties * change session options check for null in BuilderHelper which affected other tests * add recordings for create sessions, change time to testResource time * add requestInspectionPolicy and overloaded getoauth client in base test * fix null sessionsoptions issue * add fix in storagesessioncredntial for query params * add SessionTokenCredPolicy to checks for anonymousAccess * remove constructor for BlobSessionClients that uses parse url * linting issues, remove SessionOptions from service methods * add comments to policyrefreshNearExpiry test * fix linting issues * add check for container name * fix return javadoc for SessionMode * add LOGGER and appropriate error throwing in BlobSessionClient * changes based on feedback from isabelle * add single retry for all 401 errors * remove unused imports * add suppression for SessionTokenPolicy for linting * fix ubuntu tests hanging by removing local dns bypass * add session for blob client with tests * create unified http transport between data requests and session request, add ResourceLock for live tests * test multiple concurrency * add branching if for bearer policy * refactor validateSessionMode to builderhelper * add revert to bearertoken for 400 errors * remove unused imports * readd comment to bearertokechallenge --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * add forceRefreshSessionInBackground() to SessionTokenCredentialPolicy * remove .env initialization for sessions * change SessionCredentialCache to AutoRefreshingCache<T> * Regenerate azure-storage-blob from updated swagger spec for STG105 Data Locality Points swagger/README.md at the Data Locality fork spec (nickliu-msft/azure-rest-api-specs@5c678e4, 2026-10-06/blob.json) matching .NET PR Azure/azure-sdk-for-net#57554, and runs autorest to regenerate the implementation layer: new GetLayout REST operation (comp=layout), BlobLayout/BlobLayoutRanges/BlobLayoutEndpoints models, BlobsGetLayoutHeaders, and the new x-ms-download-hint header on BlobsDownloadHeaders. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Implement Data Locality getLayout API for STG105 in azure-storage-blob Adds the paged getLayout / getLayoutAsync API for BlobClient/BlobAsyncClient (via BlobClientBase/BlobAsyncClientBase), returning BlobLayoutInfo with the blob's layout ranges/endpoints plus standard properties. New public types: BlobLayoutInfo, BlobLayoutRange, BlobGetLayoutOptions, DownloadHint. Wires the new x-ms-download-hint response header into BlobDownloadHeaders. ModelHelper.transformBlobLayoutInfo bridges the generated BlobLayout/BlobsGetLayoutHeaders response into the public BlobLayoutInfo model, reusing the existing getObjectReplicationDestinationPolicyId/getObjectReplicationSourcePolicies helpers for parity with other blob property transforms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add tests for getLayout API in azure-storage-blob (STG105) Sync (BlobClientBaseGetLayoutApiTests) and async (BlobClientBaseGetLayoutAsyncApiTests) coverage for BlobClientBase/BlobAsyncClientBase.getLayout: basic success, empty blob, ranged request, page-size limiting, continuation token, request-condition success/failure (mirroring PageBlobApiTests listPageRanges AC/ACFail pattern via BlobTestBase allConditionsSupplier/allConditionsFailSupplier), and a not-found error case. All tests gated with RequiredServiceVersion min 2027-03-07. No recordings yet -- fail as expected in playback with recording does not exist, which is normal at this stage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add implementation notes for STG105 Data Locality getLayout API Comprehensive explanation of what was built (getLayout paged API and DownloadHint signal, scoped to azure-storage-blob only), how it works (generated-layer wire protocol, hand-written model/client bridge, test coverage), and why key decisions were made (eager endpoint resolution in BlobLayoutRange, matching the existing listPageRanges pattern, deferring the data-locality routing infrastructure and DataLake module as follow-on work, using the unmerged fork spec to match the still-open .NET PR). Lists outstanding items: missing test recordings due to no credentials in this environment, the unmerged spec dependency, deferred routing/DataLake work, and a pending CHANGELOG entry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Fix BlobLayoutInfo missing blob-content fields and async test bug The generated BlobsGetLayoutHeaders exposes both generic (Content-Length, Content-Type, etc from the layout response body) and blob-specific (x-ms-blob-content-length, x-ms-blob-content-type, etc reflecting the blobs actual properties) header pairs, matching the .NET model split between BlobLayoutInfo.ContentLength and BlobLayoutInfo.BlobContentLength/BlobContentType/BlobContentEncoding/BlobContentMD5/BlobCreatedOn. The initial Java BlobLayoutInfo only mapped the generic set, so getContentLength() returned null against a real service response. Adds the missing blobContentLength/blobContentType/blobContentEncoding/blobContentMd5/blobCreatedOn fields end to end (BlobLayoutInfo, ModelHelper.transformBlobLayoutInfo) and updates tests to assert on the correct getBlobContentLength(). Also fixes a StepVerifier logic bug in getLayoutAC (async): a successful conditional request legitimately emits one item, so the assertion needs response.then() before verifyComplete() to only assert on completion, not zero emissions -- matching the pattern already used by getLayoutContinuationToken/getLayoutRange in the same file. Found and fixed by actually recording against a real (preprod) storage account with AZURE_LIVE_TEST_SERVICE_VERSION=V2026_02_06. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Fix BlobTestBase afterTest cleanup client ignoring AZURE_LIVE_TEST_SERVICE_VERSION afterTest built its cleanup BlobServiceClient directly from BlobServiceClientBuilder (httpClient/credential/endpoint only), bypassing the existing getServiceClient(TestAccount) helper that beforeTest already uses for every other client in this class. That helper routes through getServiceClientBuilder -> StorageCommonTestUtils.instrument, which is the only place that reads the AZURE_LIVE_TEST_SERVICE_VERSION environment variable and applies it via builder.serviceVersion(...). As a result, the cleanup listBlobContainers call always used the SDK's default/latest service version regardless of any environment override, causing every test to fail during teardown when running against an account that only supports an older pinned version (discovered while recording STG105 getLayout tests against a preprod account limited to V2026_02_06 -- the actual test bodies passed, only cleanup failed). Fix: reuse getServiceClient(ENVIRONMENT.getPrimaryAccount()) like the rest of the class. Pre-existing bug, unrelated to the STG105 feature itself, but blocking for anyone using AZURE_LIVE_TEST_SERVICE_VERSION in this module. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add DataLocalityPolicy to azure-storage-common (STG105) Java port of .NET's DataLocalityPolicy (Azure.Storage.Common/src/Shared/DataLocalityPolicy.cs): a no-op HttpPipelinePolicy unless a caller opts a request into locality-aware routing by setting DataLocalityPolicy.LAYOUT_ENDPOINT_KEY on the per-call Context to a non-empty endpoint URL. When set, rewrites the outgoing request's host/port to that endpoint (for routing to an alternate storage node returned by GetLayout) while preserving the original host as the Host header, so server-side virtual-hosting/TLS-SNI routing still works. Runs PER_RETRY (mirroring .NET's use on the raw HttpMessage before each send/retry). Malformed endpoint values are logged and skipped rather than failing the request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add tests for DataLocalityPolicy (STG105) Real HttpPipeline-based tests (no live service dependency) using a captured NoOpHttpClient and MockHttpResponse: no-op when context data is absent, rewrites host/port and preserves the original Host header when a layout endpoint is set, and no-op for an empty-string endpoint value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add enableDataLocality option to BlobDownloadToFileOptions and BlobInputStreamOptions Adds the EnableDataLocality-equivalent toggle (mirroring .NET's BlobDownloadToOptions/BlobOpenReadOptions) so callers can opt in to layout-aware routing for parallel file downloads and streamed reads. Wiring of the flag into the download/read paths is tracked separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add BlobLayoutCacheValue and BlobLayoutRangeResolver for data locality routing Mirrors .NET's BlobLayoutSegmentCacheValue (cache wrapper distinguishing populated/empty/failed layout state) and the binary-search GetLayoutEndpoint helper from PR Azure/azure-sdk-for-net#57554, adapted to reuse the existing public BlobLayoutRange model (endpoint indices are already resolved by ModelHelper.transformBlobLayoutInfo) and the AutoRefreshingCache<T> added to azure-storage-common. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add tests for BlobLayoutCacheValue and BlobLayoutRangeResolver Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Wire DataLocalityPolicy and layout cache into Blob download paths Registers DataLocalityPolicy in the Blob HTTP pipeline (previously built but unregistered) and wires locality-aware routing into the two download entry points, opt-in via BlobDownloadToFileOptions / BlobInputStreamOptions.isEnableDataLocality(): - BlobAsyncClientBase.downloadToFileImpl: after the first chunk resolves, if enabled and the initial response's DownloadHint is LAYOUT, builds one AutoRefreshingCache<BlobLayoutCacheValue> per download (etag-locked, scoped to the remaining un-downloaded range) and resolves an endpoint per chunk via BlobLayoutRangeResolver, routing only chunks 1+ (chunk 0 already completed before the cache exists, matching the .NET PartitionedDownloader design). - BlobClientBase.openInputStream / BlobInputStream.dispatchRead: same pattern, scoped to the full requested stream range since reads can seek to arbitrary offsets. - Added a context-propagating internal getLayout(options, context) overload and fetchLayoutCacheValueAsync helper on BlobAsyncClientBase; soft-fails (caches a null/no-routing value) on BlobStorageException rather than failing the download. Mirrors Azure/azure-sdk-for-net#57554's PartitionedDownloader.cs and BlobBaseClient.DownloadStreamingInternal wiring. Default (enableDataLocality=false) behavior is unchanged; verified via BlobApiTests/BlobAsyncApiTests (738 tests) and BlobClientBaseGetLayout*ApiTests (38 tests), all passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add getLayout and data locality wiring for DataLake file client Mirrors Azure/azure-sdk-for-net#57554's DataLakeFileClient.GetLayout and locality-aware routing, taking advantage of DataLakeFileClient already wrapping a BlockBlobClient internally: - New DataLakeFileLayoutInfo/DataLakeFileLayoutRange public models and DataLakeFileGetLayoutOptions options bag (DataLake-domain shapes, field-mapping conventions matched to Transforms.toPathProperties). - DataLakeFileClient/DataLakeFileAsyncClient.getLayout(...) proxies the already-implemented BlockBlobClient/BlockBlobAsyncClient getLayout API and maps BlobLayoutInfo -> DataLakeFileLayoutInfo via new Transforms helpers. Documented as an explicit proxy since DataLake does not yet have its own generated layout REST operation; revisit if/when one is added. - Added enableDataLocality to ReadToFileOptions/ DataLakeFileInputStreamOptions, threaded through to the underlying BlobDownloadToFileOptions/BlobInputStreamOptions. Since readToFileWithResponse/openInputStream already delegate to BlockBlobClient (which now has full locality routing wired in), this makes DataLake downloads/streams locality-aware with no additional chunking/routing logic needed on the DataLake side. Default (enableDataLocality=false) behavior is unchanged. Verified via FileApiTest/FileAsyncApiTests/FileInputStreamTests (827 tests), all passing; compile/checkstyle/spotbugs clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add end-to-end tests for data locality download/stream wiring Adds BlobDataLocalityDownloadApiTests covering enableDataLocality on BlobDownloadToFileOptions/BlobInputStreamOptions: single-chunk and multi-chunk downloadToFileWithResponse, openInputStream (full and partial range), the enableDataLocality=false regression path, and that BlobProperties are still returned correctly. Recorded against a real (preprod) storage account with AZURE_LIVE_TEST_SERVICE_VERSION=V2026_02_06 (via the standard temporary TestEnvironment.java preprod patch, applied and reverted, not included in this commit). Finding: this account does not set x-ms-download-hint: Layout on plain download responses, so these recordings validate the wiring's regression-safety (the opt-in flag never changes downloaded bytes) end-to-end against a real service; the routing-selection logic itself (DownloadHint gating, per-chunk endpoint resolution, DataLocalityPolicy URL rewrite) remains covered by the existing unit tests for BlobLayoutRangeResolver, BlobLayoutCacheValue, AutoRefreshingCache, and DataLocalityPolicy. Verified passing in both RECORD and PLAYBACK (6/6). Recordings pushed to Azure/azure-sdk-assets as java/storage/azure-storage-blob_dfdc202e2d. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * Add CHANGELOG entries for STG105 data locality feature Documents the new AutoRefreshingCache/DataLocalityPolicy internal infrastructure (azure-storage-common), getLayout + enableDataLocality download/stream routing (azure-storage-blob), and the proxied getLayout + enableDataLocality pass-through (azure-storage-file-datalake), plus the 2027-03-07 service version support for the latter two. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7 * remove implementation notes * update swagger generatation removes session implementation * remove sessions implementation files * remove sessions implementation files * fix swagger for 105 generation * remove more sessions from testing * bring back autorefreshingcachetests, rename public functions with generic names * update autorefreshcache to match sessions implementation, update references * Revert "update autorefreshcache to match sessions implementation, update references" This reverts commit f64688c. * Revert "fix swagger for 105 generation" This reverts commit dd35386. * remove unneeded code from DataLocalityPolicy * remove unneeded Datalocality policy test * remove enableDataLocality, make datalocality on by default * move autorefreshing cache tests * undo all changes that were just for spacing or ternary changes * remove unused imports, params, and unneeded no-op changes * add caching for blobLayoutcachevalue * create and move cache factory to implmeentation * remove unused imports * fix linting * Add explicit data locality endpoints to Blob downloads * Refactor Blob layout cache retrieval * Linting fixes: add missing JavaDoc and format test method calls * Add data locality endpoint support to DataLake read APIs - Add dataLocalityEndpoint property to DataLakeFileInputStreamOptions and ReadToFileOptions - Propagate endpoint through Transforms.addDataLocalityEndpoint helper - Update DataLakeFileAsyncClient and DataLakeFileClient to pass endpoint to Blob client - Add DataLocalityPolicyTests for policy rewrite validation - Add DataLakeDataLocalityOptionsTests for option propagation * Fix linting: make valueProvider final in AutoRefreshingCache * Fix DataLake layout access via blob accessor Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix blob layout range resolution gaps * Align blob layout fetch failure handling with data locality spec * move Blob client failure tests to the base tests files * Send RFC 9110 conformant If-Match on layout continuation pages ScrubEtagPolicy strips the quotes from response ETag headers, so the ETag the SDK reads off the first getLayout page is a bare value. Sending that value straight back as If-Match produces a header that violates RFC 9110, which makes the surrounding quotes part of the entity-tag itself (section 8.8.3) and restricts If-Match to "*" or a list of entity-tags (section 13.1.1). Add StorageImplUtils.toETagHeaderValue, an idempotent helper mirroring the .NET Azure.Core ETag "H" (header) format: null, empty, "*", already quoted, and W/"..." values pass through untouched; everything else is wrapped in quotes. Apply it to the layout ETag in BlobAsyncClientBase.copyRequestConditionsWithIfMatch so getLayout continuation pages match the wire format the .NET client already sends. DataLake inherits the fix because its getLayout proxies through the blob client. The caller-supplied If-Match is deliberately left untouched. The first layout page is issued with the caller's request conditions as given, so normalizing only on continuation pages would make page one and page two send different values for the same request. Azure Storage accepts both the quoted and unquoted forms, so this is not a behavior change against the service. The same gap exists in the GA download and client-side encryption paths and is tracked separately. * add recordings for blob tests * Add Java BlobLayout customization * Regenerate BlobLayout internal model * Expose public BlobLayout model * shorten bloblayout import * remove RequestInspectionPolicy and unnecessary helpers from BuilderHelperTests * ETag lock getLayout continuation pages in the public blob layout APIs getLayout requires If-Match on the original response ETag when following a continuation marker; without it the service rejects the follow-up page request. The internal layout-cache path already did this, but the public BlobClientBase.getLayout and BlobAsyncClientBase.getLayoutWithResponse APIs re-sent the caller's original request conditions on every page, so any layout spanning more than one page failed after the first. Record the page-one ETag and send it as If-Match on subsequent pages, preserving the caller's other conditions. The async paths use PagedFlux.create so each subscription gets its own lock. ETags are re-quoted through StorageImplUtils.toETagHeaderValue because ScrubEtagPolicy strips the quotes RFC 9110 requires on the wire. DataLakeFileClient.getLayout and DataLakeFileAsyncClient.getLayout inherit the fix by delegation. * Fix two defects that silently disabled data locality routing UrlBuilder.getPort() returns Integer, but UrlBuilder only exposes setPort(String) and setPort(int). DataLocalityPolicy called setPort(endpointUrlBuilder.getPort()), which auto-unboxed and threw NullPointerException for any layout endpoint given without an explicit port. The surrounding catch (RuntimeException) downgraded that to a log warning and skipped the rewrite, so data locality silently did nothing for the endpoint format the service actually returns. The existing test only passed because it hardcoded :443. Separately, DataLocalityPolicy was never registered in the Data Lake pipeline, so every Data Lake data locality path was a no-op regardless. Register it in the same pipeline position the blob builder uses, immediately after AddHeadersFromContextPolicy and before the credential policy. Host rewriting does not affect SharedKey signing because the canonicalized resource is derived from the URL path and the credential account name, not the host. Cover both a port-less and an explicit non-default port endpoint so neither regresses. * Let Data Lake one-shot reads take a data locality endpoint ReadToFile and OpenRead could already be given a layout endpoint, but ReadStreaming could not: readWithResponse takes positional parameters with no options bag, so there was nowhere to supply one. The blob equivalents, downloadStreamWithResponse and downloadContentWithResponse, have had setDataLocalityEndpoint since the feature landed. Add FileReadOptions, mirroring BlobDownloadStreamOptions, plus readWithResponse overloads on the sync and async file clients that accept it. The existing positional overloads are untouched, so nothing breaks. Tests assert the endpoint reaches the wire: the outbound request host is rewritten to the layout endpoint while the Host header retains the original account authority, and no rewrite happens when no endpoint is supplied. Also drops the now-dead lossy toDataLakeFileLayoutInfo(BlobLayout) overload, which built its result from 26 nulls, and replaces a no-op 'assert options != null' in readToFileWithResponse with StorageImplUtils.assertNotNull, matching uploadWithResponse in the same file. Assertions are disabled at runtime by default, so the assert protected nothing. * Remove unreachable blob layout paging helpers getLayoutPage and toPublicLayoutPagedResponse were never called. getPublicLayoutPage became unreachable when the public paging APIs moved to getPublicLayoutPageWithLockedETag for the continuation ETag lock. * Document data locality in the blob and Data Lake changelogs The Data Lake changelog had no data locality entries at all, and the blob changelog covered only getLayout, omitting the locality-aware routing of buffered downloads and the setDataLocalityEndpoint options on the one-shot download paths. * Refresh the layout cache proactively so downloads never stall on it The layout is only an optimization: the ETag on each range GET guarantees correctness no matter which endpoint serves it. So an expired or unavailable layout must degrade to the client's configured endpoint, never block or fail a download. Three problems prevented that. Jitter was dead code, because BlobLayoutCacheValue.getRefreshOn returned a fixed expiry-minus-30s that overrode the cache's jittered calculation, so every cache in the process refreshed at the same instant and stampeded getLayout. Refresh was only triggered by access, so an idle openInputStream would find the value expired on its next read and block. And a failed refresh left the next attempt already due, so each subsequent chunk launched another request at a service that was already failing. Refresh now happens on a Reactor timer at a jittered point 90 to 30 seconds before expiry, failures back off 30 seconds as the SDK spec requires, and an expired value reports no ranges so callers fall back instead of stalling. The five minute TTL is unchanged. Because a cache belongs to a single download, the timer needs a lifetime: the cache is now Closeable, disposed when the download terminates and when BlobInputStream is closed, and it stops rescheduling while nothing is reading it so an abandoned stream cannot refresh forever. * Confine the data locality endpoint pass-in to one-shot reads A buffered download needs a different endpoint per chunk, so a single caller-supplied endpoint has no coherent meaning there. ReadToFileOptions and DataLakeFileInputStreamOptions accepted one anyway: it applied to the initial request and was then silently overwritten as soon as the implicit layout resolved an endpoint for a chunk, with no error and no signal to the caller. Remove it from those two buffered options, matching Blob, where BlobDownloadToFileOptions and BlobInputStreamOptions never had it. Both options are unreleased, so nothing breaks. FileReadOptions keeps setDataLocalityEndpoint because readWithResponse is one-shot. Also scopes the sync getLayout continuation ETag per enumeration. It was allocated once per getLayout call, so two concurrent enumerations of the same PagedIterable could overwrite each other's locked ETag and take a spurious 412. Building the PagedIterable from PagedFlux.create gives each subscription its own reference, mirroring the async path; the retriever still runs the blocking sendRequest on the caller's thread. * Distinguish a caller's bad endpoint from the service's DataLocalityPolicy previously wrapped the whole rewrite in catch (RuntimeException) and only logged a warning. That is how a real NullPointerException in setPort went unnoticed, and it also meant a caller who explicitly supplied a bad endpoint for a one-shot download got no signal that their routing was ignored. The policy now validates up front and throws IllegalArgumentException naming the offending value. A bare host[:port], which is the documented service format, and a full URL both remain valid; an absent or empty endpoint is still a silent no-op. That is right for caller-supplied endpoints but wrong for endpoints the SDK resolved from a layout, where failing the download would contradict the layout being only an optimization. So BlobLayoutRangeResolver now validates service-supplied endpoints and returns null for anything unusable, which routes that chunk to the original endpoint. The ETag on each range GET keeps the data correct either way. * Gate the layout tests on a service version that exists Every data locality test required a minimum service version of 2027-03-07. No service accepts that value: a signed GET ?comp=layout sending x-ms-version: 2027-03-07 comes back 400 InvalidHeaderValue, meaning the version is not recognised rather than gated. ci.system.properties runs live tests at V2026_10_06, so RequiredServiceVersion silently skipped all of these in CI. That is why the recorded sessions contain no comp=layout requests at all. getLayout is in fact already available at 2026-10-06 on an ordinary account. Verified directly: the same request at that version returns 200 with a populated BlobLayout, and range GETs on large enough blobs come back with x-ms-download-hint: layout. Retargeting the gate lets these tests actually run; the full set passes live against a real account. * Prove chunk downloads really route to the layout endpoint The download tests used a 16 KB blob. The service only returns x-ms-download-hint: layout above a threshold between 4 MB and 16 MB, so the layout path never engaged and the tests silently asserted nothing but the fallback. The class javadoc even described the wiring as a documented no-op. Live runs now use a 16 MiB blob with a 4 MiB block size, which does receive the hint. The new live-only test attaches a pipeline policy that records the host and Host header of every outbound request, then asserts that at least one chunk was sent to a host other than the account and that those requests still carry the original account authority in the Host header. Observed in a live run: requests to blob.hkg24prdstf03f.store.core.windows.net carrying Host: vd7xscnhkg24ptf03fx.blob.core.windows.net. The assertion was verified to be non-vacuous by inverting it and confirming it fails. Playback keeps the smaller blob so existing recordings stay valid, and the routing test is LiveOnly so it skips rather than fails without a recording. * add new swagger doc with arrow customizations * Point the blob swagger at the spec that actually has the STG105 operations The generation config pointed at a 2026-10-06 spec that predates several operations already committed here, so anyone who ran autorest silently deleted them: ContainersImpl lost the Apache Arrow list-blob operations, and BlockBlobsUploadHeaders and BlockBlobsPutBlobFromUrlHeaders lost x-ms-content-crc64. Regeneration was effectively a destructive operation. Repoint to the 2027-03-07 spec, which carries getLayout, x-ms-download-hint, Apache Arrow and content-crc64 together. Note this only changes what is generated, not what goes on the wire: AzureBlobStorageImpl takes its version from BlobServiceVersion at runtime, which stays at 2026-10-06. Generating from that spec failed with 'no common parent found for client models []' because Container_ListBlobHierarchySegment_ApacheArrow is marked x-ms-pageable but returns a binary Arrow stream with no item model. Drop the extension for both Arrow list operations, matching the existing directives that do the same for the other list and page-range operations. An earlier attempt at this repoint was reverted because a stray 'git status' had been pasted onto the license-header line, corrupting the header of every generated file. The swagger choice itself was correct. The generated output is now taken verbatim. Previous commits had stripped trailing whitespace from generated javadoc, which the rest of the repo keeps, so every regeneration produced spurious churn. BlobsDownloadHeaders picks up the access-tier response headers from the newer spec; the change is purely additive. * clean up ModelHelper to be more readable * remove closeable from autorefreshingcache implementation * remove BlobLayoutCacheFactory in favor of a functional ValueProvider BlobLayoutCacheFactory only adapted a Supplier<Mono<BlobLayoutCacheValue>> to AutoRefreshingCache.ValueProvider, implementing createSync() as createAsync().block(). Make ValueProvider a @FunctionalInterface with that same blocking default so both layout cache call sites construct AutoRefreshingCache directly with a lambda. Blocking is the right default here: azure-core's sync stack already blocks internally. Both HttpClient.sendSync and NettyAsyncHttpClient.sendSync are send(request, context).block(), so a hand-written synchronous layout fetch would duplicate the paging, ETag pinning and error policy of the async path only to avoid a block that the adjacent download call performs anyway. Also reorder the BlobInputStream constructor so layoutCache precedes context, matching its call site in openInputStream. * converge AutoRefreshingCache on the session implementation Adopt the AutoRefreshingCache design from the Bifrost session-credential work so both features share one cache. The class is no longer bound to an ExpiringValue interface; callers supply a Function<T, OffsetDateTime> that extracts the expiration. SessionCredential is a public model and ExpiringValue lives in an implementation package, so the bound would have forced a public-to-implementation dependency plus a duplicate getExpiration()/getExpiresAt() pair. BlobLayoutCacheValue drops the interface and passes getExpiresOn as a method reference. This also removes a real defect: creation no longer runs while holding a monitor. Ownership is claimed by compare-and-set on a Sinks.One latch, so the provider call happens outside any lock and sync and async callers join a single in-flight creation instead of serializing across a network round trip. Failed creations now arm a retry backoff, and the refresh point uses proportional rather than absolute jitter so it degrades sensibly across both the five-minute layout TTL and longer session credential lifetimes. ValueProvider keeps the functional-interface shape with a blocking default createSync(), which lets the layout call sites stay lambdas while the session provider overrides it with its generated synchronous call. Refresh is now opportunistic on access rather than scheduled on a timer, so there is nothing to cancel: AutoRefreshingCache.close(), the BlobInputStream.close() override, and the doFinally cleanup in downloadToFile are all removed. An expired layout is now re-fetched instead of being returned and degraded to a null range list; the failure sentinel still carries a full TTL, so a known-bad layout is not retried. * refactor addDatalocalityEndpoint method to StorageImplUtils * refactor getLayout to mirror the async variant * use the synchronous getLayout stack for the sync layout cache BlobInputStream.dispatchRead calls AutoRefreshingCache.getValidValueSync, which fell through to the default createSync implementation of createAsync().block(). That drove the reactive HTTP stack from a purely synchronous API. Add BlobClientBase.fetchLayoutCacheValueSync, built on the existing synchronous getLayout paging, and supply it through an anonymous ValueProvider that overrides createSync. The explicit type argument on the anonymous class is required because the library still targets Java 8. Extract the layout fetch error policy into isFatalLayoutFetchStatus and createLayoutFallbackValue so both the sync and async paths share one definition of which statuses fail a download and which fall back to the original endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * route seekable byte channel reads through data locality openSeekableByteChannelRead is a chunked random access read path structurally identical to BlobInputStream, but it never consulted the blob layout and passed a null Context to every range GET, so its chunks always went to the client's configured endpoint. Build the layout cache from the download hint on the initial response and hand it to the read behavior, which resolves an endpoint per chunk. The layout is requested for the whole blob rather than the remaining bytes, and there is no positive remaining count gate, because a channel can seek arbitrarily backward. The lookup deliberately sits outside the existing try/catch. That block maps errors onto -1 to signal end of file, so a layout failure raised inside it would silently truncate the caller's data. The five argument constructor is kept as a delegating overload so existing tests are untouched. The class is package private, so this is not an API change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * do not apply data locality to client side encrypted clients EncryptedBlobClientBuilder never installs DataLocalityPolicy, but encrypted clients delegate to the base download paths, which fetched a layout and placed a resolved endpoint in the request context that nothing could act on. Beyond the wasted round trip, the combination is unsound. The decryption policy treats any GET carrying a body as a download to decrypt, and a layout response is XML. The encrypted range type rewrites plaintext ranges into ciphertext ranges. Layout offsets are ciphertext as stored by the service while endpoint resolution uses the caller's plaintext offset, so routing would be wrong at region boundaries under client side encryption v2. The Azure SDK for .NET does not support the combination either; its client side encryption tests explicitly opt out with LayoutAwareRouting.Disabled. Whether it should ever be supported is an open design question. Gate all three layout sites on the pipeline actually containing a DataLocalityPolicy. This needs no new public or protected API, keeps the suppression automatic rather than opt out, self corrects for any other builder that omits the policy, and is trivially reversible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * add tests for the sync layout cache and channel routing Mock transport tests rather than recordings, because the recorded sessions contain no comp=layout traffic and cannot exercise this feature. fetchLayoutCacheValueSync is covered for single page, multi page, and empty layouts, and for the error policy: 403, 404, 409 and 412 propagate while 429 and 500 fall back to a null range list. A wiring test asserts the layout request goes through the synchronous pipeline by recording whether processSync or process was invoked, which is what actually proves the createSync override is used. Seekable channel tests cover layout caching across reads, routing distinct offsets to distinct endpoints, backward seeks, absence of the download hint, soft layout failure falling back to the original endpoint, caller context survival, and fatal layout errors propagating rather than being converted to end of file. The clients are built with retries disabled so request count assertions are not coupled to the retry policy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * let callers turn off locality aware routing Data locality routing was always on whenever the service advertised a layout, with no way for a caller to opt out. The Azure SDK for .NET exposes this choice as LayoutAwareRouting on its download options, so add the Java equivalent. Introduce LayoutAwareRouting with AUTO, DISABLED and ENABLED. AUTO is the default and resolves to enabled today, but is deliberately distinct from ENABLED so the library can change what the default means without a breaking change. Routing only affects which endpoint serves a chunk, never the bytes returned. The setting is available on the three chunked download paths that route: downloadToFile, openInputStream and openSeekableByteChannelRead. Data Lake gets its own mirrored enum and a Transforms conversion, matching how ConsistentReadControl is already handled there rather than reusing the blob type the way .NET does. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * add tests for the locality aware routing setting Mock transport rather than recordings, since the recorded sessions deliberately use a blob too small for the service to advertise a layout. Cover DISABLED suppressing the layout lookup on all three chunked download paths, AUTO and null still routing so the default stays on, ENABLED routing, and the Data Lake to blob enum conversion including null. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * prove data lake chunked reads route to the layout endpoint The existing data lake locality tests were all negative: they asserted that a layout is not fetched when routing is disabled. Those pass just as happily if routing were never wired up at all, so they said nothing about whether data lake actually inherits the behavior from the blob client. Add the positive direction for openInputStream, readToFile and its async counterpart. Each asserts the layout is fetched exactly once rather than once per chunk, that chunk requests are sent to the endpoint the layout advertises, and that the original account authority survives in the Host header. Sync readToFile had no chunked coverage at all before this. Also stop getDataRequestRecords from returning the layout request, which would otherwise be counted as a data request by these assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix the byte buddy version tag group id in storage common The x-version-update tags for byte-buddy and byte-buddy-agent named the group net.byte-buddy, but the group is net.bytebuddy, which is how external dependencies records it and how every other storage pom spells it. The pom version scanner could not match the tags to the declared dependencies and failed the Analyze stage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * reorganize blob data locality tests * pin resumed layout enumerations to their first observed page ETag A caller that resumes getLayout from a marker it obtained earlier never observes the layout's first page, so there was no ETag to pin its continuation requests to and every later page went out unconditioned. Set the pinned ETag from whichever page the enumeration sees first, rather than only from a marker-less first page. The resumed page itself still cannot be conditioned, since its marker came from an enumeration whose ETag this client never saw. * restore setter ordering in ReadToFileOptions setUserPrincipalName and setLayoutAwareRouting had their javadoc and bodies interleaved, leaving each setter documented as the other. Put each method back with its own documentation. * cover blob getLayout against a live service getLayout was proven only by recordings of a seven byte blob and by mock HttpClients, so nothing showed that a real layout parses, covers the blob, or names a backend endpoint distinct from the account. Add live coverage for a small blob, for a sixteen megabyte blob (the size at which the service begins sending x-ms-download-hint), and for a range scoped request. Assertions check that the returned ranges are contiguous and cover the requested window rather than checking a range count, because the service is free to paginate a layout and today returns a single range for blobs of every size tested, up to 5.66 GB. Also assert the untested half of a service requirement: a continuation must repeat the original range header, or omit it when the original request had none. The omit case rests on BlobRange.toHeaderValue returning null for a default range, which nothing exercised. Ranged enumeration is capped at its first page. Following the marker trips a known service defect, recorded in a disabled test that asserts the behavior expected once the defect is fixed. * cover data locality download routing against a live service The download side had no live proof that the layout fetch is actually gated on x-ms-download-hint, nor that the endpoint the download routes to is the one getLayout reports. Add a live test over a four megabyte blob, which is below the size at which the service sends the hint, asserting that no comp=layout request is issued, that every request stays on the account host, and that no Host header is rewritten. Its block size is smaller than the blob so the download is genuinely chunked; sized otherwise the layout would be skipped because no bytes remain after the initial chunk, and the test would pass without exercising the hint check at all. Strengthen the existing sixteen megabyte routing test to assert that the layout is fetched exactly once, since it is cached for the duration of the download, and that every routed chunk host appears among the endpoints getLayout reports. Enumerations are counted rather than requests, because continuation pages are also comp=layout calls. Requests now record their URL so layout calls can be identified and so a live failure prints what was actually sent. * Flatten Data Lake layout pagination * wire in bloblayout through ModelHelper * remove blobranges since bloblayout was laready included * remove enumerated values for properties, replace with properties models * remove pageSize, change BlobRequestConditions to a shallow copy on set * getLayout now pages directly over blobLayoutRange, change from bloblayout containing list of ranges * feat(storage): type data locality endpoints Replace raw String endpoint values across storage-common, Blob, and Data Lake with a validated DataLocalityEndpoint value type. This prevents malformed endpoint strings from reaching the request pipeline while preserving fallback for invalid service layout data. * Revert "feat(storage): type data locality endpoints" This reverts commit b9c7679. * add description for to getters for `getEndpoint` * Fix Data Lake file client Javadoc link * Reduce Data Lake timeout test flakiness * Fix storage layout Javadoc links --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
|
Azure Pipelines: Successfully started running 1 pipeline(s). 34 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run java - storage - ci |
|
No pipelines are associated with this pull request. |
browndav-msft
marked this pull request as ready for review
September 25, 2026 17:07
browndav-msft
requested review from
a team,
Alan Zimmer (alzimmermsft),
gunjansingh-msft,
Isabelle (ibrandes),
Kyle Knapp (kyleknap) and
Sean McCullough (seanmcc-msft)
as code owners
September 25, 2026 17:07
… stg105/dataLocality # Conflicts: # sdk/storage/azure-storage-file-share/assets.json
Member
Author
|
/azp run java - storage - tests |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Isabelle (ibrandes)
changed the title
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved locality-routing interactions with encryption, transport headers, and retry handling can break downloads.
4 open issues
7 resolved since last review
Review effort: Balanced
…d test for encrypted
Member
Author
|
/azp run java - storage - tests |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Member
Author
|
/azp run java - storage - tests |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
browndav-msft
merged commit Oct 28, 2026
134a3f4
into
Azure:feature/storage/stg105base
4 of 9 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cherry-pick data locality from private drop