[BUG] Service Bus Java scheduleMessage does not retry a transient server-busy response despite maxRetries=3 #51006

Describe the bug

With com.azure:azure-messaging-servicebus:7.18.1, the public scheduling API returns a transient ServiceBusException with reason SERVICE_BUSY after a single scheduling send when the broker's management response carries:

status-code: 503
error-condition: com.microsoft:server-busy
status-description: Error code: 50009. Entity is being throttled.

This occurs with AmqpRetryOptions.maxRetries=3. Our fixture would return a successful scheduling response on the next send, but the SDK does not make that second attempt. A transient failure while opening the send link is retried successfully in a separate positive control.

We encountered this while investigating retry handling for scheduled timeout messages. Applications relying on automatic SDK retry for scheduling can instead receive the transient rejection immediately and need their own operation-level policy.

The attached reproduction uses published SDK binaries with simulated broker responses. It is an isolated unit reproduction, not a live Azure throttling test or a claim that every production throttling error follows this path.

Exception and observed result

The scheduling call throws ServiceBusException; the tests assert:

assertEquals(ServiceBusFailureReason.SERVICE_BUSY, error.getReason());
assertTrue(error.isTransient());
assertEquals(1, fixture.sends.get());

The exception description contains the synthetic 50009 message above. The companion archive includes the observed expected-behaviour test failure in evidence/expected-regression-results.txt.

To reproduce

Archive SHA-256: 89d383d05ee3a533f4c2bb5767996856090ca8e20f15eca4e6d1c50b8381bfd3.

Use the attached servicebus-scheduling-retry-reproducer.zip, which contains a standalone Maven project, test source and captured evidence. It requires Java 21 and Maven. No application repository, Azure credentials, live namespace or running broker is required. Maven needs access to dependencies on the first run.

1. Extract the archive and enter servicebus-scheduling-retry-reproducer/.

2. Run the diagnostic/characterisation suite:

   mvn -B -ntp test

   Expected on the affected version: six tests pass. These tests assert the observed missing retry and the controls; their passing does not mean the scheduling retry succeeded.

3. Run the separate regression test that expects automatic retry:

   mvn -B -ntp -Dtest=SchedulingRetryExpectedBehaviourIT test

   This test fails on 7.18.1: scheduleMessage throws the first busy exception instead of consuming the fixture's second successful response and returning sequence number 42. The IT suffix keeps this expected failure out of the default diagnostic run.

The diagnostic cases cover:

Code snippet

The public sender in the reproduced scheduling case is configured as follows:

ServiceBusSenderAsyncClient client = new ServiceBusClientBuilder()
    // Deliberately fake; its connection is replaced by the fixture before use.
    .connectionString("Endpoint=sb://audit.invalid/;SharedAccessKeyName=audit;SharedAccessKey=YXVkaXQ=")
    .retryOptions(new AmqpRetryOptions()
        .setMaxRetries(3)
        .setTryTimeout(Duration.ofSeconds(2))
        .setDelay(Duration.ofMillis(1))
        .setMaxDelay(Duration.ofSeconds(5)))
    .sender()
    .queueName("audit-queue")
    .buildAsyncClient();

// No application retry surrounds the public call in the failing regression.
client.scheduleMessage(message, originalDueAt).block(Duration.ofSeconds(15));

The complete fixture is in src/test/java/com/example/servicebus/SchedulingRetryReproducerTest.java. The expected-behaviour regression is alongside it in SchedulingRetryExpectedBehaviourIT.java.

The public sender, ManagementChannel and RequestResponseChannel are real SDK implementations. Connection acquisition, authentication, channel cache, serializer, Proton links, dispatcher and handler events are test fixtures. The serializer supplies the synthetic successful sequence number 42. Response status interpretation and the retry operators being investigated execute in real SDK code. Reflection is used to instantiate internal constructors and inject the fake connection; this is white-box reproduction code, not suggested application usage.

Each send passes through the real request/response channel's encoding and correlation handling. Counts come from the fake Proton send callback, not actual network packets. Both channel and public client have three retries, a two-second try timeout, a one-millisecond configured delay and a five-second maximum delay. The overall wait permits the SDK's additional backoff. These timings, and the positive control's one-millisecond application retry delay, are test settings rather than production guidance.

Expected behaviour

For this explicit transient busy rejection, the scheduling operation should be retried according to the configured SDK policy. With a busy response followed by success, we expect the call to succeed and return the successful sequence number after two scheduling sends.

Please confirm whether automatic retry of this scheduling management response is intended SDK behaviour. If this response is intentionally excluded from automatic retry, please document that limitation and the recommended application retry boundary. This report does not propose treating a lost response or uncertain acceptance as a definite refusal.

Setup

• Library: com.azure:azure-messaging-servicebus:7.18.1
• Resolved Azure core: com.azure:azure-core:1.60.0
• Resolved Azure AMQP: com.azure:azure-core-amqp:2.12.2
• Test dependencies: JUnit Jupiter 5.11.4, Mockito 5.17.0
• Java: OpenJDK 21.0.10, Homebrew distribution
• Maven: 3.9.12
• OS: macOS 14.6.1, Apple Silicon / aarch64
• IDE: none required; reproduced from Maven CLI
• App server/framework: none in the standalone reproduction
• Broker: simulated AMQP management responses; no live Azure broker in this test

The portable project's POM configures Mockito's Java agent through Maven's local-repository setting, without machine-specific paths. Runtime Surefire classpaths were checked to verify the resolved SDK/core versions.

Earlier probes also observed the failure through the public synchronous and asynchronous APIs:

The first two rows held our resolved core dependencies constant. Those earlier probes mocked RequestResponseChannel; the attached 7.18.1 project strengthens the reproduction by executing the real request/response channel as well. The archive focuses on the standalone 7.18.1 case rather than claiming to rerun the full version matrix.

Additional context: suspected retry boundary

Inspection of the exact published 7.18.1 and core-amqp 2.12.2 sources suggests:

1. ServiceBusSenderAsyncClient.scheduleMessageInternal retries getSendLinkWithRetry(...), then invokes managementNode.schedule(...) downstream of that retry.
2. RequestResponseChannel.sendWithAck retries endpoint readiness, then performs the request/response work outside that retry.
3. RequestResponseChannel.settleMessage emits a received management response with sink.success(message), including a response carrying status 503.
4. ManagementChannel.sendWithVerify then interprets the unsuccessful status, creates an AMQP exception and maps it to ServiceBusException. There is no enclosing scheduling-operation retry at this point.

Relevant source: sender scheduling, management response verification, request/response send, and response settlement. These immutable links identify commit 30a1e8c2c765d3f016965cc26ca8cc660f32c6b6; all three Java files were verified to match the published source files exactly. The version-specific analysis used the published 7.18.1 source artifact and core-amqp 2.12.2 source artifact. The GitHub commit is a verified source reference, not a claim about the release commit.

There is a further type-boundary detail: RetryUtil.createRetry accepts transient AmqpException and timeouts, whereas management scheduling has already mapped its error to ServiceBusException. The included helper-control test demonstrates that simply placing the existing SDK retry helper outside the mapped management operation does not retry this error. This is supporting diagnosis, not a proposed production patch.

The 7.17.11 fix, PR #44920, adds retry around link creation. The attached link-failure positive control verifies that this retry works; the scheduling-response case occurs later. The 7.18.1 release notes list dependency updates rather than a scheduling-response retry change.

Related issues

• #24620 — Apply AmqpRetryPolicy to RequestResponseChannel.sendWithAck: concerns missing acknowledgement timeout/retry in the same underlying API. The maintainer discussion mentions performance considerations. This report concerns an explicit busy response after successful response delivery, rather than an absent acknowledgement.
• #34646 — Failure to send when request response channel is disposed, with no retry: similar missing-retry symptom, but its stack fails during authentication/channel disposal rather than a scheduling busy response.
• #35511 — Retry com.microsoft:operation-cancelled errors: related management-response classification, involving peekMessage and a different error condition.
• PR #44920 — Add retry for create-send-link in single-message send: related fix with a narrower retry boundary, as described above.

We did not locate an existing report precisely covering the attached scheduling busy-response reproduction. Please link or deduplicate this report if one exists.

Information checklist

• Bug description added
• Standalone reproduction and run commands added
• Setup and resolved dependency versions added
• Expected and actual behaviour distinguished
• Simulated broker boundary and limitations stated
• Related issues and earlier retry fix linked