Skip to content

[BUG] - Handler.close livelocks on FAIL_NON_SERIALIZED while holding the ReactorConnection monitor #50879

Open

Description

[BUG] Handler.close livelocks on FAIL_NON_SERIALIZED while holding the ReactorConnection monitor

Describe the bug

Handler.close() retries endpointStates.emitNext(CLOSED) forever when the result is FAIL_NON_SERIALIZED. ReactorConnection.closeConnectionWork() calls close() while holding the connection monitor (private synchronized). If reactor-executor is already inside emitNext on that same sink, the retry never succeeds, the monitor is never released, and the reactor thread blocks forever in getOrCreateConnection().

The loop uses about one core and allocates on the order of 1 GiB/s of log events. It does not time out. In the occurrence below it ran until the process was restarted, more than a day later. Message settlement on that connection stopped for the whole interval.

This is not #39999. That report is the Event Hubs buffered producer logging FAIL_NON_SERIALIZED, retrying a bounded number of times, and then failing the send. Handler.close() returns true with no limit and no backoff, and it does so while holding the connection monitor.

Exception or Stack Trace

No exception escapes. The close path logs this once per retry:

WebSocketsConnectionHandler {"az.sdk.message":"Could not emit closed endpoint state. Non-serial access. Retrying.","connectionId":"MF_b63740_...","signalType":"onNext","emitResult":"FAIL_NON_SERIALIZED"}

About 3.4e7 of those lines were written in the first 50 seconds, and about 1.1e8 in the first 2.5 minutes. The thread stayed in this state after that.

Application thread, RUNNABLE, about one core. Top of the dump, 50 seconds into the loop:

com.azure.core.util.logging.LoggingEventBuilder.log(LoggingEventBuilder.java:208)
com.azure.core.amqp.implementation.handler.Handler.lambda$close$0(Handler.java:157)
reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:32)
com.azure.core.amqp.implementation.handler.Handler.close(Handler.java:154)
com.azure.core.amqp.implementation.ReactorConnection.closeConnectionWork(ReactorConnection.java:627)
com.azure.core.amqp.implementation.ReactorConnection.lambda$closeAsync$2(ReactorConnection.java:595)
...
com.azure.core.amqp.implementation.ReactorConnection$ReactorExceptionHandler.onConnectionError(ReactorConnection.java:760)
com.azure.core.amqp.implementation.ReactorConnection.lambda$getOrCreateConnection$2(ReactorConnection.java:716)
...
com.azure.core.amqp.implementation.ReactorDispatcher.signalWorkQueue(ReactorDispatcher.java:160)
com.azure.core.amqp.implementation.ReactorDispatcher.invoke(ReactorDispatcher.java:111)
com.azure.core.amqp.implementation.ProtonSession.open(ProtonSession.java:177)

reactor-executor-3, BLOCKED entering the same monitor, for the whole interval. CPU time flat. Lock owner is the application thread above.

com.azure.core.amqp.implementation.ReactorConnection.getOrCreateConnection(ReactorConnection.java:688)
...
reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)
com.azure.core.amqp.implementation.handler.Handler.onNext(Handler.java:103)
com.azure.core.amqp.implementation.handler.ConnectionHandler.onConnectionRemoteOpen(ConnectionHandler.java:335)
org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:91)

Line numbers are azure-core-amqp 2.11.3. The same structure is still on main: close() returns true only for FAIL_NON_SERIALIZED, and both closeConnectionWork() and getOrCreateConnection() are private synchronized.

To Reproduce

Intermittent. Seen more than once on a ServiceBusProcessorClient workload that settles on an application thread pool by calling ServiceBusReceivedMessageContext.abandon() / complete() (both block()). Not reproduced on demand.

Code Snippet

Application side, one call per message, off the reactor thread:

messageContext.abandon(); // ServiceBusReceivedMessageContext.abandon -> receiverClient.abandon(...).block()

What the SDK then does, from 2.11.3:

// Handler.close — only FAIL_NON_SERIALIZED retries, and it retries without a bound
endpointStates.emitNext(EndpointState.CLOSED, (signalType, emitResult) -> {
    if (emitResult == Sinks.EmitResult.FAIL_NON_SERIALIZED) {
        logger.atInfo()...log("Could not emit closed endpoint state. Non-serial access. Retrying.");
        return true;
    }
    return false;
});
// Reactor's InternalManySink.emitNext
for (;;) {
    Sinks.EmitResult emitResult = tryEmitNext(value);
    if (emitResult.isSuccess()) {
        return;
    }
    if (failureHandler.onEmitFailure(SignalType.ON_NEXT, emitResult)) {
        continue;
    }
    // ...
}
// ReactorConnection.closeAsync — caller runs closeConnectionWork when the dispatcher rejects the schedule
if (dispatcher != null) {
    try {
        dispatcher.invoke(() -> closeConnectionWork());
    } catch (IOException | RejectedExecutionException e) {
        closeConnectionWork(); // synchronized, calls handler.close()
    }
}

ReactorDispatcher.signalWorkQueue is what raises the error that enters onConnectionError. A ClosedChannelException on the proton IO pipe, before the reactor is marked closed, becomes:

connectionId[...] IO Sink was interrupted before reactor closed.

Timeline from client logs. Host name omitted. t0 is the first failed abandon().

Time Thread Log
t0 application threads blocked in abandon() ReactiveException / InterruptedException from Mono.block at ServiceBusReceivedMessageContext.abandon
t0 + 35 ms those threads ReactorDispatcher instance is closed for connection MF_f318b1, then onConnectionError, Starting new reactor: IO Sink was interrupted before reactor closed. isTransient=false, isInitiatedByClient=false. onConnectionRemoteClose for this connection has no AMQP error condition
t0 + 579 ms reactor-executor-3 onConnectionRemoteOpen for the replacement connection MF_b63740
t0 + 624 ms application thread Same IO Sink was interrupted onConnectionError for MF_b63740, then Could not schedule closeConnection work. Manually disposing.
t0 + 626 ms onward that application thread Could not emit closed endpoint state. Non-serial access. Retrying., signalType=onNext, emitResult=FAIL_NON_SERIALIZED, until process restart

reactor-executor-3 is still inside the onConnectionRemoteOpen emit when close() emits CLOSED. tryEmitNext returns FAIL_NON_SERIALIZED. The handler returns true, so emitNext spins, still inside synchronized closeConnectionWork(). The reactor thread cannot leave its own emit, because a downstream subscriber is blocked on that same monitor in getOrCreateConnection().

No log in the window matched amqp:connection:forced, a link detach error, a timeout, or a transport error condition.

Expected behavior

Handler.close() should stop retrying. A failed CLOSED emit should not hold the ReactorConnection monitor or pin a reactor thread. Connection recovery should be able to finish, and abandon() / complete() should complete or fail instead of livelocking.

Setup (please complete the following information):

  • OS: Linux
  • Library/Libraries: com.azure:azure-messaging-servicebus:7.17.17, which pulls com.azure:azure-core-amqp:2.11.3. reactor-core 3.7.x (InternalManySink.emitNext still busy-loops when the handler returns true)
  • Java version: HotSpot, JDK 15+ lambda frame form ($$Lambda/0x...)
  • Frameworks: none. ServiceBusProcessorClient, blocking abandon() / complete() from an application executor
  • Also checked: azure-messaging-servicebus 7.18.1 / azure-core-amqp 2.12.2, and current main. Handler.close, ReactorConnection.closeConnectionWork, and getOrCreateConnection are unchanged. 7.18.1 still calls .block() from ServiceBusReceivedMessageContext.abandon() and complete()

Information Checklist

  • Bug Description Added
  • Repro Steps Added
  • Setup information Added

Activity

  1. added
    questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
    ClientThis issue points to a problem in the data-plane of the library.
    customer-reportedIssues that are reported by GitHub users external to the Azure organization.
    needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK team
    on Oct 20, 2026
  2. github-actions commented on Oct 20, 2026

    @github-actions
    Contributor

    Thank you for your feedback. Tagging and routing to the team member(s) best able to assist.

    Generated by Agentic Triage for #50879 · auto · 60.5 AIC · ⌖ 3.55 AIC · ⊞ 16.2K · ◷

  3. github-actions commented on Oct 20, 2026

    @github-actions
    Contributor

    🎯 Agentic Issue Triage

    Summary: Handler.close() retries emitNext(CLOSED) forever on FAIL_NON_SERIALIZED while holding the ReactorConnection monitor, causing a CPU-spinning livelock that blocks connection recovery and message settlement indefinitely.

    📋 Issue Details

    Package: com.azure:azure-messaging-servicebus:7.17.17 (pulls com.azure:azure-core-amqp:2.11.3), also reproduced structurally on current main
    Affected API: com.azure.core.amqp.implementation.handler.Handler.close(), ReactorConnection.closeConnectionWork() (private synchronized), ReactorConnection.getOrCreateConnection() (private synchronized)
    Scenarios:

    • ServiceBusProcessorClient workload calling blocking ServiceBusReceivedMessageContext.abandon()/complete() from an application thread pool
    • A ClosedChannelException on the proton IO pipe triggers onConnectionError → closeConnectionWork() while the reactor-executor thread is still inside emitNext on the same sink from onConnectionRemoteOpen

    Root ask: Handler.close() should not retry emitNext unboundedly on FAIL_NON_SERIALIZED, and a failed CLOSED emit should not hold the ReactorConnection monitor or pin a reactor thread, so connection recovery and abandon()/complete() calls can complete or fail instead of livelocking.

    🔎 Debugging / Reproduction Notes

    The reporter provides a detailed timeline and two thread dumps showing the application thread spinning in Handler.close() → InternalManySink.emitNext() (RUNNABLE, ~1 core, ~1 GiB/s of log allocation) while reactor-executor-3 is BLOCKED entering the same synchronized monitor via getOrCreateConnection(). The reporter distinguishes this from the previously reported #39999 (bounded retry/logging in Event Hubs buffered producer) — here the retry in Handler.close() is unbounded and occurs while holding the connection's monitor lock, producing a true livelock rather than a bounded failure path. Confirmed by the reporter to still be structurally present on azure-messaging-servicebus 7.18.1 / azure-core-amqp 2.12.2 and current main.

    Suggested investigation steps:

    1. Review Handler.close() in azure-core-amqp to bound retries on FAIL_NON_SERIALIZED or avoid retrying while the call stack holds the ReactorConnection monitor.
    2. Audit ReactorConnection.closeConnectionWork() and getOrCreateConnection() for the synchronized scope relative to blocking/looping operations like emitNext.
    3. Attempt to reproduce using a forced ClosedChannelException/reconnect race concurrent with an application thread calling blocking abandon()/complete(), to validate the proposed fix under the same interleaving described in the timeline.
    🏷️ Label Confidence
    • Category: Client — The defect is in SDK implementation code (azure-core-amqp's Handler/ReactorConnection classes), not a service/REST API behavior issue.
    • Service: Azure.Core.AMQP — The issue is explicitly scoped to com.azure:azure-core-amqp internals, with file/line references to Handler.java and ReactorConnection.java, independent of the Service Bus-specific trigger path.
    • Confidence: High — The issue unambiguously names the affected package/classes, includes precise stack traces, and the root cause is isolated to one component with no ambiguity across services.
    👥 Owner Routing
    • Matched CODEOWNERS entry: # ServiceLabel: %Azure.Core.AMQP (line 68) — predicted labels include Azure.Core.AMQP, matching on bottom-to-top scan.
    • AzureSdkOwners: @conniey
    • ServiceOwners: none listed
    • Routing action: Assigned @conniey; added needs-team-attention since the reporter is an external customer.
    • Scan notes: No later/more-specific CODEOWNERS entry (e.g., combined Service Bus + AMQP) was found; the AMQP-specific entry is the most precise and only match for the predicted labels.

    Generated by Agentic Triage for #50879 · auto · 60.5 AIC · ⌖ 3.55 AIC · ⊞ 16.2K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Azure.Core.AMQPazure-core-amqpClientThis issue points to a problem in the data-plane of the library.customer-reportedIssues that are reported by GitHub users external to the Azure organization.needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions