SocketException and after always AlreadyClosedException

[kresik3]

• RabbitMQ client version - 5.14.2
• Kotlin version - 1.6.10
• Android version - 7.0

Hello, I use a queue to transfer logs from device to local server. The device may has bad wifi connection so sometimes I catch the errors and no more restore connection to the queue.
Do I need to re-open channel or connection manually or hope for auto-recovery?

First error from rabbitmq client:
java.net.SocketException: No route to host
Long monitor contention with owner DefaultDispatcher-worker-3 (4142) at void java.net.SocketOutputStream.socketWrite0(java.io.FileDescriptor, byte[], int, int)(SocketOutputStream.java:-2) waiters=0 in void com.rabbitmq.client.impl.SocketFrameHandler.writeFrame(com.rabbitmq.client.impl.Frame) for 80.504s
Long monitor contention with owner DefaultDispatcher-worker-3 (4142) at void java.net.SocketOutputStream.socketWrite0(java.io.FileDescriptor, byte[], int, int)(SocketOutputStream.java:-2) waiters=0 in void com.rabbitmq.client.impl.AMQChannel.processShutdownSignal(com.rabbitmq.client.ShutdownSignalException, boolean, boolean) for 5.363s
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:112)
at java.net.SocketOutputStream.write(SocketOutputStream.java:157)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:154)
at com.rabbitmq.client.impl.Frame.writeTo(Frame.java:189)
at com.rabbitmq.client.impl.SocketFrameHandler.writeFrame(SocketFrameHandler.java:191)
at com.rabbitmq.client.impl.AMQConnection.writeFrame(AMQConnection.java:627)
at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:127)
at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:455)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:428)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:710)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:685)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:212)

Then I always get:
com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: com.rabbitmq.client.MissedHeartbeatException: Heartbeat missing with heartbeat = 60 seconds
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:258)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:427)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:710)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:685)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:212)
OR
com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: java.net.SocketException: No route to host
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:258)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:427)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:710)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:685)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:212)

Source Code:
loop(5 seconds) {
basicPublish()
waitForConfirms()
}

Thank you,
Best regards

[michaelklishin]

Please use GitHub Discussions for questions.

java.net.SocketException: No route to host

means the hostname did not resolve (using DNS).

com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: com.rabbitmq.client.MissedHeartbeatException: Heartbeat missing with heartbeat = 60 seconds
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:258)

means then a heartbeat timeout is detected
and it tries to close the already closed connection. This can be improved
in the client to be idempotent but it's a rare scenario to hit, and the outcome would
be no meaningfully different.

[kresik3]

Ok, thanks.

Perhaps we have troubles with wifi connection or server side, but is it auto recoverable or I have to reopen channel or connection?

[michaelklishin]

This is documented. There is nothing wrong with manual recovery if you know what you're doing but you certainly cannot combine both.

[kresik3]

Ok, I understand about possibility for manual and auto recovery. But I want to know, are the errors autorecoverable or I need to change logic and do manual recovery because there is no way to auto recovery after the errors?

[michaelklishin]

This is a socket exception followed by a missed heartbeat. The latter might have stopped or restarted the recovery, there is no way to tell. But connection failure does initiate connection recovery.

[acogoluegnes]

If you get an exception in the publishing looping, it's likely the connection died and the recovery mechanism failed or did not trigger. You should then re-open the connection and the channel.

As your connection seems very unstable, you may consider increasing the heartbeat and/or changing the default recovery condition [1] [2] to avoid this.

[1] https://rabbitmq.github.io/rabbitmq-java-client/api/current/com/rabbitmq/client/ConnectionFactory.html#setConnectionRecoveryTriggeringCondition(java.util.function.Predicate)
[2]

rabbitmq-java-client/src/main/java/com/rabbitmq/client/impl/recovery/AutorecoveringConnection.java

Lines 65 to 66 in d1efd36

	public static final Predicate<ShutdownSignalException> DEFAULT_CONNECTION_RECOVERY_TRIGGERING_CONDITION =
	cause -> !cause.isInitiatedByApplication() || (cause.getCause() instanceof MissedHeartbeatException);

[kresik3]

Thank you,

Is there any function or indicator to know recovery working or died?

[acogoluegnes]

There's ExceptionHandler#handleConnectionRecoveryException(Connection, Throwable).