Event connection listener remains in LISTEN and blocks subsequent event connections

[jhhenriquee]

Summary

We are experiencing an intermittent problem with Firebird's event notification mechanism using RemoteAuxPort = 3051.

Under normal operation, the auxiliary event connection is established and the listening socket on port 3051 disappears from LISTEN after the client connects.

However, in the problematic situation, the client starts a TCP connection to port 3051, Firebird replies with SYN/ACK, but the TCP handshake is not completed because the final ACK from the client does not reach the Firebird server.

After this happens, the Firebird process remains with port 3051 in LISTEN, and subsequent attempts to establish event connections fail with:

Unable to complete network request to host "SKY-INF-BD-01".
Timeout occurred while waiting for a secondary connection for event processing
Broken pipe

Environment

• Firebird: 4.0.6.3221
• Distribution: HQbird
• Architecture: SuperServer
• OS: Ubuntu 20.04.6 LTS
• Kernel: Linux 5.15.0-1081-oracle
• Architecture: x86-64
• Server: cloud VM (KVM)
• RemoteAuxPort = 3051

The server is running in a cloud environment.

The network connection between the client and server has approximately 16 ms latency, with no packet loss observed during testing.

Observed behavior

At approximately 13:38:20 (UTC-3), the following TCP sequence was captured on the Firebird server:

13:38:20.446139  177.222.147.26:58000 -> 10.107.93.111:3051  SYN
13:38:20.446158  10.107.93.111:3051 -> 177.222.147.26:58000  SYN/ACK

13:38:28.488768  10.107.93.111:3051 -> 177.222.147.26:58000  SYN/ACK
13:38:44.616772  10.107.93.111:3051 -> 177.222.147.26:58000  SYN/ACK

There is no final ACK from the client in the server-side packet capture.

Therefore, this particular TCP connection never reaches the established state.

Importantly, this does not appear to be a general network connectivity problem. The client/server connection normally has approximately 16 ms latency and we have not observed packet loss between the client and server.

Firebird stack trace

A stack trace taken while the problem was occurring showed a Firebird thread in:

poll()
  -> Select::select()       inet.cpp:454
  -> aux_connect()          inet.cpp:1509
  -> aux_request()          server.cpp:2691

The poll() was waiting with a timeout of approximately 180 seconds.

At the same time, multiple other Firebird threads were blocked trying to acquire GlobalPortLock, for example:

pthread_mutex_lock()
  -> Firebird::Mutex::enter()
  -> GlobalPortLock::GlobalPortLock()
  -> aux_request()

There were many threads waiting on the same GlobalPortLock.

Subsequent behavior

After the problematic connection, existing event connections on port 3051 remain established.

However, subsequent attempts to establish new secondary event connections fail with:

Unable to complete network request to host "SKY-INF-BD-01".
Timeout occurred while waiting for a secondary connection for event processing
Broken pipe

For example, at 13:44:06 the Firebird server logged this error.

The problem persists until the Firebird process is restarted.

Question

Is this behavior expected?

In particular, we would like to understand whether GlobalPortLock is intentionally held while aux_request() waits inside aux_connect() / poll() for the secondary connection.

If the client does not complete the TCP handshake, can this cause the auxiliary port listener to remain in LISTEN and prevent subsequent event connection attempts from acquiring GlobalPortLock?

Could this be a bug or a known limitation in the event connection implementation?

We would also like to know whether the event connection mechanism should recover automatically when a client starts but does not complete the TCP handshake.

We can provide the complete stack trace and packet captures if they are useful for investigation.