Skip to content

[BUG] Kubernetes discovery GetConns can leak mutex lock during concurrent cache initialization #3764

Description

@buvidk1234

OpenIM Server Version

main

Operating System and CPU Architecture

Linux (AMD)

Deployment Method

Source Code Deployment

Bug Description and Steps to Reproduce

KubernetesConnManager.GetConns uses a double-check pattern when connMap[serviceName] is not found on the first read-lock check.

If another goroutine initializes the same service connections before the second check runs, the second check returns cached connections directly while still holding k.mu.Lock().

k.mu.Lock()
// Check if another goroutine has already initialized the connections when we released the read lock
conns, exists = k.connMap[serviceName]
if exists {
    return conns, nil
}
k.mu.Unlock()

This leaves the mutex locked indefinitely. Later calls that need k.mu can block forever, including GetConns, endpoint refresh handling, AddOption, or Close.

This can affect Kubernetes deployments under concurrent startup or traffic spikes, especially gateway fanout / online push paths that call GetConns for message gateway services.

Expected behavior

GetConns should always release k.mu before returning from the second cache-check branch.

Actual behavior

The second cache-hit branch returns without unlocking k.mu.

Possible trigger scenario

  1. Goroutine A calls GetConns(serviceName) and misses the first read-lock check.
  2. Goroutine B also calls GetConns(serviceName) and misses the first read-lock check.
  3. Goroutine A initializes and stores connMap[serviceName].
  4. Goroutine B enters the second check, sees the initialized cache, and returns without unlocking.
  5. Future discovery operations block on k.mu.

Suggested fix

Unlock before returning from the second cache-hit branch:

if exists {
    k.mu.Unlock()
    return conns, nil
}

Screenshots Link

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions