Skip to content

Langfuse.shutdown() hangs forever on _score_ingestion_queue.join() — cached LangfuseResourceManager is reused after shutdown #1799

Open

Description

@bagowix

Summary

Langfuse.shutdown() (and flush()) can block forever on self._score_ingestion_queue.join() in LangfuseResourceManager.flush().

Root cause: LangfuseResourceManager instances are cached process-wide by public_key (LangfuseResourceManager._instances), but shutdown() does not remove the instance from the cache. Any Langfuse client constructed afterwards with the same public_key silently receives the already-shut-down manager: its ingestion consumer threads are dead, so any event enqueued after that point (e.g. create_score(), which checks neither _shutdown nor consumer liveness) is never consumed, Queue.unfinished_tasks never reaches 0, and the next flush()/shutdown() blocks forever on Queue.join().

Minimal reproduction

import faulthandler
import threading

from langfuse import Langfuse

faulthandler.dump_traceback_later(10, exit=True)

COMMON_KWARGS = dict(
    public_key='pk-lf-repro',
    secret_key='sk-lf-repro',
    host='http://localhost:9',  # unreachable on purpose; nothing is sent in this repro
)

client_a = Langfuse(**COMMON_KWARGS)
client_b = Langfuse(**COMMON_KWARGS)

assert client_a._resources is client_b._resources  # same cached resource manager

client_a.shutdown()  # stops the shared ingestion consumer threads for good

client_b.create_score(name='quality', value=1.0)  # enqueued, but no consumer is alive anymore

print('threads alive:', [t.name for t in threading.enumerate()], flush=True)
print('score queue unfinished tasks:', client_b._resources._score_ingestion_queue.unfinished_tasks, flush=True)

client_b.shutdown()  # hangs forever on _score_ingestion_queue.join()

print('UNREACHABLE: shutdown returned', flush=True)

Output (the watchdog kills the process after 10 s; without it the hang is indefinite):

threads alive: ['MainThread', 'OtelBatchSpanRecordProcessor', 'Thread-3']
score queue unfinished tasks: 1
Timeout (0:00:10)!
...
Thread 0x00000001f38bde80 (most recent call first):
  File ".../python3.12/threading.py", line 355 in wait
  File ".../python3.12/queue.py", line 90 in join
  File ".../langfuse/_client/resource_manager.py", line 608 in flush
  File ".../langfuse/_client/resource_manager.py", line 620 in shutdown
  File ".../langfuse/_client/client.py", line 2340 in shutdown
  File "langfuse_deadlock_repro.py", line 34 in <module>

Note that no IngestionConsumer threads are alive when client_b enqueues the score — they were joined by client_a.shutdown().

How we hit this in practice

In a pytest suite, several independently built application instances each create their own Langfuse client with the same public_key and shut it down in their teardown. Because all of them share the single cached LangfuseResourceManager, the first teardown kills the consumer threads for everyone; a later teardown then hangs the whole test process forever on _score_ingestion_queue.join(). The same pattern can hang a production graceful shutdown if two clients with one key exist in a process (e.g. app + sidecar worker, or re-initialization after a config reload).

Expected behavior

Either of:

  • shutdown() removes the instance from LangfuseResourceManager._instances, so the next Langfuse(...) with that key builds a fresh, working manager; and/or
  • add_score_task() / add_trace_task() refuse to enqueue (or __new__ refuses to return a manager) once _shutdown is set; and/or
  • flush() uses a bounded wait instead of a bare Queue.join(), so a lost consumer cannot block shutdown indefinitely.

Environment

  • langfuse 4.14.1
  • Python 3.12.x (CPython), macOS 15 (arm64) — also reproduced on Linux in CI
  • No Langfuse server involved; the hang is purely client-side

Activity

  1. hassiebp commented on Aug 6, 2026

    @hassiebp
    Collaborator

    @bagowix Thanks for raising this - why are you calling shutdown in an application that lives on? Using flush should suffice

  2. bagowix commented on Oct 20, 2026

    @bagowix
    Author

    @hassiebp Thanks for looking into it!

    We don't call shutdown() while the app is running. It sits in the app's lifespan teardown (the DI container finalizer), so every app instance calls it once, when that instance stops. We treat the client like the DB pool or the HTTP client: whoever creates it closes it.

    The process, however, outlives app instances. Our pytest suite has a session-wide app with its own container and Langfuse(...) client, and the MCP tests build extra apps via create_mcp_app(), each with its own client, torn down at the end of the test. All of them use the same public key, so they share one cached LangfuseResourceManager. The first MCP test teardown stops the consumer threads the session-wide client still relies on, and the session teardown then hangs forever.

    flush() would avoid the hang, and with a single app per process the atexit hook covers the rest. Still, nothing in the API says that Langfuse(...) returns a shared object, so calling shutdown() on "your own" client looks safe. Production can end up in the same shape: an API app and a worker in one process, or a client rebuilt after a config reload.

    For now we work around it with a unique public key per disabled client in tests and a 5 s timeout around shutdown(). I saw #1800 was closed without merging. Do you plan to fix this in the SDK, or would you rather document that shutdown() affects every client with the same key?

  3. hassiebp commented on Oct 20, 2026

    @hassiebp
    Collaborator

    Could you please retry with the latest SDK version? We have shipped improvements on flushing behavior for scores recently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions