fix(stream): remove del handlers - #65
Merged
Merged
Conversation
rparthasarathy-do
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to upstream fix: BerriAI#24929
Every provider HTTP client litellm creates gets cached and reused for an hour, then the cache lets it expire. The class behind that client has a del method: whenever Python's garbage collector notices nothing is holding onto that client object anymore, del fires and forcibly closes its network connections.
The bug is that "nothing holding onto it" can happen while a customer's streaming response is still actively reading from that exact connection. The stream doesn't keep a reference to the wrapper object, only to the response it's reading, so once the one-hour cache entry expires and nothing else references the wrapper, GC calls del and yanks the socket out from under the still-open stream.
We need to fix it because this isn't a rare edge case, it's baked into normal operation: every client in the fleet gets recycled every hour, and whichever streams happen to be mid-flight at that exact moment get their connection killed. That's what's producing the drops we've been seeing clustered at the same time each hour, roughly the point where the process (or fleet) last started plus a multiple of an hour.
From the customer's side this shows up as a stream that just stops, no error, no more bytes, until their own client eventually times out waiting for data that's never coming. It's not something the request retried its way out of and it's not caused by the model provider; it's litellm cutting its own connection.
The fix is to just remove the del handlers. httpx's own client classes don't force-close on garbage collection, so removing litellm's extra teardown means an evicted-but-still-in-use client is simply left alone until whatever's using it finishes naturally, and idle ones get cleaned up normally without anyone actively severing a live stream.
Regression risk: LOW
The del was reflexive cleanup boilerplate added in 2024, not a fix for any known bug. Removing it is low-risk: for cached clients (the actual bug), it just lets them live until truly unused instead of getting force-closed mid-stream. For the ~34 ad-hoc one-off usages elsewhere, losing the explicit close is cosmetic at worst (maybe an occasional ResourceWarning, not a real fd leak, and not something that fails CI). The explicit shutdown path (close_litellm_async_clients()) is untouched, so proper cleanup on process exit still works. Net effect: we trade an occasionally-wrong immediate close for zero risk of killing a live stream.