Skip to content

fix(stream): remove del handlers - #65

Merged
ShivaniKumar1 merged 1 commit into
release/v1.94.0-do.3from
skumar/remove-mid-stream-del
Aug 14, 2026
Merged

fix(stream): remove del handlers#65
ShivaniKumar1 merged 1 commit into
release/v1.94.0-do.3from
skumar/remove-mid-stream-del

Conversation

@ShivaniKumar1

@ShivaniKumar1 ShivaniKumar1 commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Related to upstream fix: BerriAI#24929
Every provider HTTP client litellm creates gets cached and reused for an hour, then the cache lets it expire. The class behind that client has a del method: whenever Python's garbage collector notices nothing is holding onto that client object anymore, del fires and forcibly closes its network connections.

The bug is that "nothing holding onto it" can happen while a customer's streaming response is still actively reading from that exact connection. The stream doesn't keep a reference to the wrapper object, only to the response it's reading, so once the one-hour cache entry expires and nothing else references the wrapper, GC calls del and yanks the socket out from under the still-open stream.

We need to fix it because this isn't a rare edge case, it's baked into normal operation: every client in the fleet gets recycled every hour, and whichever streams happen to be mid-flight at that exact moment get their connection killed. That's what's producing the drops we've been seeing clustered at the same time each hour, roughly the point where the process (or fleet) last started plus a multiple of an hour.

From the customer's side this shows up as a stream that just stops, no error, no more bytes, until their own client eventually times out waiting for data that's never coming. It's not something the request retried its way out of and it's not caused by the model provider; it's litellm cutting its own connection.

The fix is to just remove the del handlers. httpx's own client classes don't force-close on garbage collection, so removing litellm's extra teardown means an evicted-but-still-in-use client is simply left alone until whatever's using it finishes naturally, and idle ones get cleaned up normally without anyone actively severing a live stream.

Regression risk: LOW
The del was reflexive cleanup boilerplate added in 2024, not a fix for any known bug. Removing it is low-risk: for cached clients (the actual bug), it just lets them live until truly unused instead of getting force-closed mid-stream. For the ~34 ad-hoc one-off usages elsewhere, losing the explicit close is cosmetic at worst (maybe an occasional ResourceWarning, not a real fd leak, and not something that fails CI). The explicit shutdown path (close_litellm_async_clients()) is untouched, so proper cleanup on process exit still works. Net effect: we trade an occasionally-wrong immediate close for zero risk of killing a live stream.

@ShivaniKumar1 ShivaniKumar1 changed the title remove mid stream kill connection Remove del handlers Aug 14, 2026
@ShivaniKumar1 ShivaniKumar1 changed the title Remove del handlers fix(stream): remove del handlers Aug 14, 2026
@ShivaniKumar1
ShivaniKumar1 merged commit fa42650 into release/v1.94.0-do.3 Aug 14, 2026
3 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants