Repository navigation
fix(core): instantiate each sub-client exactly once in _spawn_completion_context - #159
Closed
201etheiss wants to merge 1 commit into
Closed
201etheiss wants to merge 1 commit into
201etheiss wants to merge 1 commit into
Conversation
…ion_context `_spawn_completion_context` previously constructed the first sub-client TWICE — once at line 206 as `other_backend_client`, again at line 213 inside the registration loop with identical args. Two independent BaseLM instances pointed at the same sub-model. Impact: - For backends that hold expensive underlying state (HTTP connection pools, async clients, OS sockets), the duplicate carried real cost. We observed a 3-minute hang at iter 2 of every multi-backend trajectory on macOS with anthropic backends — fully recovered by de-duplicating the instantiation. - Usage tracking merged via dict.update at lm_handler.py:198-205, so the duplicate's summary over-wrote one of the two instances' usage data — silently under-counted cost in `get_usage_summary()`. Fix: build a single list of sub-clients from `zip(self.other_backends, self.other_backend_kwargs)` exactly once. The first entry serves as `other_backend_client` (depth=1 default routing when LMHandler.get_client receives no `model` arg) AND is registered by its `model_name` for explicit-model routing — same instance, both paths. Routing semantics are preserved: `get_client(model, depth)` at lm_handler.py:131 still returns `clients[model]` first, then falls back to `other_backend_client` for `depth==1`. Both now point to the same object for the first sub-model. Verified end-to-end downstream in delta360-rlm harness PR alexzhang13#6: - pre-fix: trajectory hung at iter 2 for 3+ minutes (0% CPU, 300s proxy/socket timeout suspected) - post-fix: 47.3s wall, $0.058 actual cost, 2 sub-calls, depth_reached=1 No behavior change for single-backend (no-sub_model) callers — the loop body is unchanged for them. Tests in delta360-rlm (`tests/test_multibackend_patch.py`) verify get_client call counts and instance-identity invariants for: (a) single sub-model, (b) no sub-models, (c) multiple sub-models. 5/5 pass.
Owner
|
Thanks for catching this -- the PR is a bit verbose (+ unnecessary comments) but I'll fix this and merge. |
Owner
|
See #163 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_spawn_completion_contextcurrently constructs the first sub-client TWICE — once atrlm/core/rlm.py:206asother_backend_client, again at line 213 inside the registration loop with identical args. Two independentBaseLMinstances point at the same sub-model, with separate underlying state.This PR collapses construction so each
(backend, kwargs)pair is instantiated exactly once. The first sub-client serves both roles: asother_backend_client(depth=1 default routing whenLMHandler.get_clientreceives nomodelarg) AND registered by itsmodel_namefor explicit-model routing — same instance, both paths.Why this matters (impact)
For backends with cheap state, the duplicate is just wasted construction. For backends that hold expensive underlying state (HTTP connection pools, async clients, OS sockets), the duplicate carried real runtime cost. Two specific symptoms we hit downstream:
3-minute hang at iter 2 — every multi-backend trajectory on macOS with two
AnthropicClientinstances pointing at the same sub-model would block at iter 2 for ~180s before the proxy/socket layer gave up. Fully recovered by de-duplication: pre-fix smoke timed out at the wall-clock cap; post-fix completed in 47.3s.Usage tracking under-counted —
LMHandler.get_usage_summary()(lm_handler.py:198-205) merges per-client summaries viadict.update. Two instances keyed by the samemodel_nameoverwrite each other; one instance's tokens were dropped silently from the merged summary.The fix preserves all existing routing semantics in
LMHandler.get_client(model, depth)atlm_handler.py:131:model and model in self.clients→ returnsclients[model](registered entry)depth == 1 and other_backend_client is not None→ returnsother_backend_clientBoth branches now return the same object for the first sub-model, so behavior is unchanged at the call sites — only the construction path is collapsed.
Behavior delta
other_backends): zero change. Theif self.other_backends and self.other_backend_kwargs:guard still skips the loop.other_backendsentry:get_clientis now called twice total (root + 1 sub) instead of three times (root + 2× same sub).other_backendsentries:get_clientis now called1 + Ntimes (root + N sub) instead of1 + N + 1times.other_backend_clientis the first sub-client, identical toclients[other_backends[0]_model_name].Verification
Verified end-to-end downstream in our delta360-rlm harness (which patches the same codepath via
_install_multibackend_patch):Pre-fix — multi-backend
--sub-model claude-haiku-4-5:Post-fix — same query, same caps:
We also added 3 invariant tests at the harness layer (https://github.com/DeltaFuel/delta360-rlm/blob/fix/multi-backend-single-subclient/tests/test_multibackend_patch.py):
get_clientcalled 2x;other_backend_client is clients[sub_model_name]get_clientcalled 1x;other_backend_client is Noneget_clientcalled1 + Ntimes; first sub serves as depth=1 default5/5 tests pass against the patched contextmanager.
Adjacent: usage tracking accuracy
A separate change downstream (https://github.com/DeltaFuel/delta360-rlm/blob/fix/multi-backend-single-subclient/src/rlm_harness/watchdog.py) uncovered that even with de-dup, multi-backend trajectory cost was being over-written per instance instead of accumulated in our cost-cap watchdog. That fix lives at the harness layer (we accumulate per-call incremental into a shared counter under a lock) and isn't proposed for upstream — but flagging here in case
get_usage_summaryever evolves to multi-instance summing.Notes
rlmHEAD (this branch is based on commit03a1774).Closes our local KNOWN-ISSUES #9 (https://github.com/DeltaFuel/delta360-rlm/blob/fix/multi-backend-single-subclient/docs/KNOWN-ISSUES.md). Happy to address any review feedback.