Skip to content

fix(core): instantiate each sub-client exactly once in _spawn_completion_context - #159

Closed
201etheiss wants to merge 1 commit into
alexzhang13:mainfrom
201etheiss:fix/dedupe-sub-client-instantiation
Closed

201etheiss wants to merge 1 commit into
alexzhang13:mainfrom
201etheiss:fix/dedupe-sub-client-instantiation

Conversation

@201etheiss

Copy link
Copy Markdown

Summary

_spawn_completion_context currently constructs the first sub-client TWICE — once at rlm/core/rlm.py:206 as other_backend_client, again at line 213 inside the registration loop with identical args. Two independent BaseLM instances point at the same sub-model, with separate underlying state.

This PR collapses construction so each (backend, kwargs) pair is instantiated exactly once. The first sub-client serves both roles: as other_backend_client (depth=1 default routing when LMHandler.get_client receives no model arg) AND registered by its model_name for explicit-model routing — same instance, both paths.

Why this matters (impact)

For backends with cheap state, the duplicate is just wasted construction. For backends that hold expensive underlying state (HTTP connection pools, async clients, OS sockets), the duplicate carried real runtime cost. Two specific symptoms we hit downstream:

  1. 3-minute hang at iter 2 — every multi-backend trajectory on macOS with two AnthropicClient instances pointing at the same sub-model would block at iter 2 for ~180s before the proxy/socket layer gave up. Fully recovered by de-duplication: pre-fix smoke timed out at the wall-clock cap; post-fix completed in 47.3s.

  2. Usage tracking under-counted — LMHandler.get_usage_summary() (lm_handler.py:198-205) merges per-client summaries via dict.update. Two instances keyed by the same model_name overwrite each other; one instance's tokens were dropped silently from the merged summary.

The fix preserves all existing routing semantics in LMHandler.get_client(model, depth) at lm_handler.py:131:

  • model and model in self.clients → returns clients[model] (registered entry)
  • depth == 1 and other_backend_client is not None → returns other_backend_client

Both branches now return the same object for the first sub-model, so behavior is unchanged at the call sites — only the construction path is collapsed.

Behavior delta

  • Single-backend (no other_backends): zero change. The if self.other_backends and self.other_backend_kwargs: guard still skips the loop.
  • One other_backends entry: get_client is now called twice total (root + 1 sub) instead of three times (root + 2× same sub).
  • N other_backends entries: get_client is now called 1 + N times (root + N sub) instead of 1 + N + 1 times. other_backend_client is the first sub-client, identical to clients[other_backends[0]_model_name].

Verification

Verified end-to-end downstream in our delta360-rlm harness (which patches the same codepath via _install_multibackend_patch):

Pre-fix — multi-backend --sub-model claude-haiku-4-5:

Trajectory hangs at iter 2; 0% CPU for 3+ min; SIGTERM kill at 240s wall.

Post-fix — same query, same caps:

{
  "wall_clock_sec": 47.27,
  "cost_usd": 0.0583,
  "subcall_count": 2,
  "depth_reached": 1,
  "cap_breached": null
}

We also added 3 invariant tests at the harness layer (https://github.com/DeltaFuel/delta360-rlm/blob/fix/multi-backend-single-subclient/tests/test_multibackend_patch.py):

  1. Single sub-model: get_client called 2x; other_backend_client is clients[sub_model_name]
  2. No sub-models: get_client called 1x; other_backend_client is None
  3. Multiple sub-models: get_client called 1 + N times; first sub serves as depth=1 default

5/5 tests pass against the patched contextmanager.

Adjacent: usage tracking accuracy

A separate change downstream (https://github.com/DeltaFuel/delta360-rlm/blob/fix/multi-backend-single-subclient/src/rlm_harness/watchdog.py) uncovered that even with de-dup, multi-backend trajectory cost was being over-written per instance instead of accumulated in our cost-cap watchdog. That fix lives at the harness layer (we accumulate per-call incremental into a shared counter under a lock) and isn't proposed for upstream — but flagging here in case get_usage_summary ever evolves to multi-instance summing.

Notes

  • No new tests in this PR — the change is small enough that a unit test would either duplicate the codepath or require fixturing 2+ backends. Happy to add one if a maintainer prefers; we have the test pattern ready in our harness for reference.
  • No new dependencies, no API changes, no public-facing rename.
  • Tested locally against rlm HEAD (this branch is based on commit 03a1774).

Closes our local KNOWN-ISSUES #9 (https://github.com/DeltaFuel/delta360-rlm/blob/fix/multi-backend-single-subclient/docs/KNOWN-ISSUES.md). Happy to address any review feedback.

…ion_context

`_spawn_completion_context` previously constructed the first sub-client
TWICE — once at line 206 as `other_backend_client`, again at line 213
inside the registration loop with identical args. Two independent
BaseLM instances pointed at the same sub-model.

Impact:
- For backends that hold expensive underlying state (HTTP connection
  pools, async clients, OS sockets), the duplicate carried real cost.
  We observed a 3-minute hang at iter 2 of every multi-backend
  trajectory on macOS with anthropic backends — fully recovered by
  de-duplicating the instantiation.
- Usage tracking merged via dict.update at lm_handler.py:198-205, so
  the duplicate's summary over-wrote one of the two instances' usage
  data — silently under-counted cost in `get_usage_summary()`.

Fix: build a single list of sub-clients from
`zip(self.other_backends, self.other_backend_kwargs)` exactly once.
The first entry serves as `other_backend_client` (depth=1 default
routing when LMHandler.get_client receives no `model` arg) AND is
registered by its `model_name` for explicit-model routing — same
instance, both paths.

Routing semantics are preserved: `get_client(model, depth)` at
lm_handler.py:131 still returns `clients[model]` first, then falls
back to `other_backend_client` for `depth==1`. Both now point to the
same object for the first sub-model.

Verified end-to-end downstream in delta360-rlm harness PR alexzhang13#6:
- pre-fix: trajectory hung at iter 2 for 3+ minutes (0% CPU, 300s
  proxy/socket timeout suspected)
- post-fix: 47.3s wall, $0.058 actual cost, 2 sub-calls, depth_reached=1

No behavior change for single-backend (no-sub_model) callers — the
loop body is unchanged for them. Tests in delta360-rlm
(`tests/test_multibackend_patch.py`) verify get_client call counts
and instance-identity invariants for: (a) single sub-model, (b) no
sub-models, (c) multiple sub-models. 5/5 pass.
@alexzhang13

Copy link
Copy Markdown
Owner

Thanks for catching this -- the PR is a bit verbose (+ unnecessary comments) but I'll fix this and merge.

@alexzhang13

Copy link
Copy Markdown
Owner

See #163

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants