Repository navigation
Making RLM production-ready: multimodal media, budget controls, multi-model routing, LocalInterpreter, depth>1 recursion, and GEPA compatibility #9289
Description
Activity
Update: depth > 1 recursive subcalls added
The branch now also includes depth > 1 support — when
max_depth > 1,llm_query()spawns a child RLM with its own REPL instead of making a plain LM completion call. This mirrors the vanilla RLM depth>1 pattern adapted for dspy.Usage
rlm = dspy.RLM( "context, question -> answer", interpreter=LocalInterpreter(), max_depth=2, # llm_query spawns child RLMs with own REPLs max_iterations=15, max_llm_calls=30, max_time=300, max_cost=2.00, )
With
max_depth=2, the parent'sllm_query("analyze this data")spawns a child RLM that can write code, explore the data, make its own plain LM sub-calls, and iterate before returning. Withmax_depth=3, the child'sllm_querywould spawn a grandchild, and so on.Key design decisions
- Interpreter matching: child inherits parent's interpreter type (LocalInterpreter → LocalInterpreter, PythonInterpreter → PythonInterpreter) so sandboxing is preserved
- Budget propagation: children receive remaining time/cost from parent;
max_iterationsandmax_llm_callsare independent per child _subcall()as a method on the RLM class (not a closure), making it testable and matching the vanilla RLM pattern- Cleanup:
interpreter.shutdown()infinallyblock - Input validation:
max_depth < 1anddepth < 0raiseValueError - Batched:
llm_query_batchedruns children sequentially when recursive
Testing
- 30 new unit tests adapted from vanilla RLM test_subcall.py: parameter propagation, budget enforcement, interpreter isolation, model override, aforward
- E2E with DummyLM: child SUBMIT, prompt variable, tool inheritance, multi-iteration, error handling, batched, PythonInterpreter parent
- Real LLM validation: 5/5 correct on LongMemEval_S with
max_depth=2+ Gemini 3 Flash
New commits on the branch
fa458654 fix(rlm): match parent interpreter type in child RLM 06fa78f9 fix(rlm): add depth/max_depth validation + aforward depth>1 test d364cff6 test(rlm): add PythonInterpreter (Deno) parent + depth=2 integration test b212a208 feat(rlm): add depth>1 recursive subcalls via LocalInterpreterTotal branch diff: +2,825 lines across
rlm.py,local_interpreter.py,python_interpreter.py,bootstrap_trace.py, and tests.- changed the title
[-]Making RLM production-ready: multimodal media, budget controls, multi-model routing, LocalInterpreter, and GEPA compatibility[/-][+]Making RLM production-ready: multimodal media, budget controls, multi-model routing, LocalInterpreter, depth>1 recursion, and GEPA compatibility[/+]on Feb 11, 2026 came here from your X thread about memory. Great stuff, I hope it is included in dspy main! Out of curiosity, why not making a PR instead of creating an issue?
i was just trying to follow the contributor rules, but omar had the same question so i'll make a PR soon.
This is exactly the right set of production gaps to address. From deploying budget-controlled multi-model agents:
On budget controls: The most effective pattern is pessimistic allocation — when an RLM step is about to execute, deduct the maximum possible cost (based on model price × max output tokens) before the call. If the remaining budget is insufficient, fail fast with a structured error rather than starting a call that will exceed the budget. For recursive RLM (depth > 1), this means computing the worst-case total cost of the full recursion tree and reserving it upfront.
On multi-model routing: Per-step model selection is the highest-ROI optimization. We use a three-field config per step:
primary(default),fallback(if primary fails),ceiling(max model allowed). The ceiling prevents accidentally routing to an expensive model through config inheritance. In practice, ~70% of steps run fine on cheaper models; only quality-critical reasoning steps need the full model. Per-step routing reduced our model costs by ~40%.On LocalInterpreter beyond Deno: The sandbox needs session-scoped state persistence. Code execution across multiple RLM steps should share state (variables, imports, filesystem changes) within a session. We use per-agent isolated runtimes with session affinity — the same logical session always hits the same sandbox instance. Clean up happens on session end, not per-step.
On depth > 1 recursion: The key challenge is delegation chain context propagation. Each recursive call needs to carry: (1) remaining budget, (2) delegation depth, (3) parent task ID for cycle detection. Without explicit cycle detection, A→B→A loops can occur in deep recursion, especially when sub-tasks have overlapping capability domains.
On GEPA compatibility: Budget exhaustion during GEPA optimization is the timeout problem in disguise. Rather than crashing, the system should return the best result found within budget as a partial solution, with metadata indicating "optimization terminated due to budget constraint, result quality: X%".
Architecture references on budget governance + multi-model routing:
- Multi-tenant cost control: https://blog.kinthai.ai/openclaw-multi-tenancy-why-vm-per-user-doesnt-scale
- Multi-agent coordination: https://blog.kinthai.ai/221-agents-multi-agent-coordination-lessons
making RLM production-ready needs tamper-evident records of every optimization decision. when the budget controller makes a routing decision or the retry handler picks a different model, that decision should be independently verifiable.
we built Ed25519 signed receipts for every AI action. the receipt captures the decision, the model used, and the policy in effect. cross-validated across 5 implementations.
github.com/arian-gogani/nobulex
The GEPA trace fix (section 5) is the right direction but it still has a gap:
except Exceptiononly captures what's indspy.settings.traceat crash time, which means hard failures (OOM, SIGKILL from an external budget enforcer, host-level timeout) lose everything. The more durable pattern is write-ahead trace logging -- persist every LM call and tool invocation to a store as it happens, not reconstructed post-failure. We hit this exact problem building an agent memory server and ended up treating execution traces as append-only event streams with decay-weighted importance so completed-but-failed runs still feed back into optimization without polluting the high-quality trace pool. For multi-model routing (section 2), the same trace store becomes a natural feedback loop: you can run offline analysis on which routing decisions correlated with better metric outcomes across GEPA rounds. If you want to see one implementation of durable agent trace storage with hybrid BM25+vector retrieval over execution history, the deploy setup is at dakera-deploy/docker-compose.yml -- single container, no external deps.
Making RLM production-ready: multimodal media, budget controls, multi-model routing, LocalInterpreter, and GEPA compatibility
Summary
RLM is powerful but currently limited to text-only inputs, has no cost/time guardrails, can't route to different sub-models, is restricted to the Deno sandbox, and crashes GEPA when it hits timeouts or cost limits. This PR set addresses all five gaps.
These changes came out of trying to use RLM + GEPA together on real multimodal tasks. Each fix addresses a failure mode I hit in practice.
Branch:
rawwerks/dspy:feat/rlm-multimodal-media-supportThe five changes
1. Multimodal media support — Audio/Image inputs for RLM
Problem: RLM can only work with text. If you have an
AudioorImageinput field, the sandbox just sees a descriptor string — the agent can't actually perceive the content.Solution: Auto-detect
Audio/Imageinput fields and exposellm_query_with_media(prompt, *media_var_names)in the sandbox. Media objects are held in a registry outside the sandbox; when the agent callsllm_query_with_media("describe this", "image"), the media content is attached as multimodal content parts to the sub-LLM call.The agent's sandbox code can then do:
2. Multi-model sub-call routing via
sub_lmsProblem: Some tasks benefit from using different models for different sub-calls — e.g., a cheap model for initial exploration and a stronger model for verification. RLM only supports a single
sub_lm.Solution: New
sub_lmsparameter accepts a dict of named LM instances. Sandbox code selects a model withllm_query(prompt, model="name").Sandbox code:
3. LocalInterpreter — unsandboxed host-process execution
Problem: The Deno/Pyodide sandbox can't access host Python packages like PIL, soundfile, numpy, scipy, etc. For tasks that combine LLM perception with direct data processing, the sandbox is too restrictive.
Solution:
LocalInterpreterimplements theCodeInterpreterprotocol but executes viaexec()in the host process. State persists across iterations, tools are injected into the namespace, andSUBMIT()works identically.Now the agent can
import numpy,import PIL, etc. in its generated code.Security note: This is intentionally unsandboxed. The docstring and class-level documentation make this explicit. It's for local experiments and trusted workloads where the sandbox is the bottleneck.
4. Budget awareness —
budget(),max_time,max_costProblem: RLM agents have no way to know how much budget remains. They burn through iterations and LLM calls blindly, then either hit hard limits or run up unbounded costs. In optimization loops over many examples, a single runaway example can blow your API budget.
Solution: Three new parameters + a
budget()tool:max_time— wall-clock seconds perforward()call. Exceeded → extract fallback (not crash).max_cost— dollar cost perforward()call, tracked via litellm's per-call cost reporting. Also supports BYOK providers whereresponse_costis 0 butusage.cost_details.upstream_inference_costcarries the real charge.budget()— callable in the sandbox. Returns a human-readable summary of remaining iterations, LLM calls, time, and cost. Warns when any resource drops below 20%.Sandbox code:
When time or cost is exceeded,
forward()triggers the extract fallback (attempts to salvage a result from current state) rather than raising an exception. This is critical for optimization — a partial result with trace data is far more useful than a crash.5. GEPA bootstrap_trace resilience for RLM failures
Problem: When RLM hits a timeout, interpreter crash, or cost overrun, it raises an exception that
bootstrap_trace_datadoesn't catch. The existingexceptonly handles parse failures (AdapterParseError,ValueError). Any other exception type kills the training example entirely — the trace is lost and GEPA can't reflect on what the agent tried before failing.Solution: Add a broad
except Exceptionhandler that preserves the partial trace:This is a 6-line change but it's the difference between GEPA learning from failures vs. silently losing all trace data from budget-exceeded examples.
How they work together
The RLM agent can:
budget()to plan its approachllm_query_with_media()for multimodal perceptionTesting strategy
All new code is covered by offline tests (no API keys, no Deno required). The test suite runs in <1 second.
Unit tests verify each feature in isolation via
MockInterpreterand mock LMs:sub_lmsrouting: named model selection, fallback to default, unknown model errorsbudget()output: iteration/call/time/cost reporting, low-resource warningsmax_timeandmax_costtriggering extract fallback (not crash)usage.cost_details.upstream_inference_costaggregationIntegration tests verify the features work together through RLM's
forward()loop:llm_query_with_mediacontent construction: verifies multimodal content parts (text + media) are correctly assembled and sent to the LM, with single/multiple media objects and model routingmax_costmid-run fallback: cost injected into LM history after first iteration triggers extract fallback on the next iteration checkaforward()bootstrap_trace resilience tests verify the GEPA compatibility fix:
RuntimeErrorfromforward()is captured asFailedPrediction, not propagatedKeyboardInterrupt(BaseException) is NOT caught — confirms the broadexcept Exceptiondoesn't swallow signalsTotal: 20 new tests across 6 test classes, plus 33 existing unit tests for the LocalInterpreter and media serialization modules.
Implementation notes
max_time=None,max_cost=None,sub_lms={}, defaultPythonInterpreter)llm_query_with_mediaonly appears in the sandbox prompt when media input fields are detectedruff checkandgitleaks detect