fix(facade): host the dspy facade in in-process interpreters without replacing host dspy

[isaacbmiller]

📝 Changes Description

Since #10478, the sandbox dspy facade shim (_facade_shim.py) refuses any interpreter that runs code in the host's memory. dspy.Flex installs the facade with no fallback, so every forward on an exec()-based interpreter, including dspy_interpreters.InProcessInterpreter, raises CodeExecutionError: ... the dspy facade needs an isolated interpreter. RLM falls back to running without sub-agents. 3.4.0 works with these interpreters, but its shim could leave itself in the host's sys.modules["dspy"].

The refusal doesn't remove any capability: in-process code already shares the host's memory. What it actually prevented was the shim replacing the host's dspy module. Only an explicit choice reaches an in-process interpreter: the module's interpreter_factory or dspy.settings.interpreter_factory. The default is PythonInterpreter, and LocalInterpreter is a fresh subprocess.

This PR:

• makes the shim, when it runs in the host's memory, bind dspy through the interpreter namespace's own __import__ (a per-namespace copy of __builtins__) instead of sys.modules. Sandboxed import dspy gets the shim, import dspy.<sub> raises ModuleNotFoundError as it does in a real sandbox, and the host's sys.modules is untouched. Isolated sandboxes still register the shim in sys.modules.
• as a result, RLM gets sub-agents on in-process interpreters instead of logging a warning.
• adds a Flex regression test. It uses a bare exec() interpreter that doesn't save or restore sys.modules, plus a host tool that checks sys.modules["dspy"] is dspy during the forward. It also flips the RLM and dspy_interpreters conformance tests that pinned the refusal.
• documents in flex.md §7 and the Flex API page that an in-process interpreter works but isolates nothing.

Verification:

• The four new or flipped tests fail on main. A naive fix that drops the refusal and always writes sys.modules fails the new Flex test.
• pytest -n auto --deno tests/flex tests/predict/test_rlm.py tests/primitives: 547 passed, 5 skipped.
• A live gpt-4o run of a decomposed Flex (Predict + ChainOfThought + host tool) on InProcessInterpreter succeeds with the host's dspy intact.

✅ Contributor Checklist

• Pre-Commit checks are passing (locally and remotely)
• Title of your PR / MR corresponds to the required format
• Commit message follows required format {label}(dspy): {message}

⚠️ Warnings

Targeted for 3.4.1. Sibling fixes R2 (#10625) and R7 (facade opt-out) also touch _facade_shim.py and rlm.py; this diff changes only the final if block of the shim, so a rebase conflict would be small. Prepared with Amp in a maintainer-directed session.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] No outstanding blocking findings remain; the change preserves isolated interpreters’ existing behavior and documents the trust requirements for explicitly selected in-process interpreters.

Summary

Allows explicitly selected in-process interpreters to use the dspy facade without replacing the host’s module.

• In-process interpreters can use the dspy facade without replacing the host module.

_{Reviews (4) · Last reviewed commit: "test(flex): cover the in-process shim's ..." · Reviewed by Greptile}