Merge branch 'main' into cursor/fix-miprov2-compile-seed-zero-88e7

Author: Zsanz3

CONTRIBUTING.md

@@ -165,6 +165,8 @@ AI-assisted contributions are welcome under these conditions:
 
 - If you can't explain your changes without consulting an AI, don't submit the PR.
 - Disclose what AI tool you used and how in your PR description.
+- Agents must disclose in the PR description whether the contribution comes from a sweep for issues, a problem encountered during normal usage, or another origin.
+  State which applies, or provide a non-sensitive description of another origin. Do not share private or sensitive information about the use case.
 - Share your prompts. It helps reviewers and shows you thought through the problem.
 - Verify AI-found bugs yourself. Reproduce and confirm before filing. Write the report in
   your own words.

docs/docs/api/modules/Flex.md

@@ -84,7 +84,7 @@ The `program_trace` parameter is opt-in *by declaration*: only metrics that name
 
 ## Interpreter Execution
 
-`Flex` always runs its generated code through a `CodeInterpreter`. `interpreter_factory` defaults to `dspy.PythonInterpreter` (Deno/Pyodide) and must be a **zero-argument factory** returning a fresh interpreter; a bare instance is not accepted, so parallel evaluations receive isolated sessions. `dspy.configure(interpreter_factory=...)` replaces that default. A factory passed to `Flex` wins, unless it is `PythonInterpreter`. The factory is called once per interpreter session, including separate sessions requested by nested code-executing modules. With the default interpreter, optimizer-authored control flow, string work, arithmetic, and supported imports run inside the sandbox, and only provided-tool calls, predictor construction, and predictor calls bridge back to the host, which makes the real LM calls. Custom factories define their own trust boundary: `dspy.LocalInterpreter`, for example, separates process memory and stdout but retains the host user's filesystem, environment, credentials, network, and process authority.
+`Flex` always runs its generated code through a `CodeInterpreter`. `interpreter_factory` defaults to `dspy.PythonInterpreter` (Deno/Pyodide) and must be a **zero-argument factory** returning a fresh interpreter; a bare instance is not accepted, so parallel evaluations receive isolated sessions. `dspy.configure(interpreter_factory=...)` replaces that default. A factory passed to `Flex` wins, unless it is `PythonInterpreter`. The factory is called once per interpreter session, including separate sessions requested by nested code-executing modules. With the default interpreter, optimizer-authored control flow, string work, arithmetic, and supported imports run inside the sandbox, and only provided-tool calls, predictor construction, and predictor calls bridge back to the host, which makes the real LM calls. Custom factories define their own trust boundary: `dspy.LocalInterpreter`, for example, separates process memory and stdout but retains the host user's filesystem, environment, credentials, network, and process authority. An in-process interpreter that runs code with `exec()` in the host works, but isolates nothing; use one only for trusted code.
 
 Because the default builds a `PythonInterpreter`, *running* a `Flex` needs [Deno](https://deno.land/) installed; without it, the call raises.
 

docs/docs/diving-deeper/flex.md

@@ -34,7 +34,7 @@ The reflection model authors code, and code can be wrong. A candidate that doesn
 
 ### 7. Generated code always runs through an interpreter
 
-`Flex` takes its interpreter from `interpreter_factory`, else from `dspy.settings.interpreter_factory`, else from `dspy.PythonInterpreter` (Deno/Pyodide). Each of those must be a *zero-argument factory* — a bare instance or `None` is rejected. With the default, everything stays in the sandbox except provided-tool calls, predictor construction, and predictor calls, which bridge back to the host. Custom factories define their own trust boundary: `LocalInterpreter` separates process memory and stdout but retains the host user's filesystem, environment, credentials, network, and process authority. The factory creates a fresh interpreter for each session; a forward owns an outer session and nested code-executing modules may request separate sessions. Source portability between custom interpreters is not guaranteed. `max_predictor_calls` caps how many predictor calls the generated code can make in one `forward`.
+`Flex` takes its interpreter from `interpreter_factory`, else from `dspy.settings.interpreter_factory`, else from `dspy.PythonInterpreter` (Deno/Pyodide). Each of those must be a *zero-argument factory* — a bare instance or `None` is rejected. With the default, everything stays in the sandbox except provided-tool calls, predictor construction, and predictor calls, which bridge back to the host. Custom factories define their own trust boundary: `LocalInterpreter` separates process memory and stdout but retains the host user's filesystem, environment, credentials, network, and process authority. An in-process interpreter (one that runs code with `exec()` in the host, such as `dspy_interpreters.InProcessInterpreter`) works but isolates nothing: generated code shares the host's memory, so `max_predictor_calls` and the bridged LM-option allowlist bind only well-behaved code. Use one only for code you trust. The facade binds `dspy` in that interpreter's namespace and leaves the host's `dspy` module in place. The factory creates a fresh interpreter for each session; a forward owns an outer session and nested code-executing modules may request separate sessions. Source portability between custom interpreters is not guaranteed. `max_predictor_calls` caps how many predictor calls the generated code can make in one `forward`.
 
 ### 8. The declared output types are enforced at the sandbox boundary
 

dspy/adapters/utils.py

@@ -71,11 +71,11 @@ def serialize_for_json(value: Any) -> Any:
     Returns:
         The formatted value, which is serializable as a JSON string.
     """
-    # Attempt to format the value as a JSON-compatible object using pydantic, falling back to
-    # a string representation of the value if that fails (e.g. if the value contains an object
-    # that pydantic doesn't recognize or can't serialize)
+    # Format the value as a JSON-compatible object using pydantic. A nested value pydantic can't
+    # serialize (e.g. a numpy scalar) becomes its string form without affecting its siblings; the
+    # whole value falls back to its string form only if pydantic can't handle it at all.
     try:
-        return TypeAdapter(type(value)).dump_python(value, mode="json")
+        return TypeAdapter(type(value)).dump_python(value, mode="json", fallback=str)
     except Exception:
         return str(value)
 

dspy/predict/rlm.py

@@ -589,8 +589,8 @@ def _setup_facade(
         try:
             invocation.install(repl)
         except (CodeExecutionError, SyntaxError) as e:
-            # The runtime cannot host the shim (e.g. it runs code in the host's memory, or a restricted compiler
-            # rejects the shim's `_dspy_*` names): no sub-agents this call.
+            # The runtime cannot host the shim (e.g. a restricted compiler rejects the shim's `_dspy_*` names):
+            # no sub-agents this call.
             for name in (CONSTRUCT_TOOL, CALL_TOOL):
                 repl.tools.pop(name, None)
             logger.warning("RLM sub-agents are unavailable on %s: %s", type(repl).__name__, e)

dspy/primitives/_facade_shim.py

@@ -149,13 +149,24 @@ def _dspy_tool(func, **_kw):
 dspy = _dspy
 
 # Register as the importable ``dspy`` only inside the sandbox, where the registered host tools are
-# present in globals(). A sandbox whose ``dspy`` is the host's own module image (the host process or a
-# fork of it) would hand generated code the host's memory, so the facade refuses it.
+# present in globals(). An interpreter that runs code in the host's memory (the host process or a fork
+# of it) shares the host's sys.modules, so there ``import dspy`` resolves to the shim through this
+# namespace's own __import__ and the host's ``dspy`` module stays in place.
 if "__dspy_construct__" in globals():
     _dspy_host_facade = getattr(_dspy_sys, "modules", {}).get("dspy.primitives.facade")
     if getattr(_dspy_host_facade, "_HOST_PROCESS_TOKEN", None) == __dspy_host_token:  # noqa: F821 - host-injected
-        raise RuntimeError(
-            "This interpreter runs code in the host's memory (the host process or a fork of it); "
-            "the dspy facade needs an isolated interpreter."
-        )
-    _dspy_sys.modules["dspy"] = _dspy
+        _dspy_builtins = globals()["__builtins__"]
+        _dspy_builtins = dict(_dspy_builtins if isinstance(_dspy_builtins, dict) else vars(_dspy_builtins))
+        _dspy_import_orig = _dspy_builtins["__import__"]
+
+        def _dspy_import(name, globals=None, locals=None, fromlist=(), level=0):
+            if level == 0 and name.split(".")[0] == "dspy":
+                if name != "dspy":
+                    raise ModuleNotFoundError(f"No module named {name!r}; 'dspy' is not a package", name=name)
+                return _dspy
+            return _dspy_import_orig(name, globals, locals, fromlist, level)
+
+        _dspy_builtins["__import__"] = _dspy_import
+        globals()["__builtins__"] = _dspy_builtins
+    else:
+        _dspy_sys.modules["dspy"] = _dspy

dspy/primitives/code_interpreter.py

@@ -84,8 +84,9 @@ class CodeInterpreter(Protocol):
     Sandbox dspy facade:
         dspy.Flex and dspy.RLM install a ``dspy`` shim whose predictors are built and
         run on the host, so registered tools must be callable as globals in executed
-        code and state must persist across execute() calls. The shim refuses an
-        interpreter that runs code in the host's memory (the host process or a fork).
+        code and state must persist across execute() calls. In an interpreter that runs
+        code in the host's memory, the shim binds ``dspy`` in that interpreter's namespace
+        and leaves the host's ``sys.modules`` alone.
     """
 
     @property

tests/flex/test_flex_interpreter.py

@@ -13,6 +13,7 @@
 
 from __future__ import annotations
 
+import ast
 import datetime
 import enum
 import inspect
@@ -350,6 +351,63 @@ def test_shim_file_literals_match_bridge_constants() -> None:
         assert kind in shim, f"bridgeable kind {kind!r} missing from the shim file"
 
 
+class _ExecInterpreter:
+    """Runs code with exec() in the host process, in a private namespace; leaves sys.modules alone."""
+
+    def __init__(self):
+        self.tools = {}
+        self._namespace = {}
+
+    def start(self) -> None:
+        pass
+
+    def execute(self, code: str, variables: dict | None = None):
+        self._namespace.update(self.tools, **(variables or {}))
+        tree = ast.parse(code)
+        last = tree.body.pop() if tree.body and isinstance(tree.body[-1], ast.Expr) else None
+        exec(compile(tree, "<flex>", "exec"), self._namespace)
+        return eval(compile(ast.Expression(last.value), "<flex>", "eval"), self._namespace) if last else None
+
+    def shutdown(self) -> None:
+        self._namespace.clear()
+
+
+def test_in_process_interpreter_runs_the_facade_without_replacing_host_dspy() -> None:
+    host_dspy_seen = []
+
+    def louder(text: str) -> str:
+        """Uppercase the text."""
+        host_dspy_seen.append(sys.modules["dspy"] is dspy)
+        return text.upper()
+
+    flex = Flex(ShoutSig, tools=[louder], interpreter_factory=_ExecInterpreter)
+    flex._bind_code(
+        textwrap.dedent(
+            """
+            import dspy
+
+            try:
+                import dspy.predict
+            except ModuleNotFoundError:
+                pass
+            else:
+                raise AssertionError("import dspy.predict reached the host's dspy package")
+
+            class Shout(dspy.Module):
+                def __init__(self):
+                    self.draft = dspy.Predict("text -> out")
+
+                def forward(self, text):
+                    return dspy.Prediction(out=louder(text=self.draft(text=text).out))
+            """
+        )
+    )
+    with dspy.context(lm=DummyLM([{"out": "hello"}])):
+        assert flex(text="hi").out == "HELLO"
+    assert host_dspy_seen == [True]
+    assert sys.modules["dspy"] is dspy
+
+
 def test_max_predictor_calls_is_plumbed_to_bridge() -> None:
     flex = _bridged_flex_with(max_predictor_calls=7)
     assert flex._bridge._max_predictor_calls == 7

tests/predict/test_rlm.py

@@ -1977,17 +1977,17 @@ def test_call_time_factory_hosts_the_facade_and_backs_nested_sub_agents(self, us
         invocation.construct("RLM", "q -> a", "nested")
         assert invocation._predictors["nested"]._interpreter_factory is override
 
-    def test_interpreter_in_the_host_memory_runs_without_sub_agents(self, caplog):
-        # The shim refuses it rather than replace the host's `dspy` module; RLM carries on without sub-agents.
+    def test_interpreter_in_the_host_memory_hosts_the_facade(self, caplog):
+        # The shim binds `dspy` in the interpreter's namespace instead of replacing the host's module.
         rlm = RLM("query -> answer", max_iters=1, interpreter_factory=_InProcessInterpreter)
-        rlm.generate_action, seen = _recording_predictor('SUBMIT("42")')
+        rlm.generate_action, seen = _recording_predictor('import dspy\nSUBMIT(dspy.Predict("q -> a")(q="x").a)')
 
-        with caplog.at_level("WARNING", logger="dspy.predict.rlm"):
+        with caplog.at_level("WARNING", logger="dspy.predict.rlm"), dspy.context(lm=DummyLM([{"a": "42"}])):
             assert rlm(query="q").answer == "42"
 
         assert sys.modules["dspy"] is dspy
-        assert "Sub-agents (dspy)" not in seen[0].instructions
-        assert "host's memory" in caplog.text
+        assert "Sub-agents (dspy)" in seen[0].instructions
+        assert "unavailable" not in caplog.text
 
     def test_interpreter_that_rejects_the_shim_as_syntax_error_runs_without_sub_agents(self, caplog):
         # Restricted compilers (e.g. RestrictedPython) reject the shim's `_dspy_*` names at compile time.

tests/primitives/test_dspy_interpreters_conformance.py

@@ -2,8 +2,8 @@
 dependency, declared in the ``dev`` dependency group).
 
 RLM installs the sandbox dspy facade into every interpreter it creates. On real backends: an isolated
-one runs bridged sub-agents, one that executes in the host's memory runs RLM without them, and the
-library's own consumer checks pass.
+one runs bridged sub-agents, one that executes in the host's memory hosts the facade without replacing
+the host's ``dspy`` module, and the library's own consumer checks pass.
 """
 
 import pytest
@@ -42,21 +42,21 @@ def test_backends_satisfy_protocol(backend_name):
         interpreter.shutdown()
 
 
-def test_in_process_backend_runs_rlm_without_sub_agents(caplog):
+def test_in_process_backend_runs_rlm_with_sub_agents(caplog):
     seen = []
 
     class Recording:
         def __call__(self, signature=None, **kwargs):
             seen.append(signature.instructions)
-            return Prediction(reasoning="Submit", code='SUBMIT("ok")')
+            return Prediction(reasoning="Submit", code='import dspy\nSUBMIT(dspy.Predict("q -> a")(q="x").a)')
 
     rlm = RLM("query -> answer", max_iters=1, interpreter_factory=InProcessInterpreter)
     rlm.generate_action = Recording()
-    with caplog.at_level("WARNING", logger="dspy.predict.rlm"):
+    with caplog.at_level("WARNING", logger="dspy.predict.rlm"), dspy.context(lm=DummyLM([{"a": "ok"}])):
         assert rlm(query="q").answer == "ok"
 
-    assert "Sub-agents (dspy)" not in seen[0]
-    assert "sub-agents are unavailable on InProcessInterpreter" in caplog.text
+    assert "Sub-agents (dspy)" in seen[0]
+    assert "unavailable" not in caplog.text
 
 
 @pytest.mark.parametrize("backend_name", BACKENDS)
@@ -66,10 +66,9 @@ def test_library_consumer_checks_pass(backend_name):
     dspy_interpreters.check_rlm(backend).raise_for_failures()
 
 
-def test_flex_facade_needs_an_isolated_backend():
-    dspy_interpreters.check_flex_facade(SubprocessInterpreter).raise_for_failures()
-    report = dspy_interpreters.check_flex_facade(InProcessInterpreter)
-    assert not report.passed and "host's memory" in report.results[0].detail
+@pytest.mark.parametrize("backend_name", BACKENDS)
+def test_flex_facade_checks_pass(backend_name):
+    dspy_interpreters.check_flex_facade(getattr(dspy_interpreters, backend_name)).raise_for_failures()
 
 
 def test_isolated_backend_runs_bridged_sub_agents_with_host_tools():