Define a DSPy result contract for mixed MCP content blocks #10236

Activity

1. chrikrah commented on Sep 10, 2026

   

   chrikrah

   on Sep 10, 2026

   More actions

   Reproduces on ca54a85, after #10235 merged, through dspy.Tool.from_mcp_tool:

   sent	returned	lost
   text + image	'Here is the chart:'	the image
   text + audio	'Here is the chart:'	the audio
   text + embedded_resource	'Here is the chart:'	the resource
   text + resource_link	'Here is the chart:'	the resource link
   image + text, image first	'Here is the chart:'	the image
   text "" + image	[ImageContent…]	the text
   text ["",""] + image	['','']	the image
   text, image, text	['before','after']	the image, and the order

   dspy/utils/mcp.py:51 is return tool_content or non_text_contents, so the branch does not mean "text
   wins", it means "truthy text wins".

   The last three rows are defects rather than design questions, and a fix for them does not need the
   contract first: an empty text block already returns the non-text list, and the text-only result loses
   the ordering the contract intends to keep.

   result_mode="structured" does not help. With structuredContent absent it falls through the same
   line, so no mode returns the image.

   On raw blocks against a DSPy-native representation: raw blocks in a ReAct trajectory render as a
   pydantic repr with inline base64, not as a dspy.Image a vision adapter can send.

   #10258 flips the default mode and #10159 edits the same function. Should this land after both?
2. NaCr05 commented on Oct 2, 2026

   

   NaCr05

   on Oct 2, 2026

   More actions

   Following the existing reproduction, an AI-assisted local rerun at 98a4deeb (DSPy 3.4.0, Python 3.12.5, Windows 11, MCP 2.0.0) reproduces the text-plus-image loss and records ReActV2/ChatAdapter's outgoing messages:

   Path	Observed result
   Text + image through the MCP bridge and Tool.acall()	Only the text string is returned.
   Raw text/image blocks → ReActV2 + ChatAdapter (use_native_function_calling=False)	The blocks remain in the tool value, but the next LM request contains their JSON serialization, including base64, inside a user-message string.
   Raw text/image blocks → ReActV2 + ChatAdapter (use_native_function_calling=True)	The next LM request contains a role="tool" message whose content is a JSON string, including base64.

   In the last two rows, an ordinary DSPy tool returns the raw blocks directly, deliberately bypassing the MCP converter as a diagnostic control. Neither path produces a native image content part. They use scripted LM responses, so this checks request construction, not a real provider's vision behavior. This agrees with the existing comment's warning about raw blocks; the native path also matches the current _tool_result_content implementation.

   Would a first PR limited to opt-in content preservation, tests, and documentation be useful? Before implementing it, could we clarify:

   1. Return type: ordered MCP blocks or a DSPy-owned representation, while preserving existing default text and exact structuredContent semantics?
   2. Acceptance scope: may the first PR cover programmatic content fidelity, with actual image/audio mapping into model requests handled separately?
   3. Target version: should this target the current 3.4 interfaces, or be designed together with the proposed 3.5 LM/Adapter migration in DSPy 3.5: LMs, adapters and types speak lm15 Request/Response only #10401?

   AI assistance: Codex helped write and execute the local probes and prepare this comment. No real model API was called and no DSPy source was changed.