Repository navigation
Support normalized BaseLM contract - #2
Closed
MaximeRivest wants to merge 23 commits into
Closed
MaximeRivest wants to merge 23 commits into
MaximeRivest wants to merge 23 commits into
Conversation
We did not have complete end-to-end coverage for the three adapters to ensure their rendering behavior stays stable over time. This PR adds exact-message regression tests for the chat, JSON, and XML adapters to help prevent adapter rendering drift.
…ordnlp#9787) Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.5.3 to 0.5.5. - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](zizmorcore/zizmor-action@b1d7e1f...a16621b) --- updated-dependencies: - dependency-name: zizmorcore/zizmor-action dependency-version: 0.5.5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This PR expands the exact-format regression coverage for DSPy adapters. The goal is to lock current adapter rendering behavior before refactoring, including both the rendered messages and adapter-managed `lm_kwargs`. Coverage added: - More exact-message tests for `ChatAdapter`, `JSONAdapter`, and `XMLAdapter` - Exact-format tests for `BAMLAdapter` and `TwoStepAdapter` - Nested Pydantic input/output rendering - Complete and incomplete demos - Conversation history - Tools and native tool-calling `lm_kwargs` - Native reasoning `lm_kwargs` - Native citations behavior - Multimodal/custom adapter types: - `Image` - `Audio` - `File` - `Document` - `Citations` - `Code` - `Reasoning` - `Tool` - `ToolCalls` - custom `dspy.Type` - A kitchen-sink ChatAdapter regression test covering demos, history, custom types, nested Pydantic models, tools, literals, and scalar typed outputs. Testing: - Ran targeted adapter suite: `133 passed`
* Add core language model types * Fix core LM type edge cases --------- Co-authored-by: isaacbmiller <isaacbmiller@gmail.com>
Moving adapters toward normalized LM requests and responses. Adapters still format prompts the same way, and current `BaseLM` calls still receive OpenAI/LiteLLM-shaped kwargs. The difference is that the adapter call path now passes through a normalized boundary internally: ```text adapter messages -> LMRequest -> OpenAI/LiteLLM kwargs -> current BaseLM -> LMResponse -> existing adapter postprocess path ``` ## Why Today adapters talk directly in provider-shaped messages and legacy output objects. That makes it hard to move tools, reasoning, citations, multimodal inputs, and future adapter types onto one LM type system. This PR adds the seam without changing parser behavior yet. We normalize before and after the LM call, then convert back to the existing postprocess shape so this can stay behavior-preserving. ## Changes - Add `dspy.clients.openai_format` for normalized `LMRequest` / `LMResponse` conversion to and from OpenAI/LiteLLM-shaped data. - Add `dspy.adapters._legacy_type_markers` for the old custom type marker expansion. - Update `Adapter.__call__()` and `acall()` to route through `LMRequest` and `LMResponse`. - Preserve unknown legacy content blocks during round-trip conversion. - Preserve text block metadata when coercing OpenAI-style content into `LMTextPart`. - Add TODOs in `base.py` marking the follow-up adapter plan and direct `LMResponse` parsing work. ## Notes `dspy.clients.openai_format` is used instead of an adapter-local bridge because provider-format conversion belongs in the client/LM layer. Later LM backends can reuse the same formatter. The conversion back from `LMResponse` to legacy outputs is temporary. A follow-up PR should replace `_call_preprocess()` / `_call_postprocess()` with an explicit adapter plan and direct `LMResponse` parsing.
This was referenced Jun 9, 2026
MaximeRivest
added a commit
that referenced
this pull request
Jun 10, 2026
… boundary (TODOs #1 and #4) Close TODO(adapters-plan) #1: _call_preprocess's information-losing mutation is replaced by _engine/builder.build_plan, which performs the IDENTICAL lm_kwargs mutations and signature derivation in the IDENTICAL order while recording every decision onto an AdapterPlan — native tool specs, hidden fields as Hide transforms with reasons, and the lm_kwargs deltas produced by Type.adapt_to_native_lm_feature (which remains the planning hook: TODO #2 stays open until the strategies PR). Subtle semantics preserved verbatim: tool stripping happens before type hooks read lm_kwargs; the native-response-types loop iterates the snapshot of output fields as of loop start; issubclass resolves through the adapters.base module namespace so test_react's generic-alias guard monkeypatch keeps guarding the real path (plus a structural builder test that is monkeypatch-independent). Close TODO #4: _render_request accepts the plan, with a tripwire asserting plans carry no request content until the renderer PR — silent content drops are impossible by construction. __call__/acall build the plan once per attempt (never cached across the ChatAdapter->JSONAdapter fallback retry: the fallback constructs a new adapter, which re-plans) and consult the override verdict (everything routes legacy while the registry is empty, so this PR is byte-identical by construction). _call_preprocess survives as a thin wrapper preserving the legacy contract for direct callers; _call_postprocess gains a keyword-only plan parameter (positional signature pinned by the corpus) whose plan-driven backfill and ToolCalls destination are proven equal to the legacy signature lookups. Gates: all 176 golden fixtures reproduce byte-identically; tests/adapters 454 passed; tests/streaming + tests/callback green; tests/predict + tests/teleprompt green (incl. the issubclass-shadowing react test); import dspy still never loads the engine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaximeRivest
added a commit
that referenced
this pull request
Jun 10, 2026
…strategies (TODO #2) Semantic types become purely semantic; the engine owns representation planning through the strategy contracts a future exposure epic would publish unchanged. - _engine/strategy.py: per-field TypeStrategy and call-level PlanStep protocols per the design docs' vocabulary — stable name, priority, exclusive_group, side-effect-free applies(ctx), contribute(ctx) -> AdapterPatch over FieldContext/CallContext; deterministic resolution order; decisions recorded as StrategyTrace on plans. Documented legacy-compat deviation: strategies may mutate ctx.lm_kwargs exactly as the hooks did (byte parity over purity) until the normalized-config epic; everything else travels in the patch. - strategies/tools.py: native function calling as a PlanStep (it spans an input AND an output field — precisely why it is not a per-field strategy), lifted verbatim from the builder. - strategies/reasoning.py + strategies/citations.py: the gpt-5/litellm workaround, effort sourcing, and the anthropic/ provider sniff now live behind SHARED PREDICATES in the type modules (resolve_native_reasoning_effort, native_citations_supported) imported by BOTH the legacy hooks and the strategies — drift between paths is structurally impossible (source-inspection tests enforce it; a dependency test asserts type modules never import _engine). - Each strategy contributes its plan-carried parser hook (reasoning_content / citations channels), completing the IR's response side for QC-11. - Third-party types in native_response_types keep flowing through their documented adapt_to_native_lm_feature hook, silently honored with zero deprecation signaling (ratified; a registry epic owns that decision) — proven by a custom-type test that errors on DeprecationWarning. - Builder rewired to dispatch step + strategies + third-party fallback in the exact legacy order; native_response_types gates strategies exactly as it gated hooks (explicit test). Gates: full battery 1644 passed; 175 behavior fixtures reproduce exactly; existing type/streaming suites untouched (parse_stream_chunk/is_streamable not touched per the epic's streaming rule). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaximeRivest
added a commit
that referenced
this pull request
Jun 10, 2026
…strategies (TODO #2) Semantic types become purely semantic; the engine owns representation planning through the strategy contracts a future exposure epic would publish unchanged. - _engine/strategy.py: per-field TypeStrategy and call-level PlanStep protocols per the design docs' vocabulary — stable name, priority, exclusive_group, side-effect-free applies(ctx), contribute(ctx) -> AdapterPatch over FieldContext/CallContext; deterministic resolution order; decisions recorded as StrategyTrace on plans. Documented legacy-compat deviation: strategies may mutate ctx.lm_kwargs exactly as the hooks did (byte parity over purity) until the normalized-config epic; everything else travels in the patch. - strategies/tools.py: native function calling as a PlanStep (it spans an input AND an output field — precisely why it is not a per-field strategy), lifted verbatim from the builder. - strategies/reasoning.py + strategies/citations.py: the gpt-5/litellm workaround, effort sourcing, and the anthropic/ provider sniff now live behind SHARED PREDICATES in the type modules (resolve_native_reasoning_effort, native_citations_supported) imported by BOTH the legacy hooks and the strategies — drift between paths is structurally impossible (source-inspection tests enforce it; a dependency test asserts type modules never import _engine). - Each strategy contributes its plan-carried parser hook (reasoning_content / citations channels), completing the IR's response side for QC-11. - Third-party types in native_response_types keep flowing through their documented adapt_to_native_lm_feature hook, silently honored with zero deprecation signaling (ratified; a registry epic owns that decision) — proven by a custom-type test that errors on DeprecationWarning. - Builder rewired to dispatch step + strategies + third-party fallback in the exact legacy order; native_response_types gates strategies exactly as it gated hooks (explicit test). Gates: full battery 1644 passed; 175 behavior fixtures reproduce exactly; existing type/streaming suites untouched (parse_stream_chunk/is_streamable not touched per the epic's streaming rule). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaximeRivest
added a commit
that referenced
this pull request
Jun 10, 2026
…strategies (TODO #2) Semantic types become purely semantic; the engine owns representation planning through the strategy contracts a future exposure epic would publish unchanged. - _engine/strategy.py: per-field TypeStrategy and call-level PlanStep protocols per the design docs' vocabulary — stable name, priority, exclusive_group, side-effect-free applies(ctx), contribute(ctx) -> AdapterPatch over FieldContext/CallContext; deterministic resolution order; decisions recorded as StrategyTrace on plans. Documented legacy-compat deviation: strategies may mutate ctx.lm_kwargs exactly as the hooks did (byte parity over purity) until the normalized-config epic; everything else travels in the patch. - strategies/tools.py: native function calling as a PlanStep (it spans an input AND an output field — precisely why it is not a per-field strategy), lifted verbatim from the builder. - strategies/reasoning.py + strategies/citations.py: the gpt-5/litellm workaround, effort sourcing, and the anthropic/ provider sniff now live behind SHARED PREDICATES in the type modules (resolve_native_reasoning_effort, native_citations_supported) imported by BOTH the legacy hooks and the strategies — drift between paths is structurally impossible (source-inspection tests enforce it; a dependency test asserts type modules never import _engine). - Each strategy contributes its plan-carried parser hook (reasoning_content / citations channels), completing the IR's response side for QC-11. - Third-party types in native_response_types keep flowing through their documented adapt_to_native_lm_feature hook, silently honored with zero deprecation signaling (ratified; a registry epic owns that decision) — proven by a custom-type test that errors on DeprecationWarning. - Builder rewired to dispatch step + strategies + third-party fallback in the exact legacy order; native_response_types gates strategies exactly as it gated hooks (explicit test). Gates: full battery 1644 passed; 175 behavior fixtures reproduce exactly; existing type/streaming suites untouched (parse_stream_chunk/is_streamable not touched per the epic's streaming rule). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed:
helpers.
Validation: