Skip to content

fix(adapters): keep pydantic field constraints in structured outputs - #10326

Open
Rodrigo-Palma wants to merge 3 commits into
stanfordnlp:mainfrom
Rodrigo-Palma:fix/structured-outputs-keep-field-constraints
Open

Rodrigo-Palma wants to merge 3 commits into
stanfordnlp:mainfrom
Rodrigo-Palma:fix/structured-outputs-keep-field-constraints

Conversation

@Rodrigo-Palma

Copy link
Copy Markdown

📝 Changes Description

Fixes #10195.

_get_structured_outputs_response_format rebuilt every output field as a bare
(annotation, default) pair:

default = field.default if hasattr(field, "default") else ...
fields[name] = (annotation, default)

That discards the FieldInfo, and with it every constraint declared on the field —
ge/le/gt/lt, multiple_of, pattern, min_length/max_length. The derived
response format kept the type and dropped the bound, so the provider was never asked to
honor it:

class Score(dspy.Signature):
    text: str = dspy.InputField()
    score: float = dspy.OutputField(ge=0.0, le=1.0, multiple_of=0.25)

_get_structured_outputs_response_format(Score, True).model_json_schema()["properties"]
# before: {"score": {"title": "Score", "type": "number"}}
# after:  {"score": {"maximum": 1.0, "minimum": 0.0, "multipleOf": 0.25,
#                    "title": "Score", "type": "number"}}

Why this reads as a loss rather than a deliberate choice (the issue asks): DSPy
already consumes this exact metadata on the other path. move_kwargs runs
_translate_pydantic_field_constraints(**kwargs) and stores the result in
json_schema_extra["constraints"], which the prompt-side adapters render as natural
language for the model. So the same OutputField(ge=0, le=1) states its bound to the
model in JSON mode and states nothing at all in structured-outputs mode. The framework
understands the constraint; only the derivation drops it.

The change: rebuild from the FieldInfo and clear only DSPy's own prompt metadata,
which is not part of the wire schema. The default carries over with the FieldInfo, so
the required list and additionalProperties: false enforcement are unchanged, and a
field declared without constraints produces byte-for-byte the same schema as before.

Tests (tests/adapters/test_json_adapter.py, 6 added): numeric bounds, exclusive
bounds and string constraints are preserved — these three fail on main; DSPy metadata
does not leak into the schema; required/additionalProperties still enforced; an
unconstrained field's schema is unchanged.

✅ Contributor Checklist

  • Pre-Commit checks are passing (locally and remotely)
  • Title of your PR / MR corresponds to the required format
  • Commit message follows required format {label}(dspy): {message}

Full suite as CI runs it: pytest -m 'not extra and not deno' -n auto → 1256 passed,
253 skipped, 2 xfailed. ruff check --fix-only --diff --exit-non-zero-on-fix clean.

⚠️ Warnings

Structured-outputs backends differ in which JSON Schema validation keywords they accept.
The keywords emitted here (minimum, maximum, exclusiveMinimum, exclusiveMaximum,
multipleOf, pattern, minLength, maxLength) are in OpenAI's documented supported
subset, but a provider that rejects them would now fail where it previously succeeded by
silently dropping the user's constraint. If you would rather sanitize per provider than
emit them unconditionally, that split belongs at this same seam and I am happy to add it —
I did not want to guess at a provider allowlist.


Disclosure: I used an AI coding assistant to help explore the code and draft this. I
reproduced the behavior, wrote and verified the tests, and understand the change.

@greptile-apps

greptile-apps Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Fixes field constraints in structured output schemas.

No blocking failure remains.

What we checked:

  • Ran the JSONAdapter alias reproduction test to verify that the aliased OutputField is required and that the parser rejects a non-schema-valid alias response. T-Rex
  • Executed output field constraint tests against numeric, string, and transformed signatures; observed that score 1.25 is rejected by both direct and transformed schemas, confirming constraints are enforced. T-Rex
  • T-Rex produced a formal proof for a posted P1 finding. T-Rex
  • Documented the exact authored alias repro source and observed parsing behavior, including that the schema contains answer_text and that alias-key completion raises an AdapterParseError. T-Rex
  • Captured exact authored source and observed outputs for output-field constraints, including that the constraints remain enforced and the regression tests pass. T-Rex

Summary

Output-field constraints are retained in provider response schemas and continue to be enforced after signature transformations. The earlier alias-alignment report is disproved: generated schemas require the signature field name answer_text, and the existing end-to-end coverage confirms that key parses successfully; a response using the omitted alias is not schema-valid.

Reviews (3) · Last reviewed commit: "fix(adapters): forward only OutputField-..."

Comment thread dspy/adapters/json_adapter.py Outdated
@Rodrigo-Palma

Copy link
Copy Markdown
Author

Good catch, and it reproduces exactly as described: answer_text: str = dspy.OutputField(alias="answer") produced a schema whose only property was answer, while JSONAdapter.parse keys off the signature field names, so a schema-valid {"answer": ...} was discarded before validation.

Fixed in acfda15, and the root cause was broader than the alias. Copying the whole FieldInfo carried everything it holds into the wire schema, not just the constraints this PR is about. A field's description leaked the same way — OutputField(description="the answer") started emitting "description" into the schema, which is a separate change from preserving constraints and does not belong here either.

So rather than stripping attributes one by one, the fix now re-attaches only what the constraints actually live in. field.metadata holds exactly the declared constraints, so:

annotation = Annotated[(annotation, *field.metadata)] if field.metadata else annotation
fields[name] = (annotation, default)

Nothing else from the FieldInfo can reach the schema by construction — no alias, no description, no title override.

I deliberately did not take the other route you mentioned (normalizing aliases back to field names in the parser). Making the adapter alias-aware is a feature, not part of a constraints fix, and it would change what responses DSPy accepts. This PR now leaves alias behavior exactly as it was on main.

Schema after the fix, with constraints, an alias and a description all declared on the same signature:

{"answer_text": {"title": "Answer Text", "type": "string"},
 "label": {"maxLength": 10, "pattern": "^[a-z]+$", "title": "Label", "type": "string"},
 "score": {"maximum": 1.0, "minimum": 0.0, "multipleOf": 0.25, "title": "Score", "type": "number"}}
required: ["score", "label", "answer_text"]

Three tests added for this: the alias does not reach the schema or required, the description does not, and an aliased output field still parses end to end. Full suite: 1259 passed, 253 skipped, 2 xfailed.

@greptileai

@greptile-apps

greptile-apps Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Thanks for the follow-up. This addresses the concern: reattaching only field.metadata preserves the Pydantic constraints without leaking DSPy-only FieldInfo attributes such as aliases or descriptions. Keeping aliases unchanged is also the right scope for this fix, and the added schema plus end-to-end parsing tests cover the regression. No further changes needed from my side.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

@Rodrigo-Palma

Copy link
Copy Markdown
Author

The workflow run here is waiting on maintainer approval (first-time contributor), so I ran this repo's full CI on my fork to have the result available in advance:

Lint, Test, and Build run — fix/structured-outputs-keep-field-constraints

All 19 jobs green: Run Tests on 3.10 / 3.11 / 3.12 / 3.13 / 3.14, Run Extra and Deno Tests on all five, Run MCP v2 Tests, Run Tests with Real LM, Build Package on all five, Check Ruff Fix, and Lint GitHub Actions Workflows.

_get_structured_outputs_response_format rebuilt every output field as a bare
(annotation, default) pair, which discards the FieldInfo and with it every
constraint declared on the OutputField: ge/le/gt/lt, multiple_of, pattern and
min_length/max_length. The derived response format kept the type and dropped the
bound, so a provider was never asked to honor it.

DSPy already consumes this same metadata on the prompt path: move_kwargs runs
_translate_pydantic_field_constraints and stores the result in
json_schema_extra["constraints"], which the adapters render as natural language.
The same OutputField(ge=0, le=1) therefore stated its bound to the model in JSON
mode and stated nothing at all in structured-outputs mode.

Rebuild from the FieldInfo and clear only DSPy's own prompt metadata, which is
not part of the wire schema. Defaults carry over with the FieldInfo, so the
required list and additionalProperties enforcement are unchanged, and a field
declared without constraints produces the same schema as before.
Copying the FieldInfo carried more than the constraints into the wire schema.
An output field's alias replaced its name there, so a provider response that was
valid for the schema was discarded by JSONAdapter.parse, which keys off the
signature field names, and the call failed with AdapterParseError. A field's
description leaked the same way, which is a separate change from preserving
constraints and does not belong in this one.

field.metadata holds exactly the declared constraints, so re-attach those through
Annotated and leave everything else alone. The schema keeps the signature field
names, gains no description, and still carries every bound.
@Rodrigo-Palma

Copy link
Copy Markdown
Author

Rebased on main. #10463 added test_ordinary_annotated_fields_keep_existing_parsing_and_schema, which keeps constraints from Annotated[..., Field(...)] out of the structured output schema. So the schema now gets only the constraints declared as OutputField kwargs, the same ones the prompt already states, tracked through signature rebuilds such as ChainOfThought. Annotated and decision type metadata behave as on main.

@Rodrigo-Palma
Rodrigo-Palma force-pushed the fix/structured-outputs-keep-field-constraints branch from acfda15 to d5c9046 Compare October 1, 2026 12:49
@greptile-apps

greptile-apps Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings could not be posted inline.

  • P1 JSONAdapter rejects an OutputField alias returned by the model ▶

    • Bug
      • For answer_text: str = dspy.OutputField(alias="answer"), parsing {"answer":"ok"} raises AdapterParseError and produces an empty parsed result instead of returning {"answer_text": "ok"}.
    • Cause
      • JSONAdapter.parse() filters response keys against signature.output_fields, whose key is answer_text, rather than mapping accepted field aliases to their original output-field names. Separately, _get_structured_outputs_response_format() builds its Pydantic model from the original field name, so its generated schema exposes/requires answer_text rather than answer.
    • Fix
      • Build an alias-to-output-name map when parsing and normalize alias keys before filtering/type conversion. Decide and test the intended schema contract: preserve the OutputField alias in the generated Pydantic model/schema if providers should emit aliases, or explicitly document that aliases are unsupported for structured outputs.

@Rodrigo-Palma

Copy link
Copy Markdown
Author

@greptile-apps the alias finding is not introduced here: on main the schema for answer_text: str = dspy.OutputField(alias="answer") already exposes answer_text, and JSONAdapter().parse raises AdapterParseError for {"answer": "ok"} exactly as on this branch. Alias support in parsing would be a separate change.

@greptile-apps

greptile-apps Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

You’re right. This is not introduced by the PR: on main, _get_structured_outputs_response_format() already rebuilds fields under the signature name, and JSONAdapter.parse() already filters against signature.output_fields. Therefore, an OutputField(alias="answer") response has the same behavior before and after this change.

The current patch only reattaches DeclaredConstraints metadata; it does not alter the existing alias contract. Alias-aware parsing or schema generation would be a separate feature, so the alias finding should be disregarded for this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant