Skip to content

fix(python): honour include/exclude filters in model dumps - #149

Merged
2ynn merged 3 commits into
mainfrom
qynn/fix/python-model-dump-filters
Oct 27, 2026
Merged

2ynn merged 3 commits into
mainfrom
qynn/fix/python-model-dump-filters

Conversation

@2ynn

@2ynn 2ynn commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Why

Every generated Python model with optional or nullable fields has a wrap model_serializer. It rebuilds the output field by field from Pydantic's own dump and writes required fields, and nullable fields that were explicitly set, even when Pydantic left them out. A field that the caller filtered out therefore came back as None:

model = shared.FilteredFields(display_name="name", required_nullable=None)
model.model_dump(exclude={"display_name"})
# before: {'displayName': None, 'requiredNullable': None, 'kind': 'filtered', 'nullDefault': None, 'withDefault': 'fallback'}
# after:  {'requiredNullable': None, 'kind': 'filtered', 'nullDefault': None, 'withDefault': 'fallback'}

The same happened with include, exclude_unset and exclude_defaults, for required fields, const fields and nullable fields with a null default, including in nested models.

The write-back is needed in one case: request bodies are dumped with exclude_none=True, and must still send null for required fields and explicitly set nullable fields.

What changed

  • Skipping filtered fields. The serializer skips a declared field that is absent from Pydantic's dump under both its API name and its Python name, unless the dump uses exclude_none. Both spellings are checked because a nested model's serializer also runs on its own earlier output, which is keyed by API name.

  • Reading exclude_none without changing serialize_model's signature. Pydantic only passes info to a serializer whose signature requires it, and subclasses that override serialize_model and delegate with super().serialize_model(handler) must keep working. The generated code is now:

    @model_serializer(mode="wrap")
    def _serialize_model_with_info(self, handler, info):
        return serialize_with_info(self.serialize_model, handler, info)
    
    def serialize_model(self, handler):
        ...
        exclude_none = current_exclude_none()

    serialize_with_info and current_exclude_none live in types/basemodel.py. They hold info.exclude_none in a context variable for the duration of the call. serialize_model keeps its (self, handler) signature and skips filtered fields only when exclude_none is False. When serialize_model is called directly, the value is None and the output matches the previous serializer, apart from the empty-alias fix below.

  • The read expression and the additional-properties handling are unchanged, so JSON request bodies are unchanged: marshal_json always dumps with exclude_none=True, where the new skip never runs, and it is the only model dump the SDK runtime performs.

  • Fields whose API name is the empty string. Multipart file models declare their content field with alias="". The serializer computed the API name as f.alias or n, so it looked for content while Pydantic's by-alias output holds "". With by_alias=True this was already broken: the value was lost and written back as None ({'fileName': 'a', 'content': None, 'Content-Type': None}), and with the skip above the field would have disappeared entirely. The API name is now f.alias if f.alias is not None else n, so the value is kept under "" in model_dump(), model_dump(by_alias=True) and marshal_json. Plain model_dump() previously kept it under content; it now uses "", like every other field uses its API name. Multipart requests are serialized from field metadata, not from these dumps, and the multipart upload tests pass unchanged.

  • Review SDK: each of the 66 wrap serializers gains _serialize_model_with_info, the current_exclude_none() read, the skip lines and the new API-name line; types/basemodel.py and types/__init__.py gain the two helpers; gen.lock changes.

Known limitations

  • A dump that combines exclude_none=True with include, exclude, exclude_unset or exclude_defaults can still write excluded required fields back as None. The serializer cannot tell a field dropped for being None from one dropped by another filter, and request bodies rely on the former.
  • A subclass override decorated with @model_serializer that takes only (self, handler) and delegates with super().serialize_model(handler) gets the previous output, filtered fields included, because it never receives info; when the model is dumped as part of another generated model, the enclosing serializer's exclude_none applies and the filtering works. Declaring info and delegating with serialize_with_info(super().serialize_model, handler, info) restores the filtering. Overrides without the decorator keep the filtering.
  • A field excluded from the dump is still written when an additional property uses the field's other spelling (display_name next to displayName).

Test coverage

New model-dump-filters primary fragment: FilteredFields (required renamed field, required and optional nullable fields, a defaulted field, a nullable field with a null default, a const field, a nested copy and an array of copies), FilteredFieldsWithAdditionalProperties, and an error response whose body nests FilteredFields.

test_model_dump_filters_additional.py (19 tests) covers:

  • include, exclude, exclude_unset and exclude_defaults, top level and nested;
  • the exclude_none limitation above;
  • default model_dump() / model_dump_json() of nested models and TypeAdapter(List[...]) of model lists keeping API names;
  • request bodies: explicit nulls (top level and nested), a required nullable field, an omitted null additional property, an additional property under a field's Python name, and a top-level None body;
  • error-data models, which have no wrap serializer: plain Pydantic filtering, and exclude honoured by the nested model;
  • a field with an empty API name (BinaryString.content) keeping its value, and staying out when excluded;
  • subclass overrides of serialize_model: a decorated two-argument override delegating with super(), and an undecorated override keeping exclude.

Testing

  • TARGET=primary make test-pythonv2: 701 passed, including the multipart upload tests.
  • TARGET=secondary make test-pythonv2: 264 passed, 1 skipped.
  • TARGET=tertiary make test-pythonv2: 214 passed, 1 skipped.
  • TARGET=quaternary make test-pythonv2: 248 passed, 1 skipped.
  • TARGET=review make test-pythonv2: 36 passed, 6 skipped; regenerating the review SDK on main produces no further diff.
  • pyright and mypy pass on the generated primary SDK.

Stack

  1. fix(python): serialize unrecognized open enum values #145: unrecognized enum serialization (merged)
  2. This PR: honour include/exclude filters in model dumps
  3. feat(python): add preserveUnknownFields option #146: preserveUnknownFields
  4. feat(python): add modelSerializationMethods option #147: modelSerializationMethods

Public-safety check

  • This change contains no credentials, customer documents, private repository URLs, private filesystem paths, or unredacted private logs.
  • Title, body, comments, and commit messages name no customers or customer-derived identifiers, private paths or trackers, or workflow provenance, and are understandable without private context (.claude/skills/public-repo-communication/SKILL.md).
  • Generated fixtures and review SDK changes are public-safe.
  • I reviewed git diff --check.

@github-actions github-actions Bot added the python Trigger (106) snapshot tests for python label Oct 23, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 55 files

Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread templates/templates/pythonv2/model.py.stmpl Outdated
@2ynn
2ynn force-pushed the qynn/fix/python-model-dump-filters branch from 9ddfe10 to 7eca0ea Compare October 23, 2026 18:05
@speakeasy-generator-bot

speakeasy-generator-bot Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Generator Snapshot Testing

Selector(s) Result Summary
snapshot-python ✅ Snapshot run completed

Snapshot results: https://github.com/speakeasy-api/openapi-generation-snapshots/issues/136#issuecomment-6449573495

Snapshot results are summarized above. Additional run context is linked for maintainers with access.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 52 reviewed files. 1 file intentionally excluded from review.

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Shadow auto-approve: would auto-approve. Fixes generated Python model serializers to honour include/exclude/exclude_unset/exclude_defaults by skipping absent fields unless exclude_none, and corrects empty-alias handling. Bounded template/generated-code bug fix with new tests; request-body dumps remain unchanged.
Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack

View guided diff | Turn on auto-fix | Re-trigger cubic

@2ynn
2ynn marked this pull request as ready for review October 23, 2026 18:10
@2ynn
2ynn requested a review from a team as a code owner October 23, 2026 18:10
@2ynn
2ynn requested review from speakeasybot and removed request for a team October 23, 2026 18:10

@TristanSpeakEasy TristanSpeakEasy left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filtering fix looks good. Left one non-blocking comment on compatibility with existing serializer overrides, worth sorting before you merge.

Comment thread templates/templates/pythonv2/model.py.stmpl Outdated
Base automatically changed from qynn/fix/python-unrecognized-enum-serialization to main October 27, 2026 14:47
2ynn added 2 commits October 27, 2026 12:08
Generated wrap serializers wrote required and explicitly set nullable
fields back as None even when include, exclude, exclude_unset or
exclude_defaults left them out of the dump. Skip a declared field that
is absent from Pydantic's output under both its API name and its
Python name, unless the dump uses exclude_none, which request bodies
rely on to keep sending null.

Also read fields whose API name is the empty string (multipart file
content) under that name: the serializer fell back to the Python name,
so by-alias dumps lost the value.
The generated model serializer took a third `info` argument so it could
read `exclude_none`, which broke subclasses that delegate with
`super().serialize_model(handler)` and two-argument overrides under type
checking.

`serialize_model(self, handler)` keeps its previous signature and is no
longer the pydantic serializer itself. A new `_serialize_model_with_info`
wrap serializer receives `info`, exposes `exclude_none` through a context
variable for the duration of the call, and delegates to
`self.serialize_model`. Overrides decorated with `@model_serializer` that
call the parent get the previous behaviour; undecorated overrides keep the
include/exclude filtering.
@2ynn
2ynn force-pushed the qynn/fix/python-model-dump-filters branch from 67a4d38 to f132c2d Compare October 27, 2026 16:13

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed

Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread zSDKs/sdk-pythonv2/src/speakeasy/new_openapi/models/file.py
@2ynn
2ynn added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit b3b4a2e Oct 27, 2026
435 of 440 checks passed
@2ynn
2ynn deleted the qynn/fix/python-model-dump-filters branch October 27, 2026 18:49
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

python Trigger (106) snapshot tests for python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants