Skip to content

feat(python): add preserveUnknownFields option - #146

Merged
2ynn merged 2 commits into
mainfrom
qynn/feat/python-preserve-unknown-fields
Oct 27, 2026
Merged

2ynn merged 2 commits into
mainfrom
qynn/feat/python-preserve-unknown-fields

Conversation

@2ynn

@2ynn 2ynn commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Why

Pydantic drops fields a model does not declare. Python SDKs therefore lose undeclared fields in nested JSON request and response bodies: they are not sent to the API, and they disappear from response models, so a response cannot be sent back unchanged.

What changed

Add python.preserveUnknownFields (default false). When enabled:

  • generated models and error bodies allow extra fields, including schemas with additionalProperties: false; explicitly typed additional properties still validate;
  • each model serializer reads every declared field from the key Pydantic emitted (alias or field name, including the alias-keyed output a nested model is serialized from again) and removes it, then copies the remaining keys, the unknown fields, back into the output. serialize_with_info exposes the dump's by_alias option next to exclude_none (current_by_alias), so serialize_model(self, handler) keeps its two-argument signature. With the option disabled the serializer reads fields as before;
  • marshal_json keeps nested nulls and excludes only a top-level None; optional fields left unset are still omitted;
  • UNSET ignores extra fields;
  • generated test assertions compare only the fields their examples declare, through a drop_unexpected_fields test helper, because mock responses can carry more. Declared additional properties keep their exact key set, and their values are compared the same way.

Known limitations

  • When a payload contains both a field's API name and an undeclared key equal to its Python name (displayName and display_name), API-name dumps (model_dump(by_alias=True), the request body) keep both keys, but Python-name dumps (the default model_dump()) show the undeclared value in place of the declared one. Plain Pydantic behaves the same with extra="allow" and populate_by_name=True.
  • Additional properties assigned under a key the model already emits replace the declared value in the output.
  • Flattened operation arguments accept only declared fields, so undeclared top-level request fields need flattenRequests: false; nested objects keep theirs. The option description states this.
  • Unknown fields are carried in JSON bodies only; form, multipart, query and header serialization use declared fields.
  • As in fix(python): honour include/exclude filters in model dumps #149, a dump that combines exclude_none=True with include, exclude, exclude_unset or exclude_defaults can still write excluded required fields back as None.

types/basemodel.py becomes a template (basemodel.py.stmpl) that skips formatting, so SDKs with the option disabled keep the same file. The review SDK gains only preserveUnknownFields: false in gen.yaml.

Test coverage

  • New unknown-fields uber fragment: an echo operation whose body nests objects, arrays, maps, discriminated and plain unions, allOf, open/typed/closed additional properties (including additional properties typed as an object schema), empty objects, open enums and nullable fields, plus a flattened variant and an error response.
  • New test ID unknown-fields-json-round-trip, skipped for other targets. The tertiary TypeScript overlay removes the echo operation because its inline success and error bodies get the same generated name in that configuration.
  • primary (option disabled): unknown fields are still dropped.
  • secondary (enabled, flattenRequests: false): request/response round trips for dicts and models, response-only and error-body extras, dump filters, typed additional-properties validation, top-level None handling.
  • quaternary (enabled, flattenRequests: true): nested unknown fields through flattened arguments, sync and async; drop_unexpected_fields ignores unknown fields inside typed additional-property values but still reports extra map entries and changed values.

Stack

  1. fix(python): serialize unrecognized open enum values #145: unrecognized enum serialization (merged)
  2. fix(python): honour include/exclude filters in model dumps #149: honour include/exclude filters in model dumps (merged)
  3. This PR: preserveUnknownFields
  4. feat(python): add modelSerializationMethods option #147: modelSerializationMethods

Testing

  • TARGET=primary make test-pythonv2 (option disabled): 728 passed.
  • TARGET=secondary make test-pythonv2: 330 passed, 1 skipped.
  • TARGET=quaternary make test-pythonv2: 257 passed, 1 skipped. The typed additional-properties helper test fails with the previous drop_unexpected_fields.
  • TARGET=review make build-pythonv2: integrity validation passes.
  • Python model serializer snapshot tests pass unchanged.
  • prettier, make check-template-pythonv2 and make lint pass.
  • git diff --check passes.
  • TARGET=tertiary make test-pythonv2 (option disabled): 214 passed, 1 skipped.

Public-safety check

  • This change contains no credentials, customer documents, private repository URLs, private filesystem paths, or unredacted private logs.
  • Title, body, comments, and commit messages name no customers or customer-derived identifiers, private paths or trackers, or workflow provenance, and are understandable without private context (.claude/skills/public-repo-communication/SKILL.md).
  • Generated fixtures and review SDK changes are public-safe.
  • I reviewed git diff --check.

@github-actions github-actions Bot added csharp Trigger (24) snapshot tests for csharp go Trigger (90) snapshot tests for go java Trigger (40) snapshot tests for java mcp-typescript Trigger (16) snapshot tests for mcp-typescript php Trigger (22) snapshot tests for php python Trigger (106) snapshot tests for python ruby Trigger (19) snapshot tests for ruby typescript Trigger (168) snapshot tests for typescript unity labels Oct 23, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 31 files

Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread templates/templates/pythonv2/includes/tests.ts
Comment thread templates/templates/pythonv2/model.py.stmpl
@2ynn
2ynn force-pushed the qynn/fix/python-unrecognized-enum-serialization branch from e86efdc to dce293e Compare October 23, 2026 15:55
@2ynn
2ynn force-pushed the qynn/feat/python-preserve-unknown-fields branch from a47683e to 189b725 Compare October 23, 2026 15:55
@speakeasy-generator-bot

speakeasy-generator-bot Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Generator Snapshot Testing

Snapshot tests are running.

Snapshot results: https://github.com/speakeasy-api/openapi-generation-snapshots/issues/135#issuecomment-6451260182

@2ynn
2ynn force-pushed the qynn/feat/python-preserve-unknown-fields branch from 189b725 to 56a8cb5 Compare October 23, 2026 17:19
@2ynn
2ynn changed the base branch from qynn/fix/python-unrecognized-enum-serialization to qynn/fix/python-model-dump-filters October 23, 2026 17:19
@2ynn
2ynn marked this pull request as ready for review October 23, 2026 17:37
@2ynn
2ynn requested a review from a team as a code owner October 23, 2026 17:37
@2ynn
2ynn requested review from TristanSpeakEasy and removed request for a team October 23, 2026 17:37
@2ynn
2ynn force-pushed the qynn/fix/python-model-dump-filters branch from 9ddfe10 to 7eca0ea Compare October 23, 2026 18:05
@2ynn
2ynn force-pushed the qynn/feat/python-preserve-unknown-fields branch from 56a8cb5 to 876df5f Compare October 23, 2026 18:05

@TristanSpeakEasy TristanSpeakEasy left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, keeping this opt-in and retaining validation for typed additional properties makes sense.

@2ynn
2ynn force-pushed the qynn/feat/python-preserve-unknown-fields branch 2 times, most recently from 876df5f to d26e5f7 Compare October 27, 2026 15:47
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@2ynn
2ynn force-pushed the qynn/fix/python-model-dump-filters branch from 67a4d38 to f132c2d Compare October 27, 2026 16:13
Base automatically changed from qynn/fix/python-model-dump-filters to main October 27, 2026 18:49
@2ynn
2ynn force-pushed the qynn/feat/python-preserve-unknown-fields branch from d26e5f7 to 184152b Compare October 27, 2026 19:01
2ynn added 2 commits October 27, 2026 15:03
Pydantic drops fields a model does not declare, so undeclared fields in
nested JSON request and response bodies were lost on validation and on
serialization back to the API.

Add python.preserveUnknownFields (default false). When enabled:

- generated models and error bodies allow extra fields, including
  schemas with additionalProperties: false; explicitly typed additional
  properties still validate
- model serializers read each declared field from the key Pydantic
  emitted (alias or field name, including the alias-keyed output a
  nested model is serialized from again) and copy the remaining keys
  back out
- marshal_json keeps nested nulls and only excludes a top-level None
- UNSET ignores extra fields
- generated test assertions compare only the fields their examples
  declare, since mock responses can carry more; declared additional
  properties keep their exact key set and are compared recursively

basemodel.py becomes a template, so the review SDK copy now passes
through the Python formatter.

The option is covered by an echo fragment exercised in the primary
(disabled), secondary and quaternary variants.
@2ynn
2ynn force-pushed the qynn/feat/python-preserve-unknown-fields branch from 184152b to e8dd929 Compare October 27, 2026 19:03
@2ynn
2ynn added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit bb257c2 Oct 27, 2026
449 of 453 checks passed
@2ynn
2ynn deleted the qynn/feat/python-preserve-unknown-fields branch October 27, 2026 19:47
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

csharp Trigger (24) snapshot tests for csharp go Trigger (90) snapshot tests for go java Trigger (40) snapshot tests for java mcp-typescript Trigger (16) snapshot tests for mcp-typescript php Trigger (22) snapshot tests for php python Trigger (106) snapshot tests for python ruby Trigger (19) snapshot tests for ruby typescript Trigger (168) snapshot tests for typescript unity

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants