Skip to content

perf: reuse serializer wrapper models instead of rebuilding them on every call - #1335

Merged
jason-schrader-openrouter merged 2 commits into
mainfrom
devin/1794545319-cache-serializer-models
Nov 13, 2026
Merged

jason-schrader-openrouter merged 2 commits into
mainfrom
devin/1794545319-cache-serializer-models

Conversation

@jason-schrader-openrouter

@jason-schrader-openrouter jason-schrader-openrouter commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1056.

utils/serializers.py calls pydantic.create_model("Unmarshaller" | "Marshaller", body=(typ, ...), __config__=...) every time marshal_json or unmarshal runs. As a result, each chat.send / chat.send_async call builds and compiles about 19 throwaway model classes. That file is Speakeasy template code and gets overwritten on regeneration, so this PR fixes it from _hooks/, which regeneration doesn't overwrite:

  • New file _hooks/serializer_cache.py. install() replaces serializers.create_model with a version that builds one model per (name, type, config) and keeps it in a bounded lru_cache (_CACHE_SIZE = 1024).
  • The cache key is _type_key(typ), not typ alone. Annotations that compare equal can still validate differently, so the key keeps union member order and the type of each leaf:
    int | float == float | int        # True, but pydantic tries members in order
    Literal[1] == Literal[True]       # True, but pydantic checks the literal's type
    _type_key(int | float)  # (UnionType, ((type, int), (type, float)))
    Without this, unmarshal("1", int | float) followed by unmarshal("1", float | int) returned 1 (int) from the cache instead of 1.0 (float).
  • Calls in any other shape, or with arguments that can't be hashed, go straight to pydantic.create_model. If the template changes, the SDK keeps working; it just stops caching.
  • init_hooks() in registration.py calls install(), so the cache is active as soon as an OpenRouter client is created.

Measured on 1.3.173 with a local stub server (Python 3.12, pydantic 2.14.3):

Models built per call CPU per call
Before 19 15–17 ms
After 0 4–5 ms

tests/test_serializer_cache.py checks that:

  • creating a client installs the hook, and a second chat.send builds no new models;
  • for unions in both orders, literals, aliases, Optional, generics and validation errors, unmarshal and marshal_json give the same result type, value or error with the cache as with plain pydantic.create_model, after warming the cache in forward and reverse order;
  • unhashable annotations fall back without caching;
  • the cache stops growing at _CACHE_SIZE.

The equivalence and bound tests fail against the previous cache key. Delete this hook once the upstream template caches these models itself.

…very call

marshal_json and unmarshal call pydantic.create_model on every call, so each
chat request built about 19 model classes. An SDK init hook now caches one
wrapper model per type. Fixes #1056.
@devin-ai-integration

devin-ai-integration Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

@jason-schrader-openrouter
jason-schrader-openrouter merged commit e5567da into main Nov 13, 2026
1 check passed
@jason-schrader-openrouter
jason-schrader-openrouter deleted the devin/1794545319-cache-serializer-models branch November 13, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Serializers rebuild pydantic models on every call: ~19 per chat.send_async, ~25 ms of CPU per request

2 participants