Skip to content

State saved by 0.10.0a2 workers is discarded by 0.9.12 and 0.10.0a1 workers: a rolling deploy or rollback silently resets every session #7470

Closed as not planned

Description

@masenf

Severity: MEDIUM, regression vs 0.10.0a1. Independently confirmed on 0.10.0a2.

What happens

With the disk or Redis state manager, state persisted by a 0.10.0a2 worker is discarded (treated as incompatible and reset) when a 0.9.12 or 0.10.0a1 worker loads it. The other direction works: 0.10.0a2 reads 0.9.12-saved state fine. Consequences: in a rolling deploy with mixed old/new workers, or on a rollback, every user whose session lands on an older worker is silently reset (logged out, form state gone) with nothing in the logs on the new side.

Why (to confirm)

Something in the pickled payload / schema check changed between a1 and a2 (candidates: the descriptor-based Field layout from #7312/#7461, __getstate__, or the schema hash used to decide whether a saved state is compatible). The older loader rejects the newer payload; the newer loader accepts the older one.

Suggested fix

Either emit a payload older workers accept (if the difference is cosmetic / can be aliased), or, if the schema really changed, make the constraint explicit: document in the 0.10 upgrade notes that sessions saved by 0.10 cannot be read by 0.9.x workers (roll back only with a restart / state flush) and ship a breaking news fragment. The hidden-reset behaviour is the problem.

Repro and evidence

  • Verifier run (new worker writes, old worker reads; disk and Redis): reverify_core/verification/n004-schema-rollback/.``
  • Finding N-004 in FINDINGS.md (with the browser_cache_bundle cluster's mixed-version deployment matrix: same-schema stable/a1 frontends against an a2 backend work).

Activity

  1. self-assigned this
    on Oct 7, 2026
  2. added theissue type on Oct 7, 2026
  3. linear-code commented on Oct 7, 2026

    @linear-code
  4. added 2 commits that reference this issue on Oct 7, 2026
    378d403
    74140e2
  5. masenf commented on Oct 7, 2026

    @masenf
    CollaboratorAuthor

    Decision from @masenf: rather than keep state saved by 0.10 readable by 0.9.x (and 0.10.0a1) instances, this is declared a breaking change: an app cannot run 0.9.x and 0.10.x instances against the same Redis or disk state store. A 0.10 instance still loads state saved by 0.9.x; a 0.9.x instance discards state saved by 0.10, so upgrade every instance together and clear the store on a rollback. The breaking news fragment and the docs note land in #7494, which no longer changes the pickle format.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions