Skip to content

feat(xo-server,rest-api,xo-web/rpu): persist a recovery record across restarts - #10331

Open
mpiton wants to merge 2 commits into
masterfrom
feat/rpu-recovery-record
Open

feat(xo-server,rest-api,xo-web/rpu): persist a recovery record across restarts#10331
mpiton wants to merge 2 commits into
masterfrom
feat/rpu-recovery-record

Conversation

@mpiton

@mpiton mpiton commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Description

An xo-server restart in the middle of a rolling pool update currently loses everything: which hosts were done, which VMs were shut down because they cannot be migrated (PCI passthrough, vGPU, SR-IOV VIF), what the operator had asked for. The pinned VMs are the worst part: they only live in an in-memory map restored through $defer, so after a crash they stay halted with nothing left to restart them.

This PR makes xo-server persist a rpuRecovery record (LevelDB, one per pool) before the first side effect of a rolling pool update, and keep it updated during the run:

  • operator options, update variant (xcp / xs-cdn / xs-legacy), host order, initial VM placement;
  • per-host step statuses (evacuate, update, reboot, enable, restoreVms), with a strict write of each pinned VM → original host entry before the VM is shut down;
  • the last error, filtered through the existing RPU observability scrubber so credentials never reach the disk.

A record still in a live status at xo-server boot is flipped to interrupted. A successful run deletes its record. An unreadable record or an unknown schema version is reported as blocked without touching the stored value.

The status is exposed through pool.getRollingUpdateRecovery and GET /rest/v0/pools/{id}/rolling_update_recovery (filtered view, never the raw intent, reusing the pool rolling-update ACL), and the pool Patches tab shows a warning banner with per-host progress, the last error and the pinned VMs still halted.

This is the first card of the RPU resume-after-failure work: the record only reports for now, resuming an interrupted run comes separately.

Note for DevOps: this updates the generated OpenAPI spec (new GET /pools/{id}/rolling_update_recovery endpoint).

Tests

Tested against a 2-host XCP-ng 8.3 nested pool (3 running VMs, shared NFS SR), xo-server
built from this branch.

Scenario Result
Nominal RPU record running during the run with per-host steps advancing, then 404 + key gone from LevelDB + no banner
xo-server kill -9 during a host evacuation record flipped to interrupted at boot, interruptedAt = last known-alive timestamp, per-host progress preserved
Failed RPU failed + lastError, session id scrubbed, banner shown
Corrupted record both unknown schema version 99 and unreadable record reported as blocked, stored value left untouched on disk, boot not blocked
Non-admin user JSON-RPC not enough permissions, REST 403 with missingPrivileges: [{pool, rolling-update}]

Interrupted run : Patches tab

01-interrupted-full-page

The banner as an operator sees it, in context.

02-interrupted-banner

Close-up: interrupted by an xo-server restart, with the per-host progress.

Failed run

03-failed-banner

CANNOT_EVACUATE_HOST(VM_REQUIRES_SR, ...) surfaced from lastError. The failure happens in
the pre-checks, before the host order is known, so the banner degrades to the error alone.

Blocked record

04-blocked-banner

Record corrupted on purpose in LevelDB: reported as blocked, raw value kept as evidence.

Pinned VMs still halted

05-pinned-vms-banner

Not reproducible on nested hosts (needs VM_HAS_PCI_ATTACHED / VM_HAS_VGPU /
VM_HAS_SRIOV_VIF), so this one was rendered from a record seeded with haltedPinnedVms.

Checklist

  • Commit
    • Title follows commit conventions
    • Reference the relevant issue (Fixes #007, See xoa-support#42, See https://...)
    • If bug fix, add Introduced by
  • Changelog
    • If visible by XOA users, add changelog entry
    • Update "Packages to release" in CHANGELOG.unreleased.md
  • PR
    • If UI changes, add screenshots
    • If not finished or not tested, open as Draft

… restarts

A rolling pool update now writes a `rpuRecovery` record (LevelDB, one per
pool) before its first side effect and keeps it updated during the run:
operator options, update variant, host order, initial VM placement, per-host
step statuses, last filtered error, and the pinned VMs shut down for their
host's reboot (previously only tracked in memory, so an xo-server crash left
them halted with nothing to restart them). A record left in a live status at
xo-server boot is flipped to `interrupted`; a successful run deletes its
record; an unreadable or unknown-version record is reported as `blocked`
without touching the stored value.

The status is exposed through `pool.getRollingUpdateRecovery` and
`GET /rest/v0/pools/{id}/rolling_update_recovery` (filtered view: never the
raw intent, reusing the existing pool permissions), and the pool Patches tab
in XO5 shows a warning with per-host progress, the last error and the pinned
VMs still halted.

First card of the RPU resume-after-failure spike: the record only reports
for now, resuming an interrupted run comes separately.
@mpiton
mpiton marked this pull request as ready for review September 1, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant