feat(xo-server,rest-api,xo-web/rpu): persist a recovery record across restarts - #10331
Open
mpiton wants to merge 2 commits into
Open
feat(xo-server,rest-api,xo-web/rpu): persist a recovery record across restarts#10331mpiton wants to merge 2 commits into
mpiton wants to merge 2 commits into
Conversation
… restarts
A rolling pool update now writes a `rpuRecovery` record (LevelDB, one per
pool) before its first side effect and keeps it updated during the run:
operator options, update variant, host order, initial VM placement, per-host
step statuses, last filtered error, and the pinned VMs shut down for their
host's reboot (previously only tracked in memory, so an xo-server crash left
them halted with nothing to restart them). A record left in a live status at
xo-server boot is flipped to `interrupted`; a successful run deletes its
record; an unreadable or unknown-version record is reported as `blocked`
without touching the stored value.
The status is exposed through `pool.getRollingUpdateRecovery` and
`GET /rest/v0/pools/{id}/rolling_update_recovery` (filtered view: never the
raw intent, reusing the existing pool permissions), and the pool Patches tab
in XO5 shows a warning with per-host progress, the last error and the pinned
VMs still halted.
First card of the RPU resume-after-failure spike: the record only reports
for now, resuming an interrupted run comes separately.
mpiton
marked this pull request as ready for review
September 1, 2026 13:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
An xo-server restart in the middle of a rolling pool update currently loses everything: which hosts were done, which VMs were shut down because they cannot be migrated (PCI passthrough, vGPU, SR-IOV VIF), what the operator had asked for. The pinned VMs are the worst part: they only live in an in-memory map restored through
$defer, so after a crash they stay halted with nothing left to restart them.This PR makes xo-server persist a
rpuRecoveryrecord (LevelDB, one per pool) before the first side effect of a rolling pool update, and keep it updated during the run:A record still in a live status at xo-server boot is flipped to
interrupted. A successful run deletes its record. An unreadable record or an unknown schema version is reported asblockedwithout touching the stored value.The status is exposed through
pool.getRollingUpdateRecoveryandGET /rest/v0/pools/{id}/rolling_update_recovery(filtered view, never the raw intent, reusing the poolrolling-updateACL), and the pool Patches tab shows a warning banner with per-host progress, the last error and the pinned VMs still halted.This is the first card of the RPU resume-after-failure work: the record only reports for now, resuming an interrupted run comes separately.
Note for DevOps: this updates the generated OpenAPI spec (new
GET /pools/{id}/rolling_update_recoveryendpoint).Tests
Tested against a 2-host XCP-ng 8.3 nested pool (3 running VMs, shared NFS SR), xo-server
built from this branch.
runningduring the run with per-host steps advancing, then404+ key gone from LevelDB + no bannerkill -9during a host evacuationinterruptedat boot,interruptedAt= last known-alive timestamp, per-host progress preservedfailed+lastError, session id scrubbed, banner shownunknown schema version 99andunreadable recordreported asblocked, stored value left untouched on disk, boot not blockednot enough permissions, REST403withmissingPrivileges: [{pool, rolling-update}]Interrupted run : Patches tab
The banner as an operator sees it, in context.
Close-up: interrupted by an xo-server restart, with the per-host progress.
Failed run
CANNOT_EVACUATE_HOST(VM_REQUIRES_SR, ...)surfaced fromlastError. The failure happens inthe pre-checks, before the host order is known, so the banner degrades to the error alone.
Blocked record
Record corrupted on purpose in LevelDB: reported as blocked, raw value kept as evidence.
Pinned VMs still halted
Not reproducible on nested hosts (needs
VM_HAS_PCI_ATTACHED/VM_HAS_VGPU/VM_HAS_SRIOV_VIF), so this one was rendered from a record seeded withhaltedPinnedVms.Checklist
Fixes #007,See xoa-support#42,See https://...)Introduced byCHANGELOG.unreleased.md