Skip to content

fix(xo-server): auto reconnection infinite loop when pool is already connectd - #10355

Open
fbeauchamp wants to merge 5 commits into
masterfrom
fix_autoreconnect_loop
Open

fix(xo-server): auto reconnection infinite loop when pool is already connectd#10355
fbeauchamp wants to merge 5 commits into
masterfrom
fix_autoreconnect_loop

Conversation

@fbeauchamp

@fbeauchamp fbeauchamp commented Sep 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

review by commit

this PR fix multiple issues in the auto reconnection loop that can lead to an infinite reconnection loop.

  • don't rearm after a fatal error before connection setting changed ( does not affect the normal route on non fatal error like newtork issue)
  • serialize error correctly in database to ensure the "avoid a database write per auto-reconnect" corectly trigger
    *a lot of more cleanup to ensure that the connection is really dropped even if the host changed its id

introduced by #10016

journalctl logs are thousands of

Sep 04 04:28:35 xoa xo-server[16575]: 2026-09-04T08:28:35.033Z xo:xo-mixins:xen-servers INFO auto-reconnect stopped {
Sep 04 04:28:35 xoa xo-server[16575]:   serverId: '5c952216-c83e-419d-996a-2db3ae797841',
Sep 04 04:28:35 xoa xo-server[16575]:   outcome: 'permanent error'
Sep 04 04:28:35 xoa xo-server[16575]: }
Sep 04 04:28:35 xoa xo-server[16575]: 2026-09-04T08:28:35.092Z xo:xo-mixins:xen-servers WARN auto-reconnect aborted: permanent error {
Sep 04 04:28:35 xoa xo-server[16575]:   serverId: '5ba2c8f6-2ed4-418a-909e-b28a97c92913',
Sep 04 04:28:35 xoa xo-server[16575]:   error: PoolAlreadyConnected: this pool is already connected
Sep 04 04:28:35 xoa xo-server[16575]:       at XenServers._connectXenServer (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/xen-servers.mjs:473:15)
Sep 04 04:28:35 xoa xo-server[16575]:       at XenServers.connectXenServer (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/xen-servers.mjs:423:7)
Sep 04 04:28:35 xoa xo-server[16575]:       at autoReconnect (file:///usr/local/lib/node_modules/xo-server/src/_xenServerAutoReconnect.mjs:49:7) {
Sep 04 04:28:35 xoa xo-server[16575]:     poolId: 'cef643a2-5165-bd30-977a-844f8fb51b1f',
Sep 04 04:28:35 xoa xo-server[16575]:     connectedServerId: 'c236dd57-abe3-4e6c-85a0-ebcc950a5fc3',
Sep 04 04:28:35 xoa xo-server[16575]:     connectingServerId: '5ba2c8f6-2ed4-418a-909e-b28a97c92913'
Sep 04 04:28:35 xoa xo-server[16575]:   }
Sep 04 04:28:35 xoa xo-server[16575]: }
Sep 04 04:28:35 xoa xo-server[16575]: 2026-09-04T08:28:35.093Z xo:xo-mixins:xen-servers INFO auto-reconnect stopped {
Sep 04 04:28:35 xoa xo-server[16575]:   serverId: '5ba2c8f6-2ed4-418a-909e-b28a97c92913',
Sep 04 04:28:35 xoa xo-server[16575]:   outcome: 'permanent error'
Sep 04 04:28:35 xoa xo-server[16575]: }
Sep 04 04:28:35 xoa xo-server[16575]: 2026-09-04T08:28:35.525Z xo:xo-mixins:xen-servers WARN auto-reconnect aborted: permanent error {
Sep 04 04:28:35 xoa xo-server[16575]:   serverId: 'c4fa5380-d329-4fc2-9cee-f433f9d19ff7',
Sep 04 04:28:35 xoa xo-server[16575]:   error: PoolAlreadyConnected: this pool is already connected
Sep 04 04:28:35 xoa xo-server[16575]:       at XenServers._connectXenServer (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/xen-servers.mjs:473:15)
Sep 04 04:28:35 xoa xo-server[16575]:       at XenServers.connectXenServer (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/xen-servers.mjs:423:7)
Sep 04 04:28:35 xoa xo-server[16575]:       at autoReconnect (file:///usr/local/lib/node_modules/xo-server/src/_xenServerAutoReconnect.mjs:49:7) {
Sep 04 04:28:35 xoa xo-server[16575]:     poolId: '7fd3954f-016e-0104-6009-eca794cdbb3a',
Sep 04 04:28:35 xoa xo-server[16575]:     connectedServerId: '0c1b05ec-79ca-46d0-b846-9ea06784a3a6',
Sep 04 04:28:35 xoa xo-server[16575]:     connectingServerId: 'c4fa5380-d329-4fc2-9cee-f433f9d19ff7'
Sep 04 04:28:35 xoa xo-server[16575]:   }
Sep 04 04:28:35 xoa xo-server[16575]: }
Sep 04 04:28:35 xoa xo-server[16575]: 2026-09-04T08:28:35.525Z xo:xo-mixins:xen-servers INFO auto-reconnect stopped {
Sep 04 04:28:35 xoa xo-server[16575]:   serverId: 'c4fa5380-d329-4fc2-9cee-f433f9d19ff7',
Sep 04 04:28:35 xoa xo-server[16575]:   outcome: 'permanent error'
Sep 04 04:28:35 xoa xo-server[16575]: }
Sep 04 04:28:35 xoa xo-server[16575]: 2026-09-04T08:28:35.878Z xo:xo-mixins:xen-servers WARN auto-reconnect aborted: permanent error {
Sep 04 04:28:35 xoa xo-server[16575]:   serverId: 'cf2480ad-a500-4ee0-9a0d-99cee25d3dc5',
Sep 04 04:28:35 xoa xo-server[16575]:   error: PoolAlreadyConnected: this pool is already connected
Sep 04 04:28:35 xoa xo-server[16575]:       at XenServers._connectXenServer (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/xen-servers.mjs:473:15)
Sep 04 04:28:35 xoa xo-server[16575]:       at runNextTicks (node:internal/process/task_queues:64:5)
Sep 04 04:28:35 xoa xo-server[16575]:       at processImmediate (node:internal/timers:452:9)
Sep 04 04:28:35 xoa xo-server[16575]:       at process.callbackTrampoline (node:internal/async_hooks:130:17)
Sep 04 04:28:35 xoa xo-server[16575]:       at XenServers.connectXenServer (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/xen-servers.mjs:423:7)
Sep 04 04:28:35 xoa xo-server[16575]:       at autoReconnect (file:///usr/local/lib/node_modules/xo-server/src/_xenServerAutoReconnect.mjs:49:7) {
Sep 04 04:28:35 xoa xo-server[16575]:     poolId: '1a1be188-c746-8623-024e-90abf247cc2f',
Sep 04 04:28:35 xoa xo-server[16575]:     connectedServerId: '3a7d2d04-b026-424c-acf8-e436399f648d',
Sep 04 04:28:35 xoa xo-server[16575]:     connectingServerId: 'cf2480ad-a500-4ee0-9a0d-99cee25d3dc5'
Sep 04 04:28:35 xoa xo-server[16575]:   }
Sep 04 04:28:35 xoa xo-server[16575]: }

Checklist

  • Commit
    • Title follows commit conventions
    • Reference the relevant issue (Fixes #007, See xoa-support#42, See https://...)
    • If bug fix, add Introduced by
  • Changelog
    • If visible by XOA users, add changelog entry
    • Update "Packages to release" in CHANGELOG.unreleased.md
  • PR
    • If UI changes, add screenshots
    • If not finished or not tested, open as Draft

Review process

If you are an external contributor, you can skip this part. Simply create the pull request, and we'll get back to you as soon as possible.

This 2-passes review process aims to:

  • develop skills of junior reviewers
  • limit the workload for senior reviewers
  • limit the number of unnecessary changes by the author
  1. The author creates a PR.
  2. Review process:
    1. The author assigns the junior reviewer.
    2. The junior reviewer conducts their review:
      • Resolves their comments if they are addressed.
      • Adds comments if necessary or approves the PR.
    3. The junior reviewer assigns the senior reviewer.
    4. The senior reviewer conducts their review:
      • If there are no unresolved comments on the PR → merge.
      • Otherwise, we continue with 3.
  3. The author responds to comments and/or makes corrections, and we go back to 2.

Notes:

  1. The author can request a review at any time, even if the PR is still a Draft.
  2. In theory, there should not be more than one reviewer at a time.
  3. The author should not make any changes:
    • When a reviewer is assigned.
    • Between the junior and senior reviews.
  4. If the PR relates to a change in the openAPI specification, a member of the DevOps team must also participate in the review.

@fbeauchamp
fix(xo-server/autoreconnect): don't rearm autoreconnect after a fatal…
bc1eac0 
… error

for example a pool already connected like on ticket #63909
@fbeauchamp
fix(xo-server): pool already connected error losing its message
4de3932 
an `Error` cannot be stored as-is: the collection copies the record
 with `JSON.stringify` before the model gets a chance to serialize it,
 which silently drops its non-enumerable properties (`message`, `name`,
`stack`). Serializing here keeps the message readable by the user, and
 makes the stored error comparable with the next one, without which the
 check would write to the database on every single attempt.
@fbeauchamp
fix(xo-server): properly drop the right server on disconnection, even…
0ea7b41 
… if they have changed thei id
@fbeauchamp fbeauchamp changed the title Fix autoreconnect loop fix(xo-server): auto reconneciton infinite loop when pool is already connectd Sep 4, 2026
@fbeauchamp
fbeauchamp marked this pull request as ready for review September 4, 2026 08:43
@fbeauchamp
fbeauchamp requested a review from All-Ki September 4, 2026 08:43
@fbeauchamp fbeauchamp changed the title fix(xo-server): auto reconneciton infinite loop when pool is already connectd fix(xo-server): auto reconnection infinite loop when pool is already connectd Sep 4, 2026
All-Ki
All-Ki reviewed Sep 4, 2026
View reviewed changes
Comment thread packages/xo-server/src/xo-mixins/xen-servers.mjs
All-Ki
All-Ki reviewed Sep 4, 2026
View reviewed changes
Comment thread packages/xo-server/src/xo-mixins/_xen-servers.test.mjs
All-Ki
All-Ki reviewed Sep 4, 2026
View reviewed changes
Comment thread packages/xo-server/src/xo-mixins/xen-servers.mjs
@All-Ki
All-Ki requested a review from b-Nollet September 4, 2026 12:31
@fbeauchamp
fbeauchamp requested a review from All-Ki September 4, 2026 12:42
b-Nollet
b-Nollet reviewed Sep 4, 2026
View reviewed changes
Comment thread packages/xo-server/src/xo-mixins/xen-servers.mjs
xapi.xo.uninstall()

// switch server status from connected to connecting
delete serverIdsByPool[poolId]

@b-Nollet b-Nollet Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this also be replaced by _forgetXenServerPool?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@b-Nollet b-Nollet b-Nollet left review comments
@All-Ki All-Ki All-Ki approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@fbeauchamp @All-Ki @b-Nollet