Summary
Bare metal remediation reboots an unhealthy host a few times, then either reuses the host (deletes the machine so the same server is provisioned again) or retires it (sets a permanent error so the server leaves the pool). Today reuse or retire is a single setting on the remediation template, applied the same way to every node condition. This issue adds a retireConditions list so the controller retires only when one of the listed node conditions triggered the remediation, and reuses the host for every other condition.
The Problem
A MachineHealthCheck (MHC) watches a group of machines for a set of node conditions. When a node matches one of them, CAPI creates a HetznerBareMetalRemediation and the controller reboots the host up to retryLimit times. If the node is still unhealthy after the last reboot, the controller applies the template's single onExhaustion value, either Reuse or Retire.
Whether to reuse or retire depends on the failure. The host should be reused for a software problem and retired for a dead disk. The problem is that one onExhaustion applies to every condition the MHC watches. That means on the same machines it is all reuse or all retire, never reuse for one failure and retire for another.
The Solution
The controller decides reuse or retire from the node condition that triggered the remediation, so one MHC can watch every condition and still reuse for some and retire for others.
The remediation strategy gets a new onExhaustion value RetireIfUnhealthyCondition and a new retireConditions list. A template using it looks like this:
strategy:
type: Reboot
retryLimit: 2
timeout: 5m
onExhaustion: RetireIfUnhealthyCondition # new value
retireConditions: # new field, only used with the mode above
- DisksFailure
- CperHardwareErrorFatal
The MHC records which condition triggered the remediation in the Machine's HealthCheckSucceeded condition. With RetireIfUnhealthyCondition, once the reboots run out, the controller reads it and retires the host only if that condition is in retireConditions, otherwise it reuses the host. The existing Reuse and Retire modes stay the same, so current templates keep working.
A webhook rejects a strategy where the mode and the list disagree, so retireConditions is used only with RetireIfUnhealthyCondition:
- Mode without conditions.
RetireIfUnhealthyCondition with an empty retireConditions is rejected, because it would never retire.
- Conditions without the mode. A non-empty
retireConditions with Reuse, Retire, or unset is rejected, because the list would be ignored.
Expected Behavior
| Scenario |
Behavior |
Mode RetireIfUnhealthyCondition, triggering condition in retireConditions (e.g. DisksFailure) |
Host retired: permanent error, leaves the pool |
Mode RetireIfUnhealthyCondition, triggering condition not in retireConditions (e.g. Ready) |
Host reused: machine deleted, server provisioned again |
Mode RetireIfUnhealthyCondition, triggering condition cannot be read |
Host reused |
Mode Retire |
Host retired for any condition (unchanged) |
Mode Reuse or unset |
Host reused for any condition (unchanged) |
Summary
Bare metal remediation reboots an unhealthy host a few times, then either reuses the host (deletes the machine so the same server is provisioned again) or retires it (sets a permanent error so the server leaves the pool). Today reuse or retire is a single setting on the remediation template, applied the same way to every node condition. This issue adds a
retireConditionslist so the controller retires only when one of the listed node conditions triggered the remediation, and reuses the host for every other condition.The Problem
A
MachineHealthCheck(MHC) watches a group of machines for a set of node conditions. When a node matches one of them, CAPI creates aHetznerBareMetalRemediationand the controller reboots the host up toretryLimittimes. If the node is still unhealthy after the last reboot, the controller applies the template's singleonExhaustionvalue, eitherReuseorRetire.Whether to reuse or retire depends on the failure. The host should be reused for a software problem and retired for a dead disk. The problem is that one
onExhaustionapplies to every condition the MHC watches. That means on the same machines it is all reuse or all retire, never reuse for one failure and retire for another.The Solution
The controller decides reuse or retire from the node condition that triggered the remediation, so one MHC can watch every condition and still reuse for some and retire for others.
The remediation strategy gets a new
onExhaustionvalueRetireIfUnhealthyConditionand a newretireConditionslist. A template using it looks like this:The MHC records which condition triggered the remediation in the Machine's
HealthCheckSucceededcondition. WithRetireIfUnhealthyCondition, once the reboots run out, the controller reads it and retires the host only if that condition is inretireConditions, otherwise it reuses the host. The existingReuseandRetiremodes stay the same, so current templates keep working.A webhook rejects a strategy where the mode and the list disagree, so
retireConditionsis used only withRetireIfUnhealthyCondition:RetireIfUnhealthyConditionwith an emptyretireConditionsis rejected, because it would never retire.retireConditionswithReuse,Retire, or unset is rejected, because the list would be ignored.Expected Behavior
RetireIfUnhealthyCondition, triggering condition inretireConditions(e.g.DisksFailure)RetireIfUnhealthyCondition, triggering condition not inretireConditions(e.g.Ready)RetireIfUnhealthyCondition, triggering condition cannot be readRetireReuseor unset