Skip to content

Add retireConditions so bare metal remediation retires only for the chosen node conditions #2227

Description

@abdullah599

Summary

Bare metal remediation reboots an unhealthy host a few times, then either reuses the host (deletes the machine so the same server is provisioned again) or retires it (sets a permanent error so the server leaves the pool). Today reuse or retire is a single setting on the remediation template, applied the same way to every node condition. This issue adds a retireConditions list so the controller retires only when one of the listed node conditions triggered the remediation, and reuses the host for every other condition.

The Problem

A MachineHealthCheck (MHC) watches a group of machines for a set of node conditions. When a node matches one of them, CAPI creates a HetznerBareMetalRemediation and the controller reboots the host up to retryLimit times. If the node is still unhealthy after the last reboot, the controller applies the template's single onExhaustion value, either Reuse or Retire.

Whether to reuse or retire depends on the failure. The host should be reused for a software problem and retired for a dead disk. The problem is that one onExhaustion applies to every condition the MHC watches. That means on the same machines it is all reuse or all retire, never reuse for one failure and retire for another.

The Solution

The controller decides reuse or retire from the node condition that triggered the remediation, so one MHC can watch every condition and still reuse for some and retire for others.

The remediation strategy gets a new onExhaustion value RetireIfUnhealthyCondition and a new retireConditions list. A template using it looks like this:

strategy:
  type: Reboot
  retryLimit: 2
  timeout: 5m
  onExhaustion: RetireIfUnhealthyCondition  # new value
  retireConditions:  # new field, only used with the mode above
    - DisksFailure
    - CperHardwareErrorFatal

The MHC records which condition triggered the remediation in the Machine's HealthCheckSucceeded condition. With RetireIfUnhealthyCondition, once the reboots run out, the controller reads it and retires the host only if that condition is in retireConditions, otherwise it reuses the host. The existing Reuse and Retire modes stay the same, so current templates keep working.

A webhook rejects a strategy where the mode and the list disagree, so retireConditions is used only with RetireIfUnhealthyCondition:

  • Mode without conditions. RetireIfUnhealthyCondition with an empty retireConditions is rejected, because it would never retire.
  • Conditions without the mode. A non-empty retireConditions with Reuse, Retire, or unset is rejected, because the list would be ignored.

Expected Behavior

Scenario Behavior
Mode RetireIfUnhealthyCondition, triggering condition in retireConditions (e.g. DisksFailure) Host retired: permanent error, leaves the pool
Mode RetireIfUnhealthyCondition, triggering condition not in retireConditions (e.g. Ready) Host reused: machine deleted, server provisioned again
Mode RetireIfUnhealthyCondition, triggering condition cannot be read Host reused
Mode Retire Host retired for any condition (unchanged)
Mode Reuse or unset Host reused for any condition (unchanged)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions