Skip to content

Commit 75e251b

Browse files
committed
makes bootc-operator aware of karpenter
If a cluster contains both bootc-operator and Karpenter, it is possible that Karpenter could reap cluster nodes while a bootc-operator driven update is in progress. To prevent that, we detect whether the Node object is owned by a Karpenter NodeClaim object. And in that situation, we apply both the Karpenter Do Not Repair and Do Not Disrupt annotations to each node when it is cordoned for an update and we remove it after the update is completed. Signed-off-by: Zack Zlotnik <zzlotnik@redhat.com> Assisted-by: AI
1 parent 859e7bf commit 75e251b

5 files changed

Lines changed: 461 additions & 7 deletions

File tree

‎api/v1alpha1/constants.go‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,20 @@ const (
3030
// state for which the controller considered emitting an event. It is
3131
// observability bookkeeping only and is never used to drive reconciliation.
3232
AnnotationLastObservedState = "bootc.dev/last-observed-state"
33+
34+
// AnnotationOwnsKarpenterDoNotRepair is set on a BootcNode to record that
35+
// the controller added the karpenter.sh/do-not-repair annotation to the K8s
36+
// Node during slot assignment. When present, the controller removes the
37+
// karpenter.sh/do-not-repair annotation from the Node when the slot is freed.
38+
// It is not set when the annotation was already present before slot assignment
39+
// (externally set), so the controller never removes an annotation it did not add.
40+
AnnotationOwnsKarpenterDoNotRepair = "bootc.dev/owns-karpenter-do-not-repair"
41+
42+
// AnnotationOwnsKarpenterDoNotDisrupt is set on a BootcNode to record that
43+
// the controller added the karpenter.sh/do-not-disrupt annotation to the K8s
44+
// Node during slot assignment. When present, the controller removes the
45+
// karpenter.sh/do-not-disrupt annotation from the Node when the slot is freed.
46+
// It is not set when the annotation was already present before slot assignment
47+
// (externally set), so the controller never removes an annotation it did not add.
48+
AnnotationOwnsKarpenterDoNotDisrupt = "bootc.dev/owns-karpenter-do-not-disrupt"
3349
)

‎docs/src/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@
88
- [Helm Chart](helm.md)
99
- [Operations](operations/index.md)
1010
- [Managing a pool](operations/pool.md)
11+
- [Karpenter integration](operations/karpenter.md)

‎docs/src/operations/karpenter.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Karpenter integration
2+
3+
bootc-operator is aware of [Karpenter](https://karpenter.sh/) and coordinates
4+
with it during node updates to prevent unnecessary node replacement or
5+
voluntary disruption.
6+
7+
## Why this matters
8+
9+
Karpenter monitors nodes for health and may decide to replace or disrupt a
10+
node that appears unhealthy or underutilized — for example, one that is
11+
cordoned, being drained, or not yet Ready after a reboot. All of those
12+
conditions arise naturally during a bootc-operator update window, so without
13+
coordination Karpenter could race with the operator and terminate a node that
14+
is in the middle of a legitimate OS update.
15+
16+
Karpenter respects two node-level annotations:
17+
18+
- `karpenter.sh/do-not-repair: "true"` — prevents Node Auto Repair from
19+
replacing a node that has unhealthy conditions.
20+
- `karpenter.sh/do-not-disrupt: "true"` — prevents voluntary disruption
21+
actions (consolidation, drift) from selecting this node.
22+
23+
> **Note:** `karpenter.sh/do-not-repair` was introduced in
24+
> [kubernetes-sigs/karpenter#3311](https://github.com/kubernetes-sigs/karpenter/pull/3311)
25+
> and has not yet appeared in a Karpenter release. This doc will be updated
26+
> with a minimum required version once one is available.
27+
28+
bootc-operator sets both annotations automatically on Karpenter-managed nodes
29+
for the duration of the reboot slot.
30+
31+
## How it works
32+
33+
When the controller assigns a reboot slot to a node it checks whether the K8s
34+
Node object has an owner reference pointing to a Karpenter `NodeClaim`
35+
(`apiVersion: karpenter.sh/v1`, `kind: NodeClaim`). If it does, the
36+
controller takes the following steps:
37+
38+
1. For each annotation (`do-not-repair`, `do-not-disrupt`): if the annotation
39+
is **not** already present on the Node, the controller adds it (value
40+
`"true"`) and records ownership by setting the corresponding tracking
41+
annotation on the `BootcNode` object:
42+
- `bootc.dev/owns-karpenter-do-not-repair: "true"`
43+
- `bootc.dev/owns-karpenter-do-not-disrupt: "true"`
44+
2. The node proceeds through the normal update flow: drain, reboot, image
45+
verification.
46+
3. When the slot is freed (node is Ready and running the new image), the
47+
controller checks the tracking annotations on the `BootcNode`. For each
48+
annotation it owns, it removes the corresponding annotation from the Node
49+
and clears the tracking annotation from the `BootcNode`. Both removals are
50+
issued as a single Node patch.
51+
52+
## Pre-existing annotations
53+
54+
If either Karpenter annotation is already on the Node before slot assignment —
55+
set by an external actor, another controller, or a manual operation —
56+
bootc-operator leaves it completely alone:
57+
58+
- It does **not** overwrite or change the annotation on assignment.
59+
- It does **not** set the corresponding tracking annotation on the `BootcNode`.
60+
- It does **not** remove the annotation when the slot is freed.
61+
62+
The two annotations are tracked independently. If only one is pre-existing, the
63+
operator owns (and later removes) only the other.
64+
65+
This ensures bootc-operator never removes an annotation it did not add.
66+
67+
## Limitations
68+
69+
`karpenter.sh/do-not-disrupt` only blocks voluntary disruption methods
70+
(consolidation and drift). It does not protect a node against forceful
71+
termination triggered by expiration (`expireAfter`), interruption (spot
72+
instance reclaim), Node Auto Repair, or manual deletion. If any of those
73+
events occur while a node is in a bootc-operator reboot slot, Karpenter may
74+
still terminate it.
75+
76+
`karpenter.sh/do-not-repair` similarly has no effect on voluntary disruption
77+
or expiration — it only suppresses Node Auto Repair.
78+
79+
If a node is terminated by expiration, interruption, or any other forceful
80+
method while a bootc-operator update is in progress, the termination is caused
81+
by Karpenter acting on its own policy — not by the bootc-operator update. The
82+
update simply happened to be in flight at the time.
83+
84+
Refer to the [Karpenter disruption documentation](https://karpenter.sh/docs/concepts/disruption/)
85+
for a full breakdown of which controls apply to which disruption methods.
86+
87+
## Non-Karpenter nodes
88+
89+
Nodes without a `NodeClaim` owner reference are unaffected. The feature
90+
activates only when the owner reference check matches.
91+
92+
## Annotation reference
93+
94+
| Annotation | Object | Managed by | Purpose |
95+
|---|---|---|---|
96+
| `karpenter.sh/do-not-repair` | K8s Node | bootc-operator (or external) | Prevents Karpenter Node Auto Repair from replacing this node during an update |
97+
| `karpenter.sh/do-not-disrupt` | K8s Node | bootc-operator (or external) | Prevents Karpenter voluntary disruption (consolidation, drift) during an update |
98+
| `bootc.dev/owns-karpenter-do-not-repair` | BootcNode | bootc-operator | Tracks that the operator added `do-not-repair`; cleared when the annotation is removed on slot free |
99+
| `bootc.dev/owns-karpenter-do-not-disrupt` | BootcNode | bootc-operator | Tracks that the operator added `do-not-disrupt`; cleared when the annotation is removed on slot free |

‎internal/controller/rollout.go‎

Lines changed: 92 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,18 @@ import (
3434
// https://github.com/bootc-dev/bootc-operator/issues/99.
3535
const unhealthySlotHaltThreshold = 2
3636

37+
// Karpenter integration constants. The controller checks for a NodeClaim owner
38+
// reference and manages karpenter.sh/do-not-repair and
39+
// karpenter.sh/do-not-disrupt to prevent Karpenter from replacing or
40+
// voluntarily disrupting a node while it is being updated.
41+
// See docs/src/operations/karpenter.md.
42+
const (
43+
karpenterDoNotRepairAnnotationKey = "karpenter.sh/do-not-repair"
44+
karpenterDoNotDisruptAnnotationKey = "karpenter.sh/do-not-disrupt"
45+
karpenterNodeClaimAPIVersion = "karpenter.sh/v1"
46+
karpenterNodeClaimKind = "NodeClaim"
47+
)
48+
3749
// rolloutState holds the classified BootcNodes for a single reconcile
3850
// pass.
3951
type rolloutState struct {
@@ -188,6 +200,17 @@ func (r *BootcNodePoolReconciler) driveRollout(
188200
return rs, nil
189201
}
190202

203+
// nodeIsOwnedByKarpenterNodeClaim reports whether the K8s Node has an owner
204+
// reference pointing to a Karpenter NodeClaim (karpenter.sh/v1).
205+
func nodeIsOwnedByKarpenterNodeClaim(node *corev1.Node) bool {
206+
for _, ref := range node.OwnerReferences {
207+
if ref.APIVersion == karpenterNodeClaimAPIVersion && ref.Kind == karpenterNodeClaimKind {
208+
return true
209+
}
210+
}
211+
return false
212+
}
213+
191214
// assignRebootSlot marks a BootcNode as occupying a reboot slot and
192215
// cordons the corresponding K8s Node. It sets the in-reboot-slot
193216
// annotation on the BootcNode, records prior cordon state in the
@@ -200,7 +223,16 @@ func (r *BootcNodePoolReconciler) assignRebootSlot(
200223
) error {
201224
log := logf.FromContext(ctx)
202225

203-
// Set annotations on the BootcNode if not already present.
226+
// Determine Karpenter intent once so both patch blocks stay consistent.
227+
// shouldOwn* is true when the node is Karpenter-managed and the respective
228+
// annotation is not already present (set by an external actor). In that
229+
// case, the controller adds the annotation and records ownership so it can
230+
// be cleaned up on slot free.
231+
karpenterOwned := nodeIsOwnedByKarpenterNodeClaim(node)
232+
shouldOwnDoNotRepair := karpenterOwned && !metav1.HasAnnotation(node.ObjectMeta, karpenterDoNotRepairAnnotationKey)
233+
shouldOwnDoNotDisrupt := karpenterOwned && !metav1.HasAnnotation(node.ObjectMeta, karpenterDoNotDisruptAnnotationKey)
234+
235+
// Patch the BootcNode with slot annotations if not already present.
204236
if !metav1.HasAnnotation(bn.ObjectMeta, bootcv1alpha1.AnnotationInRebootSlot) {
205237
log.Info("Assigning reboot slot", "node", bn.Name)
206238
modifiedBN := bn.DeepCopy()
@@ -214,19 +246,44 @@ func (r *BootcNodePoolReconciler) assignRebootSlot(
214246
} else {
215247
modifiedBN.Annotations[bootcv1alpha1.AnnotationWasCordoned] = "false"
216248
}
249+
// Track Karpenter annotation ownership so freeRebootSlot knows
250+
// whether to remove the annotations from the Node.
251+
if shouldOwnDoNotRepair {
252+
modifiedBN.Annotations[bootcv1alpha1.AnnotationOwnsKarpenterDoNotRepair] = "true"
253+
}
254+
if shouldOwnDoNotDisrupt {
255+
modifiedBN.Annotations[bootcv1alpha1.AnnotationOwnsKarpenterDoNotDisrupt] = "true"
256+
}
217257
if err := r.Patch(ctx, modifiedBN, client.MergeFrom(bn)); err != nil {
218258
return fmt.Errorf("annotating BootcNode: %w", err)
219259
}
220260
*bn = *modifiedBN
221261
}
222262

223-
// Cordon the K8s Node if not already cordoned.
224-
if !node.Spec.Unschedulable {
225-
log.Info("Cordoning node", "node", node.Name)
263+
// Patch the K8s Node: cordon it and, when we own the Karpenter
264+
// annotations, set them. All are idempotent; we combine them into a
265+
// single patch to minimise API calls.
266+
if !node.Spec.Unschedulable || shouldOwnDoNotRepair || shouldOwnDoNotDisrupt {
226267
modifiedNode := node.DeepCopy()
227-
modifiedNode.Spec.Unschedulable = true
268+
if !node.Spec.Unschedulable {
269+
log.Info("Cordoning node", "node", node.Name)
270+
modifiedNode.Spec.Unschedulable = true
271+
}
272+
if shouldOwnDoNotRepair || shouldOwnDoNotDisrupt {
273+
if modifiedNode.Annotations == nil {
274+
modifiedNode.Annotations = map[string]string{}
275+
}
276+
if shouldOwnDoNotRepair {
277+
log.Info("Setting karpenter.sh/do-not-repair on node", "node", node.Name)
278+
modifiedNode.Annotations[karpenterDoNotRepairAnnotationKey] = "true"
279+
}
280+
if shouldOwnDoNotDisrupt {
281+
log.Info("Setting karpenter.sh/do-not-disrupt on node", "node", node.Name)
282+
modifiedNode.Annotations[karpenterDoNotDisruptAnnotationKey] = "true"
283+
}
284+
}
228285
if err := r.Patch(ctx, modifiedNode, client.StrategicMergeFrom(node)); err != nil {
229-
return fmt.Errorf("cordoning node: %w", err)
286+
return fmt.Errorf("patching node: %w", err)
230287
}
231288
*node = *modifiedNode
232289
}
@@ -272,12 +329,40 @@ func (r *BootcNodePoolReconciler) freeRebootSlot(
272329
bn *bootcv1alpha1.BootcNode,
273330
node *corev1.Node,
274331
) error {
332+
log := logf.FromContext(ctx)
333+
334+
// Remove karpenter.sh/do-not-repair and karpenter.sh/do-not-disrupt from
335+
// the Node if the controller added them. We check the tracking annotations
336+
// on the BootcNode rather than inspecting the Node directly, so an external
337+
// actor who later removes the annotations doesn't cause spurious patches.
338+
// Both removals are combined into a single Node patch when needed.
339+
ownsRepair := metav1.HasAnnotation(bn.ObjectMeta, bootcv1alpha1.AnnotationOwnsKarpenterDoNotRepair)
340+
ownsDisrupt := metav1.HasAnnotation(bn.ObjectMeta, bootcv1alpha1.AnnotationOwnsKarpenterDoNotDisrupt)
341+
if ownsRepair || ownsDisrupt {
342+
modifiedNode := node.DeepCopy()
343+
if ownsRepair {
344+
log.Info("Removing karpenter.sh/do-not-repair from node", "node", node.Name)
345+
delete(modifiedNode.Annotations, karpenterDoNotRepairAnnotationKey)
346+
}
347+
if ownsDisrupt {
348+
log.Info("Removing karpenter.sh/do-not-disrupt from node", "node", node.Name)
349+
delete(modifiedNode.Annotations, karpenterDoNotDisruptAnnotationKey)
350+
}
351+
if err := r.Patch(ctx, modifiedNode, client.StrategicMergeFrom(node)); err != nil {
352+
return fmt.Errorf("removing karpenter annotations from node: %w", err)
353+
}
354+
*node = *modifiedNode
355+
}
356+
275357
if err := r.restoreCordonState(ctx, bn, node); err != nil {
276358
return err
277359
}
278360

279-
// Remove both reboot slot annotations from the BootcNode.
361+
// Remove all reboot slot annotations from the BootcNode, including the
362+
// Karpenter ownership trackers if present.
280363
modified := bn.DeepCopy()
364+
delete(modified.Annotations, bootcv1alpha1.AnnotationOwnsKarpenterDoNotRepair)
365+
delete(modified.Annotations, bootcv1alpha1.AnnotationOwnsKarpenterDoNotDisrupt)
281366
delete(modified.Annotations, bootcv1alpha1.AnnotationWasCordoned)
282367
delete(modified.Annotations, bootcv1alpha1.AnnotationInRebootSlot)
283368
if err := r.Patch(ctx, modified, client.MergeFrom(bn)); err != nil {

0 commit comments

Comments
 (0)