Skip to content

Commit e2c1dc7

Browse files
committed
rollout: coordinate bootc updates with Karpenter
Protect Karpenter-managed nodes from repair and voluntary disruption throughout image staging, reboot, and health verification, including their NodeClaims when available. Detect active Karpenter disruptions and reboots while deferring node drains and/or reboots until completion while preserving externally managed annotations and cordon state. Assisted-by: AI Signed-off-by: Zack Zlotnik <zzlotnik@redhat.com>
1 parent 7304f7b commit e2c1dc7

9 files changed

Lines changed: 2759 additions & 42 deletions

File tree

‎api/v1alpha1/constants.go‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,28 @@ const (
3030
// state for which the controller considered emitting an event. It is
3131
// observability bookkeeping only and is never used to drive reconciliation.
3232
AnnotationLastObservedState = "bootc.dev/last-observed-state"
33+
34+
// AnnotationOwnsKarpenterDoNotRepair is set on a BootcNode to record that
35+
// the controller added the karpenter.sh/do-not-repair annotation to the K8s
36+
// Node during slot assignment. When present, the controller removes the
37+
// karpenter.sh/do-not-repair annotation from the Node when the slot is freed.
38+
// It is not set when the annotation was already present before slot assignment
39+
// (externally set), so the controller never removes an annotation it did not add.
40+
AnnotationOwnsKarpenterDoNotRepair = "bootc.dev/owns-karpenter-do-not-repair"
41+
42+
// AnnotationOwnsKarpenterDoNotDisrupt is set on a BootcNode to record that
43+
// the controller added the karpenter.sh/do-not-disrupt annotation to the K8s
44+
// Node during slot assignment. When present, the controller removes the
45+
// karpenter.sh/do-not-disrupt annotation from the Node when the slot is freed.
46+
// It is not set when the annotation was already present before slot assignment
47+
// (externally set), so the controller never removes an annotation it did not add.
48+
AnnotationOwnsKarpenterDoNotDisrupt = "bootc.dev/owns-karpenter-do-not-disrupt"
49+
50+
// AnnotationOwnsKarpenterNodeClaimDoNotRepair records that the controller
51+
// added do-not-repair to the associated Karpenter NodeClaim.
52+
AnnotationOwnsKarpenterNodeClaimDoNotRepair = "bootc.dev/owns-karpenter-nodeclaim-do-not-repair"
53+
54+
// AnnotationOwnsKarpenterNodeClaimDoNotDisrupt records that the controller
55+
// added do-not-disrupt to the associated Karpenter NodeClaim.
56+
AnnotationOwnsKarpenterNodeClaimDoNotDisrupt = "bootc.dev/owns-karpenter-nodeclaim-do-not-disrupt"
3357
)

‎config/rbac/role.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,15 @@ rules:
4949
- daemonsets
5050
verbs:
5151
- get
52+
- apiGroups:
53+
- karpenter.sh
54+
resources:
55+
- nodeclaims
56+
verbs:
57+
- get
58+
- list
59+
- patch
60+
- watch
5261
- apiGroups:
5362
- node.bootc.dev
5463
resources:

‎docs/src/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@
88
- [Helm Chart](helm.md)
99
- [Operations](operations/index.md)
1010
- [Managing a pool](operations/pool.md)
11+
- [Karpenter integration](operations/karpenter.md)

‎docs/src/operations/karpenter.md‎

Lines changed: 258 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,258 @@
1+
# Karpenter integration
2+
3+
bootc-operator is aware of [Karpenter](https://karpenter.sh/) and coordinates
4+
with it during node updates to prevent unnecessary node replacement,
5+
voluntary disruption, and concurrent reboot handling.
6+
7+
## Why this matters
8+
9+
Karpenter monitors nodes for health and may decide to replace or disrupt a
10+
node that appears unhealthy or underutilized — for example, one that is
11+
cordoned, being drained, or not yet Ready after a reboot. All of those
12+
conditions arise naturally during a bootc-operator update window, so without
13+
coordination Karpenter could race with the operator and terminate a node that
14+
is in the middle of a legitimate OS update.
15+
16+
Karpenter respects two node-level annotations:
17+
18+
- `karpenter.sh/do-not-repair: "true"` — prevents Node Auto Repair from
19+
replacing a node that has unhealthy conditions.
20+
- `karpenter.sh/do-not-disrupt: "true"` — prevents voluntary disruption
21+
actions (consolidation, drift) from selecting this node.
22+
23+
> **Note:** `karpenter.sh/do-not-repair` was introduced in Karpenter v1.15.0.
24+
> That is the minimum required version for full bootc-operator Karpenter integration.
25+
26+
bootc-operator sets both annotations automatically on Karpenter-managed nodes
27+
for the duration of the full update lifecycle — from image staging through
28+
node reboot and health verification. Where the associated `NodeClaim` exists,
29+
it also sets the annotations there. This is best-effort coordination, not a
30+
distributed lock.
31+
32+
## How it works
33+
34+
When a Karpenter-owned node enters the `Staging` state (the daemon is pulling
35+
the new image), bootc-operator immediately sets `do-not-repair` and
36+
`do-not-disrupt` on the Node and, where available, the associated NodeClaim.
37+
This protection persists through slot assignment, drain, reboot, and post-boot
38+
health verification.
39+
40+
Before assigning a reboot slot or starting a drain, the controller checks
41+
whether Karpenter has already committed any lifecycle operation that would
42+
conflict with the bootc update. It checks:
43+
44+
- Node taints `karpenter.sh/rebooting:NoSchedule` and
45+
`karpenter.sh/disrupted:NoSchedule`
46+
- Node `DeletionTimestamp` (node is being terminated)
47+
- NodeClaim `DeletionTimestamp` (NodeClaim is being deleted)
48+
- NodeClaim `status.conditions`: `Rebooting=True`, `DisruptionReason=True`,
49+
or `InstanceTerminating=True`
50+
51+
If any of these signals is active, bootc-operator defers and retries. It
52+
never removes a Karpenter taint.
53+
54+
If the Karpenter NodeClaim owner reference is present but the NodeClaim object
55+
returns a 404, bootc-operator treats this as unsafe (the NodeClaim may be
56+
mid-deletion) and defers with a short requeue rather than proceeding.
57+
58+
If none of those signals is active, the controller takes the following steps:
59+
60+
1. For each annotation (`do-not-repair`, `do-not-disrupt`): if the annotation
61+
is **not** already set to `"true"`, the controller sets it (value `"true"`)
62+
on the Node and, where available, the NodeClaim. It records ownership
63+
separately on the `BootcNode` object:
64+
- `bootc.dev/owns-karpenter-do-not-repair: "true"`
65+
- `bootc.dev/owns-karpenter-do-not-disrupt: "true"`
66+
- `bootc.dev/owns-karpenter-nodeclaim-do-not-repair: "true"`
67+
- `bootc.dev/owns-karpenter-nodeclaim-do-not-disrupt: "true"`
68+
2. The controller proceeds through the normal update flow: drain, reboot, and
69+
image verification.
70+
3. When the slot is freed (node is Ready and running the new image), the
71+
controller removes only annotations that it still owns and whose value is
72+
still `"true"`. It never removes annotations it did not add.
73+
74+
## NodeClaim annotations are supplemental
75+
76+
NodeClaim annotations (`do-not-repair`, `do-not-disrupt`) are a supplemental
77+
signal, not a distributed lock. By the time bootc-operator writes them,
78+
Karpenter may already have selected or queued a disruption for the registered
79+
Node. The authoritative disruption checks are the Node-level taint, the
80+
NodeClaim deletion timestamp, and the NodeClaim status conditions — the
81+
controller checks these before both slot assignment and drain.
82+
83+
## NodeClaim watch registration
84+
85+
When the controller starts, it checks whether the Karpenter `NodeClaim` API is
86+
available. If it is, the controller registers a watch for NodeClaim events and
87+
uses changes to the associated NodeClaim to requeue its owning
88+
`BootcNodePool`. This allows changes such as Karpenter starting or completing
89+
a reboot to be observed promptly.
90+
91+
The watch is registered only during startup. If Karpenter is installed after
92+
bootc-operator has started, the operator does not add the watch dynamically;
93+
restart the bootc-operator controller after installing Karpenter. As a fallback,
94+
the controller schedules a periodic requeue (every 30 seconds) whenever
95+
Karpenter-owned nodes are active, so that missed NodeClaim events do not
96+
stall progress indefinitely.
97+
98+
The controller represents NodeClaims with Kubernetes `Unstructured` objects
99+
instead of importing Karpenter's Go API types. This keeps Karpenter an optional
100+
integration and avoids adding a Karpenter API dependency or scheme registration
101+
to bootc-operator. It also lets clusters without the NodeClaim CRD continue to
102+
run normally. The tradeoff is that the small set of fields used here, such as
103+
`status.nodeName` and `status.conditions`, is checked at runtime rather than
104+
being enforced by Go types.
105+
106+
## Pre-existing annotations
107+
108+
If a Karpenter annotation is already present on the Node with a value of
109+
`"true"` before protection is applied — set by an external actor, another
110+
controller, or a manual operation — bootc-operator leaves it alone:
111+
112+
- It does **not** set the corresponding tracking annotation on the `BootcNode`.
113+
- It does **not** remove the annotation when the slot is freed.
114+
115+
If the annotation is present but set to any value other than `"true"` (for
116+
example `"false"`), bootc-operator overwrites it with `"true"` and records
117+
ownership. This is required because Karpenter only honors the literal value
118+
`"true"`; any other value leaves the node unprotected.
119+
120+
The two annotations are tracked independently. If only one is already set to
121+
`"true"`, the operator owns (and later removes) only the other.
122+
123+
## Temporary overprovisioning during node drains
124+
125+
During a bootc-operator update, the following sequence can cause temporary
126+
overprovisioning:
127+
128+
1. bootc-operator evicts workloads from the node being updated.
129+
2. The evicted workloads become pending.
130+
3. Karpenter provisions new capacity to satisfy the pending pods.
131+
4. The original node reboots into the new image and rejoins the cluster.
132+
5. The newly provisioned capacity may become excess and be consolidated later.
133+
134+
This is expected behavior. The `do-not-disrupt` annotation suppresses
135+
Karpenter's voluntary disruption of the node being updated, but it does not
136+
suppress Karpenter provisioning new capacity for pending workloads. Karpenter
137+
provisioning is driven by pending pod demand rather than disruption eligibility.
138+
139+
## Post-rollout consolidation and administrator policy
140+
141+
bootc-operator's protection is intentionally limited to the update lifecycle.
142+
When a node is Ready and the new image has been verified, the operator frees its
143+
reboot slot and removes only the Karpenter annotations that it added. If the
144+
workload has moved to newly provisioned capacity and the original node is now
145+
empty, Karpenter may then consolidate that node with an `Empty` disruption. This
146+
is normal Karpenter behavior after the bootc update has completed; it is not an
147+
OS update failure.
148+
149+
Cluster administrators who need a node to remain in service after the rollout
150+
must configure that steady-state policy in Karpenter. bootc-operator should not
151+
permanently disable consolidation by default, because doing so can leave unused
152+
capacity running indefinitely.
153+
154+
This is an intentional separation of responsibilities. bootc-operator protects
155+
only the Karpenter-owned nodes that are participating in its update lifecycle;
156+
it does not apply protection to every node in the `BootcNodePool`.
157+
158+
| Responsibility | bootc-operator | Cluster administrator / Karpenter |
159+
|---|---|---|
160+
| Protect a node during image staging, drain, reboot, and health verification | Applies temporary node and NodeClaim annotations and checks for conflicting Karpenter operations | — |
161+
| Provision capacity for pending workloads | — | Karpenter provisions nodes according to its NodePool and workload requirements |
162+
| Consolidate or drift unrelated nodes | — | Karpenter manages nodes according to the configured disruption policy |
163+
| Retain a node after the rollout becomes complete | Removes only annotations it owns; does not freeze unrelated nodes | Configure `do-not-disrupt`, disruption budgets, or `consolidateAfter` as needed |
164+
| Protect an entire Karpenter pool during maintenance | — | Configure an explicit Karpenter-wide maintenance policy |
165+
| Handle expiration, interruption, manual deletion, or other forceful termination | Detects some conflicts and defers its own update when possible | Configure and operate the applicable Karpenter and infrastructure policies |
166+
167+
If an entire pool must be held stable, configure that as an explicit Karpenter
168+
maintenance policy rather than relying on bootc-operator to freeze unrelated
169+
nodes.
170+
171+
Options include:
172+
173+
1. **Keep a node protected explicitly.** Set
174+
`karpenter.sh/do-not-disrupt: "true"` on the Node and its NodeClaim. An
175+
annotation that was already present before bootc-operator starts protecting
176+
the node is treated as externally owned and is not removed when the rollout
177+
completes. This blocks voluntary disruption such as consolidation and drift,
178+
but not expiration, interruption, manual deletion, or other forceful
179+
termination.
180+
2. **Temporarily block disruption for the whole Karpenter NodePool.** During a
181+
rollout, an administrator can set a zero disruption budget:
182+
183+
```yaml
184+
spec:
185+
disruption:
186+
budgets:
187+
- nodes: "0"
188+
```
189+
190+
Restore the normal budget after the rollout. This protects all nodes in the
191+
pool, not just the node being updated.
192+
3. **Delay consolidation.** Configure a suitable `consolidateAfter` interval so
193+
that a rebooted node has time to become schedulable and receive workloads
194+
before Karpenter considers it empty. This reduces the race but does not
195+
guarantee that the node will be retained.
196+
197+
All three options are administrator-owned policy. The first is appropriate for
198+
selected nodes that must remain stable; the second is appropriate when an
199+
entire pool must be held stable during a controlled maintenance window; and the
200+
third is appropriate when consolidation should remain enabled but be delayed.
201+
202+
## Limitations
203+
204+
`karpenter.sh/do-not-disrupt` only blocks voluntary disruption methods
205+
(consolidation and drift). It does not protect a node against forceful
206+
termination triggered by expiration (`expireAfter`), interruption (spot
207+
instance reclaim), Node Auto Repair, or manual deletion. If any of those
208+
events occur while a node is in a bootc-operator reboot slot, Karpenter may
209+
still terminate it.
210+
211+
`karpenter.sh/do-not-repair` similarly has no effect on voluntary disruption
212+
or expiration — it only suppresses Node Auto Repair.
213+
214+
If a node is terminated by expiration, interruption, or any other forceful
215+
method while a bootc-operator update is in progress, the termination is caused
216+
by Karpenter acting on its own policy — not by the bootc-operator update. The
217+
update simply happened to be in flight at the time.
218+
219+
## Residual race and coordination guarantees
220+
221+
The annotations and disruption checks are not an authoritative mutex. Karpenter
222+
does not currently acknowledge `do-not-repair`, expose a shared Lease, or
223+
coordinate its reboot commit with bootc-operator's slot assignment. Karpenter
224+
can therefore commit a disruption immediately before or after bootc-operator's
225+
last check. Setting `do-not-repair` does not cancel a disruption that Karpenter
226+
has already committed.
227+
228+
The integration narrows this window substantially by:
229+
230+
- Applying protection from the start of image staging rather than only at
231+
slot assignment.
232+
- Checking all disruption signals (reboot taint, disrupted taint, deletion
233+
timestamp, and NodeClaim conditions) before both slot assignment and drain.
234+
235+
Full serialization requires Karpenter support for a shared ownership protocol
236+
or an admission/provider mechanism that prevents a competing operation from
237+
being committed.
238+
239+
Refer to the [Karpenter disruption documentation](https://karpenter.sh/docs/concepts/disruption/)
240+
for a full breakdown of which controls apply to which disruption methods.
241+
242+
## Non-Karpenter nodes
243+
244+
Nodes without a `NodeClaim` owner reference are unaffected. The feature
245+
activates only when the owner reference check matches.
246+
247+
## Annotation reference
248+
249+
| Annotation | Object | Managed by | Purpose |
250+
|---|---|---|---|
251+
| `karpenter.sh/do-not-repair` | K8s Node | bootc-operator (or external) | Prevents Karpenter Node Auto Repair from replacing this node during an update |
252+
| `karpenter.sh/do-not-disrupt` | K8s Node | bootc-operator (or external) | Prevents Karpenter voluntary disruption (consolidation, drift) during an update |
253+
| `karpenter.sh/do-not-repair` | NodeClaim | bootc-operator (or external) | Best-effort suppression of Karpenter repair selection before an operator reboot |
254+
| `karpenter.sh/do-not-disrupt` | NodeClaim | bootc-operator (or external) | Best-effort suppression of Karpenter voluntary disruption before an operator reboot |
255+
| `bootc.dev/owns-karpenter-do-not-repair` | BootcNode | bootc-operator | Tracks that the operator added `do-not-repair`; cleared when the annotation is removed on slot free |
256+
| `bootc.dev/owns-karpenter-do-not-disrupt` | BootcNode | bootc-operator | Tracks that the operator added `do-not-disrupt`; cleared when the annotation is removed on slot free |
257+
| `bootc.dev/owns-karpenter-nodeclaim-do-not-repair` | BootcNode | bootc-operator | Tracks ownership of NodeClaim `do-not-repair` |
258+
| `bootc.dev/owns-karpenter-nodeclaim-do-not-disrupt` | BootcNode | bootc-operator | Tracks ownership of NodeClaim `do-not-disrupt` |

0 commit comments

Comments
 (0)