Skip to content

Commit 8e5f5b8

Browse files
cheesesashimicheesesashimi
committed
rollout: coordinate bootc updates with Karpenter
Protect Karpenter-managed nodes from repair and voluntary disruption throughout image staging, reboot, and health verification, including their NodeClaims when available. Detect active Karpenter disruptions and reboots while deferring node drains and/or reboots until completion, preserving externally managed annotations and cordon state. This also handles the case where there is drift between the boot image for a given node and the desired bootc node image, provided that the node has the well-known taint, bootc.dev/os-not-ready:NoSchedule Assisted-by: AI Signed-off-by: Zack Zlotnik <zzlotnik@redhat.com>
1 parent 4bdc3d4 commit 8e5f5b8

11 files changed

Lines changed: 3519 additions & 51 deletions

File tree

‎api/v1alpha1/constants.go‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,51 @@ const (
3030
// state for which the controller considered emitting an event. It is
3131
// observability bookkeeping only and is never used to drive reconciliation.
3232
AnnotationLastObservedState = "bootc.dev/last-observed-state"
33+
34+
// AnnotationOwnsKarpenterDoNotRepair is set on a BootcNode to record that
35+
// the controller added the karpenter.sh/do-not-repair annotation to the K8s
36+
// Node during slot assignment. When present, the controller removes the
37+
// karpenter.sh/do-not-repair annotation from the Node when the slot is freed.
38+
// It is not set when the annotation was already present before slot assignment
39+
// (externally set), so the controller never removes an annotation it did not add.
40+
AnnotationOwnsKarpenterDoNotRepair = "bootc.dev/owns-karpenter-do-not-repair"
41+
42+
// AnnotationOwnsKarpenterDoNotDisrupt is set on a BootcNode to record that
43+
// the controller added the karpenter.sh/do-not-disrupt annotation to the K8s
44+
// Node during slot assignment. When present, the controller removes the
45+
// karpenter.sh/do-not-disrupt annotation from the Node when the slot is freed.
46+
// It is not set when the annotation was already present before slot assignment
47+
// (externally set), so the controller never removes an annotation it did not add.
48+
AnnotationOwnsKarpenterDoNotDisrupt = "bootc.dev/owns-karpenter-do-not-disrupt"
49+
50+
// AnnotationOwnsKarpenterNodeClaimDoNotRepair records that the controller
51+
// added do-not-repair to the associated Karpenter NodeClaim.
52+
AnnotationOwnsKarpenterNodeClaimDoNotRepair = "bootc.dev/owns-karpenter-nodeclaim-do-not-repair"
53+
54+
// AnnotationOwnsKarpenterNodeClaimDoNotDisrupt records that the controller
55+
// added do-not-disrupt to the associated Karpenter NodeClaim.
56+
AnnotationOwnsKarpenterNodeClaimDoNotDisrupt = "bootc.dev/owns-karpenter-nodeclaim-do-not-disrupt"
57+
58+
// TaintKeyOsNotReady is a well-known taint key that cluster administrators
59+
// can apply to Karpenter NodePools as a startupTaint to gate workload
60+
// scheduling until bootc-operator has applied and verified the pool's target
61+
// OS image. The full taint is "bootc.dev/os-not-ready:NoSchedule".
62+
//
63+
// When a node joins with this taint, bootc-operator records ownership via
64+
// AnnotationOwnsOsNotReadyTaint and removes the taint once the node is
65+
// UpToDate and Ready. If the node is already running the target image when
66+
// it joins, the taint is removed without requiring a reboot slot.
67+
//
68+
// This taint MUST be declared as a startupTaint in the Karpenter NodePool.
69+
// If it is configured only as a regular taint, Karpenter will treat nodes
70+
// with the taint as unable to satisfy pending pods and will provision
71+
// additional nodes in a loop.
72+
TaintKeyOsNotReady = "bootc.dev/os-not-ready"
73+
74+
// AnnotationOwnsOsNotReadyTaint is set on a BootcNode to record that the
75+
// controller found and owns the bootc.dev/os-not-ready:NoSchedule taint on
76+
// the K8s Node at join time. When present, the controller removes the taint
77+
// once the node is UpToDate and Ready. The controller never removes a taint
78+
// it did not record ownership of.
79+
AnnotationOwnsOsNotReadyTaint = "bootc.dev/owns-os-not-ready-taint"
3380
)

‎config/rbac/role.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,15 @@ rules:
4949
- daemonsets
5050
verbs:
5151
- get
52+
- apiGroups:
53+
- karpenter.sh
54+
resources:
55+
- nodeclaims
56+
verbs:
57+
- get
58+
- list
59+
- patch
60+
- watch
5261
- apiGroups:
5362
- node.bootc.dev
5463
resources:

‎docs/src/SUMMARY.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,5 @@
88
- [Helm Chart](helm.md)
99
- [Operations](operations/index.md)
1010
- [Managing a pool](operations/pool.md)
11+
- [OS initialization taint](operations/os-not-ready-taint.md)
12+
- [Karpenter integration](operations/karpenter.md)

‎docs/src/operations/karpenter.md‎

Lines changed: 301 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,301 @@
1+
# Karpenter integration
2+
3+
bootc-operator is aware of [Karpenter](https://karpenter.sh/) and coordinates
4+
with it during node updates to prevent unnecessary node replacement,
5+
voluntary disruption, and concurrent reboot handling.
6+
7+
## Why this matters
8+
9+
Karpenter monitors nodes for health and may decide to replace or disrupt a
10+
node that appears unhealthy or underutilized — for example, one that is
11+
cordoned, being drained, or not yet Ready after a reboot. All of those
12+
conditions arise naturally during a bootc-operator update window, so without
13+
coordination Karpenter could race with the operator and terminate a node that
14+
is in the middle of a legitimate OS update.
15+
16+
Karpenter respects two node-level annotations:
17+
18+
- `karpenter.sh/do-not-repair: "true"` — prevents Node Auto Repair from
19+
replacing a node that has unhealthy conditions.
20+
- `karpenter.sh/do-not-disrupt: "true"` — prevents voluntary disruption
21+
actions (consolidation, drift) from selecting this node.
22+
23+
> **Note:** `karpenter.sh/do-not-repair` was introduced in Karpenter v1.15.0.
24+
> That is the minimum required version for full bootc-operator Karpenter integration.
25+
26+
bootc-operator sets both annotations automatically on Karpenter-managed nodes
27+
for the duration of the full update lifecycle — from image staging through
28+
node reboot and health verification. Where the associated `NodeClaim` exists,
29+
it also sets the annotations there. This is best-effort coordination, not a
30+
distributed lock.
31+
32+
## How it works
33+
34+
When a Karpenter-owned node enters the `Staging` state (the daemon is pulling
35+
the new image), bootc-operator immediately sets `do-not-repair` and
36+
`do-not-disrupt` on the Node and, where available, the associated NodeClaim.
37+
This protection persists through slot assignment, drain, reboot, and post-boot
38+
health verification.
39+
40+
Before assigning a reboot slot or starting a drain, the controller checks
41+
whether Karpenter has already committed any lifecycle operation that would
42+
conflict with the bootc update. It checks:
43+
44+
- Node taints `karpenter.sh/rebooting:NoSchedule` and
45+
`karpenter.sh/disrupted:NoSchedule`
46+
- Node `DeletionTimestamp` (node is being terminated)
47+
- NodeClaim `DeletionTimestamp` (NodeClaim is being deleted)
48+
- NodeClaim `status.conditions`: `Rebooting=True`, `DisruptionReason=True`,
49+
or `InstanceTerminating=True`
50+
51+
If any of these signals is active, bootc-operator defers and retries. It
52+
never removes a Karpenter taint.
53+
54+
If the Karpenter NodeClaim owner reference is present but the NodeClaim object
55+
returns a 404, bootc-operator treats this as unsafe (the NodeClaim may be
56+
mid-deletion) and defers reboot-slot assignment and drain with a short requeue
57+
rather than proceeding. During `Staging`, it still applies the Node-level
58+
annotations, but skips NodeClaim annotation changes while the NodeClaim is
59+
unavailable.
60+
61+
If none of those signals is active, the controller takes the following steps:
62+
63+
1. For each annotation (`do-not-repair`, `do-not-disrupt`): if the annotation
64+
is **not** already set to `"true"`, the controller sets it (value `"true"`)
65+
on the Node and, where available, the NodeClaim. It records ownership
66+
separately on the `BootcNode` object:
67+
- `bootc.dev/owns-karpenter-do-not-repair: "true"`
68+
- `bootc.dev/owns-karpenter-do-not-disrupt: "true"`
69+
- `bootc.dev/owns-karpenter-nodeclaim-do-not-repair: "true"`
70+
- `bootc.dev/owns-karpenter-nodeclaim-do-not-disrupt: "true"`
71+
2. The controller proceeds through the normal update flow: drain, reboot, and
72+
image verification.
73+
3. When the slot is freed (node is Ready and running the new image), the
74+
controller removes only annotations that it still owns and whose value is
75+
still `"true"`. It never removes annotations it did not add.
76+
77+
## NodeClaim annotations are supplemental
78+
79+
NodeClaim annotations (`do-not-repair`, `do-not-disrupt`) are a supplemental
80+
signal, not a distributed lock. By the time bootc-operator writes them,
81+
Karpenter may already have selected or queued a disruption for the registered
82+
Node. The authoritative disruption checks are the Node-level taint, the
83+
NodeClaim deletion timestamp, and the NodeClaim status conditions — the
84+
controller checks these before both slot assignment and drain.
85+
86+
## NodeClaim watch registration
87+
88+
When the controller starts, it checks whether the Karpenter `NodeClaim` API is
89+
available. If it is, the controller registers a watch for NodeClaim events and
90+
uses changes to the associated NodeClaim to requeue its owning
91+
`BootcNodePool`. This allows changes such as Karpenter starting or completing
92+
a reboot to be observed promptly.
93+
94+
The watch is registered only during startup. If Karpenter is installed after
95+
bootc-operator has started, the operator does not add the watch dynamically;
96+
restart the bootc-operator controller after installing Karpenter. As a fallback,
97+
the controller schedules a periodic requeue (every 30 seconds) whenever
98+
Karpenter-owned nodes are active, so that missed NodeClaim events do not
99+
stall progress indefinitely.
100+
101+
The controller represents NodeClaims with Kubernetes `Unstructured` objects
102+
instead of importing Karpenter's Go API types. This keeps Karpenter an optional
103+
integration and avoids adding a Karpenter API dependency or scheme registration
104+
to bootc-operator. It also lets clusters without the NodeClaim CRD continue to
105+
run normally. The tradeoff is that the small set of fields used here, such as
106+
`status.nodeName` and `status.conditions`, is checked at runtime rather than
107+
being enforced by Go types.
108+
109+
## Pre-existing annotations
110+
111+
If a Karpenter annotation is already present on the Node with a value of
112+
`"true"` before protection is applied — set by an external actor, another
113+
controller, or a manual operation — bootc-operator leaves it alone:
114+
115+
- It does **not** set the corresponding tracking annotation on the `BootcNode`.
116+
- It does **not** remove the annotation when the slot is freed.
117+
118+
If the annotation is present but set to any value other than `"true"` (for
119+
example `"false"`), bootc-operator overwrites it with `"true"` and records
120+
ownership. This is required because Karpenter only honors the literal value
121+
`"true"`; any other value leaves the node unprotected.
122+
123+
The two annotations are tracked independently. If only one is already set to
124+
`"true"`, the operator owns (and later removes) only the other.
125+
126+
## Temporary overprovisioning during node drains
127+
128+
During a bootc-operator update, the following sequence can cause temporary
129+
overprovisioning:
130+
131+
1. bootc-operator evicts workloads from the node being updated.
132+
2. The evicted workloads become pending.
133+
3. Karpenter provisions new capacity to satisfy the pending pods.
134+
4. The original node reboots into the new image and rejoins the cluster.
135+
5. The newly provisioned capacity may become excess and be consolidated later.
136+
137+
This is expected behavior. The `do-not-disrupt` annotation suppresses
138+
Karpenter's voluntary disruption of the node being updated, but it does not
139+
suppress Karpenter provisioning new capacity for pending workloads. Karpenter
140+
provisioning is driven by pending pod demand rather than disruption eligibility.
141+
142+
## Post-rollout consolidation and administrator policy
143+
144+
bootc-operator's protection is intentionally limited to the update lifecycle.
145+
When a node is Ready and the new image has been verified, the operator frees its
146+
reboot slot and removes only the Karpenter annotations that it added. If the
147+
workload has moved to newly provisioned capacity and the original node is now
148+
empty, Karpenter may then consolidate that node with an `Empty` disruption. This
149+
is normal Karpenter behavior after the bootc update has completed; it is not an
150+
OS update failure.
151+
152+
Cluster administrators who need a node to remain in service after the rollout
153+
must configure that steady-state policy in Karpenter. bootc-operator should not
154+
permanently disable consolidation by default, because doing so can leave unused
155+
capacity running indefinitely.
156+
157+
This is an intentional separation of responsibilities. bootc-operator protects
158+
only the Karpenter-owned nodes that are participating in its update lifecycle;
159+
it does not apply protection to every node in the `BootcNodePool`.
160+
161+
| Responsibility | bootc-operator | Cluster administrator / Karpenter |
162+
|---|---|---|
163+
| Protect a node during image staging, drain, reboot, and health verification | Applies temporary node and NodeClaim annotations and checks for conflicting Karpenter operations | — |
164+
| Provision capacity for pending workloads | — | Karpenter provisions nodes according to its NodePool and workload requirements |
165+
| Consolidate or drift unrelated nodes | — | Karpenter manages nodes according to the configured disruption policy |
166+
| Retain a node after the rollout becomes complete | Removes only annotations it owns; does not freeze unrelated nodes | Configure `do-not-disrupt`, disruption budgets, or `consolidateAfter` as needed |
167+
| Protect an entire Karpenter pool during maintenance | — | Configure an explicit Karpenter-wide maintenance policy |
168+
| Handle expiration, interruption, manual deletion, or other forceful termination | Detects some conflicts and defers its own update when possible | Configure and operate the applicable Karpenter and infrastructure policies |
169+
170+
If an entire pool must be held stable, configure that as an explicit Karpenter
171+
maintenance policy rather than relying on bootc-operator to freeze unrelated
172+
nodes.
173+
174+
Options include:
175+
176+
1. **Keep a node protected explicitly.** Set
177+
`karpenter.sh/do-not-disrupt: "true"` on the Node and its NodeClaim. An
178+
annotation that was already present before bootc-operator starts protecting
179+
the node is treated as externally owned and is not removed when the rollout
180+
completes. This blocks voluntary disruption such as consolidation and drift,
181+
but not expiration, interruption, manual deletion, or other forceful
182+
termination.
183+
2. **Temporarily block disruption for the whole Karpenter NodePool.** During a
184+
rollout, an administrator can set a zero disruption budget:
185+
186+
```yaml
187+
spec:
188+
disruption:
189+
budgets:
190+
- nodes: "0"
191+
```
192+
193+
Restore the normal budget after the rollout. This protects all nodes in the
194+
pool, not just the node being updated.
195+
3. **Delay consolidation.** Configure a suitable `consolidateAfter` interval so
196+
that a rebooted node has time to become schedulable and receive workloads
197+
before Karpenter considers it empty. This reduces the race but does not
198+
guarantee that the node will be retained.
199+
200+
All three options are administrator-owned policy. The first is appropriate for
201+
selected nodes that must remain stable; the second is appropriate when an
202+
entire pool must be held stable during a controlled maintenance window; and the
203+
third is appropriate when consolidation should remain enabled but be delayed.
204+
205+
## OS initialization startup taint
206+
207+
The lifecycle-manager-neutral behavior and taint lifecycle are documented in
208+
[OS initialization taint](os-not-ready-taint.md). With Karpenter, configure
209+
the taint as a `startupTaint` on the `NodePool`.
210+
211+
Add `bootc.dev/os-not-ready` to the `startupTaints` list in the Karpenter
212+
`NodePool`:
213+
214+
```yaml
215+
apiVersion: karpenter.sh/v1
216+
kind: NodePool
217+
metadata:
218+
name: default
219+
spec:
220+
template:
221+
spec:
222+
startupTaints:
223+
- key: bootc.dev/os-not-ready
224+
effect: NoSchedule
225+
```
226+
227+
No changes to the `BootcNodePool` spec are required. bootc-operator activates
228+
the feature automatically when it detects the taint on a newly joined node.
229+
230+
### `startupTaints` vs `taints`
231+
232+
This taint **must** be declared under `startupTaints`, not `taints`, in the
233+
Karpenter `NodePool`.
234+
235+
Karpenter uses `startupTaints` to mark nodes that require external
236+
initialization before workloads are scheduled. It factors the taint into its
237+
bin-packing decisions and does not treat nodes with the taint as unable to
238+
satisfy pending pods.
239+
240+
If the taint is instead declared under `taints` (permanent taints), Karpenter
241+
will observe that pending pods cannot be scheduled on nodes carrying the taint
242+
and will provision additional nodes in a loop — each of which will also carry
243+
the taint, perpetuating the loop.
244+
245+
## Limitations
246+
247+
`karpenter.sh/do-not-disrupt` only blocks voluntary disruption methods
248+
(consolidation and drift). It does not protect a node against forceful
249+
termination triggered by expiration (`expireAfter`), interruption (spot
250+
instance reclaim), Node Auto Repair, or manual deletion. If any of those
251+
events occur while a node is in a bootc-operator reboot slot, Karpenter may
252+
still terminate it.
253+
254+
`karpenter.sh/do-not-repair` similarly has no effect on voluntary disruption
255+
or expiration — it only suppresses Node Auto Repair.
256+
257+
If a node is terminated by expiration, interruption, or any other forceful
258+
method while a bootc-operator update is in progress, the termination is caused
259+
by Karpenter acting on its own policy — not by the bootc-operator update. The
260+
update simply happened to be in flight at the time.
261+
262+
## Residual race and coordination guarantees
263+
264+
The annotations and disruption checks are not an authoritative mutex. Karpenter
265+
does not currently acknowledge `do-not-repair`, expose a shared Lease, or
266+
coordinate its reboot commit with bootc-operator's slot assignment. Karpenter
267+
can therefore commit a disruption immediately before or after bootc-operator's
268+
last check. Setting `do-not-repair` does not cancel a disruption that Karpenter
269+
has already committed.
270+
271+
The integration narrows this window substantially by:
272+
273+
- Applying protection from the start of image staging rather than only at
274+
slot assignment.
275+
- Checking all disruption signals (reboot taint, disrupted taint, deletion
276+
timestamp, and NodeClaim conditions) before both slot assignment and drain.
277+
278+
Full serialization requires Karpenter support for a shared ownership protocol
279+
or an admission/provider mechanism that prevents a competing operation from
280+
being committed.
281+
282+
Refer to the [Karpenter disruption documentation](https://karpenter.sh/docs/concepts/disruption/)
283+
for a full breakdown of which controls apply to which disruption methods.
284+
285+
## Non-Karpenter nodes
286+
287+
Nodes without a `NodeClaim` owner reference are unaffected. The feature
288+
activates only when the owner reference check matches.
289+
290+
## Annotation reference
291+
292+
| Annotation | Object | Managed by | Purpose |
293+
|---|---|---|---|
294+
| `karpenter.sh/do-not-repair` | K8s Node | bootc-operator (or external) | Prevents Karpenter Node Auto Repair from replacing this node during an update |
295+
| `karpenter.sh/do-not-disrupt` | K8s Node | bootc-operator (or external) | Prevents Karpenter voluntary disruption (consolidation, drift) during an update |
296+
| `karpenter.sh/do-not-repair` | NodeClaim | bootc-operator (or external) | Best-effort suppression of Karpenter repair selection before an operator reboot |
297+
| `karpenter.sh/do-not-disrupt` | NodeClaim | bootc-operator (or external) | Best-effort suppression of Karpenter voluntary disruption before an operator reboot |
298+
| `bootc.dev/owns-karpenter-do-not-repair` | BootcNode | bootc-operator | Tracks that the operator added `do-not-repair`; cleared when the annotation is removed on slot free |
299+
| `bootc.dev/owns-karpenter-do-not-disrupt` | BootcNode | bootc-operator | Tracks that the operator added `do-not-disrupt`; cleared when the annotation is removed on slot free |
300+
| `bootc.dev/owns-karpenter-nodeclaim-do-not-repair` | BootcNode | bootc-operator | Tracks ownership of NodeClaim `do-not-repair` |
301+
| `bootc.dev/owns-karpenter-nodeclaim-do-not-disrupt` | BootcNode | bootc-operator | Tracks ownership of NodeClaim `do-not-disrupt` |

0 commit comments

Comments
 (0)