Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion STYLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
- Define every piece of jargon at first use, in one clause: "the informer (a client-side cache that watches the API server)".
- High signal, low noise: no filler ("It is important to note that…", "In the world of Kubernetes…"), no marketing tone, no repetition of what another section already said — cross-reference instead ("see Flow 1").
- Present tense. Active voice. "The scheduler writes a Binding" not "A Binding is written".
- Lists for parallel items: if a paragraph enumerates two or more parallel items — kinds, options, defenses — keep the lead-in sentence as prose and put each item on its own bullet with a bold lead term: `- **Predicate:** filters the nodes that can run the pod.` Use a table instead when the items share the same attributes (columns).
- Never invent facts. Version-sensitive claims MUST match `notes/research-notes.md`. Baseline is Kubernetes v1.36.

## File format
Expand All @@ -34,7 +35,13 @@ Contents, in order:
1. One-sentence setup ("You run `kubectl cordon node-1`.").
2. **Numbered steps.** Each step starts with the acting component in bold: `1. **kubectl** sends a PATCH …`. One actor-action per step. 6–15 steps. If a step hides interview-relevant depth, add an indented sub-bullet, max one per step.
3. **A mermaid diagram** in a fenced ` ```mermaid ` block, immediately after the steps, followed by an italic caption line: `*Figure N.M — one line saying what to notice.*` (N = chapter, M = running count within chapter).
4. **Where this can fail** — 3–6 bullets: `- **Symptom:** … **Cause:** … **Where to look:** …`.
4. **Where this can fail** — 3–6 symptom bullets. Each bullet carries the symptom, then two indented sub-bullets (4 spaces — the PDF build flattens 2-space indents):

```markdown
- **Symptom:** endless stream of 409s in controller logs.
- **Cause:** hot object plus retry without jitter, or retrying from a stale cache.
- **Where to look:** controller logs; `managedFields` for who else writes.
```

## Mermaid rules (build breaks if you're clever)

Expand Down
16 changes: 12 additions & 4 deletions chapters/ch01.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,18 @@ sequenceDiagram

**Where this can fail**

- **Symptom:** 403 Forbidden. **Cause:** RBAC denies the verb/resource pair. **Where to look:** `kubectl auth can-i`, audit log.
- **Symptom:** create hangs, fails after ~10–30s. **Cause:** a matching webhook is down with `failurePolicy: Fail` (Flow 4). **Where to look:** API server logs, webhook configurations.
- **Symptom:** all writes fail cluster-wide; reads may still work. **Cause:** etcd lost quorum — no raft leader, no commits. **Where to look:** etcd member health.
- **Symptom:** object persisted but nothing happens. **Cause:** the write succeeded; a downstream controller is broken — the API cannot tell you that. **Where to look:** object events and status, controller logs.
- **Symptom:** 403 Forbidden.
- **Cause:** RBAC denies the verb/resource pair.
- **Where to look:** `kubectl auth can-i`, audit log.
- **Symptom:** create hangs, fails after ~10–30s.
- **Cause:** a matching webhook is down with `failurePolicy: Fail` (Flow 4).
- **Where to look:** API server logs, webhook configurations.
- **Symptom:** all writes fail cluster-wide; reads may still work.
- **Cause:** etcd lost quorum — no raft leader, no commits.
- **Where to look:** etcd member health.
- **Symptom:** object persisted but nothing happens.
- **Cause:** the write succeeded; a downstream controller is broken — the API cannot tell you that.
- **Where to look:** object events and status, controller logs.

## Questions

Expand Down
48 changes: 36 additions & 12 deletions chapters/ch02.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,18 @@ sequenceDiagram

**Where this can fail**

- **Symptom:** endless stream of 409s in controller logs. **Cause:** hot object plus retry without jitter, or retrying from a stale cache. **Where to look:** controller logs; `managedFields` for who else writes.
- **Symptom:** a field flips between two values. **Cause:** two controllers each believe they own it and "correct" the other. **Where to look:** `managedFields` managers and timestamps.
- **Symptom:** SSA apply fails naming another manager. **Cause:** genuine shared-field ownership. **Where to look:** decide the rightful owner; only then consider `force`.
- **Symptom:** update succeeds but changes vanish. **Cause:** a writer sent a full-object UPDATE with no resourceVersion — last-write-wins over changes it never saw. **Where to look:** audit log for the overwriting request.
- **Symptom:** endless stream of 409s in controller logs.
- **Cause:** hot object plus retry without jitter, or retrying from a stale cache.
- **Where to look:** controller logs; `managedFields` for who else writes.
- **Symptom:** a field flips between two values.
- **Cause:** two controllers each believe they own it and "correct" the other.
- **Where to look:** `managedFields` managers and timestamps.
- **Symptom:** SSA apply fails naming another manager.
- **Cause:** genuine shared-field ownership.
- **Where to look:** decide the rightful owner; only then consider `force`.
- **Symptom:** update succeeds but changes vanish.
- **Cause:** a writer sent a full-object UPDATE with no resourceVersion — last-write-wins over changes it never saw.
- **Where to look:** audit log for the overwriting request.

### Flow 3: What happens when a watch is established — and falls behind

Expand Down Expand Up @@ -116,10 +124,18 @@ sequenceDiagram

**Where this can fail**

- **Symptom:** logs full of "too old resource version" and relists. **Cause:** handlers too slow for the event rate, or churn exceeding the cache window. **Where to look:** handler latency, object churn rate.
- **Symptom:** controller acts on stale objects — 409s on every write. **Cause:** normal cache lag, or a wedged reflector. **Where to look:** informer HasSynced, reflector logs.
- **Symptom:** API server memory spikes when a controller restarts. **Cause:** full relist of a huge resource, amplified across replicas. **Where to look:** APF metrics, list sizes; mitigate with pagination and scoped watches.
- **Symptom:** events seemingly "missed". **Cause:** compacted away during disconnect; only final state is recoverable — by design. **Where to look:** nothing to recover; the controller must reconcile from current state.
- **Symptom:** logs full of "too old resource version" and relists.
- **Cause:** handlers too slow for the event rate, or churn exceeding the cache window.
- **Where to look:** handler latency, object churn rate.
- **Symptom:** controller acts on stale objects — 409s on every write.
- **Cause:** normal cache lag, or a wedged reflector.
- **Where to look:** informer HasSynced, reflector logs.
- **Symptom:** API server memory spikes when a controller restarts.
- **Cause:** full relist of a huge resource, amplified across replicas.
- **Where to look:** APF metrics, list sizes; mitigate with pagination and scoped watches.
- **Symptom:** events seemingly "missed".
- **Cause:** compacted away during disconnect; only final state is recoverable — by design.
- **Where to look:** nothing to recover; the controller must reconcile from current state.

### Flow 4: What happens when an admission webhook is down

Expand Down Expand Up @@ -155,10 +171,18 @@ sequenceDiagram

**Where this can fail**

- **Symptom:** every deploy errors "failed calling webhook". **Cause:** dead backend, bad CA bundle, or Service/port mismatch, with `failurePolicy: Fail`. **Where to look:** webhook configuration, its Service endpoints, API server logs.
- **Symptom:** all writes ~10s slower but succeeding. **Cause:** webhook timing out with `Ignore` — a silent latency and policy hole. **Where to look:** `apiserver_admission_webhook_*` metrics.
- **Symptom:** cluster cannot recover after a full outage. **Cause:** webhook pods and the workloads they gate deadlock on startup. **Where to look:** namespace exclusions; break the loop by deleting the webhook configuration temporarily.
- **Symptom:** policy violations exist despite the webhook. **Cause:** `Ignore` fired during an incident, or objects predate the webhook. **Where to look:** audit logs; add a scanning controller for existing objects.
- **Symptom:** every deploy errors "failed calling webhook".
- **Cause:** dead backend, bad CA bundle, or Service/port mismatch, with `failurePolicy: Fail`.
- **Where to look:** webhook configuration, its Service endpoints, API server logs.
- **Symptom:** all writes ~10s slower but succeeding.
- **Cause:** webhook timing out with `Ignore` — a silent latency and policy hole.
- **Where to look:** `apiserver_admission_webhook_*` metrics.
- **Symptom:** cluster cannot recover after a full outage.
- **Cause:** webhook pods and the workloads they gate deadlock on startup.
- **Where to look:** namespace exclusions; break the loop by deleting the webhook configuration temporarily.
- **Symptom:** policy violations exist despite the webhook.
- **Cause:** `Ignore` fired during an incident, or objects predate the webhook.
- **Where to look:** audit logs; add a scanning controller for existing objects.

## Questions

Expand Down
48 changes: 36 additions & 12 deletions chapters/ch03.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,18 @@ sequenceDiagram

**Where this can fail**

- **Symptom:** pod Pending with FailedScheduling event. **Cause:** no node passes Filter — resources, taints, affinity, volume topology. **Where to look:** `kubectl describe pod`; events enumerate filter failures per node.
- **Symptom:** pod scheduled but kubelet rejects it with OutOfcpu. **Cause:** scheduler raced kubelet-reported allocatable; kubelet admission is the final check. **Where to look:** pod events, node allocatable vs summed requests.
- **Symptom:** binding fails, pod re-queued. **Cause:** PreBind failure (volume provisioning error) or API conflict. **Where to look:** scheduler logs, PVC events.
- **Symptom:** scheduling latency grows with cluster size. **Cause:** heavy pod affinity rules or scoring too many nodes. **Where to look:** per-plugin scheduler latency metrics.
- **Symptom:** pod Pending with FailedScheduling event.
- **Cause:** no node passes Filter — resources, taints, affinity, volume topology.
- **Where to look:** `kubectl describe pod`; events enumerate filter failures per node.
- **Symptom:** pod scheduled but kubelet rejects it with OutOfcpu.
- **Cause:** scheduler raced kubelet-reported allocatable; kubelet admission is the final check.
- **Where to look:** pod events, node allocatable vs summed requests.
- **Symptom:** binding fails, pod re-queued.
- **Cause:** PreBind failure (volume provisioning error) or API conflict.
- **Where to look:** scheduler logs, PVC events.
- **Symptom:** scheduling latency grows with cluster size.
- **Cause:** heavy pod affinity rules or scoring too many nodes.
- **Where to look:** per-plugin scheduler latency metrics.

### Flow 6: What happens when no node fits

Expand Down Expand Up @@ -108,10 +116,18 @@ flowchart TD

**Where this can fail**

- **Symptom:** preemptor stays Pending after victims die. **Cause:** freed capacity taken by others, or a higher-priority pod claimed the node — nominatedNodeName is not binding. **Where to look:** scheduler logs, competing priorities.
- **Symptom:** unexpected evictions of workload pods. **Cause:** someone deployed a high-priorityClass pod; preemption working as designed. **Where to look:** victim pod events (Preempted), priorityClass audit.
- **Symptom:** no preemption despite a priority difference. **Cause:** `preemptionPolicy: Never` on the class, or victims' PDBs made every candidate worse. **Where to look:** PriorityClass spec, scheduler logs.
- **Symptom:** cascading preemption churn. **Cause:** many similar priorities competing for scarce capacity. **Where to look:** priority distribution; fix with capacity or clearer tiers.
- **Symptom:** preemptor stays Pending after victims die.
- **Cause:** freed capacity taken by others, or a higher-priority pod claimed the node — nominatedNodeName is not binding.
- **Where to look:** scheduler logs, competing priorities.
- **Symptom:** unexpected evictions of workload pods.
- **Cause:** someone deployed a high-priorityClass pod; preemption working as designed.
- **Where to look:** victim pod events (Preempted), priorityClass audit.
- **Symptom:** no preemption despite a priority difference.
- **Cause:** `preemptionPolicy: Never` on the class, or victims' PDBs made every candidate worse.
- **Where to look:** PriorityClass spec, scheduler logs.
- **Symptom:** cascading preemption churn.
- **Cause:** many similar priorities competing for scarce capacity.
- **Where to look:** priority distribution; fix with capacity or clearer tiers.

### Flow 7: What happens when a taint is applied to a node

Expand Down Expand Up @@ -146,10 +162,18 @@ sequenceDiagram

**Where this can fail**

- **Symptom:** DaemonSet pods survive the node-condition NoExecute taints (`node.kubernetes.io/*`). **Cause:** the DaemonSet controller adds tolerations for exactly those taints — intended. A custom NoExecute taint like this flow's *does* evict untolerated DaemonSet pods. **Where to look:** the pod's tolerations.
- **Symptom:** pods evicted five minutes after a network blip that already healed. **Cause:** the 300s toleration expired before the taint was removed. **Where to look:** node lifecycle controller logs, taint history.
- **Symptom:** new pods still land on a tainted node. **Cause:** a broad `operator: Exists` toleration copied from a template. **Where to look:** pod spec tolerations.
- **Symptom:** taint added but evictions trickle slowly. **Cause:** the taint-eviction controller rate-limits to avoid stampedes on wide failures. **Where to look:** KCM eviction rate settings and logs.
- **Symptom:** DaemonSet pods survive the node-condition NoExecute taints (`node.kubernetes.io/*`).
- **Cause:** the DaemonSet controller adds tolerations for exactly those taints — intended. A custom NoExecute taint like this flow's *does* evict untolerated DaemonSet pods.
- **Where to look:** the pod's tolerations.
- **Symptom:** pods evicted five minutes after a network blip that already healed.
- **Cause:** the 300s toleration expired before the taint was removed.
- **Where to look:** node lifecycle controller logs, taint history.
- **Symptom:** new pods still land on a tainted node.
- **Cause:** a broad `operator: Exists` toleration copied from a template.
- **Where to look:** pod spec tolerations.
- **Symptom:** taint added but evictions trickle slowly.
- **Cause:** the taint-eviction controller rate-limits to avoid stampedes on wide failures.
- **Where to look:** KCM eviction rate settings and logs.

## Questions

Expand Down
Loading