Chapters: nest failure bullets under each symptom

Author: sanketsudake

chapters/ch01.md

@@ -86,10 +86,18 @@ sequenceDiagram
 
 **Where this can fail**
 
-- **Symptom:** 403 Forbidden. **Cause:** RBAC denies the verb/resource pair. **Where to look:** `kubectl auth can-i`, audit log.
-- **Symptom:** create hangs, fails after ~10–30s. **Cause:** a matching webhook is down with `failurePolicy: Fail` (Flow 4). **Where to look:** API server logs, webhook configurations.
-- **Symptom:** all writes fail cluster-wide; reads may still work. **Cause:** etcd lost quorum — no raft leader, no commits. **Where to look:** etcd member health.
-- **Symptom:** object persisted but nothing happens. **Cause:** the write succeeded; a downstream controller is broken — the API cannot tell you that. **Where to look:** object events and status, controller logs.
+- **Symptom:** 403 Forbidden.
+    - **Cause:** RBAC denies the verb/resource pair.
+    - **Where to look:** `kubectl auth can-i`, audit log.
+- **Symptom:** create hangs, fails after ~10–30s.
+    - **Cause:** a matching webhook is down with `failurePolicy: Fail` (Flow 4).
+    - **Where to look:** API server logs, webhook configurations.
+- **Symptom:** all writes fail cluster-wide; reads may still work.
+    - **Cause:** etcd lost quorum — no raft leader, no commits.
+    - **Where to look:** etcd member health.
+- **Symptom:** object persisted but nothing happens.
+    - **Cause:** the write succeeded; a downstream controller is broken — the API cannot tell you that.
+    - **Where to look:** object events and status, controller logs.
 
 ## Questions
 

chapters/ch02.md

@@ -75,10 +75,18 @@ sequenceDiagram
 
 **Where this can fail**
 
-- **Symptom:** endless stream of 409s in controller logs. **Cause:** hot object plus retry without jitter, or retrying from a stale cache. **Where to look:** controller logs; `managedFields` for who else writes.
-- **Symptom:** a field flips between two values. **Cause:** two controllers each believe they own it and "correct" the other. **Where to look:** `managedFields` managers and timestamps.
-- **Symptom:** SSA apply fails naming another manager. **Cause:** genuine shared-field ownership. **Where to look:** decide the rightful owner; only then consider `force`.
-- **Symptom:** update succeeds but changes vanish. **Cause:** a writer sent a full-object UPDATE with no resourceVersion — last-write-wins over changes it never saw. **Where to look:** audit log for the overwriting request.
+- **Symptom:** endless stream of 409s in controller logs.
+    - **Cause:** hot object plus retry without jitter, or retrying from a stale cache.
+    - **Where to look:** controller logs; `managedFields` for who else writes.
+- **Symptom:** a field flips between two values.
+    - **Cause:** two controllers each believe they own it and "correct" the other.
+    - **Where to look:** `managedFields` managers and timestamps.
+- **Symptom:** SSA apply fails naming another manager.
+    - **Cause:** genuine shared-field ownership.
+    - **Where to look:** decide the rightful owner; only then consider `force`.
+- **Symptom:** update succeeds but changes vanish.
+    - **Cause:** a writer sent a full-object UPDATE with no resourceVersion — last-write-wins over changes it never saw.
+    - **Where to look:** audit log for the overwriting request.
 
 ### Flow 3: What happens when a watch is established — and falls behind
 
@@ -116,10 +124,18 @@ sequenceDiagram
 
 **Where this can fail**
 
-- **Symptom:** logs full of "too old resource version" and relists. **Cause:** handlers too slow for the event rate, or churn exceeding the cache window. **Where to look:** handler latency, object churn rate.
-- **Symptom:** controller acts on stale objects — 409s on every write. **Cause:** normal cache lag, or a wedged reflector. **Where to look:** informer HasSynced, reflector logs.
-- **Symptom:** API server memory spikes when a controller restarts. **Cause:** full relist of a huge resource, amplified across replicas. **Where to look:** APF metrics, list sizes; mitigate with pagination and scoped watches.
-- **Symptom:** events seemingly "missed". **Cause:** compacted away during disconnect; only final state is recoverable — by design. **Where to look:** nothing to recover; the controller must reconcile from current state.
+- **Symptom:** logs full of "too old resource version" and relists.
+    - **Cause:** handlers too slow for the event rate, or churn exceeding the cache window.
+    - **Where to look:** handler latency, object churn rate.
+- **Symptom:** controller acts on stale objects — 409s on every write.
+    - **Cause:** normal cache lag, or a wedged reflector.
+    - **Where to look:** informer HasSynced, reflector logs.
+- **Symptom:** API server memory spikes when a controller restarts.
+    - **Cause:** full relist of a huge resource, amplified across replicas.
+    - **Where to look:** APF metrics, list sizes; mitigate with pagination and scoped watches.
+- **Symptom:** events seemingly "missed".
+    - **Cause:** compacted away during disconnect; only final state is recoverable — by design.
+    - **Where to look:** nothing to recover; the controller must reconcile from current state.
 
 ### Flow 4: What happens when an admission webhook is down
 
@@ -155,10 +171,18 @@ sequenceDiagram
 
 **Where this can fail**
 
-- **Symptom:** every deploy errors "failed calling webhook". **Cause:** dead backend, bad CA bundle, or Service/port mismatch, with `failurePolicy: Fail`. **Where to look:** webhook configuration, its Service endpoints, API server logs.
-- **Symptom:** all writes ~10s slower but succeeding. **Cause:** webhook timing out with `Ignore` — a silent latency and policy hole. **Where to look:** `apiserver_admission_webhook_*` metrics.
-- **Symptom:** cluster cannot recover after a full outage. **Cause:** webhook pods and the workloads they gate deadlock on startup. **Where to look:** namespace exclusions; break the loop by deleting the webhook configuration temporarily.
-- **Symptom:** policy violations exist despite the webhook. **Cause:** `Ignore` fired during an incident, or objects predate the webhook. **Where to look:** audit logs; add a scanning controller for existing objects.
+- **Symptom:** every deploy errors "failed calling webhook".
+    - **Cause:** dead backend, bad CA bundle, or Service/port mismatch, with `failurePolicy: Fail`.
+    - **Where to look:** webhook configuration, its Service endpoints, API server logs.
+- **Symptom:** all writes ~10s slower but succeeding.
+    - **Cause:** webhook timing out with `Ignore` — a silent latency and policy hole.
+    - **Where to look:** `apiserver_admission_webhook_*` metrics.
+- **Symptom:** cluster cannot recover after a full outage.
+    - **Cause:** webhook pods and the workloads they gate deadlock on startup.
+    - **Where to look:** namespace exclusions; break the loop by deleting the webhook configuration temporarily.
+- **Symptom:** policy violations exist despite the webhook.
+    - **Cause:** `Ignore` fired during an incident, or objects predate the webhook.
+    - **Where to look:** audit logs; add a scanning controller for existing objects.
 
 ## Questions
 

chapters/ch03.md

@@ -71,10 +71,18 @@ sequenceDiagram
 
 **Where this can fail**
 
-- **Symptom:** pod Pending with FailedScheduling event. **Cause:** no node passes Filter — resources, taints, affinity, volume topology. **Where to look:** `kubectl describe pod`; events enumerate filter failures per node.
-- **Symptom:** pod scheduled but kubelet rejects it with OutOfcpu. **Cause:** scheduler raced kubelet-reported allocatable; kubelet admission is the final check. **Where to look:** pod events, node allocatable vs summed requests.
-- **Symptom:** binding fails, pod re-queued. **Cause:** PreBind failure (volume provisioning error) or API conflict. **Where to look:** scheduler logs, PVC events.
-- **Symptom:** scheduling latency grows with cluster size. **Cause:** heavy pod affinity rules or scoring too many nodes. **Where to look:** per-plugin scheduler latency metrics.
+- **Symptom:** pod Pending with FailedScheduling event.
+    - **Cause:** no node passes Filter — resources, taints, affinity, volume topology.
+    - **Where to look:** `kubectl describe pod`; events enumerate filter failures per node.
+- **Symptom:** pod scheduled but kubelet rejects it with OutOfcpu.
+    - **Cause:** scheduler raced kubelet-reported allocatable; kubelet admission is the final check.
+    - **Where to look:** pod events, node allocatable vs summed requests.
+- **Symptom:** binding fails, pod re-queued.
+    - **Cause:** PreBind failure (volume provisioning error) or API conflict.
+    - **Where to look:** scheduler logs, PVC events.
+- **Symptom:** scheduling latency grows with cluster size.
+    - **Cause:** heavy pod affinity rules or scoring too many nodes.
+    - **Where to look:** per-plugin scheduler latency metrics.
 
 ### Flow 6: What happens when no node fits
 
@@ -108,10 +116,18 @@ flowchart TD
 
 **Where this can fail**
 
-- **Symptom:** preemptor stays Pending after victims die. **Cause:** freed capacity taken by others, or a higher-priority pod claimed the node — nominatedNodeName is not binding. **Where to look:** scheduler logs, competing priorities.
-- **Symptom:** unexpected evictions of workload pods. **Cause:** someone deployed a high-priorityClass pod; preemption working as designed. **Where to look:** victim pod events (Preempted), priorityClass audit.
-- **Symptom:** no preemption despite a priority difference. **Cause:** `preemptionPolicy: Never` on the class, or victims' PDBs made every candidate worse. **Where to look:** PriorityClass spec, scheduler logs.
-- **Symptom:** cascading preemption churn. **Cause:** many similar priorities competing for scarce capacity. **Where to look:** priority distribution; fix with capacity or clearer tiers.
+- **Symptom:** preemptor stays Pending after victims die.
+    - **Cause:** freed capacity taken by others, or a higher-priority pod claimed the node — nominatedNodeName is not binding.
+    - **Where to look:** scheduler logs, competing priorities.
+- **Symptom:** unexpected evictions of workload pods.
+    - **Cause:** someone deployed a high-priorityClass pod; preemption working as designed.
+    - **Where to look:** victim pod events (Preempted), priorityClass audit.
+- **Symptom:** no preemption despite a priority difference.
+    - **Cause:** `preemptionPolicy: Never` on the class, or victims' PDBs made every candidate worse.
+    - **Where to look:** PriorityClass spec, scheduler logs.
+- **Symptom:** cascading preemption churn.
+    - **Cause:** many similar priorities competing for scarce capacity.
+    - **Where to look:** priority distribution; fix with capacity or clearer tiers.
 
 ### Flow 7: What happens when a taint is applied to a node
 
@@ -146,10 +162,18 @@ sequenceDiagram
 
 **Where this can fail**
 
-- **Symptom:** DaemonSet pods survive the node-condition NoExecute taints (`node.kubernetes.io/*`). **Cause:** the DaemonSet controller adds tolerations for exactly those taints — intended. A custom NoExecute taint like this flow's *does* evict untolerated DaemonSet pods. **Where to look:** the pod's tolerations.
-- **Symptom:** pods evicted five minutes after a network blip that already healed. **Cause:** the 300s toleration expired before the taint was removed. **Where to look:** node lifecycle controller logs, taint history.
-- **Symptom:** new pods still land on a tainted node. **Cause:** a broad `operator: Exists` toleration copied from a template. **Where to look:** pod spec tolerations.
-- **Symptom:** taint added but evictions trickle slowly. **Cause:** the taint-eviction controller rate-limits to avoid stampedes on wide failures. **Where to look:** KCM eviction rate settings and logs.
+- **Symptom:** DaemonSet pods survive the node-condition NoExecute taints (`node.kubernetes.io/*`).
+    - **Cause:** the DaemonSet controller adds tolerations for exactly those taints — intended. A custom NoExecute taint like this flow's *does* evict untolerated DaemonSet pods.
+    - **Where to look:** the pod's tolerations.
+- **Symptom:** pods evicted five minutes after a network blip that already healed.
+    - **Cause:** the 300s toleration expired before the taint was removed.
+    - **Where to look:** node lifecycle controller logs, taint history.
+- **Symptom:** new pods still land on a tainted node.
+    - **Cause:** a broad `operator: Exists` toleration copied from a template.
+    - **Where to look:** pod spec tolerations.
+- **Symptom:** taint added but evictions trickle slowly.
+    - **Cause:** the taint-eviction controller rate-limits to avoid stampedes on wide failures.
+    - **Where to look:** KCM eviction rate settings and logs.
 
 ## Questions