commit-proxy-envs-changes.yml: leaked containerd task causes a permanent reconcile loop that restarts containerd+kubelet fleet-wide

[Danielss89]

commit-proxy-envs-changes.yml: a leaked containerd task causes a permanent reconcile loop, and its retries restart containerd+kubelet fleet-wide

Claudie: v0.14.1 (manager, ansibler, kube-eleven)
Cluster: k8s v1.34.0, containerd 1.7.29, Ubuntu 24.04.4 LTS, kernel 6.8.0-106-generic
Topology: 3 dynamic control-plane nodes (Hetzner Cloud) + 4 static worker nodes (Hetzner dedicated), Cilium with kube-proxy replacement

Summary

A single leaked containerd task on one control-plane node put our cluster into a reconcile loop that
could not clear itself. Every ~65 minutes Claudie retried proxy/commit-proxy-envs-changes.yml; each
attempt restarted containerd and kubelet on every node five times in ~4 minutes before reaching
the step that fails. That was long enough to drop every pod to NotReady, which stripped all
Service endpoints and took down every workload in the cluster for ~4 minutes, once an hour.

It ran for 4 consecutive cycles (~3.5h) until we cleared the stale task by hand. It would not have
stopped on its own — there is no backoff and no terminal state.

Throughout, the InputManifest reported WATCHING_FOR_CHANGES.

The failing step

The play removes the kube-controller-manager sandbox with:

crictl pods | grep kube-controller-manager | awk '{print $1}' \
  | xargs -I {} sh -c 'crictl stopp {} && crictl rmp {}'

One node had an old sandbox whose container was CRI-Exited but whose containerd task was still
registered as STOPPED and never reaped — the task's PID no longer existed on the host. crictl rmp
therefore failed:

E0825 19:39:49.425192 log.go:32] "RemovePodSandbox from runtime service failed"
  err="rpc error: code = FailedPrecondition desc = failed to remove container \"3a87a5c0...\":
  failed to delete containerd container \"3a87a5c0...\":
  cannot delete running task 3a87a5c0...: failed precondition"
  podSandboxID="dffd15174c7b9"

xargs exits 123, the task fails (not ignored), and the play aborts on that host with exit status 2:

ansible-playbook ../../ansible-playbooks/proxy/commit-proxy-envs-changes.yml -i inventory.ini -f 32 : exit status 2

Recap — the affected host aborts early while the others complete:

ctrl-fsn-...  : ok=4   changed=2  unreachable=0  failed=1  skipped=1  rescued=0  ignored=0
ctrl-hel-...  : ok=10  changed=7  unreachable=0  failed=0  skipped=1  rescued=0  ignored=1
ctrl-nbg-...  : ok=10  changed=7  unreachable=0  failed=0  skipped=1  rescued=0  ignored=1

The loop

ERR Task resulted in an error during the current stage: [error while running ansible to update proxy envs ...]
INF One of the clusters failed to build successfully, moving manifest to "Error" state
INF Moving to "Pending" as changes have been made to the manifest since the last build
INF Config has been successfully processed and moved to the "Pending" state
...~65 min later...
INF No task was scheduled for a while, issuing a refresh of the infrastructure

Build outcomes from the manager log:

2026-08-25T16:17:21Z SUCCESS
2026-08-25T17:24:17Z ERROR     <-- stale task appears
2026-08-25T18:32:13Z ERROR
2026-08-25T19:40:11Z ERROR
2026-08-25T20:47:58Z ERROR
                               <-- stale task cleared manually ~20:51Z
2026-08-25T21:53:12Z SUCCESS
... 14 consecutive successes since

Why the retries take down the whole cluster

Before reaching the failing step, the play runs (task names as they appear in ansibler output):

• Daemon reload
• Check if services exist
• Restart services that are active → restarts containerd and kubelet on every host
• restart scheduler

A single containerd restart is short — measured on our nodes at 2–3 seconds, which does not trip
kubelet's readiness probes and causes no visible impact. The problem is that a failing run repeats
it. Each failing cycle produced 5 containerd+kubelet restarts within ~4 minutes on all 7 nodes:

19:19:57, 19:20:31, 19:21:15, 19:22:19, 19:24:10   (+02:00)

That sustained churn does trip the probes:

Readiness probe errored and resulted in unknown state: rpc error: code = Unavailable
  desc = connection error: desc = "transport: Error while dialing:
  dial unix /run/containerd/containerd.sock: connect: no such file or directory"

Every pod goes NotReady → every Service loses all endpoints. With Cilium's socket-LB
(kube-proxy replacement), a connect() to a ClusterIP with zero backends is rejected by the
cgroup/connect4 hook, so applications get an immediate EPERM rather than a timeout. For us
that surfaced as ~20 production sites simultaneously throwing
PDOException: SQLSTATE[HY000] [2002] Operation not permitted on every request, in bursts aligned
to the second with the containerd restarts (~1000 fatals in the worst 3-minute window).

We have also seen one post-fix cycle emit two restarts 34s apart, which was enough to produce a
small burst (~40 fatals). Single-restart cycles produce none.

Our fix (node-side, manual)

ctr -n k8s.io tasks delete --force <container-id>
crictl rmp <sandbox-id>

Confirmed over the following ~16 hours: 14 consecutive successful builds, one containerd restart per
reconcile instead of five, and no recurrence of the stale sandbox on any control-plane node.

Suggested changes

1. Don't let a stale sandbox fail the play. crictl rmp failing on an already-Exited container
   isn't a reason to abort a proxy-env update. Either tolerate the failure, or
   ctr -n k8s.io tasks delete --force the leftover task before rmp. The current
   xargs -I {} sh -c 'crictl stopp {} && crictl rmp {}' also collapses per-sandbox failures into one
   opaque rc=123, which made this slow to diagnose.
2. Add backoff and a terminal state. A build that fails the same way every cycle retries forever
   at full blast radius. Exponential backoff, or halting after N identical failures, would have turned
   a 4-hour outage pattern into a single event.
3. Make the restart conditional on an actual change. Our proxy settings are unset and unchanged,
   yet Restart services that are active reports changed=3 on every single reconcile. If the
   rendered env is identical, this should be a no-op — that alone removes the failure mode.
4. Consider serialising. If a restart is genuinely needed, doing it on all hosts at once (-f 32)
   makes it cluster-wide rather than rolling.
5. Surface the failure. A cluster looping Error → Pending indefinitely still reports
   WATCHING_FOR_CHANGES on the InputManifest, so nothing alerts on it. A repeatedly-failing build
   should be visible in the CR status.
6. Minor: Check if services exist runs systemctl is-active docker, which returns rc=4 on hosts
   without Docker and prints alarming failed: [host] (item=docker) lines on every run. It's ignored,
   but it's noise in every log.

Happy to share fuller ansibler/manager logs privately — just say where.

[Despire]

This issue should be handled before the next release.

The reconciliation loop gets stuck on the crictl pods | grep kube-controller-manager | awk '{print $1}' \ | xargs -I {} sh -c 'crictl stopp {} && crictl rmp {}' command, due to stale pods.

A solution to this would be to make this idempotent and only consider ready containers, via crictl pods --state ready and also to not fail on crictl stopp and continue with the removal

There are two places within claudie where the reported error can be encountered and needs to be fixed:

claudie/services/ansibler/ansible-playbooks/apiEndpointChange.yml

Lines 86 to 90 in 7b7bac9

	- name: restart controller
	shell: "crictl pods | grep kube-controller-manager | awk '{print $1}' | xargs -I {} sh -c 'crictl stopp {} && crictl rmp {}'"
	- name: restart scheduler
	shell: "crictl pods | grep kube-scheduler | awk '{print $1}' | xargs -I {} sh -c 'crictl stopp {} && crictl rmp {}'"

claudie/services/ansibler/ansible-playbooks/proxy/commit-proxy-envs-changes.yml

Lines 34 to 39 in 7b7bac9

	- name: restart controller
	ansible.builtin.shell: |
	crictl pods | grep kube-controller-manager | awk '{print $1}' | xargs -I {} sh -c 'crictl stopp {} && crictl rmp {}'
	- name: restart scheduler
	ansible.builtin.shell: |
	crictl pods | grep kube-scheduler | awk '{print $1}' | xargs -I {} sh -c 'crictl stopp {} && crictl rmp {}'