🎙️ task - fix(keyrack): daemons orphan and never exit, ~230/hour into swap

[ehm-a-seaturtle]

🦫🎙️ dispatch to foreman

💧 task enqueued
   ├─ priority = ?
   ├─ yieldage = ?
   └─ leverage = ?

title
fix(keyrack): daemons orphan and never exit, ~230/hour into swap
description

the defect

each keyrack daemon spawned under a temp home (/tmp/pii-brain-live-*) outlives its spawner. when
the spawner exits, the daemon reparents to pid 1 and stays alive forever, and holds ~55MB of
anon memory each.

anon is the only memory that must swap — file pages get dropped instead — so a pool of orphaned
daemons goes straight into swap and stays there.

measured, 2026-08-30

moment	daemons	note
first sample	191	machine 17.4G into swap
after a prune	39
30 min later	156	refill rate ~230/hour
after a prune	40
8 min later	50	still climbs

an earlier incident on the same machine reached 641 daemons and forced disk-swap thrash.

why it is hard to see

each daemon is merely a node process whose spawner has exited, so:

• it reparents to pid 1, and carries no trace of its origin
• ps aggregation by comm reports "node ×N" — which names the binary, not the culprit
• the daemons appeared in no top-15 RSS listing, even at 191 instances

only cgroup attribution found them: cgroup membership is assigned at fork and survives
reparent, so the spawner's scope still charges them. one scope held 1,699 pids / 5.5G.

the current mitigation, and why it is not enough

a local prune operation removes a daemon when all three hold: temp home, spawner dead, and older
than a min-age gate. that bounds the pool but does not stop the leak — and the gate itself can be
out-run. at a 60m gate it caught 17 of 191, because 152 had spawned within the hour.

suggested fix

the daemon should not outlive the spawner that owns it. options, roughly in order of preference:

1. tie lifetime to the spawner — the daemon exits when its parent does. a
   prctl(PR_SET_PDEATHSIG) on linux, or a parent-liveness poll, closes this at the source.
2. idle timeout — a daemon with no client for N minutes exits on its own. simple, and
   independent of any parent relationship.
3. reuse before spawn — check for a live daemon on the socket path and attach to it rather
   than spawn a second one. the fact that 152 accumulate in an hour suggests each invocation
   spawns fresh rather than reuses.

(1) or (3) address the cause; (2) is a solid backstop either way.

note

filed from a dev-env-setup investigation into machine slowness. the daemon pool was the dominant
memory consumer at the time of the report.