Skip to content

core/amqp_leader: fix cluster cron never starting - #72

Merged
noahmehl merged 2 commits into
4.4from
fix/cluster-cron-startup
Oct 11, 2026
Merged

noahmehl merged 2 commits into
4.4from
fix/cluster-cron-startup

Conversation

@noahmehl

Copy link
Copy Markdown
Contributor

Fixes #70

Context

amqp_cron has never run a task on any node. Not "starts late" or "logs noise" — the gen_server is dead from boot and is never restarted. Two defects are responsible, and the second is only reachable once the first is fixed, which is why it stayed hidden.

Rationale

1. amqp_leader did not declare kazoo_globals.

amqp_leader_listener:ready_when_node_is_up/0 gates leader startup on kz_nodes:is_up/1, and init/1 calls kz_nodes:notify_expire/0. Both live in kazoo_globals, which was absent from amqp_leader.app.src. The release boot script therefore started amqp_leader and amqp_cron first, so the kz_nodes ETS table did not exist and ets:match/2 raised badarg where the retry loop expected [].

Declaring the dependency reorders the boot script to kazoo_globals (32) → amqp_leader (33) → amqp_cron (34). No cycle is introduced: nothing in kazoo_globals' transitive dependency set reaches amqp_leader.

2. amqp_cron exported handle_info/2 where the harness calls handle_info/3.

amqp_leader_proc:exec_callback/2 appends [ModState, Leader] to every callback's arguments. amqp_cron's export block is headed %% gen_leader callbacks and honours that for handle_cast/3, handle_call/4, handle_DOWN/3 and from_leader/3 — but not handle_info. The body is a no-op and stays one.

3. kz_nodes:is_up/1 is now total.

It answers false when the registry is not running rather than raising, which is what its only caller has always wanted — a node whose registry has not started is not up in any sense a caller can use. The unused expires = '$2' binding is dropped: it was matched and discarded, implying a recency check that is not performed. The function has exactly one caller in the tree, so the contract change is contained.

A CONTEXT.md commit adds the vocabulary this layer was missing — Node, Node registry, Leader, Cluster cron. The glossary had no name for the relationship between amqp_leader and the node registry, which is plausibly why the dependency went undeclared for so long.

Test evidence

Before, on a freshly booted dev node:

{cron_pid, undefined}
{leader_children, [{amqp_cron, undefined, supervisor, [amqp_leader_proc_sup]}]}
{kz_nodes_ets, #Ref<...>}   %% registry is up by now
{is_up, true}               %% the check would have succeeded

After:

{cron_pid, <0.1184.0>}
{leader_children, [{amqp_cron, <0.1177.0>, supervisor, [amqp_leader_proc_sup]}]}
{status, [{leader, 'kazoo_apps@...'}, {alive, ['kazoo_apps@...']},
          {down, []}, {candidates, ['kazoo_apps@...']}]}

Boot log contains zero ets:match badargs, zero reached_max_restart_intensity, and zero handle_info crashes. The pid was stable across ~10 minutes of observation; the intermediate crash loop (after fix 1, before fix 2) churned it every 3 seconds, so a stable pid is a meaningful signal here.

is_up/1 was also checked directly: is_up(node()) → true, is_up(nonexistent@nowhere) → false, no raise.

Not addressed here

The reason this failure was silent — amqp_leader_proc_sup is a transient child exiting with reason shutdown, which OTP classifies as normal and never restarts — is filed separately as #71. Changing it would have turned this bug into an unbounded restart storm across the whole leader layer, so it needs its own thinking rather than a ride-along.

amqp_cron has never started a task on any node. Two defects, the second
masked by the first.

1. amqp_leader declared no dependency on kazoo_globals, despite
   amqp_leader_listener gating readiness on kz_nodes:is_up/1. The release
   boot script therefore started amqp_leader and amqp_cron *before*
   kazoo_globals, so the kz_nodes ETS table did not exist yet and
   ets:match/2 raised badarg instead of returning []. The retry loop in
   ready_when_node_is_up/0 only handles is_up -> false, so the badarg
   escaped, burned amqp_leader_proc_sup's restart budget in ~5ms and left
   it dead: transient children are not restarted after a 'shutdown' exit.

   Declaring the dependency fixes the boot order (kazoo_globals 32 ->
   amqp_leader 33 -> amqp_cron 34). No cycle: nothing kazoo_globals
   depends on reaches amqp_leader.

2. With the gate cleared, amqp_cron reached its first AMQP message and
   died on "undefined function amqp_cron:handle_info/3".
   amqp_leader_proc:exec_callback/2 appends [ModState, Leader] to every
   callback's arguments. amqp_cron honours that for handle_cast/3,
   handle_call/4, handle_DOWN/3 and from_leader/3, but exported
   handle_info/2.

Also make kz_nodes:is_up/1 total: it now answers false when the registry
is not running rather than raising, which is what its one caller has
always wanted. The unused expires = '$2' binding is dropped -- it was
matched and discarded, implying a recency check that is not performed.

Verified on a dev node: amqp_cron:status/0 now reports an elected leader,
the pid is stable across minutes, and the boot log contains no ets:match
badarg, no reached_max_restart_intensity and no handle_info crash.
The glossary had no vocabulary for the layer this bug lives in: no term
for the node registry, for leader election, or for cluster-wide scheduled
work. That gap is plausibly why amqp_leader's dependency on kazoo_globals
went undeclared for so long -- the relationship had no name.

Adds Node, Node registry, Leader and Cluster cron. The Node registry
entry records the semantics kz_nodes actually implements: membership is
liveness, pruned by the expiry sweep, with no separate recency check.

@rdnn rdnn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@noahmehl
noahmehl added this pull request to the merge queue Oct 11, 2026
Merged via the queue into 4.4 with commit 2ef1651 Oct 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cluster cron has never started: amqp_leader missing kazoo_globals dependency, then amqp_cron:handle_info wrong arity

2 participants