Repository navigation
core/amqp_leader: fix cluster cron never starting - #72
Merged
Merged
Conversation
amqp_cron has never started a task on any node. Two defects, the second masked by the first. 1. amqp_leader declared no dependency on kazoo_globals, despite amqp_leader_listener gating readiness on kz_nodes:is_up/1. The release boot script therefore started amqp_leader and amqp_cron *before* kazoo_globals, so the kz_nodes ETS table did not exist yet and ets:match/2 raised badarg instead of returning []. The retry loop in ready_when_node_is_up/0 only handles is_up -> false, so the badarg escaped, burned amqp_leader_proc_sup's restart budget in ~5ms and left it dead: transient children are not restarted after a 'shutdown' exit. Declaring the dependency fixes the boot order (kazoo_globals 32 -> amqp_leader 33 -> amqp_cron 34). No cycle: nothing kazoo_globals depends on reaches amqp_leader. 2. With the gate cleared, amqp_cron reached its first AMQP message and died on "undefined function amqp_cron:handle_info/3". amqp_leader_proc:exec_callback/2 appends [ModState, Leader] to every callback's arguments. amqp_cron honours that for handle_cast/3, handle_call/4, handle_DOWN/3 and from_leader/3, but exported handle_info/2. Also make kz_nodes:is_up/1 total: it now answers false when the registry is not running rather than raising, which is what its one caller has always wanted. The unused expires = '$2' binding is dropped -- it was matched and discarded, implying a recency check that is not performed. Verified on a dev node: amqp_cron:status/0 now reports an elected leader, the pid is stable across minutes, and the boot log contains no ets:match badarg, no reached_max_restart_intensity and no handle_info crash.
The glossary had no vocabulary for the layer this bug lives in: no term for the node registry, for leader election, or for cluster-wide scheduled work. That gap is plausibly why amqp_leader's dependency on kazoo_globals went undeclared for so long -- the relationship had no name. Adds Node, Node registry, Leader and Cluster cron. The Node registry entry records the semantics kz_nodes actually implements: membership is liveness, pruned by the expiry sweep, with no separate recency check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #70
Context
amqp_cronhas never run a task on any node. Not "starts late" or "logs noise" — the gen_server is dead from boot and is never restarted. Two defects are responsible, and the second is only reachable once the first is fixed, which is why it stayed hidden.Rationale
1.
amqp_leaderdid not declarekazoo_globals.amqp_leader_listener:ready_when_node_is_up/0gates leader startup onkz_nodes:is_up/1, andinit/1callskz_nodes:notify_expire/0. Both live inkazoo_globals, which was absent fromamqp_leader.app.src. The release boot script therefore startedamqp_leaderandamqp_cronfirst, so thekz_nodesETS table did not exist andets:match/2raisedbadargwhere the retry loop expected[].Declaring the dependency reorders the boot script to
kazoo_globals(32) →amqp_leader(33) →amqp_cron(34). No cycle is introduced: nothing inkazoo_globals' transitive dependency set reachesamqp_leader.2.
amqp_cronexportedhandle_info/2where the harness callshandle_info/3.amqp_leader_proc:exec_callback/2appends[ModState, Leader]to every callback's arguments.amqp_cron's export block is headed%% gen_leader callbacksand honours that forhandle_cast/3,handle_call/4,handle_DOWN/3andfrom_leader/3— but nothandle_info. The body is a no-op and stays one.3.
kz_nodes:is_up/1is now total.It answers
falsewhen the registry is not running rather than raising, which is what its only caller has always wanted — a node whose registry has not started is not up in any sense a caller can use. The unusedexpires = '$2'binding is dropped: it was matched and discarded, implying a recency check that is not performed. The function has exactly one caller in the tree, so the contract change is contained.A
CONTEXT.mdcommit adds the vocabulary this layer was missing — Node, Node registry, Leader, Cluster cron. The glossary had no name for the relationship betweenamqp_leaderand the node registry, which is plausibly why the dependency went undeclared for so long.Test evidence
Before, on a freshly booted dev node:
{cron_pid, undefined} {leader_children, [{amqp_cron, undefined, supervisor, [amqp_leader_proc_sup]}]} {kz_nodes_ets, #Ref<...>} %% registry is up by now {is_up, true} %% the check would have succeededAfter:
{cron_pid, <0.1184.0>} {leader_children, [{amqp_cron, <0.1177.0>, supervisor, [amqp_leader_proc_sup]}]} {status, [{leader, 'kazoo_apps@...'}, {alive, ['kazoo_apps@...']}, {down, []}, {candidates, ['kazoo_apps@...']}]}Boot log contains zero
ets:matchbadargs, zeroreached_max_restart_intensity, and zerohandle_infocrashes. The pid was stable across ~10 minutes of observation; the intermediate crash loop (after fix 1, before fix 2) churned it every 3 seconds, so a stable pid is a meaningful signal here.is_up/1was also checked directly:is_up(node())→true,is_up(nonexistent@nowhere)→false, no raise.Not addressed here
The reason this failure was silent —
amqp_leader_proc_supis atransientchild exiting with reasonshutdown, which OTP classifies as normal and never restarts — is filed separately as #71. Changing it would have turned this bug into an unbounded restart storm across the whole leader layer, so it needs its own thinking rather than a ride-along.