Ledger (123) + AMR fold: the level>=2 freg faces ride the restrict-parent wave, s_amr_freg_wave deleted (one rendezvous per level in the fold; byte-identical; GOAL v7 item 2b step 1); timing read deferred to healthy nodes

Author: sbryngelson

docs/documentation/amr_action_plan.md

@@ -226,6 +226,18 @@ possible while AMR aborts on the target machine at 1 rank, and every increment b
 on a compiler that does not reproduce it. It also means the ladder should add a CCE arm as soon as one
 exists, or the same class of breakage will keep accumulating undetected.
 
+## 2026-09-09 (123) — THE LEVEL-2 FLUX-REGISTER WAVE RIDES THE RESTRICT-PARENT WAVE (GOAL v7 item 2b, step 1): the post-stage fold ran three sequential rendezvous per level-2 family -- the children's freg faces to the parent owner (s_amr_freg_wave, its own IRECV/ISEND set and WAITALL), then the restricted box to the same owner (s_amr_restrict_parent_wave), then the level-1 scatter -- and the per-rank timeline priced the fold at 284-358 ms of a 3.1 s np8 step with waits of ~15 / 110-140 / 45-70 ms at the three links; the faces now ride each restrict-parent transfer as [hdr | box | faces with sibling-seam weight > 0], the standalone wave is deleted (net -102 lines), one rendezvous per level remains before the scatter, and the change is byte-identical at np8 on the S0 deck where 7 of 8 level-2 children have a remote parent -- landed on the gates, with its np8/np16 reading deferred to healthy nodes for the same reason as ledger 122
+
+**Pre-registration (notes/ledger_drafts/l123_prereg.md).** Predictions (1)-(2) are timing reads (rs:wave -> 0, restr wait -P1's share, fold -10 to -15 %, np8 -1 to -2 %, np16 -2 to -3 %) and are NOT READ yet: the session node's GPU fault voids every long arm taken there since 03:00; the rung on this pin (np16_rung_fold.sbatch) is queued and the np8 pair follows on the hold node. The falsifier (restr wait unchanged because the P2 WAITALL absorbs P1's wait) is what the timeline will decide.
+
+**What the change is.** src/simulation/m_amr.fpp: per-transfer face word counts amr_fw_sfc/amr_fw_rfc; s_amr_freg_words (both sides derive the count from the sibling-seam weights on replicated metadata), s_amr_freg_pack_host (device -> host pull of the shipped faces of the child's register slot, copied into the wire in array element order, the order the standalone ISENDs carried) and s_amr_freg_unpack_host (the mirror, then the device push); s_amr_restrict_parent_wave calls s_amr_reg_prepare, adds the faces to each transfer's offset (amr_fw_pq += cnt + fcnt), packs after the box and unpacks after it; s_amr_freg_wave deleted with its export. src/simulation/m_time_steppers.fpp: the standalone wave call and its PH_RSWAVE bracket removed (the rs:wave row is now 0 by construction). The reflux applies keep their descending union walk; the finest-first level order is unchanged; the debug NaN poisoning of skipped faces (a debug aid of the deleted wave) is dropped.
+
+**Gates.** CPU AMR goldens 71/71 TOUCHED=0 (on 6d6e7697); ident2 and the GPU subset on e8e2486c (one comment line apart); the landed commit (110c1832, the formatter's line breaks aside) differs from e8e2486c only by four comment lines that named the deleted wave (m_amr.fpp, m_phase_timing.fpp, m_amr_xchg_audit.fpp), no code. ident2 e8e2486c vs 96966782 (ledger 125's pin) on the S0 deck np8: lustre_60.dat and lustre_amr_60.dat IDENTICAL -- ownership has no parent affinity, so a level-2 child's parent is remote with probability 1 - 1/8 on this deck (by the ownership design, not counted in a log), and the cross-rank merged path is exercised, closing the coverage gap the design read found (no lock-step golden splits a tower). GPU subset (6 dynamic-regrid tests incl. np2/np8 and multi-level): 6/6, TOUCHED=0.
+
+**Rendezvous count.** Per step (3 stages, level 2): rendezvous 21 -> 20 (the freg wave was one per step, not per stage; the 21 counts each stage's two base-grid halos as one rendezvous each); blocking calls 51 = 36 base-grid SENDRECVs + 15 WAITALLs -> with ledger 122's 36 -> 18 and this wave's 15 -> 14, 32.
+
+**Deferred, stated per GOAL v7 item 6:** step 2 (post the L1 -> L0 scatter's IRECVs at the top of the fold: needs per-band tag counters and a disjoint recv pool) and step 3 (pack + ISEND per batch after its stage-3 RK); the freg faces are still host-staged (the parent wave is host-staged too); the lock-step split-tower golden.
+
 ## 2026-09-09 (122) — THE TWO BASE-GRID HALOS ARE ONE (GOAL v7 item 2a, first rendezvous cut): the coarse RHS exchanged PRIM ghosts inside s_compute_rhs and the AMR fill exchanged CONS ghosts of the same stage-entry state on the same faces and peers one call later, 36 SENDRECVs per step; the cons halo is now hoisted before the coarse RHS and the RHS converts cons -> prim over the buffered domain and skips the prim MPI faces (skip_mpi), 18 SENDRECVs per step, byte-identical (pointwise conversion) -- the gates pass on both lanes and the change is landed, while its np8/np16 reading is deferred to healthy nodes because every 240-step arm on the session node since ~03:00 ran 25-55 % slow under an amdgpu kiq-fence fault (amdgpu kiq-fence timeouts in the node's kernel log, 26-60 per hour between 03:00 and 06:00; recorded in the handoff memory), so the one np8 arm taken there (logs/np8pair-halomerge2-409710, job 409710 on the faulted k004-003, against the i7 rung's np8 arm of job 409528 on k004-001, i.e. different jobs and nodes) is reported only for the row that changed by construction: the b:halo wait 44 -> 3 s per 240 steps and its call count 4320 -> 360 (the prim exchange now runs only in the 20 pre-regrid steps), while the cons-halo wait rose 20 -> 80 s -- the first rendezvous of the stage still absorbs the previous stage's skew, exactly as the pre-registration's falsifier described; every other row of that arm, including coarse 74 -> 41 s, is confounded by the node
 
 **Pre-registration (notes/ledger_drafts/l122_prereg.md).** (1) PH_HALO row 102 -> < 20 ms/step at np8: NOT MET as stated -- the hoisted exchange is still bracketed PH_HALO and now carries the wait the prim exchange used to carry (the prediction named the wrong row); the row that vanished is b:halo (201 -> ~14 ms/step). (2) b:halo does not grow: met trivially (it collapsed); the honest reading of the falsifier "the wait merely moved" is YES -- the skew moved from b:halo to halo, which is what a rendezvous count cut does when the segment imbalance is unchanged; the count is 36 -> 18 SENDRECVs and the cost is in the rows that remain. (3) step deltas: NOT READ (node fault; the np8 arm's wall, 1139 s against 737 s, is void). (4) identity and TOUCHED=0: met.