Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 888 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 888 commits into
sbryngelson wants to merge 888 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 62.81% 63.10% +0.28%
==========================================
Files 84 96 +12
Lines 21975 30211 +8236
Branches 3224 4943 +1719
==========================================
+ Hits 13804 19065 +5261
- Misses 5949 7846 +1897
- Partials 2222 3300 +1078 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
sbryngelson
changed the title
Member
Author
|
2 tasks
ffmt detects structure it cannot match and reports it as a warning on stderr while still exiting 0, and format.sh discarded that stderr. The combination meant malformed Fortran formatted clean and linted clean, so the compiler was the first thing that could say no: an unbalanced end do / end block passed both and only failed an 18-minute GPU build. Capture ffmt's stderr, always show it, and treat unmatched structure as fatal. Verified both directions: it fires on a reconstructed broken file and leaves all 303 clean files passing, so it cannot trip on well-formed code.
The level-1 tag ALLGATHERV is W4's largest term: every rank received the GLOBAL tagged-cell list, 92.77 MB per rank per regrid at np8 and doubling with every np-doubling. Delete it. s_amr_union_gtag is replaced by s_amr_local_tags, where each rank scans only its own disjoint interior, and s_amr_cluster gains a reduce mode that ALLREDUCEs one fused per-axis signature per tree node. Integer SUM is exact and order-independent, so every rank obtains a bit-identical signature and makes bit-identical trim, accept and split decisions. One reduction per node suffices for all three uses, which is why this is not three collectives: s_amr_trim_box took the per-axis MIN/MAX of the contained tags, and that IS the first and last nonzero of the signature; its emptiness test IS sum > 0; and s_amr_find_split only ever touched the tag list to build the signature. So s_amr_box_sig / s_amr_trim_from_sig / s_amr_find_split_sig replace the two tag-rescanning routines, and the node makes one pass where it previously made up to num_dims + 1. Sizing the signature on the UNTRIMMED box is what lets the same buffer serve the trim, since trim only shrinks to the tags' own bbox and no tag leaves the box. Per-axis reductions would be WRONG: find_split returns early on the first axis yielding a zero-run split, so a per-axis collective would make the collective COUNT data-dependent and therefore rank-divergent. A rank with no local tags must still walk the tree and enter every reduction, contributing zeros; the early return is now gated on .not. reduce, because returning early would deadlock the ranks that do have tags. Gates: restart output byte-identical at np8, the clustering tree reproduced exactly (lmax 8192, ldepth 31, nodes 66354 in both arms), ntag_bytes 185241240 -> 0, and 69/69 AMR goldens, which cover the 1D/2D, np=1 and multi-level paths a single np8 probe cannot. Adds a ppn=2 AMR+IB case, the suite's only distributed AMR+IB coverage. Measured after: the reduction grows 1.66x per np-doubling against the gather's 2.00x, so W4 is improved but NOT met; S3.2 and S3.3 remain.
… status, increment status The planning docs had drifted from the code in ways that would misdirect the next reader. amr_action_plan.md gains ledger (38) and leads with a CURRENT STATE block so live status precedes 2400 lines of history. amr_endstate.md's invariant table had a today column describing the collective S3.1 deleted; all seven rows are rewritten against verified numbers. amr_plan_based_exchange.md gains a STATUS block: landed I0/I1a/I1b/I2a/I3/I4a/I4b/I5, outstanding I2b/I5b/I6/I7/I8. Two corrections worth stating rather than silently fixing. W5 is not one increment: 19 of 41 AMR p2p call sites still tag per box, F1 retains an unconverted path passing amr_cur as the tag, and the subcycle sites are an explicit deferral to I8 -- so the ~28k-rank tag wall does not clear until I8. And m_amr.fpp claimed that term leaves with I7, which contradicts I7's own boundary that any family left per-box keeps its tables; the comment is corrected and the contradiction recorded. Also W1-W7 -> W1-W8, and W2 marked flat in P and therefore off the scaling critical path.
…ee is rank-local (S3.2a) B0: amr_blocking_factor raises min_child from the algorithmic floor of 2, which is what stops the bisection over-generating. With the floor at 2 and amr_cluster_eff = 0.9 the recursion never converges on its own -- it splits until amr_max_blocks stops it (warning on every regrid at five different caps) and the min-separation merge then collapses the result back. Measured at np8, per regrid: bf=1 gives 38,255 nodes / 594 boxes; bf=8 gives 6,475 / 576; bf=16 gives 1,466 / 576. So 26x fewer tree nodes produce the SAME answer. Default bf=1 is exact identity, min_child = max(2,1) = 2. Deliberately NOT AMReX's blocking factor. AMReX coarsens the TAG LATTICE, which also shrinks its global tag gather; S3.1 already deleted that gather here, and coarsening a rank-local sparse list cannot dedup coarse cells straddling a rank boundary without an extra exchange -- lower-corner ownership would DROP tags rather than dedup them. A size floor is simpler and is the part that actually stops over-generation. S3.2a instrumentation: per tree node, does its box fit inside ONE rank's subdomain? If so that rank holds every tag in the subtree and the subtree needs no communication -- the property S3.2's design rests on. Counted separately for the reducing level-1 path ([amr-scope-r]), because the level-2 forest clusters inside a parent window and is rank-local by construction, which would inflate the fraction. rbytes is split the same way: rbytes_now counts only nodes that actually reduce, rbytes projects the finished design. Result at np8: 7 of 16,383 level-1 nodes per regrid are shared, 99.96 percent rank-local, shr_maxdep 3 = log2(P). It holds at bf=8 (99.7 percent), so it is not a saturation artifact. np16 gives shared 11, shr_maxdep 5, and shared BYTES growing 1.86x per doubling -- which refutes doing the shared phase as plain allreduces over COMM_WORLD and keeps the sparse per-depth scoping in the S3.2 design. Gated by bit-identity: at bf=1 the restart output is byte-identical to the S3.1 binary, the counters being read-only. The 69-test AMR suite passed on S3.1 and was re-running on this build when the node allocation expired; re-run it before relying on this beyond bf=1.
… retract the P^0.73 law Three corrections the plan needed. (1) The two-point 'rbytes ~ P^0.73' law is RETRACTED: the np8/16/32 ladder shows the forest term growing 1.97x/1.99x = exactly P while the cap-fixed level-1 term stays constant, so the exponent climbs 0.73 -> 0.87 toward 1.0. S3.1 is asymptotically O(P), a ~27x COEFFICIENT cut, and W4 is not met. Published on two points, refuted by the third. (2) S3.2's design contract, recorded before any of it is written, including the negative result that depth-fusion ALONE is still O(P) per rank: an allreduce delivers the whole buffer to every rank, so fusing a depth's nodes fixes latency and leaves the slope. The mechanism that works is a sparse per-depth exchange over the shallow shared phase, with the deep phase rank-local and free, and no MPI_Comm_split anywhere. (3) B1, a prerequisite found by reading every use of nacc: the min-separation merge consumes the accepted boxes IN INSERTION ORDER, fusing the first too-close pair and removing by swapping with the last element. So the merged box set is a function of traversal order, and S3.2 -- which completes local subtrees in parallel -- cannot be bit-identical while that holds. B1 canonicalises the merge input by a deterministic key. It is the ONLY planned change that moves goldens, so it lands alone; folding it into S3.2 would confound 'did the scoping break something' with 'did canonicalisation move the boxes'.
… that decides S3.2 The suite finished after the commit that flagged it as pending: 69 passed, 0 failed. Records B0's pricing (26x fewer tree nodes for the same box set, saturation gone at bf >= 4) and the finding that B0 is NOT a W4 lever -- it prunes leaves, which are already rank-local and free, leaving the shared set untouched. Also records why big-machine allocation should be held: the need for the scoping technique is established here by mechanism (the forest term grows exactly with P while the cap-fixed level-1 term is constant), so Frontier would refine where S3.1 breaks rather than whether. The discriminating question is whether shr_maxdep grows +1 or +2 per doubling, and np32 answers it on this machine.
…trumentation The min-separation merge scanned in list order and fused the FIRST too-close pair, so its output was a function of the order boxes were ACCEPTED -- the traversal order. Sort the accepted boxes by Morton of lo (stable insertion sort; accepted boxes are disjoint so their lo corners are distinct) before the merge, and remove a fused box by shifting down rather than swapping with the last entry. The merge now depends only on the box SET, which is the precondition for any scheme where ranks produce boxes in a different order -- both the scoped level-1 clusterer and the level-2 ownership split close with a gather whose arrival order is not deterministic. Gate: 69/69 AMR goldens pass, bit-identical -- no golden regeneration. Note this means the suite does NOT exercise the property: the order-dependence needs ambiguous chains of too-close boxes, which need many boxes (tests carry a handful, the benchmark 594-2405). The evidence B1 is needed is the reading of nacc's three uses, not the suite. Also adds amr_cl_me_nodes_r / amr_cl_me_rb_r ([amr-scope-me]): the shared clustering nodes whose box actually reaches into THIS rank's subdomain, max-reduced across ranks. Print-only and golden-neutral; it prices the sparse per-depth exchange against the allreduce form.
…t' was a cap artifact np32 closes the three-point sweep. gwin_bytes (the level>=2 window ALLGATHERV) is 360/719/1440 MB per rank at np8/16/32 -- ratios 1.997 and 2.001, exactly O(P), and larger than the 185 MB gather S3.1 deleted. Level-1's shared set grows only as O(log P) (7/11/15 nodes per regrid). Records the S3.3 design contract (ownership + Alltoallv + one box gather) and reorders S3.3 ahead of S3.2b. CORRECTION, prompted by auditing the input decks: the decks are correct (period-1 blob lattice, 8/16/32 blobs, constant 200 cells per unit, nboxes 594/1206/2405, restart files 6 vars x cells x 8 B) but amr_max_blocks = 8192 was FIXED across rungs and SATURATED in all three (lmax exactly 8192, capped warning 10/10, nodes = 2*8192-1). So 'level-1 collectives are flat in P' measured the CAP, not the clusterer. Confirmed directly: at bf=4 np8 gives 13,825 nodes and 0 capped warnings, np16 re-saturates at 16,383 with 5. gwin's O(P) survives (it is driven by tagged cells, ~40 MB/regrid at both bf=1 and bf=4); the claim that level 1 is asymptotically harmless does not. The ordering is marked provisional pending the uncapped arm. Also records B0b (amr_blocking_factor default 1 -> 4, because the cap is load-bearing at the shipped default and a private subtree cannot see global nacc), and S3.3c: S3.3a/b do NOT close W1 -- pass 1 still allocates a dense window per parent on every rank and runs an O(parents x global blocks) = O(P^2) scan.
…oint-to-point Every rank was clustering EVERY parent's nesting window. Pass 1 gathered all ranks' tagged cells with one MPI_ALLGATHERV per level -- gwin_bytes, measured at 360/719/1440 MB per rank at np8/16/32, ratios 1.997 and 2.001, i.e. exactly O(P) and larger than the 185 MB tag gather S3.1 deleted. Pass 2 then rebuilt each parent's window with 'do i = 1, ntot_g; if (gkb(i) /= kb) cycle' INSIDE 'do kb = plo, phi', an O(parents x global tags) rescan on every rank -- W1's pass-2 term. S3.3a: assign each parent an owner, powner(kb) = mod(kb - plo, num_procs), a pure function of kb so every rank agrees without communicating; pass 2 processes only its own parents. The global 'nboxes + 1 > amr_max_fine' guard cannot be evaluated per rank once children are produced in parallel, so children are emitted uncapped into a local list and replayed after ONE ALLGATHERV of the child BOXES (7 ints each, ~67 KB against the 144 MB per-cell gather). Each parent has exactly one owner emitting in order, so a stable counting sort by kb reproduces the serial append order exactly -- same box list, same truncation at amr_max_fine, same box_level. S3.3b: bucket the packed pairs by powner(skb(i)) and replace the ALLGATHERV with MPI_ALLTOALL + MPI_ALLTOALLV. Send volume becomes O(this rank's tagged cells), receive volume O(its assigned parents' tags). Pass 2's residual per-parent scan becomes O(parents/P x tags/P). Safe because pass 2 rebuilds a DENSE window and re-extracts in (k,j,i) order, so ctags does not depend on pair arrival order. Gate: 69/69 AMR goldens pass, bit-identical -- including the np=2 multi-level dynamic regrid case, which is the golden that actually exercises the cross-rank nesting path this touches. NOT closed by this: pass 1 still allocates a dense window per parent on every rank and runs 'do ob = 1, amr_num_blocks' inside 'do kb = plo, phi' (O(P^2) per rank). See the S3.3c and W1 root-cause sections in amr_action_plan.md.
…EP, and W5's wall is the tag base Arm B (block cap raised to 65536 so it never binds, three points) overturns two of my own readings, both pure cap artifact. Level-1 tree nodes = global ALLREDUCEs per regrid: 13,825 / 27,537 / 54,961 = 1.992x, 1.996x -- O(P), not the flat 16,383 reported earlier (16,383 = 2*8192-1, a full binary tree pinned by amr_max_blocks and identical in every rung by construction). Level-1 shared nodes: 11 / 23 / 47 -- O(P), not the '+4 per doubling, O(log P)' read off cap-pinned trees. So S3.2b removes an O(P) collective count, not a latency constant, and ledger 41's reordering premise (forest O(P) vs level-1 flat) is dead -- S3.3 was still right to land first, on absolute size and readiness. W1 sized from the code: 44 'do = 1, amr_num_blocks' sites across 22 routines, and the costly ones are on the RK STAGE path (m_time_steppers calls stage_fill/parent_fill/reflux_faces/freg waves), ~16 scans per stage = ~48 per step. Each walks the whole machine's block list to find this rank's ~75: ~360 M evaluations per rank per step at 1e5 ranks. Root cause is that amr_region_lo_all/hi_all/owns_all/block_level are all allocated to amr_max_blocks, so every rank holds every block (~240 MB/rank at 1e5) -- W1 cannot be closed by restructuring loops. Records W1a (owned-block index list, 10 sites grep-verified convertible) and W1b (distribute the metadata), plus a free deletion: reflux_faces_wave and freg_wave each rescan the global list purely to re-derive an order already stored in amr_fw_rblk. W5 scoped from the tag audit: 24 of 57 p2p calls are already on the (family, epoch) form; the per-object stragglers are concentrated in the L0 tile family and the chunked gather. But the binding constraint is amr_tag_base(f) = amr_max_blocks + 100*f -- every family's base carries a global-block offset, so the MPI_TAG_UB assert caps ~28k ranks even for converted families. Only dropping that term removes the wall, and its precondition is that no family uses per-box tags.
…ock list The RK stage path ran ~48 full scans of the GLOBAL block list per step (~16 per stage under RK3, from s_amr_stage_fill_wave, s_amr_parent_fill_wave, s_amr_restrict_l1_wave, s_amr_restrict_parent_wave, s_amr_reflux_faces_wave, s_amr_freg_wave and s_amr_convert_prim_batch), each walking every block in the machine to find this rank's ~75. At 1e5 ranks with the measured 75 blocks/rank that is ~360 M predicate evaluations per rank per step, linear in P while the real work is fixed -- the direct W1 violation. Six of those loops now iterate amr_my_blk, this rank's own block indices in ascending order, which visits the same set in the same order (they only skip blocks the scan would have cycled) so the paired MPI_SENDRECVs stay matched. The list is refreshed LAZILY on a dirty flag, not rebuilt in s_amr_assign_block_owners. The first version did the latter, on my assumption that the assigner is the sole authority for ownership. It is not: a tiled level-2 block inherits its parent's owner (amr_block_owner(L2) = amr_block_owner(par)) and the restart and migration paths assign directly. The list went stale and the tiled-L2 multi-level dynamic-regrid golden -- the only test that exercises the inherit path -- failed. amr_myblk_dirty is now set at all four owner writes and the list rebuilds on first use, which is robust to a fifth writer appearing later. Gate: 69/69 AMR goldens pass, bit-identical. Left unconverted: the five sites where s_amr_select_slot(k) runs BEFORE the owner filter, since that call has a side effect on amr_cur and skipping it is not provably equivalent.
… one The module comment states the tag ceiling caps global blocks near 2**21 ~ 2.1e6, about 28k ranks at 75 blocks/rank, and the plan has treated that as the second scaling wall. Measured on hpcfund Open MPI 4.1.8: MPI_TAG_UB = 2147483647 (2**31 - 1), which admits amr_max_blocks up to ~2.1e9, about 28 MILLION ranks. The 2**21 figure is an assumption the code carried as fact. Records the probe and pends the same measurement on Frontier's Cray MPICH: if it also reports >= ~2**25, W5 leaves the critical path and the demonstration does not wait on it. Also records two user answers that change the plan: subcycling IS required for the demonstration, which un-defers the I8 subcycle sites (checker-gated at np>1 today) and couples them to W5 whichever way the tag probe lands; and old checkpoints MUST stay readable, so the restart format fix keeps a legacy reader path -- version-stamp the new layout, read both, write only the new. The restart item stays first regardless: it is the only one whose failure mode is silent corruption of saved state rather than a crash.
…ANK vector The parallel_io checkpoint wrote, for EVERY block, a 3*num_procs integer extent vector -- and read it back the same way. That made the FILE FORMAT O(global blocks x ranks): 3 x 75000 x 8192 x 4 B = 7.4 GB of header per checkpoint at Frontier scale, with the matching allocation on every rank in both directions. The array is almost entirely zeros, because only the OWNER of a block has a nonzero extent triple; the information content is (owner, m, n, p) per block. v2 stores exactly that, in 4 ints, so the header is O(blocks): 128 KB where v1 wrote 7.4 GB. The benchmark restarts from file every run but never WRITES one at scale, and the two restart goldens run at np<=2, so nothing in the ledger touched this. It would have appeared for the first time on Frontier, on the first checkpoint. Versioning without changing the 3-int global header: v2 writes a NEGATIVE rank count. A v1 reader meeting a v2 file hits its existing rank-mismatch error rather than misparsing block records, and the v2 reader keeps the full v1 path, so old checkpoints stay readable. owner is stored +1 because the record is recovered with a MAX reduction and rank 0 would otherwise be indistinguishable from an unwritten slot. The v1 same-rank-count condition on the consistency check is preserved exactly, so repartition-on-restart behaves as before. Gates: 69/69 AMR goldens; an np=8 write-then-read roundtrip (the goldens run at np<=2, where a layout indexed by rank count barely differs from one that is not); a v1 checkpoint written before this change read back by the v2 binary; and header inspection confirming 8 vs -8. The size win is invisible at np=8 (5.7 KB of 1.5 GB) -- the term removed is O(blocks x RANKS), so this is justified asymptotically, like S3.3.
… deck at WENO3 (buff_size 3) has the same steady excess as at WENO5 (0.95-1.07 vs 0.94-1.04 s/step, two reps each, same node and hour) and the fill rows are flat, closing ledger 107's open question from MFC's side
…mr_snap coarse cells per face of a live block of the same level takes the live block's box, applied all-or-none under same-level disjointness and proper nesting. The [amr-keep] probe showed 6 of 10 rebuilds on the S0 deck re-create every block because the tag envelope drifted one cell and every tile shifted with it; snapped boxes are identical, and when every box snaps the existing boxes_unchanged fast path skips the rebuild. Validator bounds amr_snap <= amr_buf - 2 so two cells of tag padding survive every snap; the cadence audit is the runtime coverage check. On the S0 deck at np8, amr_snap = 2 turns 7 of 11 rebuilds into no-ops and the differenced step falls 15-18% with every skew wait roughly halved (ledger 109).
…within amr_snap cells per face of a live same-level block takes its box, so a feature drifting a cell no longer re-creates every block; on the S0 deck at np8 7 of 10 rebuilds become no-ops, the differenced step falls 15.5-17.7% with regrid -55% and every skew wait roughly halved, coverage audit escaped 0, fields differ from snap 0 by 1e-6 relative; goldens 71/71 on both lanes with the flag off; two-code rerun with the flag pending
…hing default under dynamic regrid (amr_regrid_int > 0, amr_buf >= 3); an explicit amr_snap is never overridden. Ledger 109 measured the two-cell snap at -15.5 to -17.7% per step on the S0 deck with the coverage audit at zero.
…odes, one node, one window: MFC 0.72 s/step (sd 0.10) vs AMReX 0.38, 1.90x over three reps and 1.79x over the two clean reps (my own CPU golden suite overlapped rep 3), per rep 1.61/1.96/2.13x: the 2x target is straddled, not met, by the pre-registration's own condition; caps 32/64/96 null/-16%/-19%; amr_snap = min(2, amr_buf - 2) becomes the toolchain default under dynamic regrid (goldens 71/71 on both lanes, none touched)
… of state (Mie-Gruneisen, JWL), CI fixes (Phoenix syscheck --bind-to none; faulted-node identification). Conflicts resolved in the three Riemann solvers (upstream's speed-of-sound signature, the 6-equation isentrope flux and its five new private variables carried into our opt-in kernels), m_variables_conversion's export list (both sides), m_start_up's finalize order (upstream's pressure-relaxation report before the AMR finalize block) and case.py's generated header (both case-optimization sets).
…ns of state, Phoenix syscheck bind fix, faulted-node CI identification); six conflicts carried both sides; full CPU suite 789/789 and GPU AMR goldens 71/71 with none touched
…t is the link, not the transport mode or the traffic: per-rank messages and bytes flat np8->np16, MPI wait 436->1007 s per 240 steps on the same bytes, and rdma_mpi on/off identical at np16 (163.7/163.1/164.5 s A/B/A); levers handed to the ladder (bytes across the boundary, placement, overlap); rerun the rung with amr_snap on first
…c to repeated launches -- five fresh 240-step arms back to back sit in a 1.3% band with temperatures, clocks, VRAM and the store flat -- but ledger 111's +3.4%/+5.3% climb is 6-10 of these standard deviations with no co-tenant on its AMR arms, so it stays unexplained with the interleaving of arms as the surviving candidate; item 4's clean rerun will run each code's arms consecutively
…x per doubling against the 1.20x bar, from 1.59x flag-off; the np16 step -22% with five sixths of it MPI wait on unchanged step-family bytes (migration traffic -58%, rebuilds 7 -> 4), np8 -2% across nodes; fewer rebuilds remove a third of the cross-node wait, correcting ledger 113's 'it is the link' to the link carrying rebuild-fed wait; rung with all shipped defaults queued
…ow, each code's arms consecutive: MFC 0.70 s/step (sd 0.08) vs AMReX 0.36, 1.94x -- under 2.0 at the mean on a 1.7-2.3x band, met by this run's pre-registered threshold and not by ledger 111's stricter AMReX clause; the consecutive MFC arms did not climb
…s/step and np16 6.48 -> 4.31, doubling 1.38x (snap-only 1.26x, flag-off 1.59x): the falsifier fired, batching takes proportionally more off the node-local step; gather, seam, halo and the base-grid solve carry 69% of the doubling's growth
…sured per-arm floors (ledger 120)
…ack launches per short arm put the 40-step AMR arm at sd 3.1 %, the uniform 20/60 arms at 3.8 %/2.8 %; a differenced AMR step carries 0.7 %, a differenced uniform step 5.2 %, the excess 0.050 s/step (7-8 %), with the uniform pair 96 % of that variance; the 4.96 % figure is retired at every citation
…e budget the pools stay on the host and each column is staged through one device scratch column. Fixes the device OOM at migration-heavy rebuilds (every 240-step rung arm since the device wire landed); wire bytes and order unchanged
… 2 GiB (host pools plus one device scratch column above it) -- the device-resident wire of ledger 108 pushed every 240-step rung arm past 64 GB/GCD at migration-heavy rebuilds; the fixed binary runs the arm with the pre-wire pin's 53 GiB peak; wire bytes and order unchanged, byte-identical on the S0 deck
…r the buffered domain instead of exchanging prim ghosts a second time (36 -> 18 base-grid SENDRECVs per step; byte-identical; GOAL v7 item 2a)
…nd the RHS converts over the buffered domain instead of exchanging prim ghosts again -- 36 -> 18 base-grid SENDRECVs per step, byte-identical (GOAL v7 item 2a); timing read deferred to healthy nodes
…rent wave, s_amr_freg_wave deleted (one rendezvous per level in the fold; byte-identical; GOAL v7 item 2b step 1); timing read deferred to healthy nodes
…over owners) instead of ascending box id, so a gather chunk carries every rank's boxes; chunk-local indices, the same-chunk parent test and the early-free move to walk positions; ordering only, byte-identical (GOAL v7 item 3a)
…l so a gather chunk carries every rank's boxes instead of one rank's; chunk-local indices, the same-chunk parent test and the early-free move to walk positions; ordering only, byte-identical in two stages (GOAL v7 item 3a); timing read deferred to healthy nodes
…nteriors only) with private pools, drain after the parent fills where the seam stood; byte-identical, one rendezvous fewer in the fill chain (GOAL v7 item 2b, second half)
…te pools, drain after the parent fills where the seam stood; byte-identical (GOAL v7 item 2b, second half); timing read deferred to healthy nodes
Lines of Code
|
… is a persistent rebuild-free skew plus one observed post-rebuild spike (2.5-4x for one interval; at np8 an exchange-side wait with no rhs skew); GOAL v6 item 3 (level-2 overlap copy) deferred, a stated departure from the pre-registered rule
… on the rung deck and is not byte-identical (few-ULP owner dependence in the stage-1 advance); negative result, not landed
…ld interleave (interleave -6.6 %, halo merge -7.5 %, fold merge +1.3 %; the seam-post union read unverified; afternoon repetitions void on a node with intermittent GPU stalls; pre-wire control 2.949 s/step)
… one window: MFC AMR excess 0.58 s/step vs AMReX 0.45 (1.30x; ledger 117: 1.94x); half wait, a third thin work, 0.09 residual per-block RHS inflation; the batching's fragmentation is the pre-registered lever (pad tolerance falsified, largest-first order identity-gated, wall read pending on the EOS pin)
…33x, the schedule increments do not move weak scaling); item 2(c) and item 3(c)/ledger 121 closed by written decision; the np32/np48 rungs void on an unexplained scale-dependent NaN with the mesh never refining, not node health; np64 cannot be placed on this machine
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.