AMR batching default: amr_device_pack does not ride along (its cap-32/96 A/B is owed first); the unit test's initial block uses cell indices

Author: sbryngelson

docs/documentation/case.md

@@ -734,7 +734,7 @@ To restart the simulation from $k$-th time step, see @ref running "Restarting Ca
 | `amr_subcycle`          | Logical | Advance the coarse level at the case dt and the fine level at dt/2 (two substeps; Berger-Colella refluxing). Requires `amr`; incompatible with `cfl_dt`. |
 | `amr_device_pack`       | Logical | Pack and unpack the per-stage coarse-patch gather (F1/F2) over the plan's flat transfer list instead of one launch per transfer. The sends fuse to one kernel per family per stage; the receives fuse per contiguous (box, peer) run, so measured at np=8 the pack dispatches fall about 89x and the unpack about 4.3x. Wire bytes and floating-point values are unchanged. Requires `amr`; incompatible with `amr_subcycle`; the non-polytropic QBMM pb/mv twin keeps its per-transfer path. Default F. |
 | `amr_batched_gather`    | Logical | Consume the per-stage coarse-patch gather (F1/F2) for all of a rank's blocks at once: the gathered patches live in one pool, and each wave runs one fused own-copy, one fused unpack and one batched ghost fill instead of one set of launches per block. Wire bytes and floating-point values are unchanged. Requires `amr` and `amr_device_pack`; incompatible with `amr_subcycle`. Default F. |
-| `amr_batched_advance`   | Logical | Advance owned fine blocks of equal level and extent in batches of up to 8, stacked two ghost shells apart along the last active dimension, in one RHS call per batch. Requires `amr`; lock-step, Cartesian, uniform grid only; incompatible with the per-block fine-advance hooks (relaxation, IB, QBMM, IGR, chemistry, hypoelasticity, bubbles, MHD, relativity, damage, surface tension) and with Riemann-extrapolation BCs under `null_weights`; requires `amr_max_grid_size` > 0. Bit-identical to the per-block advance on a grid whose cell spacing is bitwise uniform (stacked blocks share the batch leader's coordinate arrays); roundoff-level differences otherwise, announced once at startup. Default F. Left unset on an `amr` case, the toolchain turns it on (with `amr_device_pack` and `amr_bat_pad` = 0.1) whenever these rules admit it; set `amr_batched_advance = F` to force the per-block advance. |
+| `amr_batched_advance`   | Logical | Advance owned fine blocks of equal level and extent in batches of up to 8, stacked two ghost shells apart along the last active dimension, in one RHS call per batch. Requires `amr`; lock-step, Cartesian, uniform grid only; incompatible with the per-block fine-advance hooks (relaxation, IB, QBMM, IGR, chemistry, hypoelasticity, bubbles, MHD, relativity, damage, surface tension) and with Riemann-extrapolation BCs under `null_weights`; requires `amr_max_grid_size` > 0. Bit-identical to the per-block advance on a grid whose cell spacing is bitwise uniform (stacked blocks share the batch leader's coordinate arrays); roundoff-level differences otherwise, announced once at startup. Default F. Left unset on an `amr` case, the toolchain turns it on (with `amr_bat_pad` = 0.1) whenever these rules admit it; set `amr_batched_advance = F` to force the per-block advance. |
 | `amr_max_blocks`       | Integer | Upper bound on the GLOBAL refined-block count. Sizes replicated per-rank METADATA (~11 kB/block); block slots themselves are allocated lazily for blocks a rank owns, so this is not N x device memory. Exceeding it silently truncates the refined region (the clusterer warns). Must be >= 1 (default 1024) |
 | `amr_max_grid_size`    | Integer | Absolute cap on a refined block's coarse-cell extent per dimension, the AMReX max_grid_size concept; must be >= 2 when set (default 0). With 0 the cap is derived from the decomposition and so shrinks as ranks are added, which tiles a fixed feature into more blocks the further you scale and makes the box set depend on the rank count. Setting it pins the cap, so the box set is identical at every rank count. The value may exceed half a rank subdomain: the solver scratch is then sized to the cap rather than to the subdomain, so per-rank memory grows as the cap raised to the number of dimensions |
 | `amr_max_level`        | Integer | Maximum AMR refinement depth (number of refined levels above L0); must be >= 1 (default 1). Multi-level nesting (>= 2) is supported: static AMR (`amr_regrid_int = 0`) nests up to level 2, dynamic regrid (`amr_regrid_int > 0`) nests deeper (see @ref amr_multilevel) |

toolchain/mfc/case_validator.py

@@ -3212,11 +3212,11 @@ def _format_errors(self) -> str:
         return "\n".join(lines)
 
 
-# The batched fine advance and its two companions, turned on for an AMR case that leaves amr_batched_advance unset
+# The batched fine advance and its padding, turned on for an AMR case that leaves amr_batched_advance unset
 # whenever the case admits it. The Fortran defaults stay F on purpose: a default set there bypasses every rule above
 # (ledger 99: ten of 58 AMR cases ran the documented unsupported combinations unguarded), so the decision lives here,
 # under the same prohibitions that guard an explicit amr_batched_advance = T.
-BATCHING_DEFAULTS: Dict[str, Any] = {"amr_batched_advance": "T", "amr_device_pack": "T", "amr_bat_pad": 0.1}
+BATCHING_DEFAULTS: Dict[str, Any] = {"amr_batched_advance": "T", "amr_bat_pad": 0.1}  # amr_device_pack: A/B at cap 32/96 first
 
 
 def apply_batching_default(params: Dict[str, Any]) -> bool:

toolchain/mfc/run/input.py

@@ -124,7 +124,7 @@ def validate_constraints(self, target) -> None:
     # Generate case.fpp & [target.name].inp
     def generate(self, target) -> None:
         if case_validator.apply_batching_default(self.params):
-            cons.print("[dim]amr_batched_advance left unset and admissible: batching on (amr_batched_advance = F forces the per-block advance)[/dim]")
+            cons.print("[dim]amr_batched_advance left unset and admissible: batching on with amr_bat_pad = 0.1 (amr_batched_advance = F forces the per-block advance)[/dim]")
         # Validate constraints before generating input files
         self.validate_constraints(target)
         self.generate_inp(target)

toolchain/mfc/test_case_validator.py

@@ -390,13 +390,13 @@ def test_not_tripped_without_alt_soundspeed(self):
 class TestBatchingDefault(unittest.TestCase):
     """The batched advance turns on by default only where the validator would admit an explicit request."""
 
-    AMR = {**BASE, "amr": "T", "amr_regrid_int": 0, "amr_max_grid_size": 16, "time_stepper": 3, "amr_block_beg(1)": 0.25, "amr_block_end(1)": 0.75}
+    AMR = {**BASE, "amr": "T", "amr_regrid_int": 0, "amr_max_grid_size": 16, "time_stepper": 3, "amr_block_beg(1)": 10, "amr_block_end(1)": 30}
 
     def test_admissible_amr_case_gets_batching(self):
         p = dict(self.AMR)
         self.assertTrue(apply_batching_default(p))
         self.assertEqual(p["amr_batched_advance"], "T")
-        self.assertEqual(p["amr_device_pack"], "T")
+        self.assertNotIn("amr_device_pack", p)
         self.assertEqual(p["amr_bat_pad"], 0.1)
         validate_case_constraints(p, "simulation")