AMR batched advance: apply the static-body immersed-boundary correction per member after the batch RK update (the batched stage skipped the per-block post-RK hook, so bodies inside batched blocks were never corrected -- ledger 99's 0.1 momentum inside static bodies); a block-frame prim scratch filled from the slab per member feeds the same s_amr_ib_correct_fine the per-block path calls; the validator admits static bodies under batching and keeps moving bodies out; the four static-IB np=1 AMR goldens pin their cap so the toolchain default batches them

Author: sbryngelson

docs/documentation/case.md

@@ -734,7 +734,7 @@ To restart the simulation from $k$-th time step, see @ref running "Restarting Ca
 | `amr_subcycle`          | Logical | Advance the coarse level at the case dt and the fine level at dt/2 (two substeps; Berger-Colella refluxing). Requires `amr`; incompatible with `cfl_dt`. |
 | `amr_device_pack`       | Logical | Pack and unpack the per-stage coarse-patch gather (F1/F2) over the plan's flat transfer list instead of one launch per transfer. The sends fuse to one kernel per family per stage; the receives fuse per contiguous (box, peer) run, so measured at np=8 the pack dispatches fall about 89x and the unpack about 4.3x. Wire bytes and floating-point values are unchanged. Requires `amr`; incompatible with `amr_subcycle`; the non-polytropic QBMM pb/mv twin keeps its per-transfer path. Default F. |
 | `amr_batched_gather`    | Logical | Consume the per-stage coarse-patch gather (F1/F2) for all of a rank's blocks at once: the gathered patches live in one pool, and each wave runs one fused own-copy, one fused unpack and one batched ghost fill instead of one set of launches per block. Wire bytes and floating-point values are unchanged. Requires `amr` and `amr_device_pack`; incompatible with `amr_subcycle`. Default F. |
-| `amr_batched_advance`   | Logical | Advance owned fine blocks of equal level and extent in batches of up to 8, stacked two ghost shells apart along the last active dimension, in one RHS call per batch. Requires `amr`; lock-step, Cartesian, uniform grid only; incompatible with the per-block fine-advance hooks (relaxation, IB, QBMM, IGR, chemistry, hypoelasticity, bubbles, MHD, relativity, damage, surface tension) and with Riemann-extrapolation BCs under `null_weights`; requires `amr_max_grid_size` > 0. Bit-identical to the per-block advance on a grid whose cell spacing is bitwise uniform (stacked blocks share the batch leader's coordinate arrays); roundoff-level differences otherwise, announced once at startup. Default F. Left unset on an `amr` case, the toolchain turns it on (with `amr_bat_pad` = 0.1) whenever these rules admit it; set `amr_batched_advance = F` to force the per-block advance. |
+| `amr_batched_advance`   | Logical | Advance owned fine blocks of equal level and extent in batches of up to 8, stacked two ghost shells apart along the last active dimension, in one RHS call per batch. Requires `amr`; lock-step, Cartesian, uniform grid only; incompatible with the per-block fine-advance hooks (relaxation, moving IB, QBMM, IGR, chemistry, hypoelasticity, bubbles, MHD, relativity, damage, surface tension) and with Riemann-extrapolation BCs under `null_weights`; requires `amr_max_grid_size` > 0. Bit-identical to the per-block advance on a grid whose cell spacing is bitwise uniform (stacked blocks share the batch leader's coordinate arrays); roundoff-level differences otherwise, announced once at startup. Default F. Left unset on an `amr` case, the toolchain turns it on (with `amr_bat_pad` = 0.1) whenever these rules admit it; set `amr_batched_advance = F` to force the per-block advance. |
 | `amr_max_blocks`       | Integer | Upper bound on the GLOBAL refined-block count. Sizes replicated per-rank METADATA (~11 kB/block); block slots themselves are allocated lazily for blocks a rank owns, so this is not N x device memory. Exceeding it silently truncates the refined region (the clusterer warns). Must be >= 1 (default 1024) |
 | `amr_max_grid_size`    | Integer | Absolute cap on a refined block's coarse-cell extent per dimension, the AMReX max_grid_size concept; must be >= 2 when set (default 0). With 0 the cap is derived from the decomposition and so shrinks as ranks are added, which tiles a fixed feature into more blocks the further you scale and makes the box set depend on the rank count. Setting it pins the cap, so the box set is identical at every rank count. The value may exceed half a rank subdomain: the solver scratch is then sized to the cap rather than to the subdomain, so per-rank memory grows as the cap raised to the number of dimensions |
 | `amr_max_level`        | Integer | Maximum AMR refinement depth (number of refined levels above L0); must be >= 1 (default 1). Multi-level nesting (>= 2) is supported: static AMR (`amr_regrid_int = 0`) nests up to level 2, dynamic regrid (`amr_regrid_int > 0`) nests deeper (see @ref amr_multilevel) |

src/simulation/m_amr.fpp

@@ -237,6 +237,9 @@ module m_amr
     !! libomptarget retention plateau). Same shared-scratch pattern as amr_rhs_pb_f/amr_cg. L0 tile slots are the exception and keep
     !! per-slot arrays (see s_amr_alloc_slot).
     type(scalar_field), allocatable :: amr_scr_prim(:), amr_scr_rhs(:)
+    !> block-frame primitive scratch for the batched advance's per-member IB correction (allocated only with ib): the slab's prim
+    !! holds the members stacked along amr_bat_sd, and s_ibm_correct_state reads a block in its own frame
+    type(scalar_field), allocatable :: amr_scr_prim_blk(:)
     !> NOT device-declared. A GPU_DECLARE(create=) on a module allocatable binds a present-table entry to the descriptor at program
     !! init; the `move_alloc` below then swaps that descriptor out and every later kernel lookup misses, which failed all 36 AMR
     !! tests on Frontier CCE gpu-acc with `find_in_present_table failed`. The `move_alloc` + GPU_ENTER_DATA pair at the allocation
@@ -8664,6 +8667,16 @@ contains
             tb3 = f_amr_wtime()
             call s_phase_tic(PH_RK)
             call s_amr_fine_rk_update_batch(amr_bat_n, amr_scr_rhs, coefs(1), coefs(2), coefs(3), coefs(4), dt)
+            if (ib) then
+                ! the per-block path corrects the body/ghost cells right after each block's RK update (s_amr_fine_stage_rk);
+                ! here once per member after the batch's update, in the member's own frame -- the correction reads only the
+                ! member's own cells, so the order across members does not matter. Ledger 99 found what its absence did.
+                do ibm = 1, amr_bat_n
+                    call s_amr_select_slot(amr_bat_blk(ibm))
+                    call s_amr_bat_member_prim(ibm, amr_scr_prim, amr_scr_prim_blk)
+                    call s_amr_ib_correct_fine(amr_scr_prim_blk)
+                end do
+            end if
             call s_phase_toc(PH_RK)
             tb4 = f_amr_wtime()
             if (rank_time_wrt) then
@@ -9698,6 +9711,14 @@ contains
             @:ACC_SETUP_SFs(amr_scr_prim(i))
             @:ACC_SETUP_SFs(amr_scr_rhs(i))
         end do
+        if (ib .and. amr_batched_advance) then
+            allocate (tmp_p(1:sys_size)); call move_alloc(tmp_p, amr_scr_prim_blk)
+            $:GPU_ENTER_DATA(create='[amr_scr_prim_blk]')
+            do i = 1, sys_size
+                @:ALLOCATE(amr_scr_prim_blk(i)%sf(mbuf1_lo:mbuf1_hi, mbuf2_lo:mbuf2_hi, mbuf3_lo:mbuf3_hi))
+                @:ACC_SETUP_SFs(amr_scr_prim_blk(i))
+            end do
+        end if
 
     end subroutine s_amr_scr_init
 
@@ -9808,6 +9829,39 @@ contains
 
     end subroutine s_amr_fine_rk_update_batch
 
+    !> Copy batch member ibm's primitive state out of the slab scratch (members stacked amr_bat_w apart along amr_bat_sd) into the
+    !! block-frame scratch over the member's own buffered extent, for the per-member IB correction after the batch RK update.
+    impure subroutine s_amr_bat_member_prim(ibm, src, dst)
+
+        integer, intent(in)                                    :: ibm
+        type(scalar_field), dimension(sys_size), intent(in)    :: src
+        type(scalar_field), dimension(sys_size), intent(inout) :: dst
+        integer                                                :: i, j, k, l, h, o1, o2, o3, b1l, b1h, b2l, b2h, b3l, b3h
+
+        h = amr_bat_blk(ibm)
+        o1 = 0; o2 = 0; o3 = 0
+        select case (amr_bat_sd)
+        case (1); o1 = (ibm - 1)*amr_bat_w
+        case (2); o2 = (ibm - 1)*amr_bat_w
+        case (3); o3 = (ibm - 1)*amr_bat_w
+        end select
+        b1l = amr_slots(h)%idwbuff(1)%beg; b1h = amr_slots(h)%idwbuff(1)%end
+        b2l = amr_slots(h)%idwbuff(2)%beg; b2h = amr_slots(h)%idwbuff(2)%end
+        b3l = amr_slots(h)%idwbuff(3)%beg; b3h = amr_slots(h)%idwbuff(3)%end
+        $:GPU_PARALLEL_LOOP(collapse=4, copyin='[o1, o2, o3, b1l, b1h, b2l, b2h, b3l, b3h]')
+        do i = 1, sys_size
+            do l = b3l, b3h
+                do k = b2l, b2h
+                    do j = b1l, b1h
+                        dst(i)%sf(j, k, l) = src(i)%sf(j + o1, k + o2, l + o3)
+                    end do
+                end do
+            end do
+        end do
+        $:END_GPU_PARALLEL_LOOP()
+
+    end subroutine s_amr_bat_member_prim
+
     !> Free the flat store and the dense-index maps. Mirrors s_amr_loc_index_init: called from BOTH finalize paths, because either
     !! pool-allocation site can have created them. Idempotent.
     impure subroutine s_amr_st_finalize()
@@ -9843,6 +9897,13 @@ contains
             @:DEALLOCATE(amr_scr_prim)
             @:DEALLOCATE(amr_scr_rhs)
         end if
+        if (allocated(amr_scr_prim_blk)) then
+            do i = 1, sys_size
+                @:ACC_TEARDOWN_SFs(amr_scr_prim_blk(i))
+                @:DEALLOCATE(amr_scr_prim_blk(i)%sf)
+            end do
+            @:DEALLOCATE(amr_scr_prim_blk)
+        end if
         amr_st_cap = 0
 
     end subroutine s_amr_st_finalize

toolchain/mfc/case_validator.py

@@ -1744,9 +1744,15 @@ def check_amr(self):
                 any(self.get(k, "F") == "T" for k in ("stretch_x", "stretch_y", "stretch_z")),
                 "amr_batched_advance requires a uniform grid (no stretching)",
             )
-            for k in ("qbmm", "ib", "relax", "igr", "chemistry", "hypoelasticity", "bubbles_euler", "bubbles_lagrange", "mhd", "relativity", "cont_damage", "surface_tension"):
+            for k in ("qbmm", "relax", "igr", "chemistry", "hypoelasticity", "bubbles_euler", "bubbles_lagrange", "mhd", "relativity", "cont_damage", "surface_tension"):
                 self.prohibit(self.get(k, "F") == "T", f"amr_batched_advance is incompatible with {k} = T (per-block hook in the fine advance)")
             self.prohibit(self.get("model_eqns") == 3, "amr_batched_advance is incompatible with model_eqns = 3 (per-block pressure relaxation)")
+            # static bodies: the batched advance applies s_amr_ib_correct_fine per member after the batch update; the moving-body
+            # update (s_amr_update_mib_fine) is still a per-block hook
+            self.prohibit(
+                self.get("ib", "F") == "T" and any((self.get(f"patch_ib({i})%moving_ibm") or 0) != 0 for i in range(1, int(self.get("num_ibs") or 0) + 1)),
+                "amr_batched_advance supports static immersed bodies only (moving_ibm /= 0 is a per-block hook in the fine advance)",
+            )
             self.prohibit(
                 not (amr_max_grid_size is not None and amr_max_grid_size > 0),
                 "amr_batched_advance requires amr_max_grid_size > 0 (the batched-slab scratch is sized to the pinned cap; a derived cap would size it to the global half-extent)",

toolchain/mfc/test/cases.py

@@ -337,6 +337,12 @@ def get_dimensions():
     "AMR + L0 tiles -> 2D -> coexist multi-level -> single tile": 32,
     "AMR -> 1D -> multi-level dynamic regrid tiled L2 np=2": 16,
     "AMR -> 1D -> three levels": 128,
+    # static immersed bodies (2D 63 x 63, np=1): batched since task30; the 127 x 127 np=2 twin has per-dimension caps 32/64
+    # that no scalar pin reproduces and stays per-block
+    "AMR -> 2D -> static IBM circle": 32,
+    "AMR -> 2D -> static IBM circle -> dynamic regrid": 32,
+    "AMR -> 2D -> static IBM two circles": 32,
+    "AMR -> 2D -> multi-level IB (static cylinder, np=1)": 32,
 }
 
 

toolchain/mfc/test_case_validator.py

@@ -401,7 +401,7 @@ def test_admissible_amr_case_gets_batching(self):
         validate_case_constraints(p, "simulation")
 
     def test_prohibited_combination_stays_per_block(self):
-        for k in ("ib", "igr", "stretch_x"):
+        for k in ("igr", "stretch_x"):
             p = {**self.AMR, k: "T"}
             self.assertFalse(apply_batching_default(p), k)
             self.assertNotIn("amr_batched_advance", p, k)