Summary
With an immersed boundary that spans more than one rank's subdomain, leaving
ib_neighborhood_radius at its default (0, meaning "choose automatically") selects 1 and the run takes a GPU
memory access fault a few steps in. Setting the radius explicitly to 3 makes the same case run to completion.
The parameter is documented as being needed in this situation:
ib_neighborhood_radius controls the size of the neighborhood size. [...] This value defaults to 0, which
leaves the radius unset so that it is selected automatically. This parameter is required to strong-scale a case
when IBs eventually grow to be larger than one full processor domain wide.
So the behaviour is arguably documented. The problem is the failure mode: the automatic choice does not detect
that it is insufficient, and the run dies with a memory fault rather than an error message naming the parameter.
Reproducer
A rectangular immersed plate 1 x 2.5 x 0.05 (chord x span x thickness) on a 800 x 280 x 480 grid, 8 nodes,
64 ranks, --gpu mp on Frontier (cpe/25.03, rocm/6.3.1). The plate is larger than a rank's subdomain in the
spanwise direction at this decomposition.
Automatic choice of ib_neighborhood_radius selected: 1
[ 1%] Time step 5 of 12000 ...
Memory access fault by GPU node-7 (Agent handle: 0x...) on address (nil). Reason: Unknown.
srun: error: frontier08132: task 19: Aborted
The same case on 8 ranks runs fine, and on 64 ranks with ib_neighborhood_radius = 3 it runs to completion.
Suggested fix
Either of:
- Have the automatic selection account for the body's extent relative to the rank subdomain size, rather than
defaulting to 1; or
- Detect the insufficient radius and abort with
s_mpi_abort naming ib_neighborhood_radius, so the failure is
a message rather than a GPU fault.
The second is cheap and would have saved the hour this cost me to bisect. A fault with a nil address gives no
hint which parameter is involved.
Environment
Frontier, ./mfc.sh build --gpu mp --case-optimization, cpe/25.03, rocm/6.3.1, master 33ad77a.
Summary
With an immersed boundary that spans more than one rank's subdomain, leaving
ib_neighborhood_radiusat its default (0, meaning "choose automatically") selects 1 and the run takes a GPUmemory access fault a few steps in. Setting the radius explicitly to 3 makes the same case run to completion.
The parameter is documented as being needed in this situation:
So the behaviour is arguably documented. The problem is the failure mode: the automatic choice does not detect
that it is insufficient, and the run dies with a memory fault rather than an error message naming the parameter.
Reproducer
A rectangular immersed plate 1 x 2.5 x 0.05 (chord x span x thickness) on a 800 x 280 x 480 grid, 8 nodes,
64 ranks,
--gpu mpon Frontier (cpe/25.03, rocm/6.3.1). The plate is larger than a rank's subdomain in thespanwise direction at this decomposition.
The same case on 8 ranks runs fine, and on 64 ranks with
ib_neighborhood_radius = 3it runs to completion.Suggested fix
Either of:
defaulting to 1; or
s_mpi_abortnamingib_neighborhood_radius, so the failure isa message rather than a GPU fault.
The second is cheap and would have saved the hour this cost me to bisect. A fault with a nil address gives no
hint which parameter is involved.
Environment
Frontier,
./mfc.sh build --gpu mp --case-optimization, cpe/25.03, rocm/6.3.1, master 33ad77a.