Skip to content

Automatic ib_neighborhood_radius faults instead of erroring when a body spans several ranks #1851

Description

@sbryngelson

Summary

With an immersed boundary that spans more than one rank's subdomain, leaving
ib_neighborhood_radius at its default (0, meaning "choose automatically") selects 1 and the run takes a GPU
memory access fault a few steps in. Setting the radius explicitly to 3 makes the same case run to completion.

The parameter is documented as being needed in this situation:

ib_neighborhood_radius controls the size of the neighborhood size. [...] This value defaults to 0, which
leaves the radius unset so that it is selected automatically. This parameter is required to strong-scale a case
when IBs eventually grow to be larger than one full processor domain wide.

So the behaviour is arguably documented. The problem is the failure mode: the automatic choice does not detect
that it is insufficient, and the run dies with a memory fault rather than an error message naming the parameter.

Reproducer

A rectangular immersed plate 1 x 2.5 x 0.05 (chord x span x thickness) on a 800 x 280 x 480 grid, 8 nodes,
64 ranks, --gpu mp on Frontier (cpe/25.03, rocm/6.3.1). The plate is larger than a rank's subdomain in the
spanwise direction at this decomposition.

 Automatic choice of ib_neighborhood_radius selected:  1
 [  1%]  Time step        5 of 12000 ...
Memory access fault by GPU node-7 (Agent handle: 0x...) on address (nil). Reason: Unknown.
srun: error: frontier08132: task 19: Aborted

The same case on 8 ranks runs fine, and on 64 ranks with ib_neighborhood_radius = 3 it runs to completion.

Suggested fix

Either of:

  1. Have the automatic selection account for the body's extent relative to the rank subdomain size, rather than
    defaulting to 1; or
  2. Detect the insufficient radius and abort with s_mpi_abort naming ib_neighborhood_radius, so the failure is
    a message rather than a GPU fault.

The second is cheap and would have saved the hour this cost me to bisect. A fault with a nil address gives no
hint which parameter is involved.

Environment

Frontier, ./mfc.sh build --gpu mp --case-optimization, cpe/25.03, rocm/6.3.1, master 33ad77a.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions