Make the IB force-reduction receive buffers device resident - #1843
Open
sbryngelson wants to merge 1 commit into
Open
Make the IB force-reduction receive buffers device resident#1843sbryngelson wants to merge 1 commit into
sbryngelson wants to merge 1 commit into
Conversation
`s_communicate_ib_forces` passed `recv_forces_snap`, `recv_torques_snap`, `recv_ids` and
`recv_ft` into GPU kernels through `copy`/`copyin`, but allocated them with a plain
`allocate`, so they never entered the device present table. On Cray OpenMP offload this
aborts at the first time step of any moving-IB case run on more than one rank:
ACC: find_in_present_table failed for 'recv_forces_snap(:,:)' from m_ibm.fpp:1338
ACC: libcrayacc/acc_runtime.c:703 CRAY_ACC_ERROR - Variable not found in present table
The sibling send buffers `send_ids`/`send_ft` are already `@:ALLOCATE`d and pushed with
`GPU_UPDATE`, so the receive side was simply inconsistent with them. Allocate the four
receive arrays the same way and update them to the device after the host writes (the
zeroing before each accumulation pass, and each `MPI_UNPACK`), which also lets the kernels
drop the `copy`/`copyin` of those arrays.
Reproduced on Frontier (`./mfc.sh build --gpu mp`, cpe/25.03, rocm/6.3.1) with a 2D moving
flat plate on 4 ranks; the same case runs to completion with this change, and an OpenACC
build was unaffected either way.
Fixes #1840
Claude-Session: https://claude.ai/code/session_01HMJ7cycfo7kTFSFq5yhHLG
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
This PR fixes a GPU offload crash in multi-rank immersed boundary runs by ensuring receive-side force/torque buffers are device-resident and explicitly synchronized for OpenMP offload runtimes (e.g., Cray).
Changes:
- Switch receive buffers to
@:ALLOCATE/@:DEALLOCATEso they become device-resident like the send-side twins. - Add
GPU_UPDATE(device=...)after host writes torecv_*receive buffers. - Remove
copy/copyinclauses for receive buffers in GPU parallel loops to avoid present-table failures.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+1339
to
+1340
| $:GPU_UPDATE(device='[recv_ids, recv_ft]') | ||
| $:GPU_PARALLEL_LOOP(private='[i, j]', copy='[forces, torques]') |
| call MPI_UNPACK(ib_force_recv_buf, buf_size, unpack_pos, recv_ft, 6*recv_count, mpi_p, MPI_COMM_WORLD, ierr) | ||
| $:GPU_PARALLEL_LOOP(private='[i, j]', copyin='[recv_ft, recv_ids]', copy='[forces, torques, & | ||
| & recv_forces_snap, recv_torques_snap]') | ||
| $:GPU_UPDATE(device='[recv_ids, recv_ft]') |
Comment on lines
+1385
to
+1386
| $:GPU_UPDATE(device='[recv_ids, recv_ft]') | ||
| $:GPU_PARALLEL_LOOP(private='[i, j]', copy='[forces, torques]') |
Lines of Code
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1840.
s_communicate_ib_forceshandsrecv_forces_snap,recv_torques_snap,recv_idsandrecv_ftto GPU kernels viacopy/copyin, but they are allocated with a plainallocateand so are never present on the device. Under Cray OpenMP offload this aborts at the first time step for any moving immersed boundary run on more than one rank:The send-side twins (
send_ids,send_ft) are already@:ALLOCATEd and pushed withGPU_UPDATE, so this only makes the receive side consistent with them:@:ALLOCATE/@:DEALLOCATE, an explicitGPU_UPDATE(device=...)after each host write, and the correspondingcopy/copyinclauses dropped.Testing. Frontier,
--gpu mp(cpe/25.03, rocm/6.3.1), 2D moving flat plate withib_state_wrt, 4 ranks: aborts at step 1 on master, runs to completion with this patch. An OpenACC build of the same case was unaffected before and after../mfc.sh precheckpasses.One file, 8 insertions, 6 deletions.
https://claude.ai/code/session_01HMJ7cycfo7kTFSFq5yhHLG