Upgrade openmpi to 5.0.8 - #264
Conversation
|
The model version in the
|
|
🚀 Attempted to deploy 🖥️
|
|
🚀 Attempted to deploy 🖥️
|
|
!update-configs |
|
🔧 Opening/Updating Model Configuration PRs in ❌ One or more errors occurred with the workflow Configurations RequestedConfigurations requested from profile
Pull Requests Opened/UpdatedThe following PRs were opened/updated: More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/33501482275 |
Apparently, this is meant to update the |
@manodeep this branch has been renamed to |
@ezhilsabareesh8 Might mean that a corresponding manifest needs to also be renamed; otherwise the |
@ezhilsabareesh8 I couldn't find out where the name update needed to be - but it's on this line |
|
🚀 Attempted to deploy 🖥️
|
|
Confirming that the stack VLA -> heap array patch was applied |
|
Ran with the latest deployment but there was an error for invalid free immediately on startup, and no further output (in either of the |
…ck arrays in write paths too
|
Claude has "patched" the patch. The previous patch only covered the pio reads; the pio writes still had the same pattern of stack arrays and needed to be converted into heap arrays |
|
🚀 Attempted to deploy 🖥️
|
|
Hi @manodeep , If you want to test parallelio 2.6.9 or 2.6.10, I've cherry-picked those commits into: You can add the above snippet in the |
|
Thanks @harshula! There's some black magic y'all can do with spack recipes 😄 That read/write error is potentially fixed but now I have a different error (invalid free and no stack trace, and a hanging + resource-consuming MPI job 😞) |
|
Running with env vars for checking malloc, I get a crash with the released version of access-om3 for 25k-IAF config - so the bug (bugs?) affect both configs, and pre-existing. |
|
🚀 Attempted to deploy 🖥️
|
|
!update-configs |
|
🔧 Opening/Updating Model Configuration PRs in Configurations RequestedConfigurations requested from profile
Pull Requests Opened/UpdatedThe following PRs were opened/updated:
More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/33965413223 |
|
All the configs pass the repro tests except for |
|
Big boost (~18% faster) in runtime performance for a 1-yr 25k RYF run - 2h45mins vs 3h22mins for the released model with Printing runtime results (all times are in mins, sorted by 'Simulated Years per wallclock-day'):
Bitwise repro passes Caveats: Single runs (not min. of repeated runs), and the two experiments were done at different times - the patched expt was yesterday, and the released one was this morning |
|
Nice work @manodeep! Will you open issues/PRs for the parallelio and esmf fixes? |
|
That's the goal - assuming we all agree that the PIO and ESMF patches are correct (or at least in the ballpark). Are you able to take a look at the two patches? I also have a stockpile of PRs that I need to work through (the CICE runtime changes, this one with the OpenMPI upgrade, and the repro-preserving non-blocking update for MOM6, plus the yet-to-be-pushed-to-GH ACCESS-FMS/ESM1.6 collective netcdf-io upgrade) |
… code changes (relevant for the upcoming 8k run)
|
🚀 Attempted to deploy 🖥️
|
|
More perf tests IAF (30-day runs only)
RYF (Mix of 30-days and 1-yr)
|
|
This PR is now being used for deploying the 8k run this week. Will have to redo once we decide to merge in the OpenMPI changes (or switch out to a different branch with only the OpenMPI v5 commits) |
|
🚀 Attempted to deploy 🖥️
|
…e + openmpi v5 changes)
|
🚀 Attempted to deploy 🖥️
|
PIO issue and ESMF issue |
|
🚀 Attempted to deploy 🖥️
|
|
🚀 Attempted to deploy 🖥️
|
Previously this had segfaulted but I can't find the original crash directory. Testing again
🚀 The latest prerelease
access-om3/pr264-11at 57f767b is here: #264 (comment) 🚀