Skip to content

Upgrade openmpi to 5.0.8 - #264

Draft
manodeep wants to merge 11 commits into
mainfrom
build-with-ompi508
Draft

Upgrade openmpi to 5.0.8#264
manodeep wants to merge 11 commits into
mainfrom
build-with-ompi508

Conversation

@manodeep

@manodeep manodeep commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Previously this had segfaulted but I can't find the original crash directory. Testing again


🚀 The latest prerelease access-om3/pr264-11 at 57f767b is here: #264 (comment) 🚀

@manodeep
manodeep requested a review from a team as a code owner September 1, 2026 09:55
@manodeep
manodeep marked this pull request as draft September 1, 2026 09:55
@manodeep
manodeep removed the request for review from a team September 1, 2026 09:55
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

The model version in the spack.yaml has not been updated.
You can update it manually.
Alternatively, comment the following to have it updated and committed automatically:

  • !bump major for feature releases
  • !bump minor for bugfixes

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-1 with commit cd824a4

🖥️ Gadi Deployment ❌

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-1 with commit cd824a4

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr264-1 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr264-1

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr264-1 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.08.005
  • access-nri/spack-config version 2026.08.001

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

@manodeep

manodeep commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

!update-configs

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om3-configs

❌ One or more errors occurred with the workflow

Configurations Requested

Configurations requested from profile default:

  • dev-MCW_100km_jra_iaf

  • dev-MC_100km_jra_ryf

  • dev-MC_100km_jra_ryf+wombatlite

  • dev-MC_25km_jra_ryf

Pull Requests Opened/Updated

The following PRs were opened/updated:

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/33501482275

@manodeep

manodeep commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om3-configs

❌ One or more errors occurred with the workflow
Configurations Requested

Configurations requested from profile default:

* `dev-MCW_100km_jra_iaf`

* `dev-MC_100km_jra_ryf`

* `dev-MC_100km_jra_ryf+wombatlite`

* `dev-MC_25km_jra_ryf`

Pull Requests Opened/Updated

The following PRs were opened/updated:

* [Auto update dev-MC_100km_jra_ryf to use access-om3/pr264-1 access-om3-configs#1637](https://github.com/ACCESS-NRI/access-om3-configs/pull/1637)

* [Auto update dev-MC_100km_jra_ryf+wombatlite to use access-om3/pr264-1 access-om3-configs#1638](https://github.com/ACCESS-NRI/access-om3-configs/pull/1638)

* [Auto update dev-MC_25km_jra_ryf to use access-om3/pr264-1 access-om3-configs#1636](https://github.com/ACCESS-NRI/access-om3-configs/pull/1636)

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/33501482275

Apparently, this is meant to update the dev-MCW_100km_jra_iaf branch, but the actions shows the following error:

error: pathspec 'dev-MCW_100km_jra_iaf' did not match any file(s) known to git

@ezhilsabareesh8 ☝🏾

@ezhilsabareesh8

Copy link
Copy Markdown
Collaborator

Apparently, this is meant to update the dev-MCW_100km_jra_iaf branch, but the actions shows the following error:

error: pathspec 'dev-MCW_100km_jra_iaf' did not match any file(s) known to git

@manodeep this branch has been renamed to dev-MCW_100km_era_iaf

@manodeep

manodeep commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Apparently, this is meant to update the dev-MCW_100km_jra_iaf branch, but the actions shows the following error:

error: pathspec 'dev-MCW_100km_jra_iaf' did not match any file(s) known to git

@manodeep this branch has been renamed to dev-MCW_100km_era_iaf

@ezhilsabareesh8 Might mean that a corresponding manifest needs to also be renamed; otherwise the !update-configs command is looking for the original branch.

@manodeep

manodeep commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Apparently, this is meant to update the dev-MCW_100km_jra_iaf branch, but the actions shows the following error:

error: pathspec 'dev-MCW_100km_jra_iaf' did not match any file(s) known to git

@manodeep this branch has been renamed to dev-MCW_100km_era_iaf

@ezhilsabareesh8 Might mean that a corresponding manifest needs to also be renamed; otherwise the !update-configs command is looking for the original branch.

@ezhilsabareesh8 I couldn't find out where the name update needed to be - but it's on this line

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-2 with commit db8b3c8

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr264-2 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr264-2

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr264-2 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

@manodeep

manodeep commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Confirming that the stack VLA -> heap array patch was applied

@manodeep

manodeep commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Ran with the latest deployment but there was an error for invalid free immediately on startup, and no further output (in either of the out or err) for the next 7-8 mins. So I qdel'ed the job - need to fix the patch

@manodeep

manodeep commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Claude has "patched" the patch. The previous patch only covered the pio reads; the pio writes still had the same pattern of stack arrays and needed to be converted into heap arrays

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-3 with commit 534cba0

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr264-3 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr264-3

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr264-3 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

@harshula

harshula commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Hi @manodeep , If you want to test parallelio 2.6.9 or 2.6.10, I've cherry-picked those commits into:

  builtin:
    git: https://github.com/ACCESS-NRI/upstream-spack-packages.git
    branch: access/v1.1-next
    destination: $env/package-repos/spack-packages

You can add the above snippet in the repos: section of the spack.yaml. Whichever repository comes first has higher precedence.

@manodeep

manodeep commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @harshula! There's some black magic y'all can do with spack recipes 😄

That read/write error is potentially fixed but now I have a different error (invalid free and no stack trace, and a hanging + resource-consuming MPI job 😞)

@manodeep

manodeep commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator Author

Running with env vars for checking malloc, I get a crash with the released version of access-om3 for 25k-IAF config - so the bug (bugs?) affect both configs, and pre-existing.

env:
  MALLOC_CHECK_: 3
  MALLOC_PERTURB_: 42
  MALLOC_ARENA_MAX: 1

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-6 with commit 45dcff7

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr264-6 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr264-6

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr264-6 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

@manodeep

manodeep commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator Author

!update-configs

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om3-configs

Configurations Requested

Configurations requested from profile default:

  • dev-MCW_100km_era_iaf

  • dev-MC_100km_jra_ryf

  • dev-MC_100km_jra_ryf+wombatlite

  • dev-MC_25km_jra_ryf

Pull Requests Opened/Updated

The following PRs were opened/updated:

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/33965413223

@manodeep

manodeep commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator Author

All the configs pass the repro tests except for test_repro_payu_setup - which is expected since I am changing the exe

@manodeep

manodeep commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Big boost (~18% faster) in runtime performance for a 1-yr 25k RYF run - 2h45mins vs 3h22mins for the released model with openmpi/4.1.7 and the PIO bug; 14.9 kSU vs 18.3 kSU 🚀. Bitwise identical results.

[~/perf-om3/25km/test-ompi-508 @gadi-login-08]$ ~/perf-om3/om3_summary.sh 25km_jra_ryf_Sep2_1yr_cloned_from_patched_ompi508 25km_jra_ryf_Sep2_1yr_ompi_508_pio_patched_write_esmf_zoltan_patched
Using command-line directory pattern: '25km_jra_ryf_Sep2_1yr_cloned_from_patched_ompi508 25km_jra_ryf_Sep2_1yr_ompi_508_pio_patched_write_esmf_zoltan_patched'
Progress : [########################################] 100%

Printing runtime results (all times are in mins, sorted by 'Simulated Years per wallclock-day'):

yrs/Wday WTime (max ocn/wtime) SUs CPUs yrs/core/Wday Directory
8.8 164.5 (73.9%) 14825 2704 3.24e-03 25km_jra_ryf_Sep2_1yr_ompi_508_pio_patched_write_esmf_zoltan_patched
7.1 202.3 (74.5%) 18231 2704 2.63e-03 25km_jra_ryf_Sep2_1yr_cloned_from_patched_ompi508

Bitwise repro passes

[~/perf-om3/25km/test-ompi-508 @gadi-login-08]$ compare-exp-tests --dirs "25km_jra_ryf_Sep2_1yr_cloned_from_patched_ompi508 25km_jra_ryf_Sep2_1yr_ompi_508_pio_patched_write_esmf_zoltan_patched"
Test session starts (platform: linux, Python 3.12.1, pytest 9.1.1, pytest-sugar 1.1.1)
rootdir: /home/593/ms2335/codes/utils/model-config-tests/src/model_config_tests/compare_exp_tests
configfile: ../../../pyproject.toml
plugins: cov-7.0.0, anyio-4.11.0, sugar-1.1.1, hypothesis-6.92.6
collected 1 item                                                                                                                                                                                      

 test_repro.py ✓                                                                                                                                                                        100% ██████████

Results (1.54s):
       1 passed

Caveats: Single runs (not min. of repeated runs), and the two experiments were done at different times - the patched expt was yesterday, and the released one was this morning

@dougiesquire

Copy link
Copy Markdown
Collaborator

Nice work @manodeep! Will you open issues/PRs for the parallelio and esmf fixes?

@manodeep

manodeep commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator Author

That's the goal - assuming we all agree that the PIO and ESMF patches are correct (or at least in the ballpark). Are you able to take a look at the two patches?

I also have a stockpile of PRs that I need to work through (the CICE runtime changes, this one with the OpenMPI upgrade, and the repro-preserving non-blocking update for MOM6, plus the yet-to-be-pushed-to-GH ACCESS-FMS/ESM1.6 collective netcdf-io upgrade)

… code changes (relevant for the upcoming 8k run)
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-7 with commit f4c1ba2

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr264-7 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr264-7

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr264-7 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

@manodeep

manodeep commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator Author

More perf tests

IAF (30-day runs only)

yrs/Wday WTime (max ocn/wtime) SUs CPUs yrs/core/Wday Directory
8.4 14.1 (63.1%) 1271 2704 3.10e-03 25km_jra_iaf_Sep6_30days_pr264-7_with_mom6_perf-rho-branch_deflate_1
8.1 14.6 (62.0%) 1320 2704 2.99e-03 25km_jra_iaf_Sep6_30days_pr264-7_with_mom6_perf-rho-branch
7.9 15.0 (68.0%) 1353 2704 2.91e-03 25km_jra_iaf_Sep6_30days_pr264-5_patched_pio_patched_Zoltan
7.8 15.3 (58.0%) 1375 2704 2.87e-03 25km_jra_iaf_Sep6_30days_pr264-7_with_mom6_perf-rho-branch_deflate_1_flush_nc_false
7.4 15.9 (56.0%) 1433 2704 2.75e-03 25km_jra_iaf_Sep6_30days_pr264-7_with_mom6_perf-rho-branch_flush_nc_false
6.1 19.3 (66.0%) 1736 2704 2.27e-03 25km_jra_iaf_Sep6_30days_cloned_from_patched_ompi508 (released)

RYF (Mix of 30-days and 1-yr)

yrs/Wday WTime (max ocn/wtime) SUs CPUs yrs/core/Wday Directory
8.8 164.5 (73.9%) 14825 2704 3.24e-03 25km_jra_ryf_Sep2_1yr_ompi_508_pio_patched_write_esmf_zoltan_patched
8.7 165.8 (73.5%) 14943 2704 3.21e-03 25km_jra_ryf_Sep2_1yr_ompi_508_pio_patched_write_esmf_zoltan_patched_deflate_1
8.2 175.7 (61.7%) 15840 2704 3.03e-03 25km_jra_ryf_Sep2_1yr_ompi_508_pr264-7_patched_pio_zoltan_mom6_perf-rho-remap
7.1 202.3 (74.5%) 18231 2704 2.63e-03 25km_jra_ryf_Sep2_1yr_cloned_from_patched_ompi508 (released)
6.3 18.7 (67.2%) 1689 2704 2.33e-03 25km_jra_ryf_Sep2_30days_deflate_1_shuffle_false
6.3 18.8 (67.5%) 1696 2704 2.33e-03 25km_jra_ryf_Sep2_30days_deflate_1
6.2 19.1 (67.8%) 1718 2704 2.30e-03 25km_jra_ryf_Sep2_30days_deflate_0
6.0 19.6 (64.1%) 1771 2704 2.23e-03 25km_jra_ryf_Sep2_30days (released)

@manodeep

manodeep commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

This PR is now being used for deploying the 8k run this week. Will have to redo once we decide to merge in the OpenMPI changes (or switch out to a different branch with only the OpenMPI v5 commits)

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-8 with commit fcddadd

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr264-8 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr264-8

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr264-8 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-9 with commit 5a17f63

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr264-9 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr264-9

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr264-9 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

@manodeep

manodeep commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Nice work @manodeep! Will you open issues/PRs for the parallelio and esmf fixes?

PIO issue and ESMF issue

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-10 with commit 73c3366

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr264-10 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr264-10

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr264-10 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

@manodeep
manodeep deployed to Gadi Prerelease September 10, 2026 11:26 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr264-11 with commit 57f767b

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr264-11 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr264-11

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr264-11 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants