Add online mixed-layer tracer budget diagnostics

[rmholmes]

Integrates three-dimensional tracer budget process terms across the mixed layer at every model time step and divides by the mixed layer depth, enabling accurate online computation of the mixed-layer tracer budget (Holmes and Malan, article submitted to JAMES). Uses some existing mixed layer routines in ocean_tracer_diag_mod.

Two core routines added to ocean_tracer_diag_mod:
compute_budget_mld — MLD-averages a 3D tendency field
compute_tracer_at_mlb — linearly interpolates a tracer to the MLD base

Mixed layer budget tracer diagnostics:
temp/salt_in_mld — MLD-averaged tracer concentration (times rho0)
temp/salt_at_mlb — tracer at the MLD base (C_ent)

MLD-averaged tracer budget process diagnostics:
temp/salt_tendency_in_mld — total tracer tendency
temp/salt_advection_in_mld — resolved advective convergence
temp/salt_submeso_in_mld — submesoscale parameterized advection
temp/salt_vdiffuse_diff_cbt_in_mld — vertical diffusion
temp/salt_vdiffuse_diff_cbt_conv_in_mld — convective vertical diffusion
temp/salt_vdiffuse_k33_in_mld — along-isopycnal K33 mixing
temp/salt_vdiffuse_sbc_in_mld — surface boundary condition flux
temp/salt_nonlocal_KPP_in_mld — nonlocal KPP
temp/salt_rivermix_in_mld — river mixing
mixdownslope_temp/salt_in_mld — downslope mixing parameterization
temp/salt_sigma_diff — laplacian diffusion in BBL mixing parameterization
neutral_physics_ndiffuse_in_mld — neutral diffusion
neutral_physics_gm_in_mld — GM mesoscale eddy parameterization
frazil_3d_in_mld — frazil ice formation
sw_heat_in_mld — shortwave penetration below surface
swflx/lw_heat/sens_heat/evap_heat_in_mld — individual surface heat flux components
net_sfc_heating_in_mld — total surface heat flux
sfc_hflux_pme_in_mld — P-E+R tracer flux at surface
pme_river_in_mld — P-E+R mass flux within MLD
temp/salt_eta_smooth_in_mld — SSH smoother tendency within MLD

GVC-to-Eulerian correction terms (see Holmes and Malan appendix):
eta_t_tendency_times_temp/salt_in_mld — ∂η/∂t × MLD-avg tracer
pme_river_times_temp/salt_in_mld — P-E+R × MLD-avg tracer
eta_smoother_times_temp/salt_in_mld — SSH smoother × MLD-avg tracer
s_surf_ent_temp/salt — z* coordinate entrainment at MLD
base: (1-H/(D+η)) × ∂η/∂t × C_ent

Testing diagnostics:
eta_t_tendency_times_temp/salt_at_mlb — ∂η/∂t × tracer at MLD base

For more information including analysis code see:

• https://github.com/rmholmes/access-om2-sst-budget, in particular the Theory_and_Diagnostics.ipynb notebook.
• Holmes and Malan, journal article submitted to JAMES.

The changes in this commit were developed on an older version of MOM5. Claude Sonnet 4.6 assisted in porting them onto the current ACCESS-NRI MOM5 codebase (upstream/upstream-master) by extracting a patch from the development branch, applying it with conflict resolution, and improving inline Fortran documentation in the new code.

Addresses the mixed layer budget diagnostics part (not the hat-averaging part) of #77

[rmholmes]

Ok @dougiesquire , how's this?

[dougiesquire]

Thanks @rmholmes - looks good. I've kicked off a "prerelease" build of ACCESS-OM2 with your MOM5 changes here. With luck, that will create a set of executables we can test shortly. Note, if you wish to use the prerelease infrastucture to do this sort of thing yourself in the future, there are some instructions here.

(Note, I'm working on the assumption that we want a version of ACCESS-OM2 with your changes to test etc before we review and merge your code)

[rmholmes]

Thanks @dougiesquire . Yes - we definitely want to test it before merging. For that, I will just want to run a few years of a 1-degree IAF simulation (with a modified diag_table) and do some analysis on it to check that everything is working.

[dougiesquire]

The prerelease build has completed and I've just pushed a branch to the access-om2-configs repo that's based on the 1deg_jra55_iaf configuration but uses the prerelease version. For future reference, changing the ACCESS-OM2 version in the configuration is as simple as modifying a few lines in the config.yaml file:

modules:
  use:
      - /g/data/vk83/modules
+     - /g/data/vk83/prerelease/modules
  load:
-     - access-om2/2026.05.000
+     - access-om2/pr151-1
      - model-tools/fre-nctools/2024.05-1

...

manifest:
  reproduce:
-   exe: True
+   exe: False

You can clone the configuration branch using Payu, something like:

payu clone \
    --branch rmholmes-diag-1deg_jra55_iaf \
    https://github.com/ACCESS-NRI/access-om2-configs.git \
    <name-of-exp-dir>

[rmholmes]

@dougiesquire I added a few more changes to account for diagnostics that are not active in the 1/4-degree config but are in the 1-degree config. How would I test a config with these updates (I had a quick look but it wasn't obvious to me).

[dougiesquire]

Hi @rmholmes. We can just change the MOM5 commit hash in this PR to your most recent one. That will trigger a new prerelease build that you can then use in your configuration.

I can make the change now to show you what I mean, but in the future if you want to do this yourself I can give you permission (let me know).

[dougiesquire]

Change made. Pushing that change has triggered a new OM2 build Once that completes the CI will add a new comment to the PR like this one.

[dougiesquire]

The new build has now completed. To use it in your config, change the following in your config.yaml:

modules:
  use:
      - /g/data/vk83/modules
      - /g/data/vk83/prerelease/modules
  load:
-     - access-om2/pr151-1
+     - access-om2/pr151-3
      - model-tools/fre-nctools/2024.05-1

[rmholmes]

Perfect, thanks @dougiesquire ! Hopefully I don't need too many more updates, but if I do I'll ask for the appropriate permissions.

[rmholmes]

@dougiesquire sorry for the silence.

I've fixed a bug. Can you give me permissions to push to ACCESS-NRI/ACCESS-OM2#151 so I can update the exe and test again?

I've just cloned the repo and commited a change to the git-hash for the mom5 exectuable in spack.yaml, so I think all I need to do is push that up?

[dougiesquire]

I've just cloned the repo and commited a change to the git-hash for the mom5 exectuable in spack.yaml, so I think all I need to do is push that up?

Yup. It's best/tidiest if you checked out and added your change to the rmholmes-diagnostic branch. Hopefully that's what you've done? I've just sent you an invite for write access to the ACCESS-OM2 repo

[dougiesquire]

The build is ready to go @rmholmes. Change to access-om2/pr151-4 in your config.yaml.

[rmholmes]

@dougiesquire ok this is now working. I have confirmed that I am getting consistent results with my existing simulations - both in an ACCESS-NRI 1-degree configuration and an ACCESS-NRI 1/4-degree configuration (which I just have as a local branch).

The testing I performed was in this notebook.

The delay was because I had to add two new diagnostics for parameterizations that are active in the 1-degree configuration but not the 1/4-degree (or 1/10-degree) configurations (mixdownslope and sigma_diff). I have added these to the description at the top of this PR. I haven't tested this at 1/10-degree, but I don't think it is necessary as there are no new parameterisations active at that resolution compared to 1/4-degree.

The code is now working. However, it might be good if someone who knows a bit more about Fortran could have a quick look at the changes I'm proposing. In particular, I think my use of wrk arrays may not be the most memory efficient, although it is working fine in these configurations.

Other than that, I think this is good to get merged into the main code branch.

[dougiesquire]

Thanks @rmholmes, I'll take a look when I can find some time

[dougiesquire]

Thanks @rmholmes. Sorry for the delay.

Firstly, thanks for taking the time to make this contribution. It's looking great.

I think the hidden wrk1-altering side effect of the compute_budget_mld/compute_tracer_at_mlb routines is a little dangerous, given that the routines are called inside blocks that work on wrk1. I see you've tried to work around it with copies to other wrk arrays, but there are still remaining examples I've flagged below of unintended bugs.

I think the safest/simplest option is to allocate private wrk arrays for the new routines in ocean_tracer_diag. We need only do the allocation when a relevant diagnostic is requested to avoid unnecessary allocation.

I can put together a PR to your branch showing what I mean and you can see what you think.

[dougiesquire]

@rmholmes, I've opened a PR that will address the comments from my previous review, should we choose to merge it.

Below are some more review comments.

[rmholmes]

Thanks @dougiesquire !!! I've implemented all your suggestions.

Let me know if you think there is something easy that can be done about avoiding calling calc_mixed_layer_depth many times. Once that is resolved I can test it again.

[dougiesquire]

Thanks @rmholmes. I've replied in the thread about calc_mixed_layer_depth (TLDR; let's leave as is).

At some point I'd like to rebase this PR so that the CI can run. Doing so will rewrite the history (i.e. the commit hashes will change) and you'll need to update your local clone. To do this, make sure you don't have any local uncommitted work and then:

git fetch
git reset --hard origin/mixed_layer_tracer_diagnostics_nri

Is now an okay time for me to do the rebase?

[rmholmes]

@dougiesquire . Yes go for it. Once that's done I can test it.

[dougiesquire]

Thanks, I've now rebased. Feel free to update your clone and do your testing

[rmholmes]

Well unfortunately now the 1-degree test is failing with a "salinity out of range" error and the 1/4-degree test is creating an access-om2.err file which is multiple GB's in size and blowing up my /home disk quota.... I think there are too many out-of-range warnings for the diagnostics now. I don't really understand why anything has changed, but I'll have another look later this week.

[rmholmes]

Actually, I think the problem is with the PME flux because 1) this explains the salinity out of range in the 1-degree model and 2) the diagnostic-out-of-range errors in the 1/4-degree are all to do with the sfc_hflux_pme diagnostic. So something we have done, likely in src/mom5/ocean_core/ocean_sbc.F90, has broken something real (not diagnostic) in the fluxes.

[dougiesquire]

Hm, that's odd. Could you please share the test configs you are running @rmholmes?

I ran our latest 1deg_jra55_iaf config using access-om2/pr151-6 for 20 years without issue.

[rmholmes]

Pretty sure the issue will only come up when the diagnostics are active, because that is all we've changed.

My 1-degree configuration: https://github.com/rmholmes/access-om2-configs/tree/rmholmes-diag-1deg_jra55_iaf

My 1/4-degree configuration: https://github.com/rmholmes/access-om2-configs/tree/rmholmes_025deg_jra55_iaf

[dougiesquire]

Thanks @rmholmes. I've worked out what's going on here. There is an additional hidden side effect that I didn't see at first. calc_mixed_layer_depth calls density_delta_sfc which also ends up modifying wrk1 and wrk2 (curses on whoever decided public scratch arrays were a good idea).

The specific issue is here, where wrk1 is being overwritten by the compute_budget_mld call but it is needed for subsequent iterations of the do n=1,num_prog_tracers loop.

We can easily fix this, but I'm a little worried about the performance impact of all my changes, so I'd like to do some tests.

[dougiesquire]

I'm a little worried about the performance impact of all my changes

This was just Gadi misbehaving.

I hope you don't mind @rmholmes, I've just pushed a fix directly to your branch. I'll kick off a new prerelease that should now work for your final testing.

[dougiesquire]

Please try with access-om2/pr151-7 when you get a chance

[rmholmes]

Excellent detective work @dougiesquire ! I seem to remember running into similar errors when I first implemented this, which is why I ended up being extra defensive by allocating lots of new arrays...

Seems to be running fine now at 1-degree. I will confirm once its done and I've compared the diagnostics.

[rmholmes]

@dougiesquire it looks like everything is working fine. I have rerun my testing script with the new 1-degree run and nothing looks amiss. I don't think it's worth wasting the compute to do the same test in the 1/4-degree?

So all good to go!?

Thanks heaps for the help!

[dougiesquire]

So all good to go!?

Yup! Thanks again for doing this (and for your patience when I broke it 😅)