Fix nvfortran two-pass IPO causing full rebuild on every build

[sbryngelson]

Summary

When building MFC with nvfortran and GPU flags (--gpu acc or --gpu mp for non-OpenMP targets), every rebuild recompiled all source files even when nothing changed. This turned what should be a no-op rebuild into a multi-minute wait (e.g., ~10 minutes for simulation).

Root Cause

The two-pass IPO mechanism (lines 497–506) creates an OBJECT library (<target>_lib) that compiles all sources with -Mextract=lib:<target>_lib to build an inline library. The main target then uses -Minline=lib:<target>_lib to inline from that library.

The problem: nvfortran's -Mextract flag does not produce .o files — it only writes inline library data to a directory. CMake's OBJECT library model expects .o outputs from each compilation. Since those .o files never exist, CMake considers every source file out of date and recompiles the entire _lib target on every build.

This only affects nvfortran GPU builds because:

• NVHPC_USE_TWO_PASS_IPO is only TRUE for nvfortran 23.11–24.11
• CPU builds on Phoenix use gfortran, which doesn't use this code path
• For --gpu mp + simulation, the IPO is disabled by the NOT(MFC_OpenMP AND ARGS_OpenMP) guard, but all other GPU target combinations are affected

Fix

A small wrapper shell script is set as RULE_LAUNCH_COMPILE on the _lib OBJECT library target. The wrapper:

1. Runs nvfortran with all its arguments (including -Mextract) normally
2. Touches the expected .o output file afterward

This gives CMake the .o file it needs for dependency tracking. On subsequent builds, CMake sees the .o exists and is newer than the source, so it correctly skips recompilation. The inline library data in the <target>_lib/ directory is still produced as before.

Before

$ ./mfc.sh build -t simulation --gpu acc   # second build, no changes
(build took 618.1s)   # recompiles everything

After

$ ./mfc.sh build -t simulation --gpu acc   # second build, no changes
gmake[3]: Nothing to be done for 'CMakeFiles/simulation_lib.dir/build'.
gmake[3]: Nothing to be done for 'CMakeFiles/simulation.dir/build'.
# instant

Test plan

• Clean build + rebuild with --gpu acc -t pre_process: second build is no-op
• Clean build + rebuild with --gpu acc -t simulation: second build is no-op
• Clean build + rebuild with --gpu mp -t pre_process: second build is no-op
• All prechecks pass
• CI

[github-actions]

Claude Code Review

Incremental review from: e6e992d
Head SHA: d992dc4

New findings since last Claude review:

• src/simulation/m_riemann_solvers.fpp — Species arrays marked private in GPU parallel loops (potential stack overflow / correctness risk)

  Both the HLLC (~line 2205) and HLLD (~line 2424) GPU_PARALLEL_LOOP private lists now include Ys_L, Ys_R, Cp_iL, Cp_iR, Xs_L, Xs_R, Gamma_iL, Gamma_iR, Yi_avg, Phi_avg, h_iL, h_iR, h_avg_2. In thermochem builds these are per-species arrays (dimension num_species), not scalars. Declaring fixed-length or allocatable arrays as private in an OpenACC parallel loop triggers per-thread device-heap allocation; with even a modest number of species across the full domain this can silently overflow the device heap or degrade throughput significantly. If these are genuinely thread-local scratch arrays the declaration site (module-level or subroutine-local) determines whether listing them in private is required or redundant. If they are subroutine-local temporaries, OpenACC already privatizes them automatically and the explicit listing is unnecessary. Worth confirming whether any of these are module-level scratch buffers (requiring explicit private) and whether the device heap/stack limit is raised accordingly in the test matrix runs.

• CMakeLists.txt — IPO guard for NVHPC 24.11–25.9 removed; only two files excluded from inlining

  The removed guard (VERSION_GREATER "24.11" AND VERSION_LESS "25.9") previously disabled two-pass IPO globally for those compiler versions due to a fort2 ICE. The replacement excludes only m_start_up and m_cbc via per-file -Mnoinline. The new CI matrix adds 25.1/25.3/25.5/25.7 cpu+gpu entries, which will surface any remaining ICE sites. However, if the ICE is triggered by inlining depth rather than a specific caller-file, other hot-path files could also hit it under 25.x. The new matrix entries are the right validation, but the PR description should document why only these two files are excluded (i.e., confirm they are the only observed ICE sites) so future maintainers know the scope of the workaround.

[coderabbitai]

📝 Walkthrough

Walkthrough

The CMakeLists.txt change adds support for NVHPC's -Mextract behavior in the NVHPC_USE_TWO_PASS_IPO build path. When not building with both MFC_OpenMP and ARGS_OpenMP, the build generates an on-disk shell wrapper script that forwards compiler arguments and touches expected output paths. This wrapper script is registered as the RULE_LAUNCH_COMPILE target property for the NVHPC OBJECT library target, altering the compile command execution. Compilation flags including -Mextract and -Minline remain unchanged, as do dependency relationships.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately and concisely summarizes the main change: fixing an nvfortran issue causing unnecessary full rebuilds on every build.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check	✅ Passed	The PR description is comprehensive and well-structured, covering motivation, root cause analysis, implementation details, and testing results.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7ba0ce1c-c59c-407e-99fc-3ccfda86999c

📥 Commits

Reviewing files that changed from the base of the PR and between 336487c and e6e992d.

📒 Files selected for processing (1)

• CMakeLists.txt

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.67%. Comparing base (6e239b6) to head (d992dc4).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1345   +/-   ##
=======================================
  Coverage   64.67%   64.67%           
=======================================
  Files          70       70           
  Lines       18251    18251           
  Branches     1504     1504           
=======================================
  Hits        11804    11804           
  Misses       5492     5492           
  Partials      955      955           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.