Skip to content

Latest commit

 

History

History
612 lines (516 loc) · 23.3 KB

DESIGN.md

File metadata and controls

612 lines (516 loc) · 23.3 KB

rsofuncpp — C++17 reimplementation of the rsofun Fortran core

Drop-in R package replacement for rsofun that removes the Fortran 90 dependency, keeps the same user-facing API, and validates every C++ routine against a pure-R reference implementation.


1. Goals and non-goals

1.1 Goals

  • Replace the ~12,926 lines of Fortran 90 in rsofun with C++17/Rcpp.
  • Preserve the existing R API: run_pmodel_f_bysite(), runread_pmodel_f(), run_biomee_f_bysite(), runread_biomee_f(), calib_sofun(), and the cost functions.
  • Re-use the same validation data sets shipped with rsofun (p_model_drivers, biomee_p_model_drivers, biomee_gs_leuning_drivers, etc.).
  • Achieve numerical parity with rsofun on all bundled examples (tolerances documented per output variable).
  • Add a modern test suite, CI, documentation, and CRAN-ready packaging.

1.2 Non-goals

  • Do not change the scientific equations or published calibration targets.
  • Do not introduce new user-facing parameters without a compatibility shim.
  • Do not delete the original _r reference functions once the C++ versions are active — they document the math and act as regression ground-truth.

2. Design principles

These principles are taken from the SSDLC used in xsdm, nicher, xclim, maxentcpp, and ucminfcpp:

  1. Science first. Each C++ routine must match a pure-R reference line-by-line before it is used in a production wrapper.
  2. R as orchestrator, C++ as engine. The R layer validates, reshapes, and dispatches; the C++ layer computes.
  3. Defensive input validation. All exported R functions use checkmate.
  4. Numerical parity before performance. Parallelism and micro-optimizations are introduced only after cross-language tests pass.
  5. CRAN hygiene from day one. R CMD check --as-cran must pass with 0 errors and 0 warnings at the end of every phase.

3. Package layout

The tree below is the target state. Empty directories are not committed; the first file in a directory makes it appear after git clone.

rsofuncpp/
├── DESCRIPTION
├── NAMESPACE                     # roxygen2-generated
├── NEWS.md
├── cran-comments.md
├── README.Rmd / README.md
├── _pkgdown.yml
├── .Rbuildignore
├── .github/
│   └── workflows/
│       ├── R-CMD-check.yaml
│       ├── test-coverage.yaml
│       └── pkgdown.yaml
│
├── R/                            # User-facing R wrappers + reference fns
│   ├── rsofuncpp-package.R
│   ├── run_pmodel_cpp_bysite.R
│   ├── runread_pmodel_cpp.R
│   ├── run_biomee_cpp_bysite.R
│   ├── runread_biomee_cpp.R
│   ├── calib_sofun.R
│   ├── cost_functions.R
│   ├── init_dates_dataframe.R
│   ├── validate_inputs.R         # shared checkmate helpers
│   └── reference/                 # Stage 1 pure-R implementations (_r suffix)
│       ├── pmodel_photosynth_r.R
│       ├── pmodel_plant_traits_r.R
│       ├── pmodel_gpp_r.R
│       ├── pmodel_soilmstress_r.R
│       ├── pmodel_temp_response_r.R
│       ├── splash_r.R
│       ├── biomee_gpp_r.R
│       ├── biomee_demography_r.R
│       └── biomee_soil_r.R
│
├── src/
│   ├── Makevars
│   ├── Makevars.win
│   ├── .gitignore                 # *.o, *.so, *.dll, RcppExports.cpp auto
│   ├── RcppExports.cpp             # auto-generated
│   ├── rcpp_pmodel.cpp             # Rcpp bridge for P-model
│   ├── rcpp_biomee.cpp           # Rcpp bridge for BiomeE
│   └── cpp/                        # standalone C++17 core
│       ├── CMakeLists.txt          # optional standalone build + Catch2
│       ├── include/rsofuncpp/
│       │   ├── types.hpp
│       │   ├── params_core.hpp     # constants & shared types
│       │   ├── utils.hpp           # sofunutils replacements
│       │   ├── forcing_pmodel.hpp
│       │   ├── forcing_biomee.hpp
│       │   ├── pmodel/
│       │   │   ├── photosynth.hpp    # Farquhar biochemistry
│       │   │   ├── plant_traits.hpp  # leaf traits from optimality
│       │   │   ├── gpp.hpp           # P-model GPP / LUE
│       │   │   ├── temp_response.hpp # ftemp_kphio
│       │   │   ├── soilm_stress.hpp  # empirical soil moisture stress
│       │   │   └── biosphere.hpp     # annual P-model driver loop
│       │   ├── splash/
│       │   │   └── water_balance.hpp # SPLASH soil water balance
│       │   └── biomee/
│       │       ├── gpp.hpp           # gs_leuning / pmodel photosynthesis
│       │       ├── vegetation.hpp    # cohort demography
│       │       ├── soil.hpp          # multi-layer soil hydraulics
│       │       └── biosphere.hpp     # annual BiomeE driver loop
│       └── src/
│           └── *.cpp                 # translation units listed in Makevars
│
├── tests/
│   ├── testthat.R
│   └── testthat/
│       ├── helper-*.R              # fixtures, tolerances, comparison helpers
│       ├── test-input-validation.R
│       ├── test-pmodel-cpp-vs-r.R
│       ├── test-biomee-cpp-vs-r.R
│       ├── test-pmodel-vs-rsofun.R
│       ├── test-biomee-vs-rsofun.R
│       ├── test-calibration.R
│       └── test-performance-regression.R
│
├── data/                           # .rda objects inherited from rsofun
├── data-raw/                       # scripts to regenerate data/
├── vignettes/
│   ├── pmodel_use.Rmd
│   ├── biomee_use.Rmd
│   └── migrating_from_rsofun.Rmd
├── inst/
│   └── benchmarks/
│       ├── bench_pmodel.R
│       └── bench_biomee.R
└── man/
    └── ...                         # roxygen2 Rd files

4. Fortran-to-C++ module mapping

Fortran module (rsofun src/) C++ header namespace Responsibility
md_params_core rsofuncpp::params Constants, physical constants, compile-time sizes
md_sofunutils rsofuncpp::utils dampen_variability, running, patm, median, trig helpers
md_forcing_pmodel rsofuncpp::pmodel::forcing Build annual climate/CO2/fAPAR vectors
md_forcing_biomee rsofuncpp::biomee::forcing BiomeE climate/CO2 staging
md_params_siml_pmodel rsofuncpp::pmodel::simulation getsteering, spin-up/recycle logic
md_params_siml_biomee rsofuncpp::biomee::simulation BiomeE steering
md_interface_pmodel rsofuncpp::pmodel::state myinterface-style state object
md_interface_biomee rsofuncpp::biomee::state BiomeE state object
md_photosynth rsofuncpp::pmodel::photosynth Farquhar Vcmax, Jmax, gammastar, kmm
md_plant_pmodel rsofuncpp::pmodel::plant_traits Optimal chi, ci, LUE
md_gpp_pmodel rsofuncpp::pmodel::gpp Daily GPP, ftemp_kphio, soilmstress
md_biosphere_pmodel rsofuncpp::pmodel::biosphere Annual loop + output packing
md_waterbal (SPLASH) rsofuncpp::splash::water_balance Soil water balance, AET/PET
md_gpp_biomee rsofuncpp::biomee::gpp gpp() / gs_leuning()
md_vegetation_biomee rsofuncpp::biomee::vegetation Cohort growth, mortality, reproduction
md_soil_biomee rsofuncpp::biomee::soil Layered soil water uptake
md_biosphere_biomee rsofuncpp::biomee::biosphere Annual BiomeE loop
sofun_r_mod / wrappersc.c src/rcpp_*.cpp Rcpp .Call() entry points

5. R API contract

User code written for rsofun should continue to work after substituting rsofuncpp:: for rsofun:::

run_pmodel_cpp_bysite(
  sitename,
  params_siml,
  site_info,
  forcing,
  params_modl,
  makecheck = TRUE,
  verbose = TRUE
)

runread_pmodel_cpp(
  drivers,
  par,
  makecheck = TRUE,
  parallel = FALSE,
  ncores = 1
)

run_biomee_cpp_bysite(
  sitename,
  params_siml,
  site_info,
  forcing,
  params_tile,
  params_species,
  params_soil,
  init_cohort,
  init_soil,
  makecheck = TRUE
)

runread_biomee_cpp(
  drivers,
  makecheck = TRUE,
  parallel = FALSE,
  ncores = 2
)

calib_sofun(
  drivers,
  obs,
  settings,
  ...
)
  • Function names mirror the _f_bysite / _f convention but use _cpp.
  • Column order in forcing data frames is standardized inside the R wrapper, so callers are not responsible for Fortran column order.
  • Output names and units are identical to rsofun.

6. Scientific Software Development Lifecycle (SSDLC)

Every migrated routine follows the four-stage pattern used in xsdm and nicher:

Stage 1 — Pure-R reference (_r)

  • File: R/reference/<name>_r.R
  • Suffix: _r
  • @keywords internal, never exported.
  • Base R only; speed is irrelevant, correctness is everything.
  • Matches the published equations line-by-line.

Example:

calc_optimal_chi_r <- function(kmm, gammastar, ns_star, ca, vpd) {
  # Eq. from Stocker et al. 2020 GMD
  chi <- (gammastar / ca) + (1.0 - gammastar / ca) * ns_star / (ns_star + sqrt(vpd))
  ci <- chi * ca
  list(chi = chi, ci = ci)
}

Stage 2 — C++ implementation (_cpp)

  • File: src/cpp/src/<name>.cpp + src/cpp/include/rsofuncpp/<name>.hpp
  • Suffix: _cpp in the R wrapper.
  • R wrapper validates with checkmate, reshapes, calls C++, resets threads, and returns.
  • If using RcppParallel, always reset threads with on.exit().
calc_optimal_chi_cpp <- function(kmm, gammastar, ns_star, ca, vpd) {
  checkmate::assert_numeric(kmm, finite = TRUE, any.missing = FALSE, len = 1)
  # ...
  on.exit(RcppParallel::setThreadOptions(numThreads = RcppParallel::defaultNumThreads()),
          add = TRUE)
  RcppParallel::setThreadOptions(numThreads = 1L)
  .Call("_rsofuncpp_calc_optimal_chi_cpp", kmm, gammastar, ns_star, ca, vpd)
}

Stage 3 — Cross-language parity test (QA gate)

  • File: tests/testthat/test-<name>-cpp-vs-r.R
  • Test: expect_equal(res_cpp, res_r, tolerance = 1e-10).
  • Must pass before the _r function can be "retired" from the public API; the _r function is never deleted.
  • Test edge cases: NA, Inf, zero-length inputs, single time steps, boundary parameters.

Stage 4 — Optimization / integration

  • Replace the _r default with _cpp behind the same public name.
  • Add performance regression test in inst/benchmarks/.
  • Run R CMD check --as-cran.

7. Testing strategy

7.1 Unit / cross-language tests

Test file What it checks
test-pmodel-cpp-vs-r.R Each P-model kernel (photosynth, plant_traits, gpp, soil stress, temp response, SPLASH) against pure-R reference
test-biomee-cpp-vs-r.R Each BiomeE kernel (gpp, cohort growth, mortality, reproduction, soil water) against pure-R reference
test-input-validation.R checkmate assertions fire with informative messages
test-pmodel-vs-rsofun.R Full run_pmodel_cpp_bysite output equals rsofun::run_pmodel_f_bysite on bundled drivers
test-biomee-vs-rsofun.R Full run_biomee_cpp_bysite output equals rsofun::run_biomee_f_bysite on bundled drivers
test-calibration.R calib_sofun returns same structure and converges with same optimum as rsofun
test-performance-regression.R C++ version is measurably faster than Fortran baseline on a 100-site synthetic workload
test-parallel.R parallel = TRUE with ncores > 1 reproduces parallel = FALSE results

7.2 Tolerance policy

  • Kernel-level cross-language tests: 1e-10 relative or absolute.
  • Full P-model run vs rsofun: variable-specific tolerance derived from mean(abs(cpp - fortran)) / mean(abs(fortran)).
  • The reference tolerance table for each output column lives in tests/testthat/helper-tolerances.R so it can be updated as migration progresses.

7.3 Test data

  • Bundle the same .rda validation objects as rsofun.
  • Use data-raw/ to document how validation objects are generated.
  • Avoid downloading data in tests; all tests must run offline and on CI.

8. Build configuration

8.1 DESCRIPTION

Package: rsofuncpp
Type: Package
Title: C++17 Reimplementation of the 'rsofun' Ecosystem Model
Version: 0.0.0.9000
Authors@R: c(
    person("Angel Luis", "Robles Fernandez",
           email = "a.l.robles.fernandez@gmail.com",
           role = c("aut", "cre"),
           comment = c(ORCID = "0000-0002-4674-4270")),
    person("Benjamin", "Stocker", role = "aut"),
    person("Koen", "Hufkens", role = "aut"),
    person("...", ..., role = "ctb")
  )
Description:
    A drop-in C++17 replacement for the 'rsofun' R package
    (Stocker et al. 2020 <doi:10.5194/gmd-13-1545-2020>),
    removing the Fortran 90 dependency while preserving the same R API
    and validation data. Provides the P-model and BiomeE ecosystem
    simulators with 'Rcpp' and 'RcppParallel' backends.
License: GPL (>= 3)
Encoding: UTF-8
Depends: R (>= 4.1.0)
SystemRequirements: GNU make, C++17
Imports:
    checkmate,
    dplyr,
    purrr,
    tidyr,
    tibble,
    stats,
    utils,
    GenSA,
    BayesianTools,
    Rcpp (>= 1.0.0)
LinkingTo:
    Rcpp (>= 1.0.0),
    RcppParallel (>= 5.1.10)
Suggests:
    testthat (>= 3.0.0),
    rsofun,
    knitr,
    rmarkdown,
    ggplot2,
    covr,
    microbenchmark
VignetteBuilder: knitr
Config/testthat/edition: 3
URL: https://alrobles.github.io/rsofuncpp/, https://github.com/alrobles/rsofuncpp
BugReports: https://github.com/alrobles/rsofuncpp/issues
RoxygenNote: 7.3.3

8.2 src/Makevars

CXX_STD = CXX17
PKG_CXXFLAGS += $(SHLIB_OPENMP_CXXFLAGS) -Icpp/include
PKG_LIBS     += $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
PKG_LIBS     += $(shell ${R_HOME}/bin/Rscript -e "RcppParallel::RcppParallelLibs()")

OBJECTS = \
  RcppExports.o \
  rcpp_pmodel.o \
  rcpp_biomee.o \
  cpp/src/utils.o \
  cpp/src/forcing_pmodel.o \
  cpp/src/forcing_biomee.o \
  cpp/src/pmodel_photosynth.o \
  cpp/src/pmodel_plant_traits.o \
  cpp/src/pmodel_gpp.o \
  cpp/src/pmodel_biosphere.o \
  cpp/src/splash_water_balance.o \
  cpp/src/biomee_gpp.o \
  cpp/src/biomee_vegetation.o \
  cpp/src/biomee_soil.o \
  cpp/src/biomee_biosphere.o

src/Makevars.win mirrors this for Windows.

8.3 .Rbuildignore

^.*\.Rproj$
^\.Rproj\.user$
^README\.Rmd$
^cran-comments\.md$
^\.github$
^data-raw$
^inst/benchmarks/.*\.pdf$
^src/cpp/CMakeLists\.txt$
^src/cpp/tests$

9. Parallelism

  1. Within-site: Use RcppParallel and OpenMP only for kernels that are embarrassingly parallel across time steps or cohorts (e.g., daily GPP loop, cohort photosynthesis).
  2. Across-site: Keep runread_*_cpp using future/furrr or RcppParallel over sites, matching rsofun's parallel argument.
  3. Thread safety: Pure functions only; no global mutable state in the C++ core. The Fortran myinterface module becomes an explicit state object passed by reference.
  4. Reset threads: every R wrapper that changes RcppParallel thread count must restore it on exit.

10. Migration phases and quality gates

Phase Deliverable Quality gate
0. Skeleton DESCRIPTION, NAMESPACE, R/, src/, tests/, CI workflows R CMD check passes with 0 errors/warnings on an empty but valid package
1. P-model kernels calc_*_r + calc_*_cpp for photosynthesis, plant traits, GPP, soil stress, temp response, SPLASH All test-pmodel-cpp-vs-r.R tests pass
2. P-model integration run_pmodel_cpp_bysite, runread_pmodel_cpp Output equal to rsofun::run_pmodel_f_bysite on p_model_drivers and p_model_drivers_vcmax25
3. BiomeE kernels Demography, soil, GPP, annual loop as _r + _cpp All test-biomee-cpp-vs-r.R tests pass
4. BiomeE integration run_biomee_cpp_bysite, runread_biomee_cpp Output equal to rsofun::run_biomee_f_bysite on biomee_p_model_drivers and biomee_gs_leuning_drivers
5. Calibration calib_sofun + cost functions Same calibration results as rsofun on bundled examples
6. Polish Vignettes, pkgdown, benchmarks, CRAN preflight R CMD check --as-cran clean; vignettes build; site deploys

No phase is merged until its gate passes. The _r reference functions from Stage 1 remain in R/reference/ permanently as executable documentation and as regression anchors.


11. Documentation and CI

11.1 Documentation

  • All exported functions use roxygen2 with runnable examples.
  • Vignettes:
    • pmodel_use.Rmd — drop-in replacement example.
    • biomee_use.Rmd — BiomeE drop-in example.
    • migrating_from_rsofun.Rmd — API differences, parameter mapping, performance tips.
  • README.Rmd builds a benchmark table against rsofun.
  • pkgdown site auto-deployed via GitHub Actions.

11.2 Continuous integration

  • R-CMD-check.yaml — Ubuntu, macOS, Windows; R release and R-devel.
  • test-coverage.yaml — covr with upload to Codecov.
  • pkgdown.yaml — build and deploy site on merge to main.

12. Comparison with reference packages

Lessons from previous migrations:

Lesson source What to reuse in rsofuncpp
ucminfcpp Header-only C++ core, standalone cpp/tests with CMake, numerical parity vs reference package
maxentcpp Method-by-method Fortran/Java → C++ mapping table, _cpp wrappers with checkmate, per-feature cross-language tests
xsdm _r reference implementations + test-*_cpp_vs_r.R tolerance gates, RcppParallel thread reset pattern
nicher Ellipsoidal _r math written line-by-line, multi-start tryCatch safety, S3 class wrappers
xclim Bioclim kernels tested against pure R, matrix/vector dispatch in Rcpp, OpenMP via Makevars

13. Remote validation on reumanlab

The bundled data sets are small enough for local CI. For larger regression campaigns (e.g., 100-site ensembles, long-term benchmarks, CRAN preflight on multiple platforms):

  • Use hermes.ecoseek.org / OpenClaw as an execution gateway.
  • Store nightly results in inst/benchmarks/ or a separate rsofuncpp-results repository.
  • Keep the CI workflow in .github/workflows/ but allow a manual dispatch that offloads to the reumanlab runner.

14. Attribution and license

  • Original rsofun authors (Stocker, Hufkens, and co-authors) are credited in Authors@R with role = "ctb".
  • References and DOIs are listed in DESCRIPTION and inst/CITATION.
  • License: GPL (>= 3) to remain compatible with rsofun.
  • This is an independent reimplementation; no Fortran source is copied or translated verbatim.

15. Calibration and Bayesian modernization

15.1 Current rsofun calibration architecture

  • calib_sofun(drivers, obs, settings, ...) is the dispatcher.
    • settings$method == "gensa" calls GenSA::GenSA() on a scalar cost.
    • settings$method == "bayesiantools" builds a BayesianTools::createBayesianSetup() and runs BayesianTools::runMCMC() with the DEzs sampler.
  • cost_likelihood_pmodel() and cost_likelihood_biomee() compute a Gaussian log-likelihood of observations given model predictions, with an iid error err_<target> per target.
  • cost_rmse_pmodel() and cost_rmse_biomee() compute RMSE for deterministic (GenSA) optimization.
  • The cost functions currently depend on dplyr, tidyr, and purrr to unnest and join nested obs/drivers data frames. For rsofuncpp these must be rewritten in base R so that rsofuncpp does not import the tidyverse.

15.2 Why BayesianTools is a modernization target

  • BayesianTools is maintenance-heavy and pulls DHARMa, coda, numDeriv, and other dependencies.
  • The DEzs sampler is robust but slow and does not use gradient information.
  • HMC/NUTS via Stan gives better exploration in the 9–12 dimensional P-model parameter space, richer diagnostics (divergences, treedepth, rhat, ess), and a clear path to hierarchical / partial-pooling extensions.

15.3 Proposed Stan integration

Because the P-model likelihood depends on a deterministic simulation, the posterior cannot be written entirely inside Stan without a major rewrite. The recommended, least-risk path is:

  1. Expose the C++ time-series kernel as a callable function (e.g. rsofuncpp::run_pmodel_bysite_matrix(params, forcing, site_info)). This is the same code used by run_pmodel_cpp_bysite, just with a C++ signature that accepts raw double* buffers.
  2. Provide an external C++ log-likelihood that Stan can link against. The function signature is approximately:
    double pmodel_log_lik(const Eigen::VectorXd& par,
                          const Eigen::MatrixXd& obs,
                          const Eigen::MatrixXd& forcing,
                          const Eigen::VectorXd& site_info);
    It builds params_modl from par, calls the exposed P-model routine, and sums normal_lpdf(obs | mod, err) for each target.
  3. Generate a tiny Stan model per calibration problem with cmdstanr. The .stan file declares the external function and only describes the parameter block and priors:
    functions {
      real pmodel_log_lik(vector par, matrix obs, matrix forcing, vector site_info);
    }
    parameters {
      real<lower=0.04, upper=0.09> kphio;
      real<lower=0.01, upper=4> err_gpp;
      // ... other calibrated parameters and errors
    }
    model {
      vector[N_par] par;
      par[1] = kphio;
      par[2] = err_gpp;
      // ...
      target += pmodel_log_lik(par, obs, forcing, site_info);
    }
    cmdstanr::cmdstan_model(..., allow_undefined = TRUE) links the rsofuncpp object or a user-supplied glue file.
  4. R wrapper calib_sofun(..., method = "stan") collects drivers and obs, writes the external Stan glue, compiles once per machine, and returns a CmdStanMCMC object together with a cleaned parameter summary.

15.4 Phased implementation plan

  1. Rewrite cost_rmse_pmodel() and cost_likelihood_pmodel() in base R using rsofuncpp::runread_pmodel(). Add parity tests against the pre-computed rsofun output.
  2. Reimplement calib_sofun() with GenSA support and a lightweight parameter builder that no longer depends on BayesianTools for the setup phase.
  3. Add method = "stan" behind a Suggests: cmdstanr dependency once the C++17 time-series P-model backend (run_pmodel_cpp_bysite) is solid.
  4. For now, keep BayesianTools as a transitional Suggests only if needed for strict API compatibility; the long-term default sampler becomes Stan.

16. First concrete steps

  1. Create package skeleton (DESCRIPTION, NAMESPACE, .Rbuildignore, CI, tests/testthat.R).
  2. Port md_params_core and md_sofunutils to params.hpp and utils.hpp.
  3. Implement calc_optimal_chi_r → calc_optimal_chi_cpp as the first SSDLC demonstration, including test-calc-optimal-chi-cpp-vs-r.R.
  4. Expand outward through P-model kernels, then full P-model integration, then BiomeE.