Skip to content

Add a shared time-varying parameter interface for Rt and infections - #1451

Draft
sbfnk-bot wants to merge 60 commits into
mainfrom
worktree-time-varying-params
Draft

Add a shared time-varying parameter interface for Rt and infections#1451
sbfnk-bot wants to merge 60 commits into
mainfrom
worktree-time-varying-params

Conversation

@sbfnk-bot

Copy link
Copy Markdown
Collaborator

Description

This PR makes first steps towards #600; it does not close it. It introduces a general interface for time-varying model parameters and uses it to express the two existing estimate_infections() models through a single, shared latent-state implementation. The interface is applied to the reproduction number and to the back-calculation infections, but is not yet extended to all parameters.

Time-varying parameter interface

  • New GP() and RW() constructors declare a parameter as a Gaussian process or random walk, in either a mean-reverting (mean = ) or first-difference (init = ) form. The anchor accepts a prior (a <dist_spec>) or a known trajectory supplied as a numeric vector.
  • A shared param_spec superclass underlies dist_spec and the new state_spec, so a parameter's prior can be a constant or a time-varying state.
  • Time-varying parameters vary over the observed period and hold their last estimated value through the forecast horizon.

Reproduction number

  • rt_opts(prior = ...) now accepts a constant value or a GP()/RW() state. The reproduction number is built through the generic state machinery, replacing the bespoke Rt Gaussian-process code path (update_Rt, the separate R0/init-prior handling). The state parameter is named R.

Back-calculation (non-mechanistic) model

  • Latent infections are now estimated as a Gaussian process on the log scale, anchored at an initial value, through the same state machinery. This retires the shared "main" Gaussian process, the deconvolution step, and the smoothed shifted-case prior. The infections process is configured with backcalc_opts(prior = GP(...)), mirroring rt_opts(prior = ...).
  • Note that the previous deconvolution behaviour is not preserved: the shifted-case prior and the multiple backcalc prior modes proved difficult to express within the new state interface, so they have been deliberately retired rather than carried over. The back-calculation results therefore differ from before, and this is a deliberate behaviour change to keep both models on a single shared implementation.

Gaussian process configuration

  • All Gaussian process settings (ls, alpha, kernel, basis_prop, boundary_scale, matern_order, w0) are now arguments of GP(), so a process is fully specified in one place.

Deprecations

  • gp_opts(); supply settings to GP() instead.
  • The gp argument of estimate_infections(), epinow(), and regional_epinow(); configure the process through the model prior.
  • The rw argument and breakpoints (use_breakpoints / the breakpoint data column) of rt_opts(); use rt_opts(prior = RW(...)).
  • The prior (now a GP() specification) and prior_window arguments of backcalc_opts().

Initial submission checklist

  • My PR is based on a package issue and I have explicitly linked it.
  • I have tested my changes locally (using devtools::test() and devtools::check()).
  • I have added or updated unit tests where necessary.
  • I have updated the documentation if required and rebuilt docs if yes (using devtools::document()).
  • I have followed the established coding standards (and checked using lintr::lint_package()).
  • I have added a news item linked to this PR.

sbfnk-bot and others added 29 commits June 16, 2026 21:43
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Avoids overloaded Stan functions, which rstan::expose_stan_functions cannot handle, and exposes state.stan in the test setup.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Non-stationary GP reuses the shared basis via a centred cumulative sum.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: epiforecasts-workflows[bot] <epiforecasts-workflows[bot]@users.noreply.github.com>
States vary over the observed window and hold their last value in the horizon (the 'latest' projection), matching Rt's default.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
A param_spec is any (constant or time-varying) parameter value specification; used wherever a value may be either.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
rt_opts(prior=) now takes a dist_spec or a state_spec (GP/RW); a time-varying Rt prior errors explicitly until the model wiring lands.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Rt now flows through the generic state machinery: rt_opts(prior=) takes a constant value or a GP/RW state; the bespoke update_Rt path, R_mean and init_priors are gone. Verified identical to the previous Rt within MC error. rw is deprecated (use prior = RW()).

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
…-params

# Conflicts:
#	inst/stan/estimate_infections.stan
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Retire the bespoke main Gaussian process: latent infections are now a
Gaussian process on log-infections anchored at an initial value, built
through the same time-varying state machinery as the reproduction number.
Removes the deconvolution step, the shifted-case prior, and the associated
Stan functions and data, and deprecates the backcalc_opts prior arguments.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
…ment

All Gaussian process settings (ls, alpha, kernel, basis_prop, ...) are now
arguments of GP(), so a process is fully specified in one place. gp_opts() is
deprecated. The back-calculation infections process is configured through
backcalc_opts(prior = GP(...)), mirroring rt_opts(prior = ...), and the gp
argument of estimate_infections()/epinow()/regional_epinow() is deprecated.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Update simulate_infections() to register the R and I state parameters, and
exclude these state levels (sampled scaffolding for the time-varying
trajectories) from the scalar-parameter extraction so the reproduction-number
trajectory is no longer overwritten when forecasting. Update tests for the
GP()/RW() interface (random walk via RW(), drop unsupported periodic-kernel and
deprecated breakpoint cases).

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Configure Gaussian processes and random walks through rt_opts(prior = ...) and
backcalc_opts(prior = ...), express a fixed reproduction number as a constant
prior, and drop references to the deprecated gp_opts()/gp argument and rw
argument.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
plot() on a state specification draws sample trajectories from the prior so
the implied time-varying behaviour can be visualised before fitting.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
RW(period = n) holds the value constant for n time steps between random walk
steps, so the value changes every n steps (e.g. period = 7 for a weekly random
walk). This restores the step interval of the deprecated rt_opts(rw = n) within
the time-varying state interface. Vignettes use a weekly random walk where they
previously set rw = 7.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Wrap reproduction-number priors that relied on the implicit default Gaussian
process in GP() (a bare distribution prior is now a constant Rt), use weekly
random walks via RW(period = 7) where rw = 7 was used, and update prose for the
GP(init)/GP(mean) and RW interfaces. Re-rendered all affected vignettes.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
The Gaussian process variant is now chosen through the prior: GP(mean = ...)
for a mean-reverting (stationary) process or GP(init = ...) for one on first
differences. Removes the now-unused stationary flag from the Rt stan data.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
The initial value of each time-varying state (used for the init-anchor prior)
now reuses the trajectory already computed for R / infections / the observation
parameters instead of recomputing it, removing a redundant Gaussian process
evaluation from the autodiff path.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
@sbfnk-bot
sbfnk-bot requested a review from sbfnk June 18, 2026 15:23
sbfnk-bot and others added 2 commits July 20, 2026 17:16
…rk (#1482)

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
…-params

# Conflicts:
#	R/create.R
#	inst/stan/estimate_infections.stan
#	inst/stan/functions/params.stan
#	vignettes/EpiNow2.Rmd.orig
@github-actions

Copy link
Copy Markdown
Contributor

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 4bc10d9 is merged into main:

  • ✔️estimate_dist: 635ms -> 634ms [-1.87%, +1.4%]
  • 🚀estimate_infections_gp: 14.4s -> 3.65s [-78.68%, -70.81%]
  • 🚀estimate_infections_rw: 7.42s -> 3.7s [-55.88%, -44.47%]
  • ❗🐌estimate_secondary: 1.38s -> 1.6s [+6.59%, +25.4%]
  • ❗🐌estimate_truncation: 1.33s -> 1.51s [+2.15%, +24.76%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

sbfnk-bot and others added 2 commits August 25, 2026 11:05
…-params

# Conflicts:
#	NAMESPACE
#	inst/stan/estimate_truncation.stan
Comment thread R/opts.R Outdated
sbfnk-bot and others added 4 commits August 25, 2026 11:39
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
@github-actions

Copy link
Copy Markdown
Contributor

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 9416634 is merged into main:

  • ✔️estimate_dist: 899ms -> 893ms [-2.31%, +0.87%]
  • 🚀estimate_infections_gp: 15.2s -> 3.67s [-78.55%, -73.08%]
  • 🚀estimate_infections_rw: 8.29s -> 3.68s [-72.89%, -38.38%]
  • ✔️estimate_secondary: 1.74s -> 1.75s [-6.33%, +7.46%]
  • ✔️estimate_truncation: 1.6s -> 13.1s [-938.39%, +2375.83%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

sbfnk and others added 5 commits September 2, 2026 16:56
…-params

# Conflicts:
#	NAMESPACE
#	R/dist_spec.R
#	R/utilities.R
#	_pkgdown.yml
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>

@sbfnk-review-bot sbfnk-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review (fresh-context reviewers, four slices: Stan state machinery, Stan model integration, R interface, R periphery + tests). The two Stan slices verified dimensions, indexing, Jacobians and the data contract with no findings. Three findings on the R side follow. Line-anchored comments are not possible here — the diff exceeds GitHub's size limit and is truncated — so findings are listed with file:line.

1. R/create.R:419 (medium) — fully-fixed configuration crashes at initialisation.
When every registered parameter is fixed (e.g. estimate_infections(data, rt = rt_opts(prior = Fixed(1)), obs = obs_opts(family = "poisson"))), n_params_variable == 0, so out$params <- array(rtruncnorm(0L, a = numeric(0), ...)) aborts with length(a) > 0 is not TRUE. The sibling state vectors use the rtruncnorm0() zero-guard; out$params did not. Before the GP hyperparameters moved into per-state settings there were always ≥2 variable params, so this path was unreachable — a regression. Fixed by routing out$params through rtruncnorm0().

2. R/create.R:1065 (low) — GP lengthscale max truncation dropped, needs author intent.
gp_rho_params <- c(gp_rho_params, numeric_params(gp$ls, "lengthscale", name)) emits only the two distribution parameters; the upper bound of the default GP(ls = LogNormal(mean = 21, sd = 7, max = 60)) is not passed to Stan, so the lengthscale prior is effectively untruncated. Whether the max is meant to be a hard Stan truncation or only an R-side default hint is a design decision — left for @sbfnk rather than changed.

3. R/utilities.R:493 (low) — make_init_priors() is dead code.
This PR removed both call sites (estimate_infections, simulate_infections) and the init-prior Stan data blocks its roxygen references, leaving the function orphaned. Fixed by removing it and its generated man page.

@sbfnk-bot

Copy link
Copy Markdown
Collaborator Author

Addressed the review:

  • Finding 1 (create.R:419) — fixed in b7d0d5b1: out$params now uses the rtruncnorm0() zero-guard, so a fully-fixed configuration initialises instead of aborting. Verified: estimate_infections(rt = rt_opts(prior = Fixed(1)), obs = obs_opts(family = "poisson")) now reaches sampling.
  • Finding 3 (utilities.R:493) — fixed in 760caa83: removed the orphaned make_init_priors() and its man page.
  • Finding 2 (create.R:1065) — left open for a decision: whether the GP lengthscale max (default 60) should be a hard truncation passed to Stan, or is only an R-side default. No behavioural change made.

devtools::test() on the affected suites (state, opts, create, process) is green; package loads cleanly.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 760caa8 is merged into main:

  • ✔️estimate_dist: 775ms -> 805ms [-0.09%, +7.72%]
  • 🚀estimate_infections_gp: 21.4s -> 4.28s [-89.24%, -70.79%]
  • 🚀estimate_infections_rw: 11.4s -> 4.37s [-93.71%, -29.75%]
  • ✔️estimate_secondary: 2.04s -> 2.18s [-7.09%, +21.56%]
  • ✔️estimate_truncation: 1.88s -> 2s [-0.5%, +13.99%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
@sbfnk-bot

Copy link
Copy Markdown
Collaborator Author

Finding 2 (create.R:1065) — now fixed in 3300f164, superseding the earlier "left open".

On closer inspection this was a genuine regression, not a design question: on main the GP lengthscale bound flowed through generically (params_upper <- vapply(dist, max, ...)T[lb, ub]), but the per-state rewrite applied every state hyperparameter prior with a hardcoded positive_infinity() upper, silently discarding any user-specified max on the GP lengthscale, GP magnitude, or RW step-sd priors.

The fix carries each hyperparameter dist's max() through create_state_data() into data/states.stan (rw_sd_upper, gp_alpha_upper, gp_rho_upper) and into the apply_prior_lp truncation, mirroring how state_init_upper already works. Verified: GP(ls = LogNormal(mean = 14, sd = 7, max = 30)) now yields gp_rho_upper = 30; the default (max = 60, alpha unbounded → Inf) compiles and fits. 80 tests green, lint clean.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 3300f16 is merged into main:

  • 🚀estimate_dist: 552ms -> 536ms [-5.61%, -0.2%]
  • 🚀estimate_infections_gp: 12.4s -> 3.13s [-80.44%, -69.16%]
  • 🚀estimate_infections_rw: 6.29s -> 2.95s [-60.19%, -45.85%]
  • ✔️estimate_secondary: 1.31s -> 1.38s [-1.76%, +12.21%]
  • ✔️estimate_truncation: 1.27s -> 1.33s [-1.14%, +12.07%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Comment thread R/create.R
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 7e47dfd is merged into main:

  • ✔️estimate_dist: 800ms -> 779ms [-6.03%, +0.97%]
  • 🚀estimate_infections_gp: 17.8s -> 6.85s [-98.2%, -24.78%]
  • 🚀estimate_infections_rw: 8.94s -> 4.14s [-58.26%, -49.2%]
  • ✔️estimate_secondary: 2.01s -> 1.95s [-14.05%, +7.89%]
  • ✔️estimate_truncation: 1.85s -> 1.81s [-10.5%, +6.48%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 08f278f is merged into main:

  • ✔️estimate_dist: 594ms -> 609ms [-2.72%, +7.75%]
  • 🚀estimate_infections_gp: 13s -> 3.16s [-80.18%, -71.04%]
  • 🚀estimate_infections_rw: 6.55s -> 3.29s [-55.43%, -44.08%]
  • ✔️estimate_secondary: 1.37s -> 1.39s [-11.98%, +14.86%]
  • ✔️estimate_truncation: 10.8s -> 1.43s [-287.25%, +113.65%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants