Default warmup of 250 leaves an intermittent non-convergence rate
The default warmup = 250 in stan_sampling_opts() is well below Stan's own default of 1000. At 250, a small fraction of fits end up with one chain that never adapts: it saturates the maximum treedepth on every post-warmup draw and never mixes, so R-hat blows up while the other chains look fine. More warmup makes it go away.
Evidence
A sweep of example_confirmed[1:60] with future = "project", 200 seeds per warmup, run on the LSHTM HPC from a single build:
| warmup |
n |
failures |
median clean fit |
median failed fit |
| 250 |
200 |
4 (2.0%) |
80s |
368s |
| 500 |
200 |
0 |
92s (+15%) |
— |
| 1000 |
200 |
0 |
107s (+34%) |
— |
Fisher's exact test, 250 against pooled 500+1000: p = 0.012.
This is the one-chain failure flagged in #1467, where it was put down to the approximate GP tweaks. It is not new: a run at the pre-#1396 commit shows the same signature (one chain saturating treedepth while the others mix), so the GP work surfaced it rather than caused it. Because the vignettes run several fits per render without a fixed seed, a per-fit rate this size surfaces as the odd chunk failing on any given render, and a different chunk on a re-run, which is most likely why it read as newly appearing.
Cost
Warmup cost is front-loaded. The first ~250 iterations run at roughly 3.5s each (deep trees until the step size settles), later ones at roughly 0.9s each, so raising the warmup costs less than a flat per-iteration estimate would suggest. Once the ~4.6x cost of a failed fit is counted, expected time per fit is about the same at 250 and 500.
Proposal
Raise the default warmup in stan_sampling_opts() from 250 to 500.
Caveat
All of this is one data set. A flat or sparse series could have a different base rate and want a different warmup, so this is worth checking across a few more series before it becomes a firm default.
Default warmup of 250 leaves an intermittent non-convergence rate
The default
warmup = 250instan_sampling_opts()is well below Stan's own default of 1000. At 250, a small fraction of fits end up with one chain that never adapts: it saturates the maximum treedepth on every post-warmup draw and never mixes, so R-hat blows up while the other chains look fine. More warmup makes it go away.Evidence
A sweep of
example_confirmed[1:60]withfuture = "project", 200 seeds per warmup, run on the LSHTM HPC from a single build:Fisher's exact test, 250 against pooled 500+1000: p = 0.012.
This is the one-chain failure flagged in #1467, where it was put down to the approximate GP tweaks. It is not new: a run at the pre-#1396 commit shows the same signature (one chain saturating treedepth while the others mix), so the GP work surfaced it rather than caused it. Because the vignettes run several fits per render without a fixed seed, a per-fit rate this size surfaces as the odd chunk failing on any given render, and a different chunk on a re-run, which is most likely why it read as newly appearing.
Cost
Warmup cost is front-loaded. The first ~250 iterations run at roughly 3.5s each (deep trees until the step size settles), later ones at roughly 0.9s each, so raising the warmup costs less than a flat per-iteration estimate would suggest. Once the ~4.6x cost of a failed fit is counted, expected time per fit is about the same at 250 and 500.
Proposal
Raise the default
warmupinstan_sampling_opts()from 250 to 500.Caveat
All of this is one data set. A flat or sparse series could have a different base rate and want a different warmup, so this is worth checking across a few more series before it becomes a firm default.