🐛[BUG]: StormScope 3km_10min intermittently returns all-NaN forecasts from finite inputs

[mordisdevone]

Version

Source installation at commit 076242b0d135374dbe5e1858d55335c55e946c9f
(version reports 0.17.0a0).

We have not yet reproduced this against the 0.17.0 release tag.

Installation method

Source

Describe the issue

We observed two StormScope 3km_10min inference runs that completed
without raising an exception but returned entirely non-finite GOES and
MRMS forecast arrays.

Affected forecast initialization times:

• 2026-08-27T06:00:00Z
• 2026-08-28T06:00:00Z

Input validation immediately before inference showed:

• GOES channel finite fractions: approximately 0.98
• MRMS reflectivity, base reflectivity, and GLM finite fractions: 1.00

The completed forecast arrays contained:

• GOES: finite fraction 0.0 for all eight output channels
• MRMS: finite fraction 0.0 for reflectivity, base reflectivity, and GLM

Other initialization times have produced finite forecasts in the same
environment.

The coupled rollout follows the public StormScope example. Relevant
configuration:

• model: 3km_10min
• batch size: 1
• num_steps=25
• S_churn=10
• amp=True
• compile=False
• no explicit RNG seed
• 36 autoregressive 10-minute forecast steps

Environment:

• Linux x86_64
• NVIDIA L40S
• Python 3.12
• PyTorch 2.10.0 / CUDA 12.8
• NATTEN 0.21.6 CUDA build

We did not instrument the original runs deeply enough to determine
whether the first non-finite value originated in GOES, MRMS,
conditioning construction, or the diffusion sampler.

Could you advise whether this is a known instability involving AMP,
stochastic sampling, or reduced sampler steps? Would you recommend
testing amp=False, the default 100 sampler steps, or a specific fixed
seed first?

It may also be useful for Earth2Studio to fail explicitly when
conditioning or model output becomes non-finite, rather than completing
the rollout with all-NaN arrays.

We can run a targeted configuration matrix and provide redacted tensor
statistics or checkpoint hashes if useful.

[pzharrington]

Thanks for reporting. Reducing the number of sampling steps this much is relatively untested in terms of NaN occurrence, and I wouldn't be surprised if the composition of that with amp led to some cases where the numerics get messed up. If you can manage to determine the exact seed(s) on those initializations which can deterministically reprocuce the NaNs, that would be helpful for further investigation (e.g. to see if amp has an impact or not)

Internal testing on the number of sampling steps found that around 48 steps is where we started to notice more substantial degradation in model skill, so I'm not aware of much testing that was done lower than that. However, that was with the fixed S_churn=10 setting; if reducing the number of sampling steps, it also makes sense to reduce S_churn, so if you wanted to experiment in the num_steps<48 regime I'd recommend trying to reduce S_churn as well.

[mordisdevone]

Thanks — we've now instrumented the failing path and can narrow this considerably.

The non-finite values originate before diffusion sampling, in StormScope.build_condition on the GOES side — not in the sampler. A fail-closed guard wrapped around build_condition (installed immediately before sampling) reports, e.g. for init 2026-09-01T06:00:00Z, 940723 non-finite values out of 97255424 in the GOES condition tensor at forecast step 1. Pre-guard runs with the same signature (2026-08-27T06Z, 2026-08-28T06Z — the two cases in the original report) completed and returned all-NaN arrays.

The pattern is deterministic by initialization time. Since 2026-08-27, every 06:00Z initialization has hit this (6 of 6 through 2026-09-01); no 00Z, 12Z, or 18Z initialization in the same window has. 06Z initializations on 2026-08-20 and 2026-08-26 were fine.

What differs in the inputs. Our input check measures per-channel finite fractions over the full fetched ABI L2 MCMIPC grid. On healthy runs every channel reads exactly 0.9874 (the fixed off-domain portion). On the failing 06Z runs the fractions vary by channel, e.g. abi01c=0.9724, abi02c=0.9686, abi03c=0.9756, abi07c=0.9696, abi08c=0.9733, abi09c=0.9769, abi10c=0.9646, abi13c=0.9771 — i.e. a small (~1%), channel-dependent set of NaN pixels inside the domain. MRMS/GLM channels are 1.0000 in both cases.

Why it propagates (reading stormscope.py at 076242b). build_input_interpolator / build_conditioning_interpolator derive valid_mask purely geometrically (nearest-neighbour distance vs max_dist_km), and _forward fills only ~valid_mask with _INPUT_INVALID_FILL_CONSTANT (x_norm = torch.where(self.valid_mask, x_norm, fill); same for conditioning). A NaN value at a geometrically valid gridpoint therefore survives normalization into build_condition. The only nan_to_num on this path is in interpolate_glm, so GLM is protected but ABI/MRMS are not.

So this appears independent of num_steps / amp (we'll still run your suggested matrix with lower S_churn for the skill question, but it shouldn't be needed to explain the NaNs).

Attribution of the input NaNs (hypothesis, not yet confirmed against the ABI DQF flags): the 06Z-only timing and late-August onset are consistent with GOES-East eclipse / keep-out-zone season around satellite local midnight (~05–07Z).

Suggested fix on the library side: treat non-finite input pixels like invalid gridpoints (mask/fill before normalization) and fail explicitly if the condition tensor is non-finite rather than completing the rollout. We can share the per-channel statistics and the exact ABI file lists for the affected initializations if useful.