Round-level convergence criteria for sequential NPE - is this still on the roadmap?

[vitorbborges]

🚀 Feature Request

Is your request related to a problem?

The sequential loop in sbi is entirely user-specified - the how-to guide for sequential methods is a bare for _ in range(num_rounds) with a fixed number of simulations per round. As far as I can tell, there's nothing in the package that indicates whether running another round is buying anything, or whether the posterior estimate has stopped moving.

Describe the solution you'd like

Mostly I'd like to ask whether anything of this kind is on your radar, before I invest effort in it.

As a concrete example of the sort of thing I mean: #840 suggested tracking the KL divergence between the proposal and the posterior across rounds as a stopping criterion, and the response at the time was that it sounded reasonable - but the issue was closed without an implementation. Something in that direction (KL between successive rounds' posterior estimates, and/or between the current posterior and the prior) is cheap to compute for NPE.

So, concretely:

1. Is a round-level convergence/stopping diagnostic something you'd want in sbi at all, or do you consider it out of scope and better left to the user's own loop?
2. Where would you want it to live? Given New organization of SNPE methods #1241 and the loss-strategy refactor currently in flight in Refactor NPE-C loss logic to Strategy Pattern #1755, where round-level concerns are being pulled out into composable objects and train() is gaining explicit strategy arguments, a round-level diagnostic looks like it might be a similarly separable concern. A self-contained utility in sbi.diagnostics that users call from their own loop would be the least invasive option.

Describe alternatives you've considered

Alternatively this could stay purely a documentation matter: a short note in the sequential how-to guide on how to choose the number of rounds and simulations per round would already help.

📌 Additional Context

I'd be glad to help implement this, however I'd rather agree on the design first than open a PR that cuts across the refactor currently in progress. I'm happy either to start with a standalone diagnostic utility plus tests, or to wait until #1755 / #1241 have settled. Whatever fits your plans.

[janfb]

Thanks for the request @vitorbborges !

you are raising an important point here. regarding your questions:

1. yes, that's something useful we should add
2. given the sequential loop is now user-owned and will be become more composable in the future as you noticed correctly (we are a bit behind with reviewing there), I agree that a "self-contained utility in sbi.diagnostics" (or in the train module if this is more trainer-specific) would be the best approach here.

As metrics, KL would be suitable for NPE, C2ST might be better for sampling-based posteriors, for vector field posteriors one would have to check whether it's useful at all. That said, starting with one for NPE as a MVP would be a good approach.

[vitorbborges]

Thanks @janfb, that's clear.

I'll put together the NPE MVP pull request as an utility in sbi.diagnostics.