Diagnostics for detecting amortized-inference generalization failures

Develop robust general diagnostics for detecting when neural epidemic forecasting methods have encountered difficulties in generalising beyond their training data or when new observations lie outside the training distribution.

Background

The paper identifies two related difficulties for neural methods in inference: uncertainty about whether a neural network can generalise from its training data, and uncertainty about whether a finite training sample adequately covers a new observation. Because GENIE is trained on simulated epidemic trajectories, diagnostics that identify out-of-distribution conditions or unreliable forecasts would be important for assessing its use in real-time epidemic forecasting.

The authors explicitly classify the development of such diagnostics as an open problem, making this an unresolved methodological problem rather than a general recommendation for future work.

References

More generally, this demonstrates fundamental difficulties in the use of neural methods in inference: it is not clear if a neural network will be able to generalise from its training data; and for a finite sample of training data, outside of heavily constrained problems, it is not clear whether a new observation will lie near to that training data. Developing robust general diagnostics for detecting when these difficulties have been encountered is an open problem for amortized inference.

— GENIE: Generative Neural Inference for Epidemics  (2608.20253 - Guzmán-Rincón et al., 20 Aug 2026) in Discussion, paragraph beginning “In the simulated epidemics used to train our model…”

Scaling to larger and more diverse geographies will introduce additional challenges, both computational and modelling related, and the performance of GENIE under such conditions remains to be established.

— GENIE: Generative Neural Inference for Epidemics  (2608.20253 - Guzmán-Rincón et al., 20 Aug 2026) in Discussion, second paragraph under “limitations of this study”