Statistical Accuracy of Approximate Filtering Methods (2402.01593v2)
Abstract: Estimating the statistics of the state of a dynamical system, from partial and noisy observations, is both mathematically challenging and finds wide application. Furthermore, the applications are of great societal importance, including problems such as probabilistic weather forecasting and prediction of epidemics. Particle filters provide a well-founded approach to the problem, leading to provably accurate approximations of the statistics. However these methods perform poorly in high dimensions. In 1994 the idea of ensemble Kalman filtering was introduced by Evensen, leading to a methodology that has been widely adopted in the geophysical sciences and also finds application to quite general inverse problems. However, ensemble Kalman filters have defied rigorous analysis of their statistical accuracy, except in the linear Gaussian setting. In this article we describe recent work which takes first steps to analyze the statistical accuracy of ensemble Kalman filters beyond the linear Gaussian setting. The subject is inherently technical, as it involves the evolution of probability measures according to a nonlinear and nonautonomous dynamical system; and the approximation of this evolution. It can nonetheless be presented in a fairly accessible fashion, understandable with basic knowledge of dynamical systems, numerical analysis and probability.
- P. Bickel, B. Li and T. Bengtsson “Sharp failure rates for the bootstrap particle filter in high dimensions” In Pushing the limits of contemporary statistics: contributions in honor of Jayanta K. Ghosh 3, Inst. Math. Stat. (IMS) Collect., 2008 DOI: 10.1214/074921708000000228
- E. Calvello, S. Reich and A.M. Stuart “Ensemble Kalman Methods: A Mean Field Perspective” In Acta Numerica, 2025
- “The Mean Field Ensemble Kalman Filter: Near-Gaussian Setting” In arXiv preprint 2212.13239, 2022
- D. Crişan, P. Del Moral and T.J. Lyons “Interacting particle systems approximations of the Kushner-Stratonovitch equation” In Adv. in Appl. Probab. 31.3, 1999, pp. 819–838 DOI: 10.1239/aap/1029955206
- Pierre Del Moral “Nonlinear filtering: interacting particle resolution” In C. R. Acad. Sci. Paris Sér. I Math. 325.6, 1997, pp. 653–658 DOI: 10.1016/S0764-4442(97)84778-7
- Pierre Del Moral and Alice Guionnet “On the stability of interacting processes with applications to filtering and genetic algorithms” In Ann. Inst. H. Poincaré Probab. Statist. 37.2, 2001, pp. 155–194 DOI: 10.1016/S0246-0203(00)01064-5
- “Sequential Monte Carlo methods in practice”, Statistics for Engineering and Information Science Springer-Verlag, New York, 2001 DOI: 10.1007/978-1-4757-3437-9
- Geir Evensen “Data assimilation” The Ensemble Kalman Filter Springer-Verlag, Berlin, 2009 DOI: 10.1007/978-3-642-03711-5
- Geir Evensen “Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics” In J. Geophys. Res. Oceans 99.C5 Wiley Online Library, 1994, pp. 10143–10162
- Geir Evensen, Femke C. Vossepoel and Peter Jan Leeuwen “Data assimilation fundamentals: a unified formulation of the state and parameter estimation problem”, Springer Textbooks in Earth Sciences, Geography and Environment Springer, Cham, 2022 DOI: 10.1007/978-3-030-96709-3
- M.A. Iglesias, K.J.H. Law and A.M. Stuart “Ensemble Kalman methods for inverse problems” In Inverse Problems 29.4, 2013, pp. 045001\bibrangessep20 DOI: 10.1088/0266-5611/29/4/045001
- R.E. Kalman “A new approach to linear filtering and prediction problems” In Trans. ASME Ser. D. J. Basic Engrg. 82.1, 1960, pp. 35–45
- Eugenia Kalnay “Atmospheric modeling, data assimilation and predictability” Cambridge university press, 2003
- Matt J Keeling and Ken TD Eames “Networks and epidemic models” In Journal of the royal society interface 2.4 The Royal Society London, 2005, pp. 295–307
- Kody Law, Andrew Stuart and Konstantinos Zygalakis “Data Assimilation” 62, Texts in Applied Mathematics Springer, Cham, 2015 DOI: 10.1007/978-3-319-20325-6
- F. Le Gland, V. Monbet and V.-D. Tran “Large sample asymptotics for the ensemble Kalman filter” In The Oxford handbook of nonlinear filtering Oxford Univ. Press, Oxford, 2011, pp. 598–631
- Jan Mandel, Loren Cobb and Jonathan D. Beezley “On the convergence of the ensemble Kalman filter” In Appl. Math. 56.6, 2011, pp. 533–541 DOI: 10.1007/s10492-011-0031-2
- “Can local particle filters beat the curse of dimensionality?” In Ann. Appl. Probab. 25.5, 2015, pp. 2809–2866 DOI: 10.1214/14-AAP1061
- “Probabilistic Forecasting and Bayesian Data Assimilation” Cambridge University Press, New York, 2015 DOI: 10.1017/CBO9781107706804
- Daniel Sanz-Alonso, Andrew Stuart and Armeen Taeb “Inverse Problems and Data Assimilation” Cambridge University Press, 2023
- “Obstacles to high-dimensional particle filtering” In Mon. Weather Rev. 136.12, 2008, pp. 4629–4640
- “Particle filters for high-dimensional geoscience applications: A review” In Q. J. R. Meteorol. Soc. 145.723 Wiley Online Library, 2019, pp. 2335–2365
- C. Villani “Optimal transport” Old and new 338, Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] Springer-Verlag, Berlin, 2009 DOI: 10.1007/978-3-540-71050-9