Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Approximation and bounding techniques for the Fisher-Rao distances between parametric statistical models (2403.10089v3)

Published 15 Mar 2024 in cs.IT, cs.CV, cs.LG, and math.IT

Abstract: The Fisher-Rao distance between two probability distributions of a statistical model is defined as the Riemannian geodesic distance induced by the Fisher information metric. In order to calculate the Fisher-Rao distance in closed-form, we need (1) to elicit a formula for the Fisher-Rao geodesics, and (2) to integrate the Fisher length element along those geodesics. We consider several numerically robust approximation and bounding techniques for the Fisher-Rao distances: First, we report generic upper bounds on Fisher-Rao distances based on closed-form 1D Fisher-Rao distances of submodels. Second, we describe several generic approximation schemes depending on whether the Fisher-Rao geodesics or pregeodesics are available in closed-form or not. In particular, we obtain a generic method to guarantee an arbitrarily small additive error on the approximation provided that Fisher-Rao pregeodesics and tight lower and upper bounds are available. Third, we consider the case of Fisher metrics being Hessian metrics, and report generic tight upper bounds on the Fisher-Rao distances using techniques of information geometry. Uniparametric and biparametric statistical models always have Fisher Hessian metrics, and in general a simple test allows to check whether the Fisher information matrix yields a Hessian metric or not. Fourth, we consider elliptical distribution families and show how to apply the above techniques to these models. We also propose two new distances based either on the Fisher-Rao lengths of curves serving as proxies of Fisher-Rao geodesics, or based on the Birkhoff/Hilbert projective cone distance. Last, we consider an alternative group-theoretic approach for statistical transformation models based on the notion of maximal invariant which yields insights on the structures of the Fisher-Rao distance formula which may be used fruitfully in applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (103)
  1. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131–142, 1966.
  2. S Amari. Finsler geometry of non-regular statistical models. RIMS Kokyuroku (in Japanese), Non-Regular Statistical Estimation, Ed. M. Akahira, 538:81–95, 1984.
  3. Shun-ichi Amari. Finsler geometry of non-regular statistical models. RIMS Kokyuroku (in Japanese), Non-Regular Statistical Estimation, Ed. M. Akahira, 538:81–95, 1984.
  4. Shun-ichi Amari. Information Geometry and Its Applications. Applied Mathematical Sciences. Springer Japan, 2016.
  5. Curvature of Hessian manifolds. Differential Geometry and its Applications, 33:1–12, 2014.
  6. Attila Andai. On the geometry of generalized Gaussian distributions. Journal of Multivariate Analysis, 100(4):777–793, 2009.
  7. The Pontryagin Forms of Hessian Manifolds. In International Conference on Geometric Science of Information, pages 240–247. Springer, 2015.
  8. Christoph Arndt. Information measures: information and its description in science and engineering. Springer Science & Business Media, 2001.
  9. Colin Atkinson and Ann F. S. Mitchell. Rao’s distance measure. Sankhyā: The Indian Journal of Statistics, Series A, pages 345–365, 1981.
  10. Dually flat manifolds and global information geometry. Open Systems & Information Dynamics, 9(2):195–200, 2002.
  11. Elliptical Wishart Distribution: Maximum Likelihood Estimator from Information Geometry. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  12. Miroslav Bacák. Convex analysis and optimization in Hadamard spaces, volume 22. Walter de Gruyter GmbH & Co KG, 2014.
  13. Exponential transformation models. Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences, 379(1776):41–65, 1982.
  14. Ole Barndorff-Nielsen. Information and exponential families. John Wiley & Sons, 2014.
  15. Maurice S Bartlett. Approximate confidence intervals. II. More than one unknown parameter. Biometrika, 40(3/4):306–317, 1953.
  16. Michèle Basseville. Divergence measures for statistical data processing: An annotated bibliography. Signal Processing, 93(4):621–633, 2013.
  17. Geodesic estimation in elliptical distributions. Journal of Multivariate Analysis, 63(1):35–46, 1997.
  18. Anil Bhattacharyya. On a measure of divergence between two multinomial populations. Sankhyā: the indian journal of statistics, pages 401–406, 1946.
  19. Garrett Birkhoff. Extensions of Jentzsch’s theorem. Transactions of the American Mathematical Society, 85(1):219–227, 1957.
  20. N Bouhlel and D Rousseau. Exact Rényi and Kullback-Leibler Divergences Between Multivariate t𝑡titalic_t-Distributions. IEEE Signal Processing Letters, 2023.
  21. Kullback–Leibler divergence between multivariate generalized Gaussian distributions. IEEE Signal Processing Letters, 26(7):1021–1025, 2019.
  22. Lev M. Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200–217, 1967.
  23. Metric spaces of non-positive curvature, volume 319. Springer Science & Business Media, 2013.
  24. The information metric for univariate linear elliptic models. Statistics & Risk Modeling, 6(3):209–222, 1988.
  25. Entropy differential metric, distance and divergence measures in probability spaces: A unified approach. Journal of Multivariate Analysis, 12(4):575–596, 1982.
  26. Geometric modeling in probability and statistics, volume 121. Springer, 2014.
  27. A distance between elliptical distributions based in an embedding into the Siegel group. Journal of Computational and Applied Mathematics, 145(2):319–334, 2002.
  28. An explicit solution of information geodesic equations for the multivariate normal model. Statistics & Risk Modeling, 9(1-2):119–138, 1991.
  29. Measure, integral and probability, volume 14. Springer, 2004.
  30. Mahalanobis Prasanta Chandra. On the generalised distance in statistics. In Proceedings of the National Institute of Sciences of India, volume 2, pages 49–55, 1936.
  31. Upper bounds for Rao distance on the manifold of multivariate elliptical distributions. Automatica, 129:109604, 2021.
  32. Stochastic control liaisons: Richard Sinkhorn meets Gaspard Monge on a Schrodinger bridge. Siam Review, 63(2):249–313, 2021.
  33. Nikolai Nikolaevich Chentsov. Statiscal decision rules and optimal inference. Monog, 53, 1982.
  34. Margaret Ann Chmielewski. Elliptically symmetric distributions: A review and bibliography. International Statistical Review/Revue Internationale de Statistique, pages 67–74, 1981.
  35. Hamilton-Jacobi approach to potential functions in information geometry. Journal of Mathematical Physics, 58(6), 2017.
  36. Imre Csiszár. Information-type measures of difference of probability distributions and indirect observation. studia scientiarum Mathematicarum Hungarica, 2:229–318, 1967.
  37. Fast linear algebra is stable. Numerische Mathematik, 108(1):59–91, 2007.
  38. Encyclopedia of distances. Springer, 2009.
  39. James G Dowty. Chentsov’s theorem for exponential families. Information Geometry, 1:117–135, 2018.
  40. Faster matrix multiplication via asymmetric hashing. In 2023 IEEE 64th Annual Symposium on Foundations of Computer Science (FOCS), pages 2129–2138. IEEE, 2023.
  41. Analytical properties of generalized Gaussian distributions. Journal of Statistical Distributions and Applications, 5(1):1–40, 2018.
  42. Morris L Eaton. A characterization of spherical distributions. Journal of Multivariate Analysis, 20(2):272–276, 1986.
  43. Morris L Eaton. Group invariance applications in statistics. IMS, 1989.
  44. Hyperbolic distributions in finance. Bernoulli, pages 281–299, 1995.
  45. Minimum Divergence Methods in Statistical Machine Learning. Springer, 2022.
  46. P. S. Eriksen. Geodesics connected with the Fisher metric on the multivariate normal manifold. In Proceedings of the GST Workshop, Lancaster, UK, pages 28–31, 1987.
  47. Kai Wang Fang. Symmetric multivariate and related distributions. CRC Press, 2018.
  48. On choosing and bounding probability metrics. International statistical review, 70(3):419–435, 2002.
  49. An introduction to Riemannian geometry. With Applications, 2012.
  50. Geometry and fixed-rate quantization in Riemannian metric spaces induced by separable Bregman divergences. In 4th International Conference on Geometric Science of Information (GSI), pages 351–358. Springer, 2019.
  51. Multivariate exponential power distributions as mixtures of normal distributions with bayesian applications. Communications in Statistics—Theory and Methods, 37(6):972–985, 2008.
  52. The geometry of complex domains, volume 291. Springer Science & Business Media, 2011.
  53. Elliptically contoured models in statistics and Portfolio theory. Springer, 2016.
  54. Harold Hotelling. Spaces of statistical parameters. Bull. Amer. Math. Soc, 36:191, 1930. First mention hyperbolic geometry for Fisher-Rao metric of location-scale family.
  55. Hiroto Inoue. Group theoretical study on geodesics for the elliptical models. In International Conference on Geometric Science of Information, pages 605–614. Springer, 2015.
  56. Alan Treleven James. The variance information manifold and the functions on it. In Multivariate Analysis–III, pages 157–169. Elsevier, 1973.
  57. Jürgen Jost. Mathematical concepts. Springer, 2015.
  58. Douglas Kelker. Distribution theory of spherical distributions and a location-scale parameter generalization. Sankhyā: The Indian Journal of Statistics, Series A, pages 419–430, 1970.
  59. Shimpei Kobayashi. Geodesics of multivariate normal distributions and a Toda lattice type Lax pair. Physica Scripta, 98(11):115241, oct 2023. arXiv preprint 2304.12575.
  60. The Laplace distribution and generalizations: a revisit with applications to communications, economics, engineering, and finance. Number 183. Springer Science & Business Media, 2001.
  61. Multivariate t𝑡titalic_t-distributions and their applications. Cambridge University Press, 2004.
  62. WJ Krzanowski. Rao’s distance between normal populations that have common principal components. Biometrics, pages 1467–1471, 1996.
  63. Bayesian sensitivity analysis with the Fisher–Rao metric. Biometrika, 102(3):601–616, 2015.
  64. Parametric information geometry with the package Geomstats. ACM Transactions on Mathematical Software, 49(4):1–26, 2023.
  65. Birkhoff’s version of Hilbert’s metric and its applications in analysis. 2014.
  66. Fisher-Rao metric, geometry, and complexity of neural networks. In The 22nd international conference on artificial intelligence and statistics, pages 888–896. PMLR, 2019.
  67. Prasanta Chandra Mahalanobis. On the generalized distance in statistics. National Institute of Science of India, 1936.
  68. Interpretable scientific discovery with symbolic regression: a review. Artificial Intelligence Review, 57(1):2, 2024.
  69. Rao distances. Journal of Multivariate Analysis, 92(1):97–115, 2005.
  70. Geomstats: a Python package for Riemannian geometry in machine learning. The Journal of Machine Learning Research, 21(1):9203–9211, 2020.
  71. Ann FS Mitchell. Statistical manifolds of univariate elliptic distributions. International Statistical Review, pages 1–16, 1988.
  72. Ann FS Mitchell. The information matrix, skewness tensor and α𝛼\alphaitalic_α-connections for the general multivariate elliptic distribution. Annals of the Institute of Statistical Mathematics, 41:289–304, 1989.
  73. The Mahalanobis distance and elliptic distributions. Biometrika, 72(2):464–467, 1985.
  74. Keiji Miura. An introduction to maximum likelihood estimation and information geometry. Interdisciplinary Information Sciences, 17(3):155–174, 2011.
  75. On Closed-Form expressions for the Fisher-Rao Distance. arXiv preprint arXiv:2304.14885, 2023.
  76. Differential geometry with extreme eigenvalues in the positive semidefinite cone. arXiv preprint arXiv:2304.07347, 2023.
  77. Robb J Muirhead. Aspects of multivariate statistical theory. John Wiley & Sons, 2009.
  78. Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2):429–443, 1997.
  79. Generalizing point embeddings using the Wasserstein space of elliptical distributions. Advances in Neural Information Processing Systems, 31, 2018.
  80. Yoshimasa Nakamura. Algorithms associated with arithmetic, geometric and harmonic means and integrable systems. Journal of computational and applied mathematics, 131(1-2):161–174, 2001.
  81. Frank Nielsen. A Simple Approximation Method for the Fisher–Rao Distance between Multivariate Normal Distributions. Entropy, 25(4):654, 2023.
  82. The hyperbolic Voronoi diagram in arbitrary dimension. arXiv preprint arXiv:1210.8234, 2012.
  83. On f𝑓fitalic_f-divergences between Cauchy distributions. IEEE Transactions on Information Theory, 69(5):3150–3171, May 2023.
  84. On the f𝑓fitalic_f-divergences between hyperboloid and Poincaré distributions. In International Conference on Geometric Science of Information, pages 176–185. Springer, 2023.
  85. On the f𝑓fitalic_f-divergences between densities of a multivariate location or scale family. Statistics and Computing, 34(1):60, 2024.
  86. Roger D Nussbaum. Finsler structures for the part metric and Hilbert’s projective metric and applications to ordinary differential equations. 1994.
  87. The Fisher–Rao distance between multivariate normal distributions: Special cases, bounds and applications. Entropy, 22(4):404, 2020.
  88. C Radhakrishna Rao. Information and accuracy attainable in the estimation of statistical parameters. Bulletin of the Calcutta Mathematical Society, 37(3):81–91, 1945.
  89. C Radhakrishna Rao. Information and the accuracy attainable in the estimation of statistical parameters. In Breakthroughs in Statistics: Foundations and basic theory, pages 235–247. Springer, 1992.
  90. Computing the Rao distance for Gamma distributions. Journal of computational and applied mathematics, 157(1):155–167, 2003.
  91. Rafael Schmidt. Credit risk modelling and estimation via elliptical copulae. In Credit Risk: Measurement, Evaluation and Management, pages 267–289. Springer, 2003.
  92. Hirohiko Shima. The geometry of Hessian structures. World Scientific, 2007.
  93. Tomer Shushi. Generalized skew-elliptical distributions are closed under affine transformations. Statistics & Probability Letters, 134:1–4, 2018.
  94. CL Siegel. Symplectic geometry. Am. J. Math., 65:1–86, 1964.
  95. Lene Theil Skovgaard. A Riemannian geometry of the multivariate normal model. Scandinavian journal of statistics, pages 211–223, 1984.
  96. Stephen M Stigler. The epic story of maximum likelihood. Statistical Science, pages 598–620, 2007.
  97. Is affine-invariance well defined on SPD matrices? A principled continuum of metrics. In 4th International Conference on Geometric Science of Information (GSI), pages 502–510. Springer, 2019.
  98. Harmonic exponential families on homogeneous spaces. Information Geometry, 4(1):215–243, 2021.
  99. Geodesics on the manifold of multivariate generalized Gaussian distributions with an application to multicomponent texture discrimination. International Journal of Computer Vision, 95:265–286, 2011.
  100. On the geometry of multivariate generalized Gaussian models. Journal of mathematical imaging and vision, 43:180–193, 2012.
  101. Statistical tests for the inverse Gaussian distribution based on Rao distance. Sankhya: The Indian Journal of Statistics, Series A, pages 80–103, 1993.
  102. A comprehensive survey of loss functions in machine learning. Annals of Data Science, pages 1–26, 2020.
  103. A new class of symmetric distributions including the elliptically symmetric logistic. Communications in Statistics-Theory and Methods, 51(13):4537–4558, 2022.
Citations (2)
View on Semantic Scholar

Summary

  • The paper addresses the computational difficulty of calculating exact Fisher-Rao distances by presenting novel approximation and bounding techniques for parametric statistical models.
  • Key methods introduced include the Fisher-Manhattan upper bound using 1D sums, curve length approximations via f-divergences, and algorithms leveraging known (pre)geodesics for precision.
  • Additional insights explore Hessian metrics, bounds via Jeffreys-Bregman divergences, analysis for elliptical models, and group-theoretic approaches using maximal invariants.

Overview of Approximation and Bounding Techniques for the Fisher-Rao Distances

The paper "Approximation and Bounding Techniques for the Fisher-Rao Distances" by Frank Nielsen discusses the intrinsic nature of Fisher-Rao distances and presents novel approximation and bounding techniques for evaluating these distances within parametric statistical models. The Fisher-Rao metric, recognized for its appeal in statistical inference due to its invariance under diffeomorphisms, defines a Riemannian geometric structure on statistical models, which poses challenges for its computation. Particularly, the calculation of the Fisher-Rao distance isn't straightforward for most statistical models, including the prevalent multivariate normal distributions, necessitating approximation methods.

Key Contributions

  1. Singular Cases in Fisher-Rao Distances:
    • Uniparametric Models: The paper elaborates on the tractable computation of the Fisher-Rao distances for uniparametric models by deriving it as an infimum over all length curves connecting two distributions, pointing to a relatively straightforward one-dimensional scenario.
    • Product of Independent Models: Fisher-Rao distances are developed for product models by treating them as Euclidean distances across product spaces, which simplifies the otherwise complex calculations into more manageable components.
  2. Fisher-Manhattan Upper Bound: The paper introduces a Fisher-Manhattan upper bound, an innovative method for approximating higher-dimensional Fisher-Rao distances using the sum of 1D Fisher-Rao distances along the hypercube edges. This approach leverages the simplicity of submodel distances to construct an upper bound for multidimensional models.
  3. Approximation Techniques:
    • Curve Length Approximation: For models where Fisher-Rao geodesics are unknown, the paper suggests using curve length approximations combined with the properties of f-divergences (notably Jeffreys divergence) to estimate lengths efficiently.
    • Geodesic and Pregeodesic-Based Approximations: Algorithms are proposed for leveraging known (pre)geodesics to ensure arbitrary precision in Fisher-Rao approximations, demonstrating applicability when certain metric properties are verifiable.
  4. Hessian Metrics and Bounds: A detailed paper into Hessian metrics is presented, establishing upper bounds for Fisher-Rao distances using Jeffreys-Bregman divergences which, in turn, rely on verifying Hessian property conditions. This delimitation between bounds proves instrumental for models fitting Hessian traits.
  5. Elliptical and General Models: For elliptical distributions, the paper offers an analysis of both Fisher metric specifics and geodesic equations. This sector considers practical distance bounds through Calvo and Oller's isometric works, offering ways to navigate more complex covariance-driven models.
  6. Group-Theoretic Insights and Maximal Invariants: The research suggests a new direction by examining group actions on parameter measures to elucidate invariants that can characterize Fisher-Rao distances. This abstract algebraic view enhances understanding where traditional geometrical tactics encounter limits.

Implications and Future Work

The proposed techniques promise substantial benefits in statistical computing, especially in model evaluation and hypothesis testing where exact Fisher-Rao calculations are computationally infeasible. The advancement in approximations for models like multivariate Gaussians and their modifications can translate directly to improvements in fields such as machine learning, where model efficiency is crucial.

In terms of theoretical advancements, the exploration of connections between information geometry and Hessian manifolds opens opportunities for enhanced algorithmic frameworks. Future challenges remain in refining these methods for arbitrary complex models and analyzing the applicability of maximal invariant techniques across varying parametric forms.

Moreover, verifying uniqueness in geodesic paths relative to Fisher-Rao distances presents a compelling problem, extending the work beyond current computational confines to theoretical determination. By leveraging diverse mathematical structures, Nielsen's work underscores critical pathways for both statistical theory development and applied computation advancements in geometric analysis of models.