Information Geometry

• Information Geometry is a field that uses differential geometry to describe statistical manifolds via the Fisher information metric and divergence functions.
• It employs dual affine connections and α-connections to characterize estimation efficiency and optimization in both parametric and nonparametric models.
• Its practical applications include natural gradient descent, Cramér-Rao efficiency in estimation, and quantum Fisher metrics for advanced statistical inference.

Information geometry generalizes the differential geometric treatment of spaces of probability distributions by equipping these spaces—statistical manifolds—with a Riemannian metric and dual affine connections canonically derived from statistical inference principles. The foundational metric is the Fisher information metric, which encodes statistical distinguishability and underpins the geometry of parametric and nonparametric models, estimation efficiency, optimization, and inference. The dualistic structure, via α-connections (notably the exponential and mixture connections), admits global dually flat coordinates for exponential families and more generally extends to quantum systems, stochastic dynamical models, compositional data, optimization algorithms, and connections with optimal transport.

1. Statistical Manifolds, Fisher Metric, and Dual Connections

A statistical manifold $\mathcal{M}$ is a smooth parameterized family $\{p(x;\theta)\}$, with the parameter domain $\Theta\subset\mathbb{R}^d$ serving as the manifold coordinates. The Fisher information metric $g$ on the tangent space $T_\theta\mathcal{M}$ is defined by

$$g_{ij}(\theta) = \mathbb{E}_{p(\cdot;\theta)} \left[ \partial_{i}\log p(X;\theta) \partial_{j}\log p(X;\theta) \right]$$

where $\partial_{i}=\partial/\partial\theta^i$, and expectation is under $p(\cdot;\theta)$ (Caticha, 2014, Pistone, 4 Feb 2025, Mishra et al., 2023, Ay et al., 2012).

Dual affine connections $\nabla^{(\alpha)}$ parameterized by $\alpha\in\mathbb{R}$ are defined via Christoffel symbols: $$\Gamma^{(\alpha)\,k}_{ij} = \mathbb{E}_\theta \left[ \partial_{i}\partial_{j} \log p \ \partial^{k} \log p \right] + \frac{1-\alpha}{2} \mathbb{E}_\theta \left[ \partial_{i}\log p \ \partial_{j}\log p \ \partial^{k}\log p \right]$$ with $\nabla^{(1)}$ (exponential connection) and $\nabla^{(-1)}$ (mixture connection) being dual with respect to $g$ (Pistone, 4 Feb 2025, Nielsen, 2018, Mishra et al., 2023, Itoh et al., 2022).

Exponential families $$p(x;\theta)=h(x)\exp\{\theta^iT_i(x)-\psi(\theta)\}$$ imbue the manifold with dually flat geometry: exponential ($\nabla^{(1)}$) coordinates $\theta$, and mixture ($\nabla^{(-1)}$) coordinates $\eta=\mathbb{E}_\theta[T(X)]$, related by Legendre duality (Mishra et al., 2023, Caticha, 2014).

The unique status of the Fisher metric is established by Chentsov’s theorem: it is invariant under sufficient statistics and Markov morphisms on finite and infinite sample spaces, up to scale, and thus is the canonical metric for statistical models (Ay et al., 2012, Erb et al., 2020).

2. Divergences, Bregman Geometry, Projections, and Geodesics

Divergence functions $D(P\|Q)$ generalize distances on manifold $\mathcal{M}$, yielding a metric and dual connections via derivatives: $$g_{ij}(\theta)= \left. \partial_{\theta_i}\partial_{\theta_j'} D(\theta\|\theta') \right|_{\theta'=\theta}$$

$$\Gamma_{ijk} = -\left. \partial_{\theta_i}\partial_{\theta_j}\partial_{\theta_k'} D(\theta\|\theta') \right|_{\theta'=\theta}$$

(Mishra et al., 2023, Nielsen, 2018, Wong et al., 2019).

Bregman divergences for strictly convex $\psi:\Theta\to\mathbb{R}$

$$D_\psi(x\|y) = \psi(x) - \psi(y) - \langle\nabla\psi(y), x-y\rangle$$

induce the Riemannian metric $g(x)=\nabla^2\psi(x)$ and dual coordinates via Legendre transform $\phi$ (Raskutti et al., 2013, Nielsen, 2018, Mishra et al., 2023, Erb et al., 2020).

KL-divergence $D_{\mathrm{KL}}(p\|q)=\int p(x)\log(p(x)/q(x))dx$ is central, with the second derivative yielding the Fisher metric (Caticha, 2014, Mishra et al., 2023, Erb et al., 2020, Ay et al., 2012).

The generalized Pythagorean theorem for Bregman/KL divergence: $D_\psi(x''\|x') + D_\psi(x'\|x) = D_\psi(x''\|x)$ holds when primal/dual geodesics are orthogonal with respect to $g$—a property with extensive implications for information projection, statistical inference, and variational optimization (Caticha, 2014, Nielsen, 2018, Amari et al., 2017, Oizumi et al., 2015).

Geodesics under dual connections are straight lines in corresponding coordinates: $\theta_t=(1-t)\theta_0+t\theta_1$ (exponential), $\eta_t$ (mixture) (Mishra et al., 2023, Nielsen, 2018, Caticha, 2014). On dually flat manifolds, geodesics allow closed-form solutions. For Riemannian (Levi-Civita) geodesics, solution is curve interpolating between exponential and mixture geodesics (Itoh et al., 2022).

3. Sufficient Statistics, Invariance, and Canonicality

A sufficient statistic $\kappa:\Omega\to\Omega'$ for $(M, \Omega, \mu_0, p(\cdot))$ ensures that the Fisher metric and Amari-Chentsov tensor are preserved under pushforward models: $g_F^p = g_F^{\kappa_*p}$

$T_{AC}^p = T_{AC}^{\kappa_*p}$

(Ay et al., 2012). For any non-sufficient statistic, monotonicity holds: $g_F^{\kappa_*p}(V,V) \leq g_F^p(V,V)$ Monotonicity follows from invariance under sufficient statistics, extending Chentsov's finite-sample uniqueness result to infinite dimensions (Ay et al., 2012).

The only Riemannian metric on the space of discrete probability distributions (simplex) that is invariant under sufficient statistics (Markov morphisms, coarse graining, reparametrization) is the Fisher information metric (Erb et al., 2020, Ay et al., 2012).

4. Applications: Estimation, Optimization, Inference, Quantum Generalizations

Efficient Estimation and Cramér-Rao Bound

The inverse Fisher metric yields the Cramér-Rao lower bound for unbiased estimators (asymptotically achieved by MLE) (Mishra et al., 2023, Caticha, 2014).

Optimization Algorithms

Mirror descent—using Bregman divergences—is equivalent to natural gradient descent on the dual Riemannian manifold: $\mu_{t+1} = \mu_t - \eta \nabla f(x_t)$ is steepest descent under the dual metric $G(\mu)=\nabla^2\phi(\mu)$ (Raskutti et al., 2013).

The natural gradient update: $$\theta_{t+1} = \theta_t - \eta [I(\theta_t)]^{-1} \nabla f(\theta_t)$$ parallels mirror descent; for exponential families, mirror descent with Bregman divergence is precisely the first-order implementation of natural gradient descent. These methods achieve Fisher asymptotic efficiency (Cramér-Rao bound) (Malagò et al., 2014, Raskutti et al., 2013, Mishra et al., 2023).

Quantum Information Geometry

Quantum Fisher information yields monotone Riemannian metrics (Petz classification). The Bures metric for mixed states is the quantum analogue of the 2-Wasserstein distance, and quantum Cramér-Rao bounds govern precision in estimation (Bohra et al., 2021, Mishra et al., 2023). Information geometry also reconstructs quantum theory from Fisher geometry plus physical postulates (complementarity, measurement simulability, gauge invariance), leading to complex Hilbert space and unitary dynamics (0805.2770).

Other Domains

Applications include compositional data analysis (CoDA), evolutionary game theory (Shahshahani geometry), information integration quantification, statistical physics, radar signal processing, machine learning, and analysis of compositional mixtures (Erb et al., 2020, 0911.1383, Oizumi et al., 2015, Amari et al., 2017, Malagò et al., 2014, Caticha, 2015).

5. Geometry of Probability Measures and Infinite-dimensional Extensions

On a compact manifold $M$, the space of smooth probability densities $P^+(M)$ admits a Fisher-Rao metric: $$G_\mu(\tau_1,\tau_2) = \int_M \frac{h_1(x) h_2(x)}{f(x)} d\lambda(x)$$ with tangent vectors $\tau=h\lambda$ (Itoh et al., 2022). The Fisher metric is the Hessian of the KL-divergence, and the Levi-Civita connection yields Riemannian geodesics expressible as closed-form curves via the square-root map embedding into $L^2$ spheres.

Dualistic α-connections extend to infinite dimensions; the geometry encodes statistical distinguishability, invariance under diffeomorphisms, and characterizes barycenter maps on the ideal boundary of Hadamard manifolds (Itoh et al., 2022).

Nonparametric information geometry generalizes finite-dimensional concepts using statistical bundles—pairs (density, score)—parallel transport, and exponential charts. KL-divergence is generated via affine-geometric transformations and the second-order Taylor expansion recovers the Fisher metric as the Hessian (Pistone, 4 Feb 2025).

6. Interactions, Information Integration, and Hierarchies

Information geometry provides a unified language for interaction measures via KL-projection onto constraint manifolds. Integrated information, transfer entropy, and stochastic interaction quantify "broken" causal links as divergence from full joint to split-model submanifolds: $\Phi_C(P) := \min_{q\in\mathcal{M}_C} D(P\|q)$ for constraints $C$ encoding independence/conditional independence relations. Hierarchies of measures arise via inclusion of submanifolds, with properties deduced from geometry (e.g., nonnegativity, bounds by mutual information) (Oizumi et al., 2015, Amari et al., 2017).

In compositional data analysis, the Fisher metric is uniquely suitable for measuring statistical distance, respecting invariance under coarse graining, and is foundational for maximum-entropy and projection theorems. It contrasts with Euclidean (Aitchison) metric, and monotonicity of statistical divergence under amalgamation is rigorously established (Erb et al., 2020).

7. Curvature, Criticality, and Connections with Optimal Transport

The scalar curvature of the information metric (Ruppeiner metric, quantum fidelity susceptibility) encodes correlation volume (criticality) in statistical and quantum systems: $R \sim \xi^D$ Scaling relations among curvature, geodesic distance, and correlation length are prescribed by RG beta-functions and parameter manifold geometry (Maity et al., 2015). Notably, in quantum statistical models, scalar curvature need not diverge at phase transitions, as demonstrated in the geometry of Bose-Einstein condensation (Pessoa et al., 2021).

Optimal transport and information geometry intersect via c-divergence and the pseudo-Riemannian metric, with dual connections and curvature tensors arising from the Levi-Civita connection of the ambient pseudo-metric. Bregman and $L^{(\alpha)}$-divergences correspond to dually flat and constant-curvature structures, respectively, with geodesic-induced curvature expressing the Ma-Trudinger-Wang (MTW) regularity condition (Wong et al., 2019).

---

Table: Key Information-Geometric Objects

Object	Definition/Formulation	Domain/Significance
Fisher metric	$$g_{ij}=\mathbb{E}_\theta[\partial_i \log p \partial_j \log p]$$	Statistical distinguishability
α-connections	$\Gamma^{(\alpha)}$ via score and cubic tensor	Dualistic structure
KL/Bregman divergence	$$D_\psi(x\|y)=\psi(x)-\psi(y)-\langle\nabla\psi(y), x-y\rangle$$	Statistical "distance"/projection
Exponential/mixture coordinates	$\theta$/$\eta$ linked via Legendre duality	Dually flat geometry
Sufficient statistics	Compression preserving Fisher metric and cubic tensor	Canonicality, monotonicity
Mirror/natural gradient	First-order update via dual coordinates or Hessian inversion	Efficient optimization

Information geometry is thus a canonical, rigorous framework for understanding statistical inference, learning, dynamical inference, and physical models from the perspective of differential geometry and metric theory, with deep unifying links to quantum mechanics, data analysis, optimization, and geometry of probability spaces.