Symmetry Guarantees Statistic Recovery in Variational Inference

Abstract: Variational inference (VI) is a central tool in modern machine learning, used to approximate an intractable target density by optimising over a tractable family of distributions. As the variational family cannot typically represent the target exactly, guarantees on the quality of the resulting approximation are crucial for understanding which of its properties VI can faithfully capture. Recent work has identified instances in which symmetries of the target and the variational family enable the recovery of certain statistics, even under model misspecification. However, these guarantees are inherently problem-specific and offer little insight into the fundamental mechanism by which symmetry forces statistic recovery. In this paper, we overcome this limitation by developing a general theory of symmetry-induced statistic recovery in variational inference. First, we characterise when variational minimisers inherit the symmetries of the target and establish conditions under which these pin down identifiable statistics. Second, we unify existing results by showing that previously known statistic recovery guarantees in location-scale families arise as special cases of our theory. Third, we apply our framework to distributions on the sphere to obtain novel guarantees for directional statistics in von Mises-Fisher families. Together, these results provide a modular blueprint for deriving new recovery guarantees for VI in a broad range of symmetry settings.

• The paper introduces a theoretical framework showing that symmetry in both the target distribution and variational family guarantees exact recovery of statistics.
• It demonstrates that under even and elliptical symmetries, conditions like uniqueness of the variational optimum and specific f-divergence choices ensure precise recovery of measures such as the mean and covariance.
• The work extends these guarantees to non-Euclidean settings, proving that rotational symmetry on the sphere allows variational inference using the vMF family to accurately recover the true axis of symmetry.

Symmetry-Induced Statistic Recovery in Variational Inference

Overview

This paper provides a rigorous and unified theoretical framework for understanding when and why variational inference (VI) can exactly recover certain statistics of a target distribution—even under severe model misspecification—by leveraging symmetries inherent in both the target and the variational family. Moving beyond ad hoc, problem-specific guarantees, the authors systematize the phenomenon of symmetry-induced statistic recovery for general $f$-divergence-based VI, including the reverse KL objective and its variants.

Theoretical Framework

The central contribution is a general theorem characterizing conditions under which a unique minimizer of the variational problem,

$Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$

inherits symmetries from the target $P$ whenever (i) the variational family $\mathcal{Q}$ is stable under those symmetries, and (ii) the statistic of interest respects a natural equivariance under the group action.

The authors specify a triplet of structural conditions:

1. Target invariance: $P$ is invariant under a group of measurable bijections $G$.
2. Variational family stability: $\mathcal{Q}$ is stable under pushforward by $G$.
3. Statistic compatibility: The statistic's domain is $G$-stable, and its value depends equivariantly on the pushforward action.

These imply that both the target statistic $S(P)$ and the variational statistic $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$0 are pinned to the unique point in the fixed set under the induced $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$1-action on the statistic's codomain. When this fixed set is a singleton, the statistic is recovered exactly.

Figure 1: Unique variational minimisers $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$2 from a Gaussian family exactly recover symmetry-determined statistics of highly non-Gaussian targets $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$3. Left: under even symmetry, the mean is recovered exactly. Right: under elliptical symmetry, the mean and correlation are recovered exactly, while the covariance is matched only up to a scalar, as per the theoretical corollaries.

Unification and Generalization of Previous Work

The theory recovers and subsumes the main results of Margossian and Saul on even and elliptical symmetry for location–scale families (cf. [margossian_variational_2025], [margossian_generalized_2025]), and also clarifies the precise mechanism by which statistic recovery arises. The results hold for general $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$4-divergences (including but not limited to reverse KL, forward KL, $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$5-divergences, and Rényi divergences).

For example, in the case of even symmetry (distribution symmetric about a point $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$6), if the variational family is a location–scale class induced by an even base measure, and the statistic is the mean, then the fixed set is the singleton $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$7, ensuring exact mean recovery as long as the VI optimum is unique. In the case of elliptical symmetry, the framework shows that both mean and covariance are transferred up to an indeterminacy in scaling; exact correlation recovery follows when the base measure is spherically symmetric.

New Recovery Guarantee: Rotational Symmetry on the Sphere

Going beyond Euclidean settings, the framework enables novel guarantees for targets and variational families supported on manifolds. Specializing to distributions on the unit sphere $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$8, the authors consider rotationally symmetric targets and the von Mises–Fisher (vMF) variational family, showing that VI exactly recovers the axis of symmetry (modulo sign, i.e., the equivalence class in projective space).

A crucial point is that for non-uniform rotationally symmetric distributions, the symmetry uniquely identifies the axis direction (as a line), and the vMF family respects this symmetry. Provided uniqueness of the variational optimum, the recovered vMF mean direction aligns with the true axis of symmetry.

Figure 2: Log-density contours of the target and variational posterior on the sphere, using a Lambert azimuthal projection centered at the ground truth axis. Left: for $Q^* = \arg\min_{Q \in \mathcal{Q}} D_f(P \| Q),$9, reverse KL minimization yields a unique optimum at the correct direction. Right: for $P$0, multiple minimizers appear, forming a latitude circle, and no minimizer recovers the true axis.

The phase transition (demonstrated numerically) in the minimizer structure as the target departs from the vMF form illustrates the necessity of uniqueness for exact recovery, and identifies a sharp boundary beyond which symmetric statistics are no longer matched.

Proof Technique

The argument leverages invariance of $P$1-divergences under pushforward and orbit structure under group actions. The uniqueness of the minimizer ensures the pushforward does not generate multiplicities, and conditions on the statistic ensure the group action propagates to the codomain, constraining the possible values.

Generalizations to parameterized families and implications for identifiability and optimization-induced local minima are discussed.

Implications and Future Directions

This paper provides a standardized pathway to deducing which statistics are robust to misspecification in VI, as dictated by symmetry. In practice, this enables stronger guarantees in settings where the full distributional approximation is poor but certain invariants (e.g., location, direction, correlation structure) are reliably computed.

On the theoretical side, this clarifies the limitations and scope of variational inference, and provides a modular blueprint for further generalizations, such as:

• Extension to approximate symmetries,
• Incorporation of higher-order or non-classical statistics (e.g., manifold-valued statistics),
• Analysis under non-uniqueness and multimodal posterior structure,
• Connections to equivariant inference and geometric statistics.

Conclusion

The symmetry-based framework developed in this work provides necessary and sufficient structure to deduce exact or partial statistic recovery in variational inference across a wide range of group actions, divergence measures, and variational families. The results unify prior ad hoc guarantees and establish a modular approach for constructing new ones, with implications for both theory and robust probabilistic modeling.