A Tutorial on Bregman Projection in Statistics

Abstract: A single geometric operation -- projecting a reference onto a constrained family under a Bregman divergence -- underlies a striking range of statistical methods. This tutorial develops the operation first as pure convex geometry, with no statistics attached. A strictly convex generator $G$ and its conjugate $F$ furnish two coordinate systems, a projection theorem with existence and uniqueness, and a Pythagorean {theorem}; the Pythagorean theorem itself produces {two} dual projections -- the information (e-) projection onto moment-constrained families and the moment (m-) projection onto exponential families -- exchanged by the conjugacy $G\leftrightarrow F$, so a single theorem governs both. Part~II reads off the statistics. The generalized linear model is treated in detail as the concrete carrier of the two projections: {under the canonical link,} the score equation is exactly the Pythagorean orthogonality, and the fit is simultaneously an e-projection in the natural coordinate and an m-projection in the mean coordinate. Maximum entropy, survey calibration, over-identified moment models, the EM algorithm, variational inference, autoencoders, and expectation propagation then fall into place as instances of the same construction -- exactly where the underlying families are flat, and as controlled approximations or neighboring-divergence analogies where they are not. The mathematics of Part~I is self-contained; the statistical sections presume only familiarity with the methods being unified.

• The paper synthesizes Bregman projection theory and statistical applications by describing how projecting a point onto constrained sets under Bregman divergence underpins many estimation methods.
• It develops a dual-coordinate framework with e- and m-projections, proving existence, uniqueness, and a Pythagorean decomposition in convex settings.
• Practical implications are illustrated through applications to maximum likelihood, survey calibration, EM, and variational inference in modern statistical and machine learning tasks.

Succinct Overview

The paper "A Tutorial on Bregman Projection in Statistics" (2606.21714) systematically synthesizes Bregman projection theory and its statistical applications, providing a rigorous delineation of how a single geometric operation—projecting a reference point onto a constrained set under a Bregman divergence—underpins a considerable fraction of estimation methodologies in modern statistics and machine learning. It develops the mathematical foundation from convex geometry, dual coordinate systems, existence and uniqueness results, and the Pythagorean decomposition, then elucidates how this structure governs exponential family models, maximum entropy, survey calibration, moment estimation, EM and variational inference, autoencoders, and expectation propagation—either exactly or as controlled approximations.

Mathematical Foundation and Projection Geometry

Bregman divergence $D_G(p\|q)$, defined for a strictly convex generator $G$, quantifies discrepancy as $G(p)$ minus the tangent at $q$. The dual coordinate systems arise from the Legendre transform: $G$ and its conjugate $F$, linked via $\nabla G$ and $\nabla F$, yield the "mean" (m-) and "natural" (e-) coordinates. The core constructs are:

• e-projection: Minimizes $D_G(p\|p_0)$ over an affine constraint set (moment family), landing in a generalized exponential family.
• m-projection: Minimizes $D_G(p_0\|p)$ over an affine exponential family, conjugate to the e-projection.
• Pythagorean theorem: Decomposes divergence into orthogonal components, with duality depending on argument order and generator conjugacy.

Existence and uniqueness follow from strict convexity and integrability, with solutions given in closed form as $G$0 for suitably chosen Lagrange multipliers.

Statistical Impact and Duality

The statistical interpretation leverages these geometric facts. Canonical-link GLMs instantiate the e- and m-projections, with negative log-likelihood equating to a Bregman divergence and score equations reflecting orthogonality. The maximum-entropy principle reflects an e-projection of a reference (e.g., uniform or prior) onto m-flat sets (moment constraints), generating exponential families. Maximum likelihood is the m-projection of empirical distributions onto e-flat families, with strong duality: intersection points coincide, and both decompositions hold.

Survey calibration is formulated as a Bregman projection of design weights onto moment constraints, with prototype solutions depending on the generator (Shannon, quadratic, or general $G$1), yielding calibrated weights via $G$2. Over-identified moment estimation is built as slice-wise e-projections, with the overall estimator minimizing calibration cost across parameter slices.

EM and latent-variable methods (VI, autoencoders, expectation propagation) are analyzed as alternating e/m projections under KL or general Bregman divergences, subject to tractability and flatness assumptions. EM is exact for flat exponential families, VI and autoencoders approximate the projection by restricting variational families, and expectation propagation performs local m-projections via moment matching.

Extensions and Neighboring Geometries

The tutorial rigorously demarcates where the projection theorem governs methods directly, and where modern algorithms operate in "neighboring" geometries. Score matching, diffusion models, and flow matching minimize quadratic Bregman divergences on scores or velocity fields, not directly on densities; kernel MMD estimators impose discrepancies after embedding distributions into RKHSs. Adversarial generative models ($G$3-GANs, Wasserstein GANs) operate via $G$4-divergence or transport metrics, outside the Bregman projection landscape.

These distinctions are itemized in the final summary table, specifying projection status (exact, approximate, neighboring, or outside the theorem) and generator geometry.

Numerical and Structural Results

Key structural claims include:

• Existence and uniqueness: For the e-projection and its m-dual, under convexity and integrability assumptions.
• Pythagorean decomposition: Exact for flat families, giving additive divergence decompositions.
• Duality: Maximum-entropy and maximum-likelihood coincide at the e/m intersection point, both statistically and geometrically.
• Generalized linear models: Score equations manifest the orthogonality central to Bregman projection; negative log-likelihood is a divergence.
• Numerical forms: Closed-form expressions for projections are given via dual coordinate systems for all admissible generators.

Theoretical and Practical Implications

The unification clarifies the geometric origin of estimation in exponential family models, M-estimation, maximum entropy, calibration, and alternating-projection algorithms. It enables principled tuning of robustness/efficiency trade-offs by generator selection (e.g., Shannon for efficiency, power/Tsallis for robustness), and provides a geometric lens through which to interpret divergence minimization in deep generative modeling. Practical calibration can use held-out divergence costs for generator selection, while theoretical work can systematically characterize generator-induced robustness and efficiency, identifiability in under-constrained settings, and generalize deep amortized inference well beyond KL-VI.

Anticipated future developments include systematic generator selection for robustness, generalization of variational inference to Bregman divergences, and expansion of calibration and moment estimation methods to nonstandard divergence families. These directions will further stratify the unified geometric theory built here.

Conclusion

The paper presents a definitive geometric and statistical synthesis of Bregman projection, organizing a wide array of statistical and machine learning estimators under the dual-coordinate, projection, and Pythagorean identities of convex geometry. Exact and approximate projection, duality, and neighboring discrepancies are rigorously delineated for both classical and modern methods, offering theoretical clarity and practical guidance for robust estimation, divergence minimization, and generative modeling.