A Generalized Bias-Variance Decomposition for Bregman Divergences (2511.08789v1)

Abstract: The bias-variance decomposition is a central result in statistics and machine learning, but is typically presented only for the squared error. We present a generalization of the bias-variance decomposition where the prediction error is a Bregman divergence, which is relevant to maximum likelihood estimation with exponential families. While the result is already known, there was not previously a clear, standalone derivation, so we provide one for pedagogical purposes. A version of this note previously appeared on the author's personal website without context. Here we provide additional discussion and references to the relevant prior literature.

• The paper introduces a generalized bias-variance decomposition for Bregman divergences that extends traditional analysis beyond the mean-squared error loss.
• It establishes that the minimizer of the expected divergence is achieved through convex gradient alignment, linking estimation theory with information geometry.
• The findings offer practical insights for model selection and algorithm comparison in contexts using cross-entropy and other proper scoring rules.

Generalized Bias-Variance Decomposition for Bregman Divergences

Background and Motivation

The bias-variance decomposition is a fundamental result for characterizing the expected generalization error of statistical learning algorithms. Traditionally, such decomposition has been formulated with respect to the mean-squared error (MSE) loss, which is suitable for regression problems in Euclidean spaces. However, a wide range of contemporary machine learning scenarios, including density estimation, classification, and generalized linear models (GLMs), employ loss functions derived from the negative log-likelihood. For exponential family distributions, these losses can be formulated as Bregman divergences, exploiting the duality between convex functions and exponential family likelihoods.

This paper provides a formal derivation and presentation of a generalized bias-variance decomposition for prediction error measured by Bregman divergences. Such generalization is crucial for statistical estimation and learning tasks where predictions are probability distributions, particularly in cases where cross-entropy or other proper scoring rules are used. The results connect foundational work in bias-variance theory with information geometry and exponential family theory.

Formal Definitions and Analytical Results

Bregman Divergence

Let $F:\mathcal{S}\to\mathbb{R}$ be strictly convex and differentiable. The Bregman divergence between $x$ and $y$ is given by: $$D_F(x\,||\,y) = F(x) - F(y) - \langle \nabla F(y), x-y \rangle$$

Bregman divergences encompass squared error, Kullback-Leibler divergence, and other loss functions standard in likelihood-based inference.

Optimization and Expected Divergence

The key result is that the point minimizing expected Bregman divergence with respect to an RV $X$ is given by a solution of $\nabla F(x^*) = E[\nabla F(X)]$ when expectation is taken over the second argument, and by $E[X]$ when expectation is over the first argument, assuming $F$ is strictly convex.

Decomposition Theorem

For any fixed $s \in \mathcal{S}$ and RV $X$, the expected Bregman divergence admits the following decomposition: $E[D_F(s, X)] = D_F(s, x^*) + E[D_F(x^*, X)]$ where $x^*$ minimizes the expected divergence as above. Analogously, $E[D_F(X, s)]$ decomposes with $x^* = E[X]$.

Generalized Bias-Variance Decomposition

In the context of statistical prediction, let $Y$ be a target variable dependent on input $X$, algorithm $f_D$ predicts $Y$ from $X$ using training data $D$. The expected prediction loss (averaged over data and targets) satisfies: $$E_{D, Y}[D_F(Y, f_D(X))] = E_Y[D_F(Y, f^*(X))] \quad \text{(Noise)} \ \quad + D_F(f^*(X), \bar{f}(X)) \quad \text{(Bias)} \ \quad + E_D[D_F(\bar{f}(X), f_D(X))] \quad \text{(Variance)}$$ Here, $f^*(X)$ is the Bayes optimal prediction (minimizing expected loss over $Y$), and $\bar{f}(X)$ minimizes expected Bregman divergence over the distribution of learned predictors $f_D(X)$ induced by random data $D$. Expectations are conditioned on $X$.

This decomposition has direct analogues to the classical bias, variance, and irreducible noise terms, but is generalized to the Bregman divergence loss structure.

Relation to Exponential Families and Proper Scoring

For distributions in the exponential family, negative log-likelihood losses can be written as a Bregman divergence with respect to the convex conjugate of the log-partition function. That is,

$$\log p(x;\eta) = -D_{A^*}(T(x)\,||\,\mu) + A^*(T(x))$$

where $T(x)$ is the sufficient statistic, $\mu$ its mean, and $A^*$ is the convex conjugate (negative entropy).

Consequently, the bias-variance decomposition expressed above applies to expected log-likelihood losses for exponential family models, providing theoretical underpinning for generalized linear model estimation, classification with cross-entropy, and training of probabilistic neural networks using proper scoring rules.

Practical and Theoretical Implications

The generalized decomposition enables rigorous characterization of estimator performance for a broad class of convex loss functions, including those relevant to probabilistic and categorical prediction. Key implications for practice include:

• Model selection: Enables explicit bias-variance analysis for non-quadratic loss, such as cross-entropy, critical for modern classification and likelihood-based training.
• Algorithm comparison: Direct quantification of overfitting (variance) versus underfitting (bias) in learning algorithms under loss functions derived from information geometry.
• Density estimation: Provides theoretical foundation for evaluating mixture models, variational inference, and density networks using KL-divergence or other Bregman-type criteria.
• Interpretation in proper scoring rule framework: Unifies disparate loss functions under a single analytic decomposition, relevant for probabilistic calibration and risk analysis.

Numerical implications include the exact characterization of expected loss decomposability even in high dimensions, given sufficient conditions on $F$.

Future Directions

Potential avenues for extension include:

• Generalization to $g$-Bregman divergences: Following recent work, bias-variance decompositions may be further generalized to non-standard divergences induced by non-linear link functions.
• Connections to risk bounds: Investigating how decomposition interacts with PAC-type guarantees and information-theoretic bounds in learning theory.
• Application to probabilistic deep learning: Adapting the framework for complex stochastic architectures, e.g., variational autoencoders and energy-based models, that naturally employ Bregman-type divergences in training.
• Extension to structured prediction: Analyzing the decomposition for models with output spaces structured beyond vectorial, leveraging information geometry concepts.

Conclusion

This paper provides a formal, generalized bias-variance decomposition for Bregman divergences, extending the classic result to settings where prediction error is not naturally measured by squared error but by more general convex losses, often arising in exponential family modeling and probabilistic inference. The theoretical contribution clarifies how bias and variance should be properly quantified and interpreted in learning systems using these loss functions, supporting a rigorous framework for model evaluation across a spectrum of applications in machine learning and statistics.