Optimal Stability of KL Divergence under Gaussian Perturbations

Abstract: We study the problem of characterizing the stability of Kullback-Leibler (KL) divergence under Gaussian perturbations beyond Gaussian families. Existing relaxed triangle inequalities for KL divergence critically rely on the assumption that all involved distributions are Gaussian, which limits their applicability in modern applications such as out-of-distribution (OOD) detection with flow-based generative models. In this paper, we remove this restriction by establishing a sharp stability bound between an arbitrary distribution and Gaussian families under mild moment conditions. Specifically, let $P$ be a distribution with finite second moment, and let $\mathcal{N}_1$ and $\mathcal{N}_2$ be multivariate Gaussian distributions. We show that if $KL(P||\mathcal{N}_1)$ is large and $KL(\mathcal{N}_1||\mathcal{N}_2)$ is at most $ε$, then $KL(P||\mathcal{N}_2) \ge KL(P||\mathcal{N}_1) - O(\sqrtε)$. Moreover, we prove that this $\sqrtε$ rate is optimal in general, even within the Gaussian family. This result reveals an intrinsic stability property of KL divergence under Gaussian perturbations, extending classical Gaussian-only relaxed triangle inequalities to general distributions. The result is non-trivial due to the asymmetry of KL divergence and the absence of a triangle inequality in general probability spaces. As an application, we provide a rigorous foundation for KL-based OOD analysis in flow-based models, removing strong Gaussian assumptions used in prior work. More broadly, our result enables KL-based reasoning in non-Gaussian settings arising in deep learning and reinforcement learning.

• The paper establishes a sharp O(√ε) stability bound for KL divergence under Gaussian perturbations, marking its main contribution.
• It employs a rigorous analytic decomposition to derive tight bounds that account for deviations in both mean and covariance.
• The findings support robust KL-based OOD detection and have implications for improving deep generative and reinforcement learning models.

Optimal Stability of KL Divergence under Gaussian Perturbations

Introduction

The analyzed paper addresses foundational questions surrounding the behavior of Kullback-Leibler (KL) divergence under Gaussian perturbations, specifically in high-dimensional and deep generative modeling scenarios. Classical results regarding KL divergence, such as relaxed triangle inequalities, typically hinge on all involved distributions being Gaussian, a restriction which curtails their utility in contemporary machine learning tasks—most notably, out-of-distribution (OOD) detection using flow-based generative models. The central contribution is the formal establishment of a sharp optimal stability bound for the KL divergence between an arbitrary finite second-moment distribution and (potentially perturbed) Gaussian distributions. The authors prove that the stability rate is $O(\sqrt{\epsilon})$ under Gaussian perturbations, and demonstrate that this rate is tight, even when restricted to Gaussian distributions.

Background and Motivation

KL divergence is central to statistical inference, ML, and information theory, but lacks key metric properties such as symmetry and the triangle inequality, making it difficult to analyze relationships among distributions. Prior work has addressed this gap for multivariate Gaussian families, establishing that the KL divergence between Gaussians is approximately symmetric and satisfies a relaxed triangle inequality [zhang2023properties, xiao2026relaxedtriangleinequalitykullbackleibler]. However, in numerous applied settings, such as flow-based OOD detection, distributions of interest are not Gaussian or are only approximately Gaussian.

A persistent practical issue arises in flow-based generative modeling: models (e.g., Glow) trained on in-distribution (ID) data frequently assign higher likelihoods to OOD samples than to true ID samples, contradicting the expectation that likelihood should robustly indicate distributional fit.

Figure 1: Distribution of model log-likelihood for Glow trained on CIFAR-10, showing higher values for SVHN and SVHN with altered contrast, illustrating the failure of naive likelihood for OOD detection.

Theoretical explanations, such as those in [zhang2023outofdistribution], often rely on strong Gaussianity assumptions for latent representations. This motivates seeking a more general stability result for KL divergence that extends beyond the Gaussian case, allowing direct application to practical, non-Gaussian settings.

Main Results

The paper’s primary contribution is a relaxed triangle inequality for KL divergence connecting an arbitrary finite second-moment distribution $P$ and two Gaussian distributions $\mathcal{N}_1$ and $\mathcal{N}_2$. The main theorem proves that if $KL(P||\mathcal{N}_1)$ is large ($> C$) and $KL(\mathcal{N}_1||\mathcal{N}_2) \leq \epsilon$ (with small $\epsilon$), then

$$KL(P||\mathcal{N}_2) \geq KL(P||\mathcal{N}_1) - O(\sqrt{\epsilon})$$

where the degradation factor $O(\sqrt{\epsilon})$ is shown to be optimal.

The technical approach decomposes the difference $P$0 and develops tight analytic bounds for the impact of small perturbations in the Gaussian reference on the KL to an arbitrary distribution, employing explicit control on both mean and covariance deviations. Through direct functional analysis of $P$1 (Figure 2) and exploitation of the quadratic structure of Gaussian likelihoods, the analysis extends the geometric intuition underlying previous work to the general setting.

Figure 2: The function $P$2 underlies KL divergence between univariate Gaussians, showing convexity and minimum at $P$3.

The tightness of the $P$4 scaling is illustrated by explicit construction of distributions where the lower bound is achieved, demonstrating that this is not an artifact of the analytical machinery but reflects the intrinsic geometry of KL divergence under Gaussian perturbation.

Figure 3: KL divergence structure for an arbitrary distribution $P$5 and two nearby Gaussians; the $P$6 gap is unavoidable even in the Gaussian case.

Implications for OOD Detection and Deep Learning

A crucial application is OOD detection with flow-based models, for which the paper removes the necessity for all involved latent representations to be Gaussian and avoids the model mismatch assumption $P$7 previously adopted in [zhang2023outofdistribution]. In practical architectures, the latent code distributions induced by data (either ID or OOD) are often arbitrarily non-Gaussian.

Figure 4: KL divergence-based analysis for OOD detection in the latent space of a flow model, illustrating relationships among ID, OOD, and prior distributions.

The present result provides a rigorous theoretical foundation justifying KL-based OOD scoring even when the OOD embedding is highly non-Gaussian, guaranteeing that as long as $P$8 between OOD latent and ID latent is large and the model fit $P$9 between ID latent and prior is small, the OOD latent distribution will remain well-separated from the prior. This directly supports algorithms such as KLODS [zhang2023outofdistribution], justifies their use in practical non-Gaussian regimes, and explains persistent empirical failures of naive likelihood-based OOD detection in flow models.

Further implications extend to reinforcement learning, where policy updates often constrain the KL between (Gaussian) parameterizations of policies. The provided result ensures that separation in KL from arbitrary reference behavior is robust to small parameter perturbations. Similarly, in regularized VAE or $\mathcal{N}_1$0-VAE learning, the result clarifies the limitation: even strong KL regularization cannot bridge an intrinsic mismatch when the true latent is far from the Gaussian prior.

Connections to Broader Divergence Theory

The established optimal stability result generalizes classical Pinsker-type (total variation) bounds and provides a new robustness principle: large KL separation from a Gaussian reference persists up to $\mathcal{N}_1$1 under Gaussian perturbations, with explicit dependence only on the first and second moments of the arbitrary distribution. This advances both the theoretical and practical analysis of statistical divergences in high-dimensional representation learning, generative modeling, and policy optimization.

Conclusion

The paper presents the first general, optimal relaxed triangle inequality for KL divergence under Gaussian perturbations, covering arbitrary finite-moment distributions. The $\mathcal{N}_1$2 stability degradation is shown to be unavoidable and captures the true sensitivity of KL to small Gaussian referencing errors. This result closes a fundamental theoretical gap in KL-based OOD detection analysis and provides a robust basis for KL reasoning in deep learning and reinforcement learning, even when standard Gaussian assumptions fail. The result is poised to inform the design and theoretical analysis of future generative models, OOD detection algorithms, and policy regularization techniques in diverse high-dimensional settings (2604.11026).