The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning

Abstract: Autonomous (noise-agnostic) generative models, such as Equilibrium Matching and blind diffusion, challenge the standard paradigm by learning a single, time-invariant vector field that operates without explicit noise-level conditioning. While recent work suggests that high-dimensional concentration allows these models to implicitly estimate noise levels from corrupted observations, a fundamental paradox remains: what is the underlying landscape being optimized when the noise level is treated as a random variable, and how can a bounded, noise-agnostic network remain stable near the data manifold where gradients typically diverge? We resolve this paradox by formalizing Marginal Energy, $E_{\text{marg}}(\mathbf{u}) = -\log p(\mathbf{u})$, where $p(\mathbf{u}) = \int p(\mathbf{u}|t)p(t)dt$ is the marginal density of the noisy data integrated over a prior distribution of unknown noise levels. We prove that generation using autonomous models is not merely blind denoising, but a specific form of Riemannian gradient flow on this Marginal Energy. Through a novel relative energy decomposition, we demonstrate that while the raw Marginal Energy landscape possesses a $1/t^p$ singularity normal to the data manifold, the learned time-invariant field implicitly incorporates a local conformal metric that perfectly counteracts the geometric singularity, converting an infinitely deep potential well into a stable attractor. We also establish the structural stability conditions for sampling with autonomous models. We identify a ``Jensen Gap'' in noise-prediction parameterizations that acts as a high-gain amplifier for estimation errors, explaining the catastrophic failure observed in deterministic blind models. Conversely, we prove that velocity-based parameterizations are inherently stable because they satisfy a bounded-gain condition that absorbs posterior uncertainty into a smooth geometric drift.

• The paper introduces a geometric analysis showing that noise-agnostic diffusion models implicitly optimize a marginal energy landscape.
• The paper demonstrates that Riemannian gradient flow preconditioning stabilizes the otherwise singular gradients inherent in autonomous models.
• The paper validates that only velocity-based parameterizations ensure stable, high-fidelity generation across canonical datasets.

The Geometry of Noise: Why Diffusion Models Don’t Need Noise Conditioning

Overview

This paper presents a comprehensive geometric and theoretical analysis of autonomous (noise-agnostic) generative models—diffusion-based models that operate without explicit noise-level conditioning. Traditional denoising diffusion probabilistic models (DDPMs) and score-based models use a time- (or noise-) dependent field, but recent evidence shows that single, time-invariant vector fields can suffice for generation. The authors rigorously identify the implicit marginal energy landscape these noise-blind models optimize and explain both their capabilities and fundamental limitations. The work introduces a Riemannian gradient flow framework and provides necessary and sufficient stability conditions, drawing strong distinctions between different parameterizations.

Marginal Energy Landscape and the Geometry of Noise

Central to the analysis is the marginal energy $E_{\mathrm{marg}}(u) = -\log p(u)$, where $p(u)$ is the mixture distribution over all possible noise strengths $t$:

$p(u) = \int p(u|t)p(t)dt.$

The paper proves that in autonomous models, the learned static field is not simply a denoiser but corresponds to an expected gradient of the marginal energy landscape. This gradient is singular near the clean data manifold—in fact, it diverges as $u$ approaches a data point, creating a deep potential well and severely ill-conditioned optimization geometry.

Figure 1: Visualization of the marginal energy landscape, revealing the singular potential well geometry near the data manifold, driving divergent gradients.

This result directly explains the observed training and inference difficulties with explicitly energy-based models and highlights a core paradox: how do bounded neural networks maintain stability when the underlying energy geometry is singular?

Riemannian Gradient Flow and Resolution of the Energy Paradox

The authors show that autonomous models, instead of following the raw gradient of the marginal energy (which would be unstable), implicitly implement a Riemannian gradient flow. The learned vector field acts as a preconditioned gradient, where a conformal metric—localized to the posterior noise variance—counteracts the singularity. The field thus stabilizes the otherwise divergent dynamics, making it compatible with bounded neural architectures. An exact decomposition demonstrates the vector field splits into three interpretable terms: a Riemannian preconditioned energy gradient, a transport (covariance) correction, and a linear drift.

In two key regimes—the global high-dimensional ($D \gg 1$) scenario, and the near-manifold ($u \sim \mathcal{X}$) limit—the correction term vanishes due to concentration-of-measure phenomena. In high dimensions, the noise level $t$ becomes tightly encoded in the observation norm, and the model’s apparent “noise-blindness” simply reflects an implicit posterior inference over $t$.

Stability Analysis: Parameterization is Crucial

The theoretical framework leads to critical practical results. The authors rigorously analyze the stability of different parameterizations:

• Noise prediction (as in DDPM) introduces a gain that diverges as $1/b(t)$ when $t \to 0$. The resulting "Jensen Gap" between expected and actual noise amplification structurally destabilizes sampling: small estimation errors are amplified to catastrophic generation failures.
• Signal prediction (EDM) also incurs a singular gain, but the estimation error decays exponentially near the data, ensuring the divergence is controlled and sampling remains stable.
• Velocity parameterization (Flow Matching, EqM) uses a bounded gain, resulting in inherently stable trajectories regardless of implicit noise estimation errors.

The paper mathematically demonstrates that only velocity-based parameterizations provide a stability guarantee for autonomous (blind) field models.

Empirical Validation

Experiments across canonical datasets (CIFAR-10, SVHN, Fashion MNIST) clearly confirm the theoretical predictions. Autonomous noise-predicting models (DDPM Blind) exhibit structural instability, generating noisy or collapsed samples, while velocity-based architectures (“Flow Matching Blind,” EqM) achieve high-fidelity generations indistinguishable from their time-conditioned counterparts.

Figure 2: Generative sample quality on CIFAR-10, contrasting unstable DDPM Blind output (top) with the robust, high-fidelity samples of Flow Matching Blind (bottom).

Figure 3: Stability of Flow Matching Blind on SVHN; DDPM Blind displays pronounced instability, as predicted by the theoretical gain analysis.

Figure 4: Fashion MNIST generations further corroborate the stability of velocity-based, noise-agnostic models.

A toy experiment embedding 2D structure in high-dimensional ambient spaces visualizes the concentration effect—autonomous models are ambiguous in low dimensions, but become increasingly more capable as $D$ rises, converging to performance-parity with conditional models only for sufficiently large $D$.

Figure 5: Generative performance on 2D concentric circles (embedded in $\mathbb{R}^D$): velocity-parameterized autonomous models lock in geometry reliably at moderate dimension, whereas noise-predicting models remain noisy until $D$ is extremely high.

Implications and Future Directions

This work formalizes why explicit time/noise conditioning is not strictly necessary in generative modeling, provided the right parameterization is used. The identification of the marginal energy landscape and the preconditioned gradient flow explains when and why noise-agnostic (autonomous) models can generalize and produce high-quality samples. The singularity problem, previously poorly understood, is rigorously resolved by the Riemannian metric embedding provided by the field itself. The necessity for velocity-based parameterizations in practical implementations of autonomous models is strongly supported, with both theoretical and empirical backing.

This geometric framework invites further research in several directions:

• Extension to non-affine or learned forward processes and their empirical energy landscapes.
• Design of new preconditioners or training protocols explicitly leveraging the insights around Riemannian flows.
• Exploration of the trade-off between stability and expressivity for architectures interpolating between velocity and noise prediction.
• Application to denoising and restoration tasks with more complex or structured noise models.
• Investigating analogous geometric phenomena in other classes of autonomous generative models, such as normalizing flows and energy-based models on manifolds.

Conclusion

The paper provides a rigorous geometric explanation for the effectiveness of noise-agnostic diffusion models, grounded in the marginal energy landscape and the implicit Riemannian preconditioning realized by the neural network. The analysis establishes both theoretical and practical guidelines for the design of stable autonomous generative models and offers a unifying geometric perspective on the interplay between model architecture, loss parameterization, and sampling dynamics.

Reference:

"The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning" (2602.18428)