Self-Regulating Annealing in Heavy-Tailed Diffusion Models

Abstract: Diffusion models have emerged as a leading framework for deep generative modeling. While the standard Gaussian formulation is theoretically convenient, its suitability for heavy-tailed datasets remains unclear. To address this, heavy-tailed diffusion models (HTDMs) extend the standard formulation by replacing the Gaussian distribution with a Student's t-distribution, thereby improving tail fidelity on heavy-tailed datasets. Although stochastic differential equation (SDE)-based sampling is possible in HTDMs, it has not been fully explored. In this paper, we propose an SDE-based sampler for HTDMs that explicitly incorporates a state-dependent diffusion coefficient. This state dependence naturally induces a self-regulating annealing mechanism by adaptively modulating the effective noise scale. We theoretically explore this mechanism and experimentally verify its necessity for reproducing samples from a heavy-tailed distribution.

• The paper introduces a novel SDE-based sampler that implements a self-regulating annealing mechanism by dynamically scaling noise using the squared Mahalanobis distance in heavy-tailed diffusion models.
• The methodology leverages Student's t-distributions to overcome the limitations of Gaussian noise, achieving better tail fidelity as validated by Q–Q plots and extreme-tail probability measures.
• Empirical results demonstrate that the adaptive noise mechanism effectively captures extreme outliers, highlighting its potential for applications in anomaly detection and modeling rare, high-impact events.

Self-Regulating Annealing in Heavy-Tailed Diffusion Models

Introduction

This work introduces a stochastic differential equation (SDE)-based generative sampling framework within Heavy-Tailed Diffusion Models (HTDMs) parameterized with Student's $t$-distributions. Unlike conventional Gaussian-based Diffusion Probabilistic Models (DPMs), HTDMs address the pronounced limitations of Gaussian noise assumptions when modeling heavy-tailed or outlier-contaminated datasets. The paper advances prior work on HTDMs by devising an SDE-based sampler that explicitly implements a state-dependent diffusion coefficient. This design choice enables a self-regulating annealing mechanism characterized by dynamic scaling of effective noise proportional to the squared Mahalanobis distance between the current generative state and the denoiser's estimate. Both theoretical analysis and empirical validation on synthetic heavy-tailed targets are provided, with particular focus on tail fidelity.

Background: Diffusion Models and Heavy-Tailed Extensions

Standard DPMs evolve data toward noise through a Markovian Gaussian forward diffusion and reverse the process via learned denoising transitions. While these methods offer tractable closed-form training objectives due to Gaussianity, their generated distributions lack fidelity in the presence of heavy tails. HTDMs [pandey2024heavytailed] address this deficiency by generalizing both the forward and reverse kernels to the Student's $t$-distribution, leading to non-Gaussian marginals, tractable posteriors, and improved tail modeling, albeit with increased non-Markovianity in the forward process.

The critical insight leveraged in this work is that the posterior in HTDMs, derived from the Student's $t$ structure, contains an explicit state-dependent covariance scaling via $(\nu + \Delta_t^2)/(\nu + d)$, where $\nu$ is the degrees of freedom and $\Delta_t^2$ is the squared Mahalanobis distance. This term, omitted in previous practical samplers, is central in implementing adaptive noise scaling during sampling.

SDE-Based Sampler with State-Dependent Diffusion

The authors derive an SDE formulation for HTDMs in the continuous-time limit by discretizing the generative process and carefully accounting for the asymptotics of the Student’s $t$-posterior. They choose the transition variance so it remains $O(\Delta t)$ for vanishing time steps, ensuring proper SDE dynamics in the limit. The resulting SDE possesses a diffusion coefficient scaled by the non-linear, state-dependent term $$\alpha(\Delta_t^2(\mathbf{x}_t), t) = \sqrt{(\nu + \Delta_t^2)/(\nu + d - 2)}$$, where the distance $\Delta_t^2$ reflects the deviation from the ideal denoised state.

Crucially, this implementation realizes a self-regulating annealing mechanism—when the current state is distant from the denoiser’s target, the effective noise increases, promoting larger updates and accelerated exploration of the generative space; when close, noise diminishes, enforcing finer refinement near data manifolds.

Figure 1: Self-regulating annealing mechanism in the variance-exploding SDE for the symmetric two-point data distribution.

The paper also demonstrates equivalence of this mechanism to adaptive effective temperature modulation in deformed-exponential associative memory models, where such adaptation enhances retrieval and storage phase transitions.

Empirical Evaluation: Tail Fidelity and Dynamics

Experimental validation is conducted on a one-dimensional Student's $t$0-distributed synthetic dataset ($t$1), supporting direct analysis of tail statistics. The SDE-based sampler with state-dependent coefficient (denoted $t$2-SDE) is compared against a Gaussian baseline (VE-SDE), an ODE sampler for $t$3-EDM, and an ablated $t$4-SDE where the adaptive noise coefficient is set to unity.

Q–Q plots reveal that the adaptive $t$5-SDE and $t$6-ODE samplers accurately reproduce empirical quantiles, especially in the heavy-tailed regime, whereas the Gaussian and ablated $t$7-SDE fail to capture extreme outliers.

Figure 2: Q–Q plots comparing generated and test samples for the four samplers, highlighting improved tail fidelity for samplers with state-dependent diffusion.

Quantitative evaluation of extreme-tail probabilities ($t$8, $t$9) further demonstrates that only $t$0-ODE and $t$1-SDE (with adaptive diffusion) achieve probabilities on the correct order of magnitude ($t$2), with the VE-SDE underestimating by nearly two orders of magnitude, and the ablated $t$3-SDE producing no extreme samples. Additionally, Wasserstein-1 distances confirm improved overall distributional matching, with $t$4-SDE outperforming all baselines except $t$5-ODE.

Theoretical Implications

The explicit inclusion of state-dependent noise in generative SDEs broadens the theoretical foundations of diffusion-based generative modeling for non-Gaussian settings. By constructing the sampler directly from the closed-form Student’s $t$6 posterior, the methodology realizes an annealing regime congruent with statistical-physics interpretations of adaptive temperature systems. It elucidates the limitations of fixed-noise SDEs for approximating targets with divergent higher-order moments or rare, large-magnitude events.

This adaptive SDE can be regarded as an instance of a broader class of stochastic processes where the diffusion term responds nonlinearly to the state, leading to nontrivial sample path properties and potentially richer expressivity in modeling diverse data modalities.

Practical Implications and Future Directions

Practically, the state-dependent SDE-based sampler advances the capacity of diffusion models to match empirical distributions with heavy-tails, mitigating a key failure mode in outlier-generative tasks. Applications susceptible to rare but high-impact events (e.g., financial, ecological, or scientific anomaly modeling) stand to benefit from these developments.

Future avenues include:

• Extension to high-dimensional data, especially natural images, assessing scalability and calibration of the degrees-of-freedom parameter $t$7.
• Investigation of more principled, potentially adaptive heuristics for selecting or learning $t$8 during training and sampling.
• Formal analysis of convergence and ergodicity properties under state-dependent diffusions in HTDMs for broader theoretical assurances.

Conclusion

This study presents a principled SDE-based sampler for heavy-tailed diffusion models leveraging Student's $t$9-distributions, with explicit state-dependent diffusion reflecting a self-regulating annealing mechanism. The approach achieves strong tail fidelity and overcomes the limitations of fixed-noise and non-adaptive generative samplers, thereby enhancing the robustness of diffusion models in the presence of heavy-tailed or outlier-rich datasets. The integration of statistical-physics insights and robust empirical methodologies positions the proposed framework as a natural extension for generative modeling of heavy-tailed phenomena.