Benign Overfitting Does Not Occur in Diffusion Models

Abstract: Benign overfitting and double descent have come to shape our understanding of generalization in deep learning, establishing that overfitting is not only compatible with good generalization but can actively benefit it. Diffusion models share much of the machinery of standard deep learning, so it is natural to assume that they also exhibit these properties. In this work, we show that this assumption is largely incorrect. We first establish fundamental impossibility results showing that, unless the sample size grows exponentially with the data dimension, overfitting and good generalization cannot occur simultaneously. Consequently, the population loss follows a classical U-shaped curve in model complexity rather than exhibiting double descent. Analyzing a simplified setting, we identify a key difference between regression and score matching: regression benefits from an alignment between the target and the empirical covariance; score matching admits no such alignment, leaving overfitting irreparably harmful. We further identify implicit regularization stemming from time-smoothness of the score and early stopping during training as mechanisms that prevent such overfitting and verify our findings with high-dimensional image generation experiments. Our results reveal that generalization in diffusion models is governed by mechanisms distinct from those of traditional regression, motivating the development of new theory.

• The paper demonstrates that score matching in diffusion models prevents benign overfitting, contrasting with the double descent observed in standard regression settings.
• Using analyses on RFNNs and CIFAR-10 experiments, it shows that perfect training data memorization leads to catastrophic test loss without implicit alignment.
• It highlights that implicit regularization via time-smoothness and early stopping is essential to maintain generalization in high-dimensional generative tasks.

Benign Overfitting and Diffusion Models: An Incompatibility

Overview

This paper rigorously investigates the compatibility of benign overfitting with the training and generalization properties of diffusion models, a principal family of generative models widely adopted in high-dimensional domains. Contrasting prevailing intuition drawn from classical deep learning phenomena such as double descent, the results demonstrate that benign overfitting and double descent are fundamentally absent from diffusion models trained via score matching. This conclusion is theoretically justified through analyses based on random feature neural networks (RFNNs), supported by population-empirical loss lower bounds, and empirically validated on standard high-dimensional image generation benchmarks.

Fundamental Limitations of Overfitting in Diffusion

The hallmark phenomenon of benign overfitting refers to the regime where models perfectly fit (interpolate) noisy training data yet generalize well to unseen samples. This is the basis for the "double descent" risk curve observed in MLPs and random feature models: as model complexity grows, test error first decreases, increases near interpolation, and then decreases again due to implicit regularization.

However, the paper proves that diffusion models optimized with score matching cannot exhibit benign overfitting at any practical sample size, especially as data dimensionality increases. The core technical result states that, unless the number of training examples grows exponentially in intrinsic data dimension, the empirical and population score matching losses cannot both be small. The proof leverages a lower bound involving relative Fisher information and intrinsic (Rényi) dimension, establishing that test generalization degrades catastrophically once overfitting is achieved.

This fundamental distinction between diffusion's score matching loss and standard least-squares regression is illustrated in direct comparison: score matching lacks the alignment between empirical and population covariances that enables implicitly regularized interpolation in regression settings.

Figure 1: (Left) The canonical double descent in regression using a 2-layer RFNN; (Right) Population and empirical score matching losses in diffusion, showing classical U-shaped generalization with no double descent.

Fine-Grained Analysis: Linear Random Feature Models

The authors sharpen their claims by studying explicit two-layer linear RFNNs within the score matching framework. The analytical approach enables characterization of empirical and population losses as a function of model width $p$, data dimension $d$, sample size $n$, and noise scale. In the overparameterized regime ($p \gg d, n$), they derive:

• The empirical risk becomes arbitrarily small—indicating full memorization.
• The population (test) score matching loss explodes as noise level or time parameter $t \to 0$.

This uncoupling of train and test errors is in stark contrast to regression, where minimum-norm interpolators can retain bounded excess risk due to "alignment" effects. In diffusion, the population loss receives unavoidable contributions from the null space that cannot be attenuated by model overparameterization or sample size, unless in the exponentially-infeasible data regime.

Figure 2: Empirical and population losses for linear RFNNs as model size and noise scale vary, emphasizing incompatibility of memorization and generalization.

Empirical Evidence: High-Dimensional Diffusion with U-Nets

Experiments on CIFAR-10 using DDPM U-Nets further reveal the consequences of these theoretical findings. By training models with increasingly large numbers of features (i.e., more parameters), the authors show:

• Empirical train loss drops to near-zero, indicating memorization past a critical model size.
• Population (test) loss does not demonstrate double descent; rather, it exhibits a classical U-shaped curve in model complexity, with growth in test loss as overfitting emerges.
• Generated samples for high-capacity models exhibit near-identical reconstructions of training data, further exposing memorization without generalization.

Figure 3: (Left, Center) Train and test errors as U-Net size increases, highlighting U-shaped test loss; (Right) Generated images and nearest train set neighbors for low and high parameter models, evidencing memorization.

Implicit Regularization: Time-Smoothness and Early Stopping

With the impossibility of benign overfitting established, the authors analyze mechanisms that enable diffusion models to avoid catastrophic overfitting in practice. Two sources of implicit regularization are identified:

1. Time-Smoothness: Training a single network to estimate the score function across a range of noise levels (times) introduces smoothness constraints that act analogously to ridge regularization. Analytical and empirical analyses demonstrate time-smoothness controls both empirical and population losses—preventing the divergence seen in non-smooth/single-time estimators.
2. Early Stopping: Investigating the training dynamics of overparameterized score networks under gradient flow, the authors show that early stopping effectively regularizes the network in a way that keeps the population score matching loss bounded, even as the empirical risk approaches zero given sufficient training.

Figure 4: Generalization benefits of time-smoothness—training a time-independent network over a range $[t, t+r]$ demonstrates that larger $r$ increases train error and reduces overfitting at fixed time $t$.

Implications and Theoretical Contributions

The findings radically depart from accepted intuitions in deep learning about the universality of benign overfitting in highly overparameterized networks. For practitioners, the results diagnose the risk of memorization in large diffusion networks, underlining the necessity of explicit or implicit regularization (time-smoothness, early stopping) to ensure robust generalization. Theoretically, the distinction highlights that score matching fundamentally alters the relationship between interpolation and generalization, requiring a new statistical understanding tailored to the geometry and dynamics of high-dimensional diffusion processes.

Potential avenues for future work include developing data-dependent generalization guarantees for diffusion models, investigating alternative loss formulations that might align empirical and population objectives more closely, and extending the analysis to manifold-structured or non-Euclidean data settings where intrinsic dimension scaling may be more favorable.

Conclusion

The paper establishes that benign overfitting and double descent do not occur in diffusion models trained with score matching, both theoretically and empirically. Instead, generalization in diffusion models is governed by mechanisms distinctly different from regression. Implicit regularization through time-smoothness and early stopping are essential to prevent overfitting, motivating the need for a revised understanding of generalization principles in generative diffusion systems.