Selective Underfitting in Diffusion Models

• The paper demonstrates that the score function fits accurately in the supervision regions around noisy training samples, while underfitting in less-visited extrapolation areas.
• Empirical analyses like the CFG paradox and direct error partitioning reveal distinct errors between supervised and unsupervised regions, informing model capacity choices.
• By balancing memorization and generalization through structured underfitting, diffusion models generate diverse, high-quality samples without simply replicating the training data.

Selective underfitting in diffusion models refers to the phenomenon where the score function learned by the model fits the empirical score very accurately in regions of input space corresponding to noisy versions of the training data (the supervision region), but underfits elsewhere (the extrapolation region). This selective behavior is essential for enabling generative diffusion models to generalize, produce high-quality samples, and not simply memorize or regenerate the training data. Understanding and characterizing selective underfitting provides critical insight into the inductive biases, architecture choices, and generalization properties of diffusion models (Song et al., 1 Oct 2025).

1. Definition and Distinction from Global Underfitting

Selective underfitting contrasts with global underfitting, where a model fails to fit the empirical score uniformly across the entire input space. In the context of diffusion models, global underfitting would manifest as the score function systematically deviating from the empirical optimum everywhere, leading to low-fidelity and poorly generalizing samples.

Selective underfitting, by comparison, is a localized phenomenon. The model is highly accurate in the "supervision region"—the high-probability sets around noisy training samples corresponding to the mixture-of-Gaussians peaks formed by the forward corruption process—but underfits (i.e., deviates from the empirical score) in "extrapolation regions" that are rarely, if ever, visited by the noisy data during training.

The supervision region $\mathcal{T}_t(\delta)$ at timestep $t$ is given by:

$$\mathcal{T}_t(\delta) = \Big\{ x_t : \exists\, i \in \{1,\dots,N\},\, \left| \|x_t-\alpha_t^{(i)}\| - \sigma_t\sqrt{d} \right| \le \sigma_t\sqrt{d\log(1/\delta)} \Big\}$$

where $\alpha_t^{(i)}$ is the mean of the corrupted $i^{th}$ sample, and $\sigma_t$ is the standard deviation of the noise.

2. Score Function Learning and Supervision Geometry

Diffusion models are trained via denoising score-matching, minimizing the loss:

$$L_{\text{DSM}}(t) = \mathbb{E}_{x_t \sim \hat{p}_t} \left\| s_\theta(x_t, t) - s_\star(x_t, t) \right\|^2$$

where $s_\star(x_t, t) = \nabla_{x_t} \log \hat{p}_t(x_t)$ is the empirical score (gradient of the log-density of the noisy data mixture).

The structure of $\hat{p}_t(x_t)$—a mixture of Gaussians centered at the training data—implies that training samples concentrate on "thin shells" in high dimension around each training point. The network is thus heavily supervised within these regions, and the learned score accurately aligns with the empirical score (i.e., points directly toward the clean data point). Outside these shells, no direct supervision occurs, and the neural network's inductive bias determines the extrapolation, hence underfitting the empirical score.

Quantitative studies show that the score error

$\| s_\theta(x_t,t)-s_\star(x_t,t)\|^2$

decreases with model capacity in the supervision region, but increases in the extrapolation region as the network relies purely on its smoothness or architectural bias.

3. Empirical Probes of Selective Underfitting

The paper employs several experimental interventions:

• CFG Paradox Experiment: Highlights that while conditional and unconditional scores match temporally due to overlapping supervision during training, they diverge during inference as they extrapolate differently, explained by different "supervision regions."
• Direct Error Partitioning: Measures score error inside vs. outside $\mathcal{T}_t(\delta)$. As capacity increases, in-supervision-region error vanishes, but extrapolation region error grows.
• Denoising Trajectory Experiments: By initializing the denoising process from a noisy training point (firmly inside the supervision shell), the model reproduces the original image, verifying memorization in supervised regions.
• Distance Ratio Analysis: A metric

$$r^{(i)} = \frac{\|x_t-\alpha_t^{(i)}\|}{\sigma_t\sqrt{d}}$$

shows that at training, $r_\star$ (minimum over $i$) is near $1$, inside the shell, but increases rapidly during denoising of new samples, indicating movement outside the supervised shell.

These diagnostics collectively establish that supervision (and thus proper fitting) is highly localized.

4. Inductive Biases and Architectural Effects

Selective underfitting is strongly governed by the structure of the score-matching objective—i.e., by the geometry of the diffusion process which produces mixtures of Gaussians in data space. Because high-dimensional Gaussians tend to concentrate on narrow shells, the diffusion model is intrinsically shown a very limited region during training.

Network architecture (e.g., convolutional versus transformer) further determines how the score function is extrapolated beyond the supervised region. Convolutional architectures may support more efficient extrapolation but potentially at the cost of reduced precision within the supervision region. The core observation is that network inductive bias controls behavior in unsupervised (extrapolation) regions, manifesting as systematic selective underfitting.

5. Implications for Generalization and Sample Diversity

Selective underfitting in the extrapolation region allows diffusion models to avoid trivial memorization of the training set. If the model perfectly fit the empirical score globally, the reverse diffusion process would revert only to original training examples.

By fitting the score exactly in the supervision region, but underfitting elsewhere, the model is able to generate novel samples during inference. This selective underfitting balances memorization and generalization. Furthermore, expanding the supervision region too much (e.g., via too much data augmentation or wider noise kernels) provably leads to generalized overfitting and loss of novelty.

Scaling laws introduced in the paper decompose metrics such as FID as:

$\text{FID} = f_{\text{extrapolation}}(L)$

where $L$ is the DSM loss (training error in supervised region), and $f_{\text{extrapolation}}$ captures the impact of extrapolated regions on sample quality.

6. Regional Characterization and Diagnostics

Quantitative demarcation between supervision and extrapolation regions enables insight into training and inference dynamics. The supervision region $\mathcal{T}_t(\delta)$ encapsulates the thin-shell geometry where the majority of the model's supervision occurs; the extrapolation region is its complement.

The ratio metric $r^{(i)}$ and its minimum $r_\star$ across training data are used in practice to determine region membership. A sample is "supervised" if $r_\star \approx 1$, unsupervised otherwise.

7. Testable Predictions and Future Research

The selective underfitting framework yields several testable hypotheses:

• Increasing the size or measure of the supervision region (e.g., via noise schedule adjustments) leads to degraded generalization and possible memorization, as the network becomes constrained to fit the data everywhere.
• Scaling network capacity reduces error within the supervision region but does not, and may even increase, error in the extrapolation region.
• The "CFG paradox"—the observed differences between conditional and unconditional denoising during inference—are traceable to the differing geometry of their respective supervision regions.
• Perception-Aligned Training (PAT), which seeks to remap model representations to perceptually meaningful distances, may improve extrapolation and generative quality—a direction open to empirical validation.

A plausible implication is that further study of selective underfitting, especially through the lens of representation geometry and network bias, could lead to refined training or architectural choices designed to optimize extrapolation and hence generative performance and sample diversity in diffusion models.

In summary, selective underfitting—accurate score learning in the data-supported region and structured underfitting elsewhere—is a central mechanism for the generalization capacity and generative diversity observed in state-of-the-art diffusion models (Song et al., 1 Oct 2025). This insight supports new, testable approaches to improving model design, supervising regimes, and understanding the balance between fitting and generalization in modern generative modeling.