Mechanism Behind Extrapolation in Diffusion Models

Determine a theoretical characterization of the mechanism by which diffusion models trained via denoising score matching extrapolate outside the supervision region during inference; specifically, analyze and describe how the learned score function is shaped in the extrapolation region under the selective underfitting framework and the identified freedom of extrapolation.

Background

The paper introduces selective underfitting: diffusion models closely approximate the empirical score within a restricted supervision region formed by thin Gaussian shells around training data, but they extrapolate outside this region during inference. Experiments show that sampling trajectories quickly leave the supervision region and that underfitting manifests predominantly in the extrapolation region.

While the empirical and learned scores coincide in the supervision region, the behavior of the learned score outside this region determines generalization and generative performance. The authors propose a decomposed analysis of performance and a Perception-Aligned Training hypothesis, but they note that the underlying mechanism shaping the extrapolated score remains theoretically uncharacterized.