Is Your Conditional Diffusion Model Actually Denoising? (2512.18736v1)

Abstract: We study the inductive biases of diffusion models with a conditioning-variable, which have seen widespread application as both text-conditioned generative image models and observation-conditioned continuous control policies. We observe that when these models are queried conditionally, their generations consistently deviate from the idealized "denoising" process upon which diffusion models are formulated, inducing disagreement between popular sampling algorithms (e.g. DDPM, DDIM). We introduce Schedule Deviation, a rigorous measure which captures the rate of deviation from a standard denoising process, and provide a methodology to compute it. Crucially, we demonstrate that the deviation from an idealized denoising process occurs irrespective of the model capacity or amount of training data. We posit that this phenomenon occurs due to the difficulty of bridging distinct denoising flows across different parts of the conditioning space and show theoretically how such a phenomenon can arise through an inductive bias towards smoothness.

• The paper presents Schedule Deviation, a novel metric quantifying the divergence between a conditional diffusion model's learned denoising process and its idealized trajectory.
• Empirical analyses demonstrate that increasing model capacity or dataset size fails to eliminate SD, underlining inherent structural inconsistencies in conditional generative modeling.
• The study reveals that self-guidance and inductive smoothness in conditional settings lead to persistent denoising deviations that affect sampler equivalence.

Denoising Consistency in Conditional Diffusion Models: The Schedule Deviation Metric

Introduction

Conditional diffusion models (CDMs), pivotal in domains such as text-conditioned image synthesis and control policy inference, are typically conceived as denoising processes where the generation at each step is consistent with the corresponding diffusion schedule. This paper exposes a fundamental inconsistency in this abstraction. Despite the theoretical formulation that diverse sampling algorithms (e.g., DDPM, DDIM) should yield equivalent marginals, empirical analysis uncovers systematic deviations in conditional settings. The authors present Schedule Deviation (SD), a rigorous metric quantifying the divergence from the idealized denoising path, and demonstrate that this divergence is not trivially eliminated by increasing model capacity or dataset size.

Figure 1: Example data from conditional MNIST, Fashion-MNIST, and maze endpoint-conditional trajectory datasets; t-SNE conditioning proxies text and context embeddings.

Schedule Deviation: Metric Formulation and Computation

The central technical contribution is the derivation of Schedule Deviation. SD measures the instantaneous discrepancy between the evolution of sample densities under the learned flow $v$ and the idealized model-consistent flow (IMCF) associated with the empirical data distribution. Mathematically, for a conditioning value $z$ and time $s$, SD is defined as: $SD(v; z, s) := \int_{X} \left|\left[ \frac{\partial p^v_{s|t}{\partial s}\right]_{t=s} - \frac{\partial _s}{\partial s}\right| dx$ where $p^v_{s|t}$ evolves according to the learned flow and $_s$ denotes the theoretical denoising process for $\hat{p}_0$. Algorithmic estimation utilizes only model samples and does not require access to ground truth score functions or additional data (Algorithm 1 in the main text).

Prevalence and Predictive Utility of Schedule Deviation

Empirical findings consistently indicate nontrivial SD values for conditional generative settings, robust to model scale and training set expansion. Notably:

• SD is not alleviated by increasing model parameter count or dataset size, and sometimes grows with more data.
• SD varies significantly between classes/contexts with equal representation, implicating dataset structure over sampling sufficiency.

  Figure 2: Schedule Deviation and empirical 1-Wasserstein distance between DDPM/DDIM samples as a function of training data size; strong structural similarity across metrics in t-SNE-conditional MNIST.

Crucially, high SD correlates with practical disagreements between samplers, notably DDPM and DDIM, despite their supposed equivalence in the limit of negligible discretization error.

Figure 3: Scatter plots demonstrate SD predicts OT-based divergence between sample sets from DDPM and DDIM across randomly drawn $z$ in multiple datasets.

Further ablations highlight the persistence and structure-dependence of SD across maze path generation and attribute-conditioned datasets.

Figure 4: Consistent SD and divergence patterns for trajectory generation (maze) and Fashion-MNIST datasets, reflecting dataset and conditioning structure.

Theoretical Analysis: Inductive Smoothness and Self-Guidance

The persistence of SD is traced to the inductive bias of neural networks and the interpolation mechanism in conditional spaces. Theoretically, even with infinite data and capacity, the need to smoothly bridge multimodal or disconnected support in the conditioning variable $z$ compels the network to interpolate using convex combinations (splines/Kernel/Fourier-weighted interpolants) of class-conditional flows. This phenomenon, termed self-guidance, is analogous in formulation to classifier-free guidance but arises intrinsically during learning rather than via explicit algorithmic intervention.

In both discrete-support (finite $z$) and continuous-support (uniform $z$), the minimization of a joint loss comprising data-fit and smoothness regularization leads to a learned flow that blends scores from nearby data points, deviating from true denoising. Explicit cubic spline solutions and Fourier-domain interpolants illustrate this principle.

Figure 5: MLP and closed-form denoiser comparison across discrete and continuous-support toy datasets; similar inductive bias and SD emerge as smoothness regularization enforces self-guidance.

Broader Dataset, Capacity, and Sampler Ablations

The analysis is extended to larger datasets (e.g., Celeb-A), confirming that Schedule Deviation remains present and is mildly impacted (rather than strongly reduced) by increased dataset size and model capacity. This indicates that SD is not simply an artifact of underfitting or insufficient data, but a structural property of conditional generative modeling with diffusion objectives.

Figure 6: Visualizations of Celeb-A conditioning and attribute clusters; no strong correlation between SD and optimal transport cost in uniformly distributed conditioning spaces.

Further ablation on per-class Schedule Deviation quantifies the variability introduced by context structure, with median and quantile values demonstrating substantial span within the same class.

Figure 7: Training dataset ablation and per-class SD distribution in Fashion-MNIST; even extensive training does not normalize SD across all contexts.

Implications, Limitations, and Future Directions

The strong numerical finding is that CDMs routinely and consistently deviate from the principle of denoising on their own model-distribution. Numerical deviations between samplers (DDPM, DDIM, GE) for the same trained network are explained directly by observed SD in conditional regions.

Implications:

• The model-consistency assumption, foundational for sampler equivalence and for distillation/acceleration approaches, is not realized in practical conditional settings.
• SD may underlie broader phenomena observed in generative modeling, including context drift, sample hallucination, and the variable efficacy of guidance heuristics.
• For algorithm designers, it serves as a caution against assuming that inference rules derived under denoising-consistency will generalize out of model ideality, notably in conditional workflows (text, context, class, property-conditioned generation).

Theoretical outlook:

• Inductive smoothness with respect to context $z$ engenders unavoidable schedule inconsistencies, and guidance emerges naturally even in the absence of explicit classifier-guidance terms.

Limitations:

• Computation of SD is resource-intensive in high-dimensional spaces, given the need to estimate divergence and scores for generated samples.
• Extension to very large-scale models (modern LDMs, multimodal transformers) will require significant computational resources or algorithmic innovation for efficient SD estimation.

Conclusion

Schedule Deviation exposes a persistent and theoretically well-founded deviation from idealized denoising in conditional diffusion models. This deviation is quantitatively robust and structurally dependent on the conditioning space and dataset, rather than on model scale or data abundance. It has direct explanatory power for the divergence of widely used samplers and calls for reconsideration of algorithmic principles predicated on denoising-consistency in conditional generative settings. Further research into mitigating, exploiting, or systematically characterizing SD may yield new advances in both practical generative performance and theoretical understanding.

Figure 8: Optimal transport distance vs Schedule Deviation across multiple datasets and samplers; cross-dataset consistency in SD as a predictor of distributional divergence.