Impact of the training trajectory on diffusion model quality

Determine whether the entire training trajectory—rather than only the final training/validation loss—affects the final generative quality of the U-Net denoising diffusion probabilistic model (DDPM) trained to learn the score function of Navier–Stokes Kolmogorov-flow trajectories, and characterize which aspects of the training dynamics (e.g., learning-rate annealing) are responsible for this effect.

Background

Across experiments, the authors observe cases where similar or even better loss values do not correspond to comparable generative quality, particularly for schedule-free AdamW and for the wsd learning-rate schedule. This suggests that final loss alone may not predict sample quality in their diffusion training setup.

Motivated by these findings, the authors posit that the path of optimization, including scheduling and cooldown, could be crucial for the eventual quality of the learned diffusion model, and explicitly flag this as an open direction for future work.

References

We conjecture that the entire training trajectory might impact the final model quality, and leave this open for future work.

— Optimization Benchmark for Diffusion Models on Dynamical Systems  (2510.19376 - Schaipp, 22 Oct 2025) in Conclusion (Section 4)

Whether the multi-step path buys the gain is a separate question: our control holds the loss fixed but still cannot separate the schedule from the variational term it carries, so we scope that claim to the path as a whole.

— Generative Semantic Scene Completion  (2608.26737 - Chen et al., 27 Aug 2026) in Section 2, Related Work, paragraph “Diffusion from a structured source”

An implementation error ran its first 12{,}140 steps as a complete cosine to the floor and then re-warmed, so the endpoint learning rate is the designed one and the path is not (Appendix~\ref{app:prereg}). The direction of the deviation's effect was not measured.

— Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs  (2610.02657 - Lu et al., 2 Oct 2026) in Appendix, Section “Interface ablations: detail,” paragraph “The 1.7\,B donor at matched tokens per trainable parameter”