Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling (2402.15170v1)

Published 23 Feb 2024 in cs.LG and cs.AI

Abstract: With the incorporation of the UNet architecture, diffusion probabilistic models have become a dominant force in image generation tasks. One key design in UNet is the skip connections between the encoder and decoder blocks. Although skip connections have been shown to improve training stability and model performance, we reveal that such shortcuts can be a limiting factor for the complexity of the transformation. As the sampling steps decrease, the generation process and the role of the UNet get closer to the push-forward transformations from Gaussian distribution to the target, posing a challenge for the network's complexity. To address this challenge, we propose Skip-Tuning, a simple yet surprisingly effective training-free tuning method on the skip connections. Our method can achieve 100% FID improvement for pretrained EDM on ImageNet 64 with only 19 NFEs (1.75), breaking the limit of ODE samplers regardless of sampling steps. Surprisingly, the improvement persists when we increase the number of sampling steps and can even surpass the best result from EDM-2 (1.58) with only 39 NFEs (1.57). Comprehensive exploratory experiments are conducted to shed light on the surprising effectiveness. We observe that while Skip-Tuning increases the score-matching losses in the pixel space, the losses in the feature space are reduced, particularly at intermediate noise levels, which coincide with the most effective range accounting for image quality improvement.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313–326, 1982.
  2. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  22669–22679, 2023.
  3. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  4. Effectively unbiased fid and inception score and where to find them. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  6070–6079, 2020.
  5. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  6. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  7. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  8. A kernel method for the two-sample-problem. Advances in neural information processing systems, 19, 2006.
  9. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723–773, 2012.
  10. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  11. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
  12. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  13. Complexity matters: Rethinking the latent space for generative modeling. arXiv preprint arXiv:2307.08283, 2023.
  14. Scalelong: Towards more stable training of diffusion model via scaling network long skip connection. arXiv preprint arXiv:2310.13545, 2023.
  15. Rethinking fid: Towards a better evaluation metric for image generation. arXiv preprint arXiv:2401.09603, 2023.
  16. Scedit: Efficient and controllable image diffusion generation via skip connection editing. arXiv preprint arXiv:2312.11392, 2023.
  17. Internalized biases in fréchet inception distance. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
  18. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852–863, 2021.
  19. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565–26577, 2022.
  20. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.02696, 2023.
  21. Variational diffusion models. arXiv preprint arXiv:2107.00630, 2021.
  22. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022.
  23. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. arXiv preprint arXiv:2305.18455, 2023.
  24. Deciphering the projection head: Representation evaluation self-supervised learning. arXiv preprint arXiv:2301.12189, 2023a.
  25. Elucidating the design space of classifier-guided diffusion generation. arXiv preprint arXiv:2310.11311, 2023b.
  26. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  11410–11420, 2022.
  27. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
  28. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.  8748–8763. PMLR, 2021.
  29. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10684–10695, 2022.
  30. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp.  234–241. Springer, 2015.
  31. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
  32. Freeu: Free lunch in diffusion u-net. arXiv preprint arXiv:2309.11497, 2023.
  33. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp.  2256–2265. PMLR, 2015.
  34. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
  35. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
  36. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
  37. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
  38. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2818–2826, 2016.
  39. Vincent, P. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011.
  40. A unified framework for u-net design and analysis. arXiv preprint arXiv:2305.19638, 2023.
  41. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp.  3–19, 2018.
  42. Bourgan: Generative networks with metric embeddings. Advances in neural information processing systems, 31, 2018.
  43. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. arXiv preprint arXiv:2309.05019, 2023.
  44. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902, 2022.
  45. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867, 2023.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now