Improving Diffusion Models's Data-Corruption Resistance using Scheduled Pseudo-Huber Loss (2403.16728v1)
Abstract: Diffusion models are known to be vulnerable to outliers in training data. In this paper we study an alternative diffusion loss function, which can preserve the high quality of generated data like the original squared $L_{2}$ loss while at the same time being robust to outliers. We propose to use pseudo-Huber loss function with a time-dependent parameter to allow for the trade-off between robustness on the most vulnerable early reverse-diffusion steps and fine details restoration on the final steps. We show that pseudo-Huber loss with the time-dependent parameter exhibits better performance on corrupted datasets in both image and audio domains. In addition, the loss function we propose can potentially help diffusion models to resist dataset corruption while not requiring data filtering or purification compared to conventional training algorithms.
- Anderson, B. D. Reverse-time Diffusion Equation Models. Stochastic Processes and their Applications, 12(3):313 – 326, 1982. ISSN 0304-4149.
- Extracting Training Data from Diffusion Models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253–5270. USENIX Association, aug 2023.
- ChiroDiff: Modelling chirographic data with Diffusion Models. In The Eleventh International Conference on Learning Representations, 2023.
- Diffusion Models Beat GANs on Image Synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780–8794. Curran Associates, Inc., 2021.
- GENIE: Higher-Order Denoising Diffusion Solvers. In Advances in Neural Information Processing Systems, volume 35, pp. 30150–30166. Curran Associates, Inc., 2022.
- Implicit Diffusion Models for Continuous Super-Resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10021–10030, June 2023.
- NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates. In Proc. Interspeech 2022, pp. 4401–4405, 2022.
- Multi-instrument Music Synthesis with Spectrogram Diffusion. In Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, pp. 598–607, 2022.
- Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, volume 33. Curran Associates, Inc., 2020.
- Lora: Low-rank adaptation of large language models, 2021.
- Huber, P. J. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1):73 – 101, 1964.
- Denoising Diffusion Restoration Models. In Advances in Neural Information Processing Systems, volume 35, pp. 23593–23606. Curran Associates, Inc., 2022.
- Variational Diffusion Models. In Advances in Neural Information Processing Systems, volume 34, pp. 21696–21707. Curran Associates, Inc., 2021.
- Flow Matching for Generative Modeling. In The Eleventh International Conference on Learning Representations, 2023.
- Statistics of Random Processes, volume 5 of Stochastic Modelling and Applied Probability. Springer-Verlag, 1978.
- DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 5775–5787. Curran Associates, Inc., 2022.
- VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10209–10218. IEEE Computer Society, jun 2023.
- SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In International Conference on Learning Representations, 2022.
- Diffusion Models for Adversarial Purification. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16805–16827. PMLR, 17–23 Jul 2022.
- Owen, A. B. A robust hybrid of lasso and ridge regression. Contemporary Mathematics, 443:59 – 72, 01 2007.
- Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8599–8608. PMLR, 2021.
- Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme. In International Conference on Learning Representations, 2022.
- Learning transferable visual models from natural language supervision, 2021.
- High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10695, June 2022.
- Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023.
- Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models. In Proceedings of the 32nd USENIX Conference on Security Symposium, SEC ’23. USENIX Association, 2023a.
- Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models, 2023b.
- A response to glaze purification via impress, 2023c.
- Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6048–6058. IEEE Computer Society, jun 2023.
- Improved Techniques for Training Consistency Models, 2023.
- Maximum Likelihood Training of Score-Based Diffusion Models. In Advances in Neural Information Processing Systems, volume 34, pp. 1415–1428. Curran Associates, Inc., 2021a.
- Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations, 2021b.
- Consistency models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
- Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data. In NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly, 2023.
- Human Motion Diffusion Model. In The Eleventh International Conference on Learning Representations, 2023.
- Diffusers: State-of-the-Art Diffusion Models. https://github.com/huggingface/diffusers, 2022.
- The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipeline. In NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly, 2023.
- DensePure: Understanding Diffusion Models for Adversarial Robustness. In The Eleventh International Conference on Learning Representations, 2023.
- Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In International Conference on Learning Representations, 2022.
- The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.