NoiseDiffusion: Correcting Noise for Image Interpolation with Diffusion Models beyond Spherical Linear Interpolation (2403.08840v1)
Abstract: Image interpolation based on diffusion models is promising in creating fresh and interesting images. Advanced interpolation methods mainly focus on spherical linear interpolation, where images are encoded into the noise space and then interpolated for denoising to images. However, existing methods face challenges in effectively interpolating natural images (not generated by diffusion models), thereby restricting their practical applicability. Our experimental investigations reveal that these challenges stem from the invalidity of the encoding noise, which may no longer obey the expected noise distribution, e.g., a normal distribution. To address these challenges, we propose a novel approach to correct noise for image interpolation, NoiseDiffusion. Specifically, NoiseDiffusion approaches the invalid noise to the expected distribution by introducing subtle Gaussian noise and introduces a constraint to suppress noise with extreme values. In this context, promoting noise validity contributes to mitigating image artifacts, but the constraint and introduced exogenous noise typically lead to a reduction in signal-to-noise ratio, i.e., loss of original image information. Hence, NoiseDiffusion performs interpolation within the noisy image space and injects raw images into these noisy counterparts to address the challenge of information loss. Consequently, NoiseDiffusion enables us to interpolate natural images without causing artifacts or information loss, thus achieving the best interpolation results.
- Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 1982.
- Invariant risk minimization. arXiv, 2019.
- Conditional image generation with score-based diffusion models. arXiv, 2021.
- Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
- Score-based generative neural networks for large-scale optimal transport. In NeurIPS, 2021.
- Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. In NeurIPS, 2021.
- Is out-of-distribution detection learnable? In NeurIPS, 2022.
- Denoising diffusion probabilistic models. In NeurIPS, 2020.
- Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics, 2022.
- A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
- Elucidating the design space of diffusion-based generative models. In NeurIPS, 2022.
- Stochastic differential equations. Springer, 1992.
- Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv, 2024.
- Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
- Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
- Friedrich Pukelsheim. The three sigma rule. The American Statistician, 1994.
- Hierarchical text-conditional image generation with clip latents. arXiv, 2022.
- High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
- Palette: Image-to-image diffusion models. In SIGGRAPH, 2022a.
- Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022b.
- Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022c.
- Denoising diffusion implicit models. In ICLR, 2021a.
- Improved techniques for training score-based generative models. In NeurIPS, 2020.
- Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
- Gan inversion: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
- Multi-stage progressive image restoration. In CVPR, 2021.
- Causaladv: Adversarial robustness through the lens of causality. In ICLR, 2022.