· Trending Papers
Emergent Mind

Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder

(2403.10255)
Published Mar 15, 2024 in cs.CV

Abstract

Super-resolution (SR) and image generation are important tasks in computer vision and are widely adopted in real-world applications. Most existing methods, however, generate images only at fixed-scale magnification and suffer from over-smoothing and artifacts. Additionally, they do not offer enough diversity of output images nor image consistency at different scales. Most relevant work applied Implicit Neural Representation (INR) to the denoising diffusion model to obtain continuous-resolution yet diverse and high-quality SR results. Since this model operates in the image space, the larger the resolution of image is produced, the more memory and inference time is required, and it also does not maintain scale-specific consistency. We propose a novel pipeline that can super-resolve an input image or generate from a random noise a novel image at arbitrary scales. The method consists of a pretrained auto-encoder, a latent diffusion model, and an implicit neural decoder, and their learning strategies. The proposed method adopts diffusion processes in a latent space, thus efficient, yet aligned with output image space decoded by MLPs at arbitrary scales. More specifically, our arbitrary-scale decoder is designed by the symmetric decoder w/o up-scaling from the pretrained auto-encoder, and Local Implicit Image Function (LIIF) in series. The latent diffusion process is learnt by the denoising and the alignment losses jointly. Errors in output images are backpropagated via the fixed decoder, improving the quality of output images. In the extensive experiments using multiple public benchmarks on the two tasks i.e. image super-resolution and novel image generation at arbitrary scales, the proposed method outperforms relevant methods in metrics of image quality, diversity and scale consistency. It is significantly better than the relevant prior-art in the inference speed and memory usage.

We're not able to analyze this paper right now due to high demand.

Please check back later (sorry!).

Sign up for a free account or log in to generate a summary of this paper:

Sign Up Now
or
Log In

We ran into a problem analyzing this paper.

Newsletter

Get summaries of trending comp sci papers delivered straight to your inbox:

Unsubscribe anytime.

References
  1. Image generators with conditionally-independent pixel synthesis. In CVPR, pages 14278–14287
  2. MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation
  3. The perception-distortion tradeoff. In CVPR
  4. iEdit: Localised Text-guided Image Editing with Weak Supervision
  5. Prompt augmentation for self-supervised text-guided image manipulation. In CVPR
  6. Glean: Generative latent bank for large-factor image super-resolution. In CVPR, pages 14245–14254
  7. Learning continuous image representation with local implicit image function. In CVPR, pages 8628–8638
  8. Toward spatially unbiased generative models. In ICCV, pages 14253–14262
  9. Implicit diffusion models for continuous super-resolution. In CVPR, pages 10021–10030
  10. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 6840–6851. Curran Associates, Inc.
  11. Cascaded Diffusion Models for High Fidelity Image Generation
  12. Meta-sr: A magnification-arbitrary network for super-resolution. In CVPR
  13. Msg-gan: Multi-scale gradients for generative adversarial networks. In CVPR
  14. Progressive growing of gans for improved quality, stability, and variation. ICLR
  15. A style-based generator architecture for generative adversarial networks. In CVPR
  16. Diffusionclip: Text-guided diffusion models for robust image manipulation. In CVPR, pages 2426–2435
  17. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, pages 4681–4690
  18. Local texture estimator for implicit representation function. In CVPR, pages 1929–1938
  19. Interhandgen: Two-hand interaction generation via cascaded reverse diffusion. In CVPR
  20. Swinir: Image restoration using swin transformer. In ICCVW, pages 1833–1844
  21. Enhanced deep residual networks for single image super-resolution. In CVPRW
  22. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In CVPR
  23. Arbitrary-scale image synthesis. In CVPR, pages 11533–11542
  24. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684–10695
  25. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241
  26. Pyramidal Denoising Diffusion Probabilistic Models
  27. Image Super-Resolution via Iterative Refinement
  28. Progressive distillation for fast sampling of diffusion models. In ICLR
  29. Adversarial generation of continuous images. In CVPR, pages 10753–10764
  30. Denoising Diffusion Implicit Models
  31. LLDiffusion: Learning Degradation Representations in Diffusion Models for Low-Light Image Enhancement
  32. Esrgan: Enhanced super-resolution generative adversarial networks. In ECCVW
  33. Unlimited-size diffusion restoration. In CVPRW, pages 1160–1167, 2023b.
  34. Positional encoding as spatial inductive bias in gans. In CVPR, pages 13569–13578
  35. Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization
  36. LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
  37. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018a.
  38. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018b.
  39. Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. In CVPR

Show All 39