Papers
Topics
Authors
Recent
2000 character limit reached

Lazy Diffusion Transformer for Interactive Image Editing (2404.12382v1)

Published 18 Apr 2024 in cs.CV, cs.AI, and cs.GR

Abstract: We introduce a novel diffusion transformer, LazyDiffusion, that generates partial image updates efficiently. Our approach targets interactive image editing applications in which, starting from a blank canvas or an image, a user specifies a sequence of localized image modifications using binary masks and text prompts. Our generator operates in two phases. First, a context encoder processes the current canvas and user mask to produce a compact global context tailored to the region to generate. Second, conditioned on this context, a diffusion-based transformer decoder synthesizes the masked pixels in a "lazy" fashion, i.e., it only generates the masked region. This contrasts with previous works that either regenerate the full canvas, wasting time and computation, or confine processing to a tight rectangular crop around the mask, ignoring the global image context altogether. Our decoder's runtime scales with the mask size, which is typically small, while our encoder introduces negligible overhead. We demonstrate that our approach is competitive with state-of-the-art inpainting methods in terms of quality and fidelity while providing a 10x speedup for typical user interactions, where the editing mask represents 10% of the image.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208ā€“18218, 2022.
  2. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2:3, 2023.
  3. Video generation models as world simulators. 2024.
  4. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08465, 2023.
  5. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650ā€“9660, 2021.
  6. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315ā€“11325, 2022.
  7. Pixart-Ī±š›¼\alphaitalic_Ī±: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
  8. Generative pretraining from pixels. In International conference on machine learning, pages 1691ā€“1703. PMLR, 2020.
  9. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022.
  10. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248ā€“255. Ieee, 2009.
  11. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780ā€“8794, 2021.
  12. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35:16890ā€“16902, 2022.
  13. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  14. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873ā€“12883, 2021.
  15. Scaling rectified flow transformers for high-resolution image synthesis, 2024.
  16. Masked diffusion transformer is a strong image synthesizer. arXiv preprint arXiv:2303.14389, 2023.
  17. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000ā€“16009, 2022.
  18. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
  19. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
  20. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  21. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840ā€“6851, 2020.
  22. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565ā€“26577, 2022.
  23. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  24. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. arXiv preprint arXiv:2309.06380, 2023.
  25. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  26. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775ā€“5787, 2022a.
  27. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022b.
  28. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
  29. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
  30. Swiftbrush: One-step text-to-image diffusion model with variational score distillation. arXiv preprint arXiv:2312.05239, 2023.
  31. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
  32. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162ā€“8171. PMLR, 2021.
  33. On buggy resizing libraries and surprising subtleties in fid calculation. arXiv preprint arXiv:2104.11222, 2021.
  34. Localizing object-level shape variations with text-to-image diffusion models. arXiv preprint arXiv:2303.11306, 2023.
  35. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195ā€“4205, 2023.
  36. Poisson image editing. In ACM SIGGRAPH 2003 Papers, pages 313ā€“318. 2003.
  37. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  38. Open world entity segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  39. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485ā€“5551, 2020.
  40. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  41. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
  42. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684ā€“10695, 2022.
  43. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479ā€“36494, 2022.
  44. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
  45. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
  46. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256ā€“2265. PMLR, 2015.
  47. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
  48. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
  49. Consistency models. 2023.
  50. stable-diffusion webui. stable-diffusion-webui. https://github.com/AUTOMATIC1111/stable-diffusion-webui, 2024. Accessed: Jan 2024.
  51. Resolution-robust large mask inpainting with fourier convolutions. arXiv preprint arXiv:2109.07161, 2021.
  52. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022.
  53. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  54. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022.
  55. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18359ā€“18369, 2023.
  56. Diffusion models as masked autoencoders. arXiv preprint arXiv:2304.03283, 2023.
  57. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22428ā€“22437, 2023.
  58. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828, 2023.
  59. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4471ā€“4480, 2019.
  60. Large scale image completion via co-modulated generative adversarial networks. In International Conference on Learning Representations (ICLR), 2021.
  61. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306.09305, 2023.
  62. Designing a better asymmetric vqgan for stablediffusion. arXiv preprint arXiv:2306.04632, 2023a.
  63. Designing a better asymmetric vqgan for stablediffusion, 2023b.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Video Overview

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free