Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor (2303.17546v3)

Published 30 Mar 2023 in cs.CV, cs.AI, and cs.LG

Abstract: Generative image editing has recently witnessed extremely fast-paced growth. Some works use high-level conditioning such as text, while others use low-level conditioning. Nevertheless, most of them lack fine-grained control over the properties of the different objects present in the image, i.e. object-level image editing. In this work, we tackle the task by perceiving the images as an amalgamation of various objects and aim to control the properties of each object in a fine-grained manner. Out of these properties, we identify structure and appearance as the most intuitive to understand and useful for editing purposes. We propose PAIR Diffusion, a generic framework that can enable a diffusion model to control the structure and appearance properties of each object in the image. We show that having control over the properties of each object in an image leads to comprehensive editing capabilities. Our framework allows for various object-level editing operations on real images such as reference image-based appearance editing, free-form shape editing, adding objects, and variations. Thanks to our design, we do not require any inversion step. Additionally, we propose multimodal classifier-free guidance which enables editing images using both reference images and text when using our approach with foundational diffusion models. We validate the above claims by extensively evaluating our framework on both unconditional and foundational diffusion models. Please refer to https://vidit98.github.io/publication/conference-paper/pair_diff.html for code and model release.

PDF HTML Abstract

PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor

The paper discusses PAIR Diffusion, a framework designed for fine-grained, object-level image editing using diffusion models. Conventional image editing methods often lack the capacity to independently manipulate distinct objects within an image at a granular level. This framework perceives images as a combination of multiple objects, with the goal of controlling properties such as structure and appearance for each object individually.

Object-Level Editing

PAIR Diffusion is predicated on the notion that each image consists of distinct objects, each characterized by structural and appearance attributes. The structural properties include shape and category, whereas appearance encompasses attributes like color and texture. The framework uses panoptic segmentation maps and pre-trained image encoders to extract these elements.

  • Structure Representation: Utilizes panoptic segmentation to capture object shapes and categories.
  • Appearance Representation: Employs convolutional and transformer-based encoders to encapsulate both low-level and high-level object features.

Editing Capabilities

The framework enables an array of editing tasks, including:

  • Appearance Editing: Modifying object appearances while retaining their structures by leveraging reference images.
  • Shape Editing: Altering the shapes of objects independently.
  • Object Addition: Introducing new objects with tailored structures and appearances into existing images.
  • Variations: Generating diverse visual renditions of objects.

Diffusion Model Integration

The paper integrates these capabilities into diffusion models, specifically leveraging them in two contexts: unconditional diffusion models and foundational text-to-image models. The approach introduces modifications to the architecture to incorporate object-level conditioning:

  • Unconditional Models: Enhancements are applied to latent diffusion models.
  • Foundational Models: Employs ControlNet to modulate Stable Diffusion models with object-level detail.

Multimodal Classifier-Free Guidance

A notable innovation is the multimodal classifier-free guidance technique, which ensures effective control over image outputs using both textual descriptions and reference images. This method allows seamless integration of these inputs to guide the editing process, ensuring that both the structure and appearance are accurately reflected in the output.

Evaluation and Implications

The authors present extensive qualitative and quantitative evaluations across several datasets, demonstrating the comprehensive editing capabilities and highlighting enhancements over existing methods. The framework offers significant implications for AI-driven image editing tools, enabling more intuitive and precise manipulations of image content.

Conclusion

PAIR Diffusion represents a significant advancement in the capability of image editing models, marking a step toward more nuanced object-level control in image synthesis. Future research may focus on expanding the framework's applicability across broader domains, improving efficiency, and exploring additional object attributes for more sophisticated editing capabilities. The development of such frameworks continues to push the boundaries of what is feasible in AI-driven content creation.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. SpaText: Spatio-Textual Representation for Controllable Image Generation, November 2022. URL http://arxiv.org/abs/2211.14305. arXiv:2211.14305 [cs].
  2. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022.
  3. End-to-end visual editing with a generatively pre-trained artist. In European Conference on Computer Vision (ECCV), 2022.
  4. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  1209–1218, 2018.
  5. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08465, 2023.
  6. Diffusiondet: Diffusion model for object detection. arXiv preprint arXiv:2211.09788, 2022.
  7. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  1290–1299, 2022.
  8. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022a.
  9. DiffEdit: Diffusion-based semantic image editing with mask guidance, October 2022b. URL http://arxiv.org/abs/2210.11427. arXiv:2210.11427 [cs].
  10. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  11. Blobgan: Spatially disentangled scene representations. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV, pp.  616–635. Springer, 2022.
  12. Diffusion self-guidance for controllable image generation. arXiv preprint arXiv:2306.00986, 2023.
  13. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032, 2022.
  14. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  3224–3234, 2019.
  15. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. arXiv preprint arXiv:2302.10109, 2023.
  16. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
  17. Classifier-Free Diffusion Guidance, July 2022. URL http://arxiv.org/abs/2207.12598. arXiv:2207.12598 [cs].
  18. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  19. Unified Discrete Diffusion for Simultaneous Vision-Language Generation, November 2022. URL http://arxiv.org/abs/2211.14842. arXiv:2211.14842 [cs].
  20. Semask: Semantically masking transformer backbones for effective semantic segmentation. arXiv, 2021.
  21. Transgan: Two transformers can make one strong gan. arXiv preprint arXiv:2102.07074, 2021.
  22. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
  23. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4401–4410, 2019.
  24. Diffusion-lm improves controllable text generation. arXiv preprint arXiv:2205.14217, 2022.
  25. MagicMix: Semantic Mixing with Diffusion Models, October 2022. URL http://arxiv.org/abs/2210.16056. arXiv:2210.16056 [cs].
  26. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp.  740–755. Springer, 2014.
  27. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34:16331–16345, 2021.
  28. Compositional Visual Generation with Composable Diffusion Models, July 2022. URL http://arxiv.org/abs/2206.01714. arXiv:2206.01714 [cs].
  29. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103.16091, 2021.
  30. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794, 2022.
  31. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421, 2023.
  32. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
  33. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
  34. Zero-shot image-to-image translation. arXiv preprint arXiv:2302.03027, 2023.
  35. Localizing object-level shape variations with text-to-image diffusion models. arXiv preprint arXiv:2303.11306, 2023.
  36. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.
  37. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116–12128, 2021.
  38. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022a.
  39. Hierarchical Text-Conditional Image Generation with CLIP Latents, April 2022b. URL http://arxiv.org/abs/2204.06125. arXiv:2204.06125 [cs].
  40. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10684–10695, 2022a.
  41. High-Resolution Image Synthesis with Latent Diffusion Models, April 2022b. URL http://arxiv.org/abs/2112.10752. arXiv:2112.10752 [cs].
  42. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, August 2022. URL http://arxiv.org/abs/2208.12242. arXiv:2208.12242 [cs].
  43. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
  44. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
  45. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
  46. Text-to-4d dynamic scene generation. arXiv preprint arXiv:2301.11280, 2023.
  47. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp.  2256–2265. PMLR, 2015.
  48. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
  49. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
  50. ObjectStitch: Generative Object Compositing, December 2022. URL http://arxiv.org/abs/2212.00932. arXiv:2212.00932 [cs].
  51. Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation, November 2022. URL http://arxiv.org/abs/2211.12572. arXiv:2211.12572 [cs].
  52. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287–11302, 2021.
  53. Sketch-Guided Text-to-Image Diffusion Models, November 2022. URL http://arxiv.org/abs/2211.13752. arXiv:2211.13752 [cs].
  54. Neurallift-360: Lifting an in-the-wild 2d photo to a 3d object with 360° views. arXiv e-prints, pp.  arXiv–2211, 2022a.
  55. Versatile diffusion: Text, images and variations all in one diffusion model, 2022b. URL https://arxiv.org/abs/2211.08332.
  56. Prompt-free diffusion: Taking” text” out of text-to-image diffusion models. arXiv preprint arXiv:2305.16223, 2023.
  57. Paint by example: Exemplar-based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18381–18391, 2023.
  58. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015.
  59. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
  60. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp.  818–833. Springer, 2014.
  61. Scenecomposer: Any-level semantic image synthesis. arXiv preprint arXiv:2211.11742, 2022a.
  62. SceneComposer: Any-Level Semantic Image Synthesis, November 2022b. URL http://arxiv.org/abs/2211.11742. arXiv:2211.11742 [cs].
  63. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
  64. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  586–595, 2018.
  65. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  633–641, 2017.
  66. Sean: Image synthesis with semantic region-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5104–5113, 2020.
  67. zllrunning. https://github.com/zllrunning/face-parsing.pytorch. github, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Vidit Goel (13 papers)
  2. Elia Peruzzo (9 papers)
  3. Yifan Jiang (79 papers)
  4. Dejia Xu (37 papers)
  5. Xingqian Xu (23 papers)
  6. Nicu Sebe (270 papers)
  7. Trevor Darrell (324 papers)
  8. Zhangyang Wang (375 papers)
  9. Humphrey Shi (97 papers)
Citations (4)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets