2000 character limit reached
Scene Graph Conditioning in Latent Diffusion (2310.10338v1)
Published 16 Oct 2023 in cs.CV
Abstract: Diffusion models excel in image generation but lack detailed semantic control using text prompts. Additional techniques have been developed to address this limitation. However, conditioning diffusion models solely on text-based descriptions is challenging due to ambiguity and lack of structure. In contrast, scene graphs offer a more precise representation of image content, making them superior for fine-grained control and accurate synthesis in image generation models. The amount of image and scene-graph data is sparse, which makes fine-tuning large diffusion models challenging. We propose multiple approaches to tackle this problem using ControlNet and Gated Self-Attention. We were able to show that using out proposed methods it is possible to generate images from scene graphs with much higher quality, outperforming previous methods. Our source code is publicly available on https://github.com/FrankFundel/SGCond
- Novelai improvements on stable diffusion — by novelai — medium. https://blog.novelai.net/novelai-improvements-on-stable-diffusion-e10d38db82ac. (Accessed on 05/25/2023).
- Image colorization: A survey and dataset, 2022.
- Multi-stage variational auto-encoders for coarse-to-fine image generation, 2017.
- Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
- P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis, 2021.
- An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
- Taming transformers for high-resolution image synthesis, 2021.
- Scenegenie: Scene graph guided diffusion models for image synthesis, 2023.
- Generative adversarial networks, 2014.
- Denoising diffusion probabilistic models, 2020.
- J. Ho and T. Salimans. Classifier-free diffusion guidance, 2022.
- Lora: Low-rank adaptation of large language models, 2021.
- Structure-clip: Enhance multi-modal language representations with structure knowledge, 2023.
- Huggingface. diffusers. https://github.com/huggingface/diffusers/. (Accessed on 05/25/2023).
- Image-to-image translation with conditional adversarial networks, 2018.
- Image generation from scene graphs, 2018.
- D. P. Kingma and M. Welling. Auto-encoding variational bayes, 2022.
- T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks, 2017.
- Relation-aware graph attention network for visual question answering, 2019.
- Gligen: Open-set grounded text-to-image generation, 2023.
- Pastegan: A semi-parametric method to generate image from scene graph, 2019.
- Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, 2022.
- T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models, 2023.
- Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
- Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents, 2022.
- K. Panguluri and K. Kamarajugadda. Image generation using variational autoencoders. IJITEE (International Journal of Information Technology and Electrical Engineering), 9, 03 2020.
- W. Peebles and S. Xie. Scalable diffusion models with transformers, 2023.
- M. A. Pinsky and S. Karlin. 3 - markov chains: Introduction. In M. A. Pinsky and S. Karlin, editors, An Introduction to Stochastic Modeling (Fourth Edition), pages 79–163. Academic Press, Boston, fourth edition edition, 2011.
- Learning transferable visual models from natural language supervision, 2021.
- Hierarchical text-conditional image generation with clip latents, 2022.
- Generating diverse high-fidelity images with vq-vae-2, 2019.
- Generative adversarial text to image synthesis, 2016.
- High-resolution image synthesis with latent diffusion models, 2022.
- Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
- Denoising diffusion implicit models, 2022.
- Score-based generative modeling through stochastic differential equations, 2021.
- Transformer-based image generation from scene graphs, 2023.
- Neural discrete representation learning, 2018.
- Attention is all you need, 2017.
- Semantic image synthesis via diffusion models, 2022.
- A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24, jan 2021.
- Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning, 2023.
- Diffusion-based scene graph to image generation with masked contrastive pre-training, 2022.
- Unsupervised image generation with infinite generative adversarial networks, 2021.
- L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
- Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.