Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LLMGA: Multimodal Large Language Model based Generation Assistant (2311.16500v4)

Published 27 Nov 2023 in cs.CV

Abstract: In this paper, we introduce a Multimodal LLM-based Generation Assistant (LLMGA), leveraging the vast reservoir of knowledge and proficiency in reasoning, comprehension, and response inherent in LLMs to assist users in image generation and editing. Diverging from existing approaches where Multimodal LLMs (MLLMs) generate fixed-size embeddings to control Stable Diffusion (SD), our LLMGA provides a detailed language generation prompt for precise control over SD. This not only augments LLM context understanding but also reduces noise in generation prompts, yields images with more intricate and precise content, and elevates the interpretability of the network. To this end, we curate a comprehensive dataset comprising prompt refinement, similar image generation, inpainting & outpainting, and instruction-based editing. Moreover, we propose a two-stage training scheme. In the first stage, we train the MLLM to grasp the properties of image generation and editing, enabling it to generate detailed prompts. In the second stage, we optimize SD to align with the MLLM's generation prompts. Additionally, we propose a reference-based restoration network to alleviate texture, brightness, and contrast disparities between generated and preserved regions during inpainting and outpainting. Extensive results show that LLMGA has promising generation and editing capabilities and can enable more flexible and expansive applications in an interactive manner.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPRW, 2017.
  2. Jointly training large autoregressive multimodal models. arXiv preprint arXiv:2309.15564, 2023.
  3. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022.
  4. Structured denoising diffusion models in discrete state-spaces. NeurIPS, 2021.
  5. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.
  6. All are worth words: A vit backbone for diffusion models. In CVPR, 2023a.
  7. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint arXiv:2303.06555, 2023b.
  8. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111.13606, 2021.
  9. Language models are few-shot learners. NeurIPS, 2020.
  10. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
  11. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  12. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
  13. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023.
  14. Diffusion models beat gans on image synthesis. NeurIPS, 2021.
  15. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499, 2023.
  16. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
  17. Frido: Feature pyramid diffusion for complex scene image synthesis. In AAAI, 2023.
  18. Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts. In CVPR, 2023.
  19. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041, 2023.
  20. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
  21. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017.
  22. Denoising diffusion probabilistic models. NeurIPS, 2020.
  23. Cascaded diffusion models for high fidelity image generation. JMLR, 2022.
  24. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045, 2023.
  25. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
  26. Diffusionclip: Text-guided diffusion models for robust image manipulation. In CVPR, 2022.
  27. Variational diffusion models. NeurIPS, 2021.
  28. Generating images with multimodal language models. arXiv preprint arXiv:2305.17216, 2023.
  29. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  30. Microsoft coco: Common objects in context. In ECCV, 2014.
  31. Visual instruction tuning. NeurIPS, 2023.
  32. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
  33. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
  34. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
  35. OpenAI. Gpt-4 technical report, 2023.
  36. Scalable diffusion models with transformers. In ICCV, 2023.
  37. Sdxl: Improving latent diffusion models for high-resolution image synthesis. NeurIPS, 2023.
  38. Learning transferable visual models from natural language supervision. In ICML, 2021.
  39. Zero-shot text-to-image generation. In ICML, 2021.
  40. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  41. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
  42. Palette: Image-to-image diffusion models. In ACM SIGGRAPH, 2022a.
  43. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022b.
  44. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022c.
  45. Improved techniques for training gans. NeurIPS, 2016.
  46. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
  47. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023.
  48. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
  49. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
  50. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023.
  51. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
  52. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPRW, 2017.
  53. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  54. Unitune: Text-driven image editing by fine tuning an image generation model on a single image. arXiv preprint arXiv:2210.09477, 2022.
  55. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In ICCVW, 2021.
  56. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023a.
  57. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023b.
  58. Diffi2i: Efficient diffusion model for image-to-image translation. arXiv preprint arXiv:2308.13767, 2023a.
  59. Diffir: Efficient diffusion model for image restoration. ICCV, 2023b.
  60. Knowledge distillation based degradation estimation for blind super-resolution. ICLR, 2023c.
  61. Raphael: Text-to-image generation via large mixture of diffusion paths. arXiv preprint arXiv:2305.18295, 2023.
  62. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023.
  63. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2023.
  64. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
  65. Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models. In ACM MM, 2023.
  66. Places: A 10 million image database for scene recognition. TPAMI, 2017.
  67. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Bin Xia (56 papers)
  2. Shiyin Wang (10 papers)
  3. Yingfan Tao (1 paper)
  4. Yitong Wang (47 papers)
  5. Jiaya Jia (162 papers)
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets