Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Customizing Text-to-Image Diffusion with Object Viewpoint Control (2404.12333v2)

Published 18 Apr 2024 in cs.CV

Abstract: Model customization introduces new concepts to existing text-to-image models, enabling the generation of these new concepts/objects in novel contexts. However, such methods lack accurate camera view control with respect to the new object, and users must resort to prompt engineering (e.g., adding ``top-view'') to achieve coarse view control. In this work, we introduce a new task -- enabling explicit control of the object viewpoint in the customization of text-to-image diffusion models. This allows us to modify the custom object's properties and generate it in various background scenes via text prompts, all while incorporating the object viewpoint as an additional control. This new task presents significant challenges, as one must harmoniously merge a 3D representation from the multi-view images with the 2D pre-trained model. To bridge this gap, we propose to condition the diffusion process on the 3D object features rendered from the target viewpoint. During training, we fine-tune the 3D feature prediction modules to reconstruct the object's appearance and geometry, while reducing overfitting to the input multi-view images. Our method outperforms existing image editing and model customization baselines in preserving the custom object's identity while following the target object viewpoint and the text prompt.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (106)
  1. A neural space-time representation for text-to-image personalization. ACM Transactions on Graphics (TOG), 2023.
  2. Domain-agnostic tuning-encoder for fast personalization of text-to-image models. In SIGGRAPH Asia 2023 Conference Papers, 2023.
  3. Multidiffusion: Fusing diffusion paths for controlled image generation. In International Conference on Machine Learning (ICML), 2023.
  4. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
  5. Zip-nerf: Anti-aliased grid-based neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2023.
  6. Nerd: Neural reflectance decomposition from image collections. In IEEE International Conference on Computer Vision (ICCV), 2021.
  7. Samurai: Shape and material from unconstrained real-world arbitrary image collections. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
  8. Ledits++: Limitless image editing using text-to-image models. arXiv preprint arXiv:2311.16711, 2023.
  9. Instructpix2pix: Learning to follow image editing instructions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  10. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In IEEE International Conference on Computer Vision (ICCV), 2023.
  11. Genvs: Generative novel view synthesis with 3d-aware diffusion models. In IEEE International Conference on Computer Vision (ICCV), 2023.
  12. ChatGPT. Chatgpt. https://chat.openai.com/chat, 2022.
  13. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 2023.
  14. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
  15. 3-sweep: Extracting editable objects from a single photo. ACM Transactions on graphics (TOG), 2013.
  16. Subject-driven text-to-image generation via apprenticeship learning. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
  17. Learning continuous 3d words for text-to-image generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  18. Depth-supervised nerf: Fewer views and faster training for free. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  19. Diffusion models beat gans on image synthesis. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  20. Vica-nerf: View-consistency-aware 3d editing of neural radiance fields. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
  21. K-planes: Explicit radiance fields in space, time, and appearance. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  22. Make-a-scene: Scene-based text-to-image generation with human priors. In European Conference on Computer Vision (ECCV), 2022.
  23. An image is worth one word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations (ICLR), 2023a.
  24. Encoder-based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG), 2023b.
  25. Expressive text-to-image generation with rich text. In IEEE International Conference on Computer Vision (ICCV), 2023.
  26. Svdiff: Compact parameter space for diffusion fine-tuning. In IEEE International Conference on Computer Vision (ICCV), 2023.
  27. Instruct-nerf2nerf: Editing 3d scenes with instructions. In IEEE International Conference on Computer Vision (ICCV), 2023.
  28. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  29. Prompt-to-prompt image editing with cross attention control. In International Conference on Learning Representations (ICLR), 2023.
  30. Denoising diffusion probabilistic models. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
  31. Viewdiff: 3d-consistent image generation with text-to-image models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  32. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
  33. Navi: Category-agnostic image collections with high-quality 3d shape and pose annotations. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
  34. Scaling up gans for text-to-image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  35. Elucidating the design space of diffusion-based generative models. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
  36. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.02696, 2023.
  37. Rendering synthetic objects into legacy photographs. ACM Transactions on graphics (TOG), 2011.
  38. Imagic: Text-based real image editing with diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  39. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
  40. Lerf: Language embedded radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2023.
  41. 3d object manipulation in a single photograph using stock 3d models. ACM Transactions on graphics (TOG), 2014.
  42. Dense text-to-image generation with attention modulation. In IEEE International Conference on Computer Vision (ICCV), 2023.
  43. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
  44. Segment anything. In IEEE International Conference on Computer Vision (ICCV), 2023.
  45. Multi-concept customization of text-to-image diffusion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  46. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
  47. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  48. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations (ICLR), 2022.
  49. Zero-1-to-3: Zero-shot one image to 3d object. In IEEE International Conference on Computer Vision (ICCV), 2023.
  50. Syncdreamer: Generating multiview-consistent images from a single-view image. In International Conference on Learning Representations (ICLR), 2024.
  51. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
  52. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations (ICLR), 2022.
  53. Object 3dit: Language-guided 3d-aware image editing. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
  54. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021.
  55. Null-text inversion for editing real images using guided diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  56. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Conference on Artificial Intelligence (AAAI), 2024.
  57. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 2022.
  58. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  59. Dinov2: Learning robust visual features without supervision. In TMLR, 2023.
  60. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.
  61. Styleclip: Text-driven manipulation of stylegan imagery. In IEEE International Conference on Computer Vision (ICCV), 2021.
  62. Localizing object-level shape variations with text-to-image diffusion models. In IEEE International Conference on Computer Vision (ICCV), 2023.
  63. Scalable diffusion models with transformers. In IEEE International Conference on Computer Vision (ICCV), 2023.
  64. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  65. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR), 2023.
  66. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  67. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
  68. Dreambooth3d: Subject-driven text-to-3d generation. In IEEE International Conference on Computer Vision (ICCV), 2023.
  69. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  70. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501, 2020.
  71. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In IEEE International Conference on Computer Vision (ICCV), 2021.
  72. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  73. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 2015.
  74. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023a.
  75. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint arXiv:2307.06949, 2023b.
  76. Simo Ryu. Lora-stable diffusion. https://github.com/cloneofsimo/lora, 2023.
  77. Photorealistic text-to-image diffusion models with deep language understanding. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
  78. Zeronvs: Zero-shot 360-degree view synthesis from a single real image. arXiv preprint arXiv:2310.17994, 2023.
  79. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In International Conference on Machine Learning (ICML), 2023.
  80. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
  81. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411, 2023.
  82. Mvdream: Multi-view diffusion for 3d generation. In International Conference on Learning Representations (ICLR), 2024.
  83. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML), 2015.
  84. Total-recon: Deformable scene reconstruction for embodied view synthesis. In IEEE International Conference on Computer Vision (ICCV), 2023.
  85. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021.
  86. Learned initializations for optimizing coordinate-based neural representations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  87. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, 2023.
  88. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023.
  89. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH 2023 Conference Proceedings, 2023.
  90. Face0: Instantaneously conditioning a text-to-image model on a face. In SIGGRAPH Asia 2023 Conference Papers, 2023.
  91. Attention is all you need. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
  92. p+limit-from𝑝p+italic_p +: Extended textual conditioning in text-to-image generation. arXiv preprint arXiv:2303.09522, 2023.
  93. Evaluating data attribution for text-to-image models. In IEEE International Conference on Computer Vision (ICCV), 2023.
  94. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In IEEE International Conference on Computer Vision (ICCV), 2023.
  95. Reconfusion: 3d reconstruction with diffusion priors. arXiv preprint arXiv:2312.02981, 2023.
  96. 3d-aware scene manipulation via inverse graphics. In Conference on Neural Information Processing Systems (NeurIPS), 2018.
  97. Featurenerf: Learning generalizable nerfs by distilling foundation models. In IEEE International Conference on Computer Vision (ICCV), 2023.
  98. pixelnerf: Neural radiance fields from one or few images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  99. Scaling autoregressive models for content-rich text-to-image generation. In International Conference on Machine Learning (ICML), 2022.
  100. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In European Conference on Computer Vision (ECCV), 2022.
  101. Cameras as rays: Sparse-view pose estimation via ray diffusion. In International Conference on Learning Representations (ICLR), 2024.
  102. Adding conditional control to text-to-image diffusion models. In IEEE International Conference on Computer Vision (ICCV), 2023.
  103. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. In International Conference on Learning Representations (ICLR), 2021.
  104. Prospect: Prompt spectrum for attribute-aware personalization of diffusion models. ACM Transactions on Graphics (TOG), 2023.
  105. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision (ECCV), 2022.
  106. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets