Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Generating images of rare concepts using pre-trained diffusion models (2304.14530v3)

Published 27 Apr 2023 in cs.CV and cs.LG

Abstract: Text-to-image diffusion models can synthesize high-quality images, but they have various limitations. Here we highlight a common failure mode of these models, namely, generating uncommon concepts and structured concepts like hand palms. We show that their limitation is partly due to the long-tail nature of their training data: web-crawled data sets are strongly unbalanced, causing models to under-represent concepts from the tail of the distribution. We characterize the effect of unbalanced training data on text-to-image models and offer a remedy. We show that rare concepts can be correctly generated by carefully selecting suitable generation seeds in the noise space, using a small reference set of images, a technique that we call SeedSelect. SeedSelect does not require retraining or finetuning the diffusion model. We assess the faithfulness, quality and diversity of SeedSelect in creating rare objects and generating complex formations like hand images, and find it consistently achieves superior performance. We further show the advantage of SeedSelect in semantic data augmentation. Generating semantically appropriate images can successfully improve performance in few-shot recognition benchmarks, for classes from the head and from the tail of the training data of diffusion models

Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. SpaText: Spatio-Textual Representation for Controllable Image Generation. CVPR.
  2. Synthetic Data from Diffusion Models Improves ImageNet Classification. ArXiv.
  3. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324.
  4. Meta-learning with differentiable closed-form solvers. ICLR.
  5. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”.
  6. Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models. SIGGRAPH.
  7. Multi-Level Semantic Feature Augmentation for One-Shot Learning. IEEE Transactions on Image Processing, 28: 4594–4605.
  8. Fine-grained Visual Classification with High-temperature Refinement and Background Suppression. ArXiv.
  9. Parametric Contrastive Learning. ICCV.
  10. ImageNet: A large-scale hierarchical image database. In cvpr.
  11. Imagenet: A large-scale hierarchical image database. In CVPR.
  12. Diffusion models beat gans on image synthesis. NeurIPS.
  13. Efron, B. 1992. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution.
  14. Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. ICLR.
  15. An image is worth one word: Personalizing text-to-image generation using textual inversion. ICLR.
  16. Designing an Encoder for Fast Personalization of Text-to-Image Models. arXiv preprint arXiv:2302.12228.
  17. Is synthetic data from generative models ready for image recognition? ICLR.
  18. Denoising diffusion probabilistic models. NeurIPS.
  19. Classifier-free diffusion guidance. NeurIPS workshop on Deep Generative Models and Downstream Applications.
  20. Label Hallucination for Few-shot Classification. In AAAI.
  21. Decoupling Representation and Classifier for Long-Tailed Recognition. ICLR.
  22. Elucidating the design space of diffusion-based generative models. NeurIPS.
  23. Supervised contrastive learning. NeurIPS.
  24. Learning multiple layers of features from tiny images.
  25. MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognition. CVPR.
  26. LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models. arXiv preprint arXiv:2305.13655.
  27. Compositional visual generation with composable diffusion models. In ECCV.
  28. Design guidelines for prompt engineering text-to-image generative models. In ACM SIGCHI.
  29. Large-scale long-tailed recognition in an open world. In CVPR.
  30. A very preliminary analysis of dall-e 2. arXiv preprint arXiv:2204.13807.
  31. Reliable Fidelity and Diversity Metrics for Generative Models. Proceedings of Machine Learning Research. PMLR.
  32. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. NeurIPS.
  33. Training language models to follow instructions with human feedback. NeurIPS.
  34. Learning transferable visual models from natural language supervision. In ICML.
  35. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125.
  36. Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment. ArXiv.
  37. On gans and gmms. NeurIPS.
  38. High-resolution image synthesis with latent diffusion models. In CVPR.
  39. U-net: Convolutional networks for biomedical image segmentation. In MICCAI.
  40. FeLMi : Few shot Learning with hard Mixup. In NeurIPS.
  41. DiffAlign : Few-shot learning using diffusion based synthesis and alignment. ArXiv, abs/2212.05404.
  42. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR.
  43. Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles. ICML.
  44. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS.
  45. From Generalized Zero-Shot Learning to Long-Tail With Class Descriptors. In WACV.
  46. Distributional Robustness Loss for Long-tail Learning. ICCV.
  47. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS.
  48. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. Data Centric AI NeurIPS Workshop.
  49. Baby steps towards few-shot learning with multiple semantics. Pattern Recognition Letters.
  50. Key-Locked Rank One Editing for Text-to-Image Personalization. SIGGRAPH.
  51. Thorndike, R. L. 1953. Who belongs in the family? Psychometrika.
  52. VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition. In European Conference on Computer Vision.
  53. MaxViT: Multi-Axis Vision Transformer. ECCV.
  54. The inaturalist species classification and detection dataset. In CVPR.
  55. Matching Networks for One Shot Learning. In NeurIPS.
  56. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology.
  57. Long-tailed Recognition by Routing Diverse Distribution-Aware Experts. ICLR.
  58. DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models. arXiv preprint arXiv:2210.14896.
  59. Generating Representative Samples for Few-Shot Classification. In CVPR.
  60. SEGA: semantic guided attention on visual prototype for few-shot learning. In WACV.
  61. Few-Shot Learning via Embedding Adaptation With Set-to-Set Functions. In CVPR.
  62. Apache spark: a unified engine for big data processing. Communications of the ACM.
  63. DeepEMD: Few-Shot Image Classification With Differentiable Earth Mover’s Distance and Structured Classifiers. In CVPR.
  64. Adding Conditional Control to Text-to-Image Diffusion Models. arXiv preprint arXiv:2302.05543.
  65. Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling. ECCV.
  66. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR.
  67. Deep long-tailed learning: A survey. arXiv preprint arXiv:2110.04596.
  68. Learning to Prompt for Vision-Language Models. International Journal of Computer Vision.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Dvir Samuel (12 papers)
  2. Rami Ben-Ari (21 papers)
  3. Simon Raviv (2 papers)
  4. Nir Darshan (14 papers)
  5. Gal Chechik (110 papers)
Citations (27)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube