Re-Imagen: Retrieval-Augmented Text-to-Image Generator (2209.14491v3)

Abstract: Research on text-to-image generation has witnessed significant progress in generating diverse and photo-realistic images, driven by diffusion and auto-regressive models trained on large-scale image-text data. Though state-of-the-art models can generate high-quality images of common entities, they often have difficulty generating images of uncommon entities, such as Chortai (dog)' orPicarones (food)'. To tackle this issue, we present the Retrieval-Augmented Text-to-Image Generator (Re-Imagen), a generative model that uses retrieved information to produce high-fidelity and faithful images, even for rare or unseen entities. Given a text prompt, Re-Imagen accesses an external multi-modal knowledge base to retrieve relevant (image, text) pairs and uses them as references to generate the image. With this retrieval step, Re-Imagen is augmented with the knowledge of high-level semantics and low-level visual details of the mentioned entities, and thus improves its accuracy in generating the entities' visual appearances. We train Re-Imagen on a constructed dataset containing (image, text, retrieval) triples to teach the model to ground on both text prompt and retrieval. Furthermore, we develop a new sampling strategy to interleave the classifier-free guidance for text and retrieval conditions to balance the text and retrieval alignment. Re-Imagen achieves significant gain on FID score over COCO and WikiImage. To further evaluate the capabilities of the model, we introduce EntityDrawBench, a new benchmark that evaluates image generation for diverse entities, from frequent to rare, across multiple object categories including dogs, foods, landmarks, birds, and characters. Human evaluation on EntityDrawBench shows that Re-Imagen can significantly improve the fidelity of generated images, especially on less frequent entities.

Evaluating and Enhancing Text-to-Image Generation with Retrieval-Augmented Models

The paper "Re-Imagen: Retrieval-Augmented Text-to-Image Generator" presents a novel approach to the task of generating high-fidelity, photorealistic images from textual descriptions, with a particular emphasis on accurately representing rare or unseen entities. Building on the foundations of existing text-to-image generation models like Imagen, DALL-E 2, and Parti, the proposed model, Re-Imagen, introduces retrieval-augmented methods to enhance the model's capability to generate images for entities that are less frequently represented in the training data.

Key Contributions and Methodological Advancements

The Re-Imagen model is distinguished by its ability to incorporate external multimodal knowledge through a retrieval process. Given a text prompt, the model accesses an external knowledge base to retrieve relevant <image, text> pairs, which are then used as references for image generation. This retrieval-augmented mechanism provides both high-level semantic information and low-level visual details, which significantly improves the generation of images involving rare entities.

The architecture of Re-Imagen is based on a cascaded diffusion model, which includes multiple stages of resolution enhancement to produce high-resolution images. The design of the retrieval mechanism leverages a pre-constructed dataset paired with a novel interleaved guidance strategy during sampling. This strategy ensures that the model achieves a balanced alignment between the text input and the retrievals during image synthesis, ultimately leading to improvements in the faithfulness and photorealism of the generated images.

Empirical Results and Evaluations

The authors have conducted extensive evaluations of Re-Imagen across various datasets, including COCO, WikiImages, and a newly introduced benchmark, EntityDrawBench. On COCO and WikiImages, Re-Imagen demonstrated superior performance, measured by the Fréchet Inception Distance (FID) scores, indicating the model's capability in generating high-quality images. When evaluated on EntityDrawBench, which focuses on generating images of rare and diverse entities, Re-Imagen outperformed other leading models like Imagen and DALL-E 2, notably in terms of faithfulness to visual details and inputs involving long-tail entities.

The model's proficiency in handling infrequent entities showcases its potential in applications that require detailed and accurate visual representations from textual descriptions, which could include fields like digital art, content creation, and virtual environments.

Implications and Future Directions

The innovations presented in the Re-Imagen model point towards a significant direction in text-to-image generation research, emphasizing the enhancement of model capabilities through retrieval-augmented methods. This approach counteracts the limitations of previously existing models that struggle with rare entity generation by integrating additional knowledge resources externally, thus reducing the need for memorization within the model parameters.

Future developments could explore scaling the retrieval system to more diverse and comprehensive multimodal databases, possibly enhancing the model's generalization capabilities across even broader domains of visual content. Additionally, improvements in retrieval strategies and integration of dynamic knowledge graphs could further advance the generation capabilities of models like Re-Imagen, providing even richer contextual knowledge for better image synthesis.

Conclusion

Re-Imagen presents a significant advancement in retrieval-augmented text-to-image generation, addressing critical challenges in visualizing rare and unseen entities. The interleaved guidance strategy and the model architecture highlight promising avenues for further enhancing the capabilities of generative models by leveraging external multimodal data retrieval. This paper provides valuable insights for researchers aiming to enhance the fidelity and diversity of generated imagery, suggesting impactful pathways for future research in generative modeling and AI-driven creative applications.