A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation (2310.16656v1)

Abstract: Text-to-image diffusion models achieved a remarkable leap in capabilities over the last few years, enabling high-quality and diverse synthesis of images from a textual prompt. However, even the most advanced models often struggle to precisely follow all of the directions in their prompts. The vast majority of these models are trained on datasets consisting of (image, caption) pairs where the images often come from the web, and the captions are their HTML alternate text. A notable example is the LAION dataset, used by Stable Diffusion and other models. In this work we observe that these captions are often of low quality, and argue that this significantly affects the model's capability to understand nuanced semantics in the textual prompts. We show that by relabeling the corpus with a specialized automatic captioning model and training a text-to-image model on the recaptioned dataset, the model benefits substantially across the board. First, in overall image quality: e.g. FID 14.84 vs. the baseline of 17.87, and 64.3% improvement in faithful image generation according to human evaluation. Second, in semantic alignment, e.g. semantic object accuracy 84.34 vs. 78.90, counting alignment errors 1.32 vs. 1.44 and positional alignment 62.42 vs. 57.60. We analyze various ways to relabel the corpus and provide evidence that this technique, which we call RECAP, both reduces the train-inference discrepancy and provides the model with more information per example, increasing sample efficiency and allowing the model to better understand the relations between captions and images.

Improving Text-to-Image Diffusion Models through Principled Recaptioning

The paper "A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation" investigates enhancing the performance of text-to-image (T2I) diffusion models by addressing the quality of training data captions. Traditional T2I models are built on datasets of (image, caption) pairs, where the captions are often substandard, impacting the models' ability to accurately interpret and render the semantics of input prompts. By focusing on the LAION dataset utilized in models like Stable Diffusion, the authors propose using a refined captioning approach termed RECAP, which has demonstrated significant improvements in both image quality and semantic alignment.

Key Contributions and Methodology

The central contribution of this work is the development and application of the RECAP method, which involves recaptioning the training dataset images using an automatic image-to-text (I2T) model. By fine-tuning PaLI, a sophisticated captioning system, the authors generated detailed and contextually rich captions for the dataset. The resultant text-to-image model, trained on this recaptioned data, displayed marked improvements over the baseline:

• FID: The Fréchet Inception Distance (FID) score improved from 17.87 to 14.84, indicating better overall image quality.
• Semantic Object Accuracy (SOA): Increased from 78.90 to 84.34, illustrating superior semantic fidelity.
• Counting Alignment Errors and Positional Alignment: Exhibited reductions in alignment errors, underscoring the enhanced model competency in handling multi-entity prompts.

The paper delineates a comprehensive methodology integrating three pivotal steps: fine-tuning a caption model on a human-curated dataset, employing this model to recaption the main dataset, and subsequently using this revised dataset for T2I model training. These recaptioned datasets consisted of both short and long variants to evaluate the diversity of expression.

Results and Implications

The quantitative assessments assert the merit of RECAP across a suite of evaluative metrics. Additionally, human evaluation further corroborated the model's enhanced capacity to adhere to prompts, showing significant improvements in prompt adherence and output quality. These findings are particularly relevant for complex prompts involving multiple entities, modifiers, and spatial arrangements.

Theoretically, this demonstrates the potential efficacy of improving caption data to bridge the train-inference skew, enhancing sample efficiency, and subsequently, the resultant image quality. Practically, the research suggests a feasible technique for boosting existing generative models without necessitating resource-intensive hardware or extended model architectures.

Challenges and Future Directions

Despite the promising outcomes, this paper leaves open several research trajectories. First, the method's adaptability to larger datasets and more extensive models remains to be explored. Second, the potential of RECAP-like strategies in other generative tasks, such as video or multimodal content synthesis, could represent an interesting inquiry. Lastly, understanding the implications of such techniques for bias in AI-generated content warrants attention, given the automatic nature of captioning and potential dataset biases.

In conclusion, through principled recaptioning, this work offers actionable insights for the advancement of text-to-image diffusion models. The approach provides a clear path to methodically enhancing training datasets, thereby empowering models to more faithfully replicate complex semantic instructions from text prompts. The implications extend beyond improved visual fidelity, suggesting foundational enhancements to generative AI's interpretative capacities.