What You See is What You Read? Improving Text-Image Alignment Evaluation (2305.10400v4)

Abstract: Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-LLMs, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation.

• The paper introduces SeeTRUE, a benchmark with 31,855 labeled examples from real and synthetic text-image pairs, addressing alignment challenges.
• It presents two novel methodologies, VQ and VNLI, that enhance evaluation by leveraging question-answering and end-to-end classification approaches.
• Experimental results show superior performance on complex image compositions, offering practical insights for refining multimodal models.

Enhancing Text-Image Alignment Evaluation with SeeTRUE and Novel Evaluation Methods

Introduction to SeeTRUE and Its Motivation

The evaluation of semantic alignment between text and corresponding images is a significant challenge in the field of vision-LLMs. This challenge persists across both text-to-image and image-to-text generation tasks, despite the recent advancements in multimodal LLMs. Recognizing this, the introduction of SeeTRUE, a comprehensive evaluation suite, marks a significant step towards addressing the nuances of text-image alignment. SeeTRUE expands the scope of evaluation by including a diverse range of real and synthetic text-image pairs, accompanied by human judgments, to assess the accuracy of this alignment. This initiative not only fills a gap in existing benchmarks, which are predominantly focused on natural images and lack challenging negative examples but also sets a new standard in the evaluation of text-image alignment models across various contexts.

Dataset Construction and Evaluation

The SeeTRUE benchmark stands out through its methodical curation that spans across four categories combining real and synthetic text and images, encapsulating a total of 31,855 labeled examples from diverse sources. This broad spectrum of data aims to enhance the models' generalization abilities while tackling the complexity of semantic alignment in a structured manner. What further distinguishes SeeTRUE is its approach to generating contradicting captions for existing aligned pairs, leveraging LLMs for this purpose. This novel strategy not only enriches the dataset with challenging examples but also introduces a refined method for capturing the essence of misalignments.

Methodological Advancements

The exploration of automatic text-image alignment evaluation methodologies presented in the paper introduces two key approaches:

1. VQ: This method leverages question generation and visual question answering models to confirm if the visual content answers the generated questions accurately, reflecting the text's semantics.
2. VNLI: An end-to-end classification approach that predicts semantic alignment by finetuning multimodal pretrained models.

Both methods have demonstrated superior performance over a range of datasets when compared to existing approaches. Particularly noteworthy is the VQ method's adeptness at tackling compositions and unnatural images, showcasing significant improvements and setting new benchmarks in previously challenging datasets.

Experimental Insights and Implications

The comprehensive evaluation against strong baselines underscores the efficacy of the proposed methodologies, especially VQ's robustness and flexibility in handling complex scenarios. Furthermore, the ability of these methods to re-rank generated image candidates for a given prompt not only enhances the practicality of text-to-image generation models but also contributes significantly to the fine-tuning of such models for improved output alignment.

Concluding Perspectives

The insights garnered from the construction of SeeTRUE and the evaluation of the VQ and VNLI methodologies provide a solid foundation for future research endeavors within the field of vision-LLMs. These contributions not only pave the way for more accurate and reliable evaluation metrics but also open avenues for refining generative models to produce semantically aligned text-image pairs. As we look forward, the integration of these methodologies in guiding the training processes of multimodal models emerges as a promising direction, potentially leading to significant advancements in the generation of coherent and contextually relevant visual-textual content.