Fine-grained Image Captioning with CLIP Reward (2205.13115v2)

Abstract: Modern image captioning models are usually trained with text similarity objectives. However, since reference captions in public datasets often describe the most salient common objects, models trained with text similarity objectives tend to ignore specific and detailed aspects of an image that distinguish it from others. Toward more descriptive and distinctive caption generation, we propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function. We also propose a simple finetuning strategy of the CLIP text encoder to improve grammar that does not require extra text annotation. This completely eliminates the need for reference captions during the reward computation. To comprehensively evaluate descriptive captions, we introduce FineCapEval, a new dataset for caption evaluation with fine-grained criteria: overall, background, object, relations. In our experiments on text-to-image retrieval and FineCapEval, the proposed CLIP-guided model generates more distinctive captions than the CIDEr-optimized model. We also show that our unsupervised grammar finetuning of the CLIP text encoder alleviates the degeneration problem of the naive CLIP reward. Lastly, we show human analysis where the annotators strongly prefer the CLIP reward to the CIDEr and MLE objectives according to various criteria. Code and Data: https://github.com/j-min/CLIP-Caption-Reward

Fine-grained Image Captioning with CLIP Reward: A Critical Evaluation

In the field of image captioning, most contemporary models are trained using text similarity objectives, which often results in the generation of captions that overlook specific details of an image. These models typically produce captions describing the most salient objects, neglecting finer details that contribute to the distinctiveness of an image. Addressing this limitation, the paper "Fine-grained Image Captioning with CLIP Reward" introduces an innovative approach employing the CLIP model to guide the generation of more descriptive and distinctive captions. The paper also proposes a new fine-grained caption evaluation dataset, FineCapEval, to assess various aspects of descriptive captions, such as background information, objects, and their relations.

Methodology and Framework

The core contribution of this paper is the utilization of CLIP, a powerful multimodal encoder, to enhance image captioning models. By leveraging CLIP's ability to compute multimodal similarity between images and text, the authors present a strategy whereby CLIP's similarity score serves as a reward function within a reinforcement learning framework. This approach recalibrates the standard reward mechanisms and reduces reliance on reference captions, which often fail to contain detailed and distinctive descriptive elements.

Additionally, to address the potential grammatical deterioration in captions generated through purely multimodal-based rewards, the authors introduce a method to fine-tune the CLIP text encoder. This fine-tuning employs synthetic negative caption augmentation to refine grammar without extra annotations, and optimizes the CLIP encoder to balance both grammaticality and semantic relevance.

Experimental Set-Up and Results

The experiments are conducted on the widely-used MS COCO dataset, with metrics encompassing n-gram based, embedding-based, text-to-image retrieval, and evaluations using the new FineCapEval dataset. The results demonstrate that CLIP-guided models proficiently generate more distinctive captions compared to models optimized solely with traditional CIDEr-based rewards. Notably, the CLIP-guided model surpasses even the reference captions in text-to-image retrieval tasks, indicating a high degree of specificity and distinctiveness in the generated captions.

Furthermore, the inclusion of grammar finetuning substantially mitigates grammatical defects such as repetition, promoting a balance between detail orientation and linguistic coherence. Human evaluations reinforce these findings, indicating a preference for CLIP-based captions across several qualitative criteria.

Theoretical and Practical Implications

The use of CLIP in image captioning underscores a significant shift from static text reference objectives towards dynamic, contextually-enriched evaluations. The methodology underscores CLIP's potential to transcend traditional benchmarks by fusing multimodal insights, thereby enhancing the semantic richness and distinctiveness of captions. Practically, this can improve applications requiring precise image descriptions, such as image search engines and assistive technologies for the visually impaired.

The introduction of FineCapEval fills a critical gap in existing evaluation benchmarks, systematically considering various descriptive aspects that previous datasets overlook. Researchers can leverage this dataset to build and evaluate models that capture a comprehensive range of details within images, guiding more nuanced advances in image captioning systems.

Future Directions

This paper opens several avenues for future research. One potential direction involves extending the versatility of CLIP-guided approaches across different languages, which would necessitate adaptations to CLIP's training to accommodate non-English datasets. Additionally, exploring different multimodal architectures and aligning them with personalized writing styles could further distinguish captions based on specific user needs or contexts.

The paper presents a well-structured approach to tackling the limitations in existing image captioning models while laying the groundwork for further enhancements in multimodal machine learning tasks. As these techniques evolve, they hold the promise of crafting more descriptive, coherent, and contextually aware narratives from images.