Papers
Topics
Authors
Recent
Search
2000 character limit reached

No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning

Published 4 Sep 2024 in cs.CV | (2409.03025v2)

Abstract: Image captioning systems are unable to generate fine-grained captions as they are trained on data that is either noisy (alt-text) or generic (human annotations). This is further exacerbated by maximum likelihood training that encourages generation of frequently occurring phrases. Previous works have tried to address this limitation by fine-tuning captioners with a self-retrieval (SR) reward. However, we find that SR fine-tuning has a tendency to reduce caption faithfulness and even hallucinate. In this work, we circumvent this bottleneck by improving the MLE initialization of the captioning system and designing a curriculum for the SR fine-tuning process. To this extent, we present (1) Visual Caption Boosting, a novel framework to instill fine-grainedness in generic image captioning datasets while remaining anchored in human annotations; and (2) BagCurri, a carefully designed training curriculum that more optimally leverages the contrastive nature of the self-retrieval reward. Jointly, they enable the captioner to describe fine-grained aspects in the image while preserving faithfulness to ground-truth captions. Our approach outperforms previous work by +8.9% on SR against 99 random distractors (RD100) (Dessi et al., 2023); and +7.6% on ImageCoDe. Additionally, existing metrics to evaluate captioning systems fail to reward diversity or evaluate a model's fine-grained understanding ability. Our third contribution addresses this by proposing self-retrieval from the lens of evaluation. We introduce TrueMatch, a benchmark comprising bags of highly similar images that uses SR to assess the captioner's ability to capture subtle visual distinctions. We evaluate and compare several state-of-the-art open-source MLLMs on TrueMatch, and find that our SR approach outperforms them all by a significant margin (e.g. +4.8% - 7.1% over Cambrian) while having 1-2 orders of magnitude fewer parameters.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. SPICE: Semantic Propositional Image Caption Evaluation. In European Conference on Computer Vision (ECCV), pp.  382–398. Springer, 2016.
  2. Inspecting the Geographical Representativeness of Images from Text-to-Image Models. In International Conference on Computer Vision (ICCV), pp.  5136–5147, 2023.
  3. Introducing our multimodal models, 2023. URL https://www.adept.ai/blog/fuyu-8b.
  4. Big vision. https://github.com/google-research/big_vision, 2022.
  5. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
  6. RedCaps: Web-curated image-text data created by the people, for the people. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  7. Cross-Domain Image Captioning With Discriminative Finetuning. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.  6935–6944, 2023.
  8. Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
  9. Describing Differences in Image Sets with Natural Language. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  10. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. In British Machine Vision Conference (BMVC), 2017.
  11. Multi-modal hallucination control by visual information grounding. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.  14303–14312, 2024.
  12. CapWAP: Image Captioning with a Purpose. In Empirical Methods in Natural Language Processing (EMNLP), pp.  8755–8768, 2020.
  13. Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  14. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Empirical Methods in Natural Language Processing (EMNLP), pp.  7514–7528, 2021.
  15. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
  16. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR), 2021.
  17. Learning to Describe Differences Between Pairs of Similar Images. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
  18. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
  19. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361, 2020.
  20. Andrej Karpathy and Li Fei-Fei. Deep Visual-Semantic Alignments for Generating Image Descriptions. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.  3128–3137, 2015.
  21. Guiding Image Captioning Models Toward More Specific Captions. In International Conference on Computer Vision (ICCV), pp.  15259–15269, 2023.
  22. Concadia: Towards Image-Based Text Generation with a Purpose. In Empirical Methods in Natural Language Processing (EMNLP), pp.  4667–4684, 2022.
  23. Image Retrieval from Contextual Descriptions. In Association of Computational Linguistics (ACL), pp.  3426–3440, 2022.
  24. From scarcity to efficiency: Improving clip training via visual-enriched captions. arXiv preprint arXiv:2310.07699, 2023.
  25. Does clip bind concepts? probing compositionality in large image models. In European Chapter of the Association for Computational Linguistics (EACL), 2022.
  26. What If We Recaption Billions of Web Images with LLaMA-3? arXiv preprint arXiv:2406.08478, 2024.
  27. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision (ECCV), pp.  740–755. Springer, 2014.
  28. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024.
  29. Improved baselines with visual instruction tuning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  30. Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data. In European Conference on Computer Vision (ECCV), pp.  338–354, 2018.
  31. Decoupled Weight Decay Regularization. In International Conference on Learning Representations (ICLR), 2018.
  32. Discriminability Objective for Training Descriptive Captions. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.  6964–6974, 2018.
  33. ClipCap: CLIP Prefix for Image Captioning. arXiv preprint arXiv:2111.09734, 2021.
  34. BLEU: a Method for Automatic Evaluation of Machine Translation. In Association of Computational Linguistics (ACL), pp.  311–318, 2002.
  35. Robust Change Captioning. In International Conference on Computer Vision (ICCV), pp.  4624–4633, 2019.
  36. Tuning Computer Vision Models with Task Rewards. In International Conference on Machine Learning (ICML), pp.  33229–33239, 2023.
  37. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. In International Conference on Computer Vision (ICCV), pp.  2641–2649, 2015.
  38. Language Models are Unsupervised Multitask Learners. 2019.
  39. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), pp.  8748–8763. PMLR, 2021.
  40. Self-Critical Sequence Training for Image Captioning. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.  7008–7024, 2017.
  41. When does bias transfer in transfer learning? arXiv preprint arXiv:2207.02842.
  42. Cider-r: Robust consensus-based image description evaluation. arXiv preprint arXiv:2109.13701, 2021.
  43. LAION-5B: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS), 35:25278–25294, 2022.
  44. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Association of Computational Linguistics (ACL), pp.  2556–2565, 2018.
  45. HowToCaption: Prompting LLMs to Transform Video Annotations at Scale. arXiv preprint arXiv:2310.04900, 2023.
  46. From Pixels to Prose: A Large Dataset of Dense Image Captions. arXiv preprint arXiv:2406.10328, 2024.
  47. A study on the distribution of social biases in self-supervised learning visual models. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.  10442–10451, 2022.
  48. From Show to Tell: A Survey on Deep Learning-Based Image Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 45(1):539–559, 2022.
  49. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016.
  50. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024a.
  51. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
  52. A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  53. CIDEr: Consensus-Based Image Description Evaluation. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.  4566–4575, 2015.
  54. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning (ICML), pp.  23318–23340. PMLR, 2022.
  55. Towards Unique and Informative Captioning of Images. In European Conference on Computer Vision (ECCV), pp.  629–644. Springer, 2020.
  56. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992.
  57. Computer Vision Datasets and Models Exhibit Cultural and Linguistic Diversity in Perception. arXiv preprint arXiv:2310.14356, 2023.
  58. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
  59. When and why vision-language models behave like bag-of-words models, and what to do about it. In International Conference on Learning Representations (ICLR), 2023.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 7 tweets with 80 likes about this paper.

Sign Up for Free