Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
11 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves? (2404.06510v2)

Published 9 Apr 2024 in cs.CV

Abstract: Enhancing semantic grounding abilities in Vision-LLMs (VLMs) often involves collecting domain-specific training data, refining the network architectures, or modifying the training recipes. In this work, we venture into an orthogonal direction and explore whether VLMs can improve their semantic grounding by "receiving" feedback, without requiring in-domain data, fine-tuning, or modifications to the network architectures. We systematically analyze this hypothesis using a feedback mechanism composed of a binary signal. We find that if prompted appropriately, VLMs can utilize feedback both in a single step and iteratively, showcasing the potential of feedback as an alternative technique to improve grounding in internet-scale VLMs. Furthermore, VLMs, like LLMs, struggle to self-correct errors out-of-the-box. However, we find that this issue can be mitigated via a binary verification mechanism. Finally, we explore the potential and limitations of amalgamating these findings and applying them iteratively to automatically enhance VLMs' grounding performance, showing grounding accuracy consistently improves using automated feedback across all models in all settings investigated. Overall, our iterative framework improves semantic grounding in VLMs by more than 15 accuracy points under noise-free feedback and up to 5 accuracy points under a simple automated binary verification mechanism. The project website is hosted at https://andrewliao11.github.io/vlms_feedback

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015.
  2. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  3. Making large multimodal models understand arbitrary visual prompts. arXiv preprint arXiv:2312.00784, 2023.
  4. Making large multimodal models understand arbitrary visual prompts. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
  5. Shikra: Unleashing multimodal llm’s referential dialogue magic, 2023.
  6. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=KuPixIqPiq.
  7. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958–979, 2024.
  8. Learning to evaluate image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  9. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  10. Improving factuality and reasoning in language models through multiagent debate, 2024. URL https://openreview.net/forum?id=QAwaaLJNCk.
  11. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Sx038qxjek.
  12. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  13. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=lL3lnMbR4WU.
  14. Regiongpt: Towards region understanding vision language model. arXiv preprint arXiv:2403.02330, 2024.
  15. CLIPScore: A reference-free evaluation metric for image captioning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.595. URL https://aclanthology.org/2021.emnlp-main.595.
  16. Large language models cannot self-correct reasoning yet, 2023a.
  17. Language is not all you need: Aligning perception with language models, 2023b.
  18. Huggingface. sentence-transformers/all-mpnet-base-v2. https://huggingface.co/sentence-transformers/all-mpnet-base-v2.
  19. Language models (mostly) know what they know. ArXiv, abs/2207.05221, 2022. URL https://api.semanticscholar.org/CorpusID:250451161.
  20. Re-evaluating automatic metrics for image captioning. In Mirella Lapata, Phil Blunsom, and Alexander Koller, editors, Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 199–209, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-1019.
  21. Language models can solve computer tasks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=M6OmjAZ4CX.
  22. Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  23. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=e2TBb5y0yFf.
  24. Collavo: Crayon large language and vision model, 2024.
  25. Otterhd: A high-resolution multi-modality model, 2023.
  26. Monkey: Image resolution and text label are important things for large multi-modal models, 2024.
  27. Vila: On pre-training for visual language models, 2023.
  28. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014. URL http://arxiv.org/abs/1405.0312.
  29. Improved baselines with visual instruction tuning, 2023a.
  30. Visual instruction tuning. In NeurIPS, 2023b.
  31. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava-vl.github.io/blog/2024-01-30-llava-next/.
  32. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation, 2023.
  33. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36, 2024.
  34. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=S37hOerQLB.
  35. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
  36. Mm1: Methods, analysis & insights from multimodal llm pre-training, 2024.
  37. Pivot: Iterative visual prompting elicits actionable knowledge for vlms, 2024a.
  38. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872, 2024b.
  39. MAF: Multi-aspect feedback for improving reasoning in large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=bNeDLx5O6w.
  40. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813, 2023.
  41. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  42. What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11987–11997, October 2023.
  43. C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 100(3/4):441–471, 1987. ISSN 00029556. URL http://www.jstor.org/stable/1422689.
  44. Gemini: A family of highly capable multimodal models, 2023.
  45. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
  46. Llama 2: Open foundation and fine-tuned chat models, 2023.
  47. Llms cannot find reasoning errors, but can correct them!, 2024.
  48. Cogvlm: Visual expert for pretrained language models, 2024.
  49. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw.
  50. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=_VjQlMeSB_J.
  51. Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation, 2024.
  52. Perils of self-feedback: Self-bias amplifies in large language models, 2024.
  53. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023a.
  54. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023b.
  55. The dawn of lmms: Preliminary explorations with gpt-4v(ision), 2023c.
  56. Tree of Thoughts: Deliberate problem solving with large language models, 2023.
  57. Ferret: Refer and ground anything anywhere at any granularity, 2023.
  58. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014.
  59. Modeling context in referring expressions, 2016.
  60. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=KRLUvxh8uaX.
  61. Can mllms perform text-to-image in-context learning?, 2024.
  62. Using large language models for hyperparameter optimization. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023a.
  63. GPT4roi: Instruction tuning large language model on region-of-interest, 2024. URL https://openreview.net/forum?id=DzxaRFVsgC.
  64. Gpt-4v (ision) as a generalist evaluator for vision-language tasks. arXiv preprint arXiv:2311.01361, 2023b.
  65. Gpt-4v(ision) as a generalist evaluator for vision-language tasks, 2023c.
  66. MMICL: Empowering vision-language model with multi-modal in-context learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=5KojubHBr8.
  67. Step-back prompting enables reasoning via abstraction in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3bq3jsvcQ1.
  68. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  69. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WZH7099tgfM.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now