ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling (2402.06118v3)

Abstract: By combining natural language understanding, generation capabilities, and breadth of knowledge of LLMs with image perception, recent large vision LLMs (LVLMs) have shown unprecedented visual reasoning capabilities. However, the generated text often suffers from inaccurate grounding in the visual input, resulting in errors such as hallucination of nonexistent scene elements, missing significant parts of the scene, and inferring incorrect attributes of and relationships between objects. To address these issues, we introduce a novel framework, ViGoR (Visual Grounding Through Fine-Grained Reward Modeling) that utilizes fine-grained reward modeling to significantly enhance the visual grounding of LVLMs over pre-trained baselines. This improvement is efficiently achieved using much cheaper human evaluations instead of full supervisions, as well as automated methods. We show the effectiveness of our approach through a variety of evaluation methods and benchmarks. Additionally, we released our human annotation (https://github.com/amazon-science/vigor) comprising 15,440 images and generated text pairs with fine-grained evaluations to contribute to related research in the community.

Enhancing Visual Grounding in Large Vision LLMs with ViGoR: A Novel Fine-Grained Reward Modeling Approach

Introduction to Visual Grounding Challenges

Recent breakthroughs in Large Vision LLMs (LVLMs) have intertwined language understanding with image perception, allowing models to engage in real-world reasoning tasks. Despite these advancements, LVLMs often struggle with accurately grounding their generated texts in visual inputs. This misalignment can lead to inaccuracies such as hallucinating nonexistent elements, overlooking significant parts of the scene, or misinterpreting object attributes and relations. Addressing these grounding issues is crucial for improving the reliability and effectiveness of LVLMs in practical applications.

ViGoR: A Solution to Visual Grounding

Enter ViGoR (Visual Grounding Through Fine-Grained Reward Modeling), a framework designed to enhance the visual grounding capabilities of LVLMs by employing fine-grained reward modeling. This approach leverages detailed human evaluations and automated methods to refine LVLM outputs, making them more accurate and contextually relevant. Notably, ViGoR demonstrates its effectiveness by presenting significant improvements over existing models on various benchmarks, with substantial advancements in generating accurate and detailed descriptions while preserving the logical reasoning and creativity inherent to LLMs.

The Mechanics of ViGoR

The ViGoR framework operates by initially sampling text outputs from a pre-trained LVLM given an input image. These outputs are then assessed by human annotators at a sentence level for inaccuracies and creativity. Using feedback as the ground truth, a reward model is trained to capture fine-grained human evaluations. This reward model subsequently guides the fine-tuning of the LVLM, considerably improving its visual grounding with a limited dataset.

Additionally, ViGoR incorporates an automated method to construct the reward model without further human intervention. This approach employs state-of-the-art object detection models to verify the presence of described entities in the images, contributing to the model's grounding capabilities.

Practical and Theoretical Implications

Practically, ViGoR represents a cost-effective solution to enhancing the visual grounding of LVLMs without the need for extensive annotated datasets. It significantly reduces the incidence of hallucinations and misconceptions in model outputs, leading to more accurate and meaningful machine-generated interpretations of visual data. Theoretically, ViGoR's approach provides insights into the efficiency of fine-grained feedback and the integration of complementary reward signals from human evaluations and automated methods. This blend of feedback sources showcases how different types of information can cohesively refine model performance.

Looking Ahead

This paper's findings underscore the importance of addressing the visual grounding issues in LVLMs and propose a promising direction for future research. The development of ViGoR paves the way for more sophisticated models capable of even closer integration of visual perception and language comprehension. Future explorations may include extending the ViGoR framework with reinforcement learning from human feedback (RLHF) and integrating explicit visual predictions for improved alignment and grounding accuracy.

Conclusion

The ViGoR framework marks an important step forward in the quest to improve the visual grounding of LVLMs. Through its novel use of fine-grained reward modeling, combined with human evaluations and automated methods, ViGoR significantly enhances the accuracy and contextual relevance of LVLM-generated texts. By addressing the challenges of hallucination and inaccurate grounding, ViGoR contributes valuable insights and tools to the field of AI, with broad implications for the future development of more perceptively aligned machine learning models.