Aligning Large Multimodal Models with Factually Augmented RLHF (2309.14525v1)

Abstract: Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in "hallucination", generating textual outputs that are not grounded by the multimodal information in context. To address the multimodal misalignment issue, we adapt the Reinforcement Learning from Human Feedback (RLHF) from the text domain to the task of vision-language alignment, where human annotators are asked to compare two responses and pinpoint the more hallucinated one, and the vision-LLM is trained to maximize the simulated human rewards. We propose a new alignment algorithm called Factually Augmented RLHF that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options, which alleviates the reward hacking phenomenon in RLHF and further improves the performance. We also enhance the GPT-4-generated training data (for vision instruction tuning) with previously available human-written image-text pairs to improve the general capabilities of our model. To evaluate the proposed approach in real-world scenarios, we develop a new evaluation benchmark MMHAL-BENCH with a special focus on penalizing hallucinations. As the first LMM trained with RLHF, our approach achieves remarkable improvement on the LLaVA-Bench dataset with the 94% performance level of the text-only GPT-4 (while previous best methods can only achieve the 87% level), and an improvement by 60% on MMHAL-BENCH over other baselines. We opensource our code, model, data at https://llava-rlhf.github.io.

Aligning Large Multimodal Models with Factually Augmented RLHF: A Technical Perspective

The paper "Aligning Large Multimodal Models with Factually Augmented RLHF" addresses the critical issue of multimodal misalignment in Large Multimodal Models (LMMs) that can lead to hallucinatory outputs—this refers to generating text that is not grounded in the corresponding multimodal data. The research adapts Reinforcement Learning from Human Feedback (RLHF), traditionally used in text domains, to enhance vision-language alignment in LMMs.

Problem Context and Significance

Large Multimodal Models have shown promise in integrating and interpreting data across different modalities, such as text and images. However, a significant challenge lies in the alignment of these modalities, as misalignment could lead to incorrect or "hallucinated" responses. Unlike the abundance of quality data available for training text-only models, the multimodal context lacks such comprehensive datasets, causing a gap that conventional supervised learning struggles to bridge.

Methodological Innovations

1. Factually Augmented RLHF: The paper introduces an innovative alignment algorithm called Factually Augmented RLHF. This method enhances the reward models by integrating additional factual data—such as image captions and ground-truth options—thereby mitigating the risk of reward hacking, a situation where models could receive high scores for outputs that are not truly aligned with human evaluations.
2. Human Feedback Integration: By having human annotators compare and prioritize less hallucinated responses, the model is trained to optimize human preferences. This provides a feedback loop that incrementally tunes the model from purely synthetic data learning to real-world grounded data perception.
3. GPT-4 Enhanced Training Data: Another contribution is the augmentation of GPT-4-generated training data with previously available human-written image-text pairs. This hybrid training helps improve the baseline capabilities of the model, leveraging the vast knowledge base of GPT-4 with the nuanced understanding provided by human data.

Evaluation and Results

The introduction of a new evaluation benchmark, MMHal-Bench, is a testament to the project's focus on real-world applicability. This benchmark specifically assesses and penalizes hallucinations, ensuring that models are evaluated on their ability to remain grounded in reality. The RLHF-trained LMM demonstrated considerable improvements, achieving a 94% performance level relative to the text-only GPT-4 on the LLaVA-Bench dataset—surpassing prior methodologies which reached only 87%. Additionally, the approach marked a 60% improvement on MMHal-Bench compared to other baselines, demonstrating its effectiveness in hallucination-prone scenarios.

Implications and Future Directions

Practically, the development of factually consistent LMMs could revolutionize fields requiring precise multimodal interactions, such as autonomous driving, medical imaging, and human-computer interaction. Theoretically, this work enriches our understanding of neural alignment across diverse data types, facilitating more nuanced machine learning models capable of reliable autonomy.

For future advancements, scaling the RLHF paradigm to include more sophisticated architectures and exploring the nuances of more dynamic multimodal interactions with environment-adaptive systems could be pivotal. Investigations could also extend into more complex data domains beyond traditional images and text, integrating auditory or sensorial inputs to create comprehensive models of perception.

Overall, this work represents a rigorous advancement in the alignment of LMMs, providing a foundation for future research dedicated to bridging the multimodal ambits of AI with human-grounded reality. The opensourcing of the model and data further underscores a commitment to collaborative progress in AI research.