TLDR: Token-Level Detective Reward Model for Large Vision Language Models (2410.04734v2)

Abstract: Although reward models have been successful in improving multimodal LLMs, the reward models themselves remain brutal and contain minimal information. Notably, existing reward models only mimic human annotations by assigning only one binary feedback to any text, no matter how long the text is. In the realm of multimodal LLMs, where models are required to process both images and texts, a naive reward model may learn implicit biases toward texts and become less grounded in images. In this paper, we propose a $\textbf{T}$oken-$\textbf{L}$evel $\textbf{D}$etective $\textbf{R}$eward Model ($\textbf{TLDR}$) to provide fine-grained annotations to each text token. We first introduce a perturbation-based method to generate synthetic hard negatives and their token-level labels to train TLDR models. Then we show the rich usefulness of TLDR models both in assisting off-the-shelf models to self-correct their generations, and in serving as a hallucination evaluation tool. We show that TLDR automatically trains a token-level likelihood optimization, and can improve the base model's performance significantly. Finally, we show that TLDR models can significantly speed up human annotation by 3 times to acquire a broader range of high-quality vision language data.

• The paper proposes TLDR to deliver token-level feedback that enhances interpretability and enables self-correction in vision-language outputs.
• It employs a PaliGemma-3B backbone with LoRA fine-tuning on synthetic datasets to simulate varied scenarios in object and attribute recognition.
• Empirical results show marked improvements in mean Average Precision and response accuracy, effectively mitigating hallucinations in generated content.

Token-Level Detective Reward Model for Large Vision LLMs

The paper "TLDR: Token-Level Detective Reward Model for Large Vision LLMs" introduces a novel approach to improving the interpretability and utility of reward models in vision-language tasks. The authors propose the Token-Level Detective Reward (TLDR) model, designed to augment large vision-LLMs by providing token-level evaluations and facilitating self-correction in generated outputs.

Vision-LLMs, while increasingly powerful, struggle with generating contextually accurate content, often producing ungrounded or hallucinated text. Traditional reward models, typically binary, offer limited interpretability and granularity, which can constrain the optimization and refinement of output quality. The TLDR model tackles these limitations by offering token-level feedback, thereby enhancing transparency and alignment with human expectations.

Model Architecture and Training

The TLDR model leverages a PaliGemma-3B backbone without the LLM head. A newly trained reward model head evaluates each token, providing detailed feedback on its accuracy or hallucination status. Notably, the TLDR model is trained on synthetic datasets created using perturbation techniques, which allow for the simulation of both positive and negative instances across multiple taxonomies such as spatial relationships, object identification, and attribute binding.

An efficient training setup involves LoRA fine-tuning to adjust the linear projection and transformer decoder modules, facilitating better alignment of visual features within the language embedding space. This strategy seeks to maintain the fidelity of the reward signals while optimizing the model's fine-tuning processes.

Performance and Evaluation

Empirical results highlight the TLDR model's ability to surpass traditional models in terms of response-level accuracy, particularly excelling in nuanced tasks that require careful distinction between visually grounded and ungrounded content. Tests on synthetic data demonstrate strong performance metrics, with the TLDR model achieving notable improvements in mean Average Precision, particularly for negative labels, over a naive binary model.

An ablation paper underscores the importance of joint training across the model's components, revealing that separate tuning of individual modules yields suboptimal outcomes. This comprehensive fine-tuning approach is vital for enhancing the model's capability to detect and rectify hallucinations effectively.

Applications and Implications

The paper explores the broader application of the TLDR model beyond mere hallucination detection. It acts as a guide for both model self-correction and augmented human annotation processes. By providing granular feedback, the TLDR model facilitates efficient human edits of AI-generated captions, significantly accelerating annotation speeds.

In a speculative sense, the TLDR model suggests pathways for improving vision-LLMs through more precise reward streamlining. It remains a promising tool for integrating strong reward signals into models for direct or proximal policy optimization.

Future Directions

While the TLDR model offers substantial advancements, the path forward involves exploring scaling strategies to larger models and refining the underlying methodologies to ensure continued enhancement of vision-language interaction. Further research could focus on the development of interfaces to streamline human-computer interaction, thus reducing the overhead associated with data annotation and increasing dataset quality and volume efficiently.

In conclusion, this paper lays the groundwork for a more nuanced approach to reward modeling within vision-language systems, positioning the TLDR model as a strategic tool to refine models' interpretability, alignment, and utility across multimodal tasks.