Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback (2412.15838v2)

Abstract: Reinforcement learning from human feedback (RLHF) has proven effective in enhancing the instruction-following capabilities of LLMs; however, it remains underexplored in the cross-modality domain. As the number of modalities increases, aligning all-modality models with human intentions -- such as instruction following -- becomes a pressing challenge. In this work, we make the first attempt to fine-tune all-modality models (i.e. input and output with any modality, also named any-to-any models) using human preference data across all modalities (including text, image, audio, and video), ensuring its behavior aligns with human intentions. This endeavor presents several challenges. First, there is no large-scale all-modality human preference data in existing open-source resources, as most datasets are limited to specific modalities, predominantly text and image. Secondly, the effectiveness of binary preferences in RLHF for post-training alignment in complex all-modality scenarios remains an unexplored area. Finally, there is a lack of a systematic framework to evaluate the capabilities of all-modality models, particularly regarding modality selection and synergy. To address these challenges, we propose the align-anything framework, which includes meticulously annotated 200k all-modality human preference data. Then, we introduce an alignment method that learns from unified language feedback, effectively capturing complex modality-specific human preferences and enhancing the model's instruction-following capabilities. Furthermore, to assess performance improvements in all-modality models after post-training alignment, we construct a challenging all-modality capability evaluation framework -- eval-anything. All data, models, and code frameworks have been open-sourced for the community. For more details, please refer to https://github.com/PKU-Alignment/align-anything.

Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

The presented paper addresses the challenging problem of aligning multimodal models with human preferences and intents. As the scope of multimodal information processing broadens, there emerges a pressing strategic demand for an efficient and unified blueprint capable of handling any-to-any modality alignment. Despite the well-established use of Reinforcement Learning from Human Feedback (RLHF) techniques to augment human-like instruction-following in LLMs, their implementation in a cross-modality context remains largely unexplored.

Core Contributions

The authors propose the "Align Anything" framework, which aims to train these all-modality models using human preference data across various modalities such as text, image, audio, and video. Their contributions can be summarized as follows:

1. All-Modality Human Preference Dataset: The introduction of align-anything-200k, the first large-scale dataset annotated with human preferences across multiple modalities, sets a new standard for aligning model behavior with human intentions.
2. Alignment Method via Language Feedback: A novel method that learns from unified language feedback to capture complex modality-specific preferences. The approach extends the RLHF framework's applicability by incorporating modality-agnostic insights through language feedback.
3. Evaluation Framework - Eval-Anything: An evaluative structure encompassing all-modality understanding and generation, essential to adequately address the intricacies of modality selection and synergistic integration.

By addressing the absence of substantial all-modality preference data, questions around the effectiveness of binary preferences in RLHF were tackled, thus providing insight into the necessity of systemic frameworks to evaluate multimodal capabilities.

Methodological Insights

The paper's methodology involves training on an unprecedented dataset - a meticulous conglomeration sourced from freely accessible multimodal resources with human feedback on various subtasks. Additionally, the innovative supervision by LLF, or Learning from Language Feedback, provides solutions to the previously assumed limitations of binary preferences.

1. LLF Pipelines: The alignment process consists of two phases—feedback modeling through Supervised Fine-Tuning (SFT), and self-improving using preferenced data synthesized from language feedback. Such a mechanism optimizes responses particularly by refining and tuning model outputs to enhance their compliance with human intentions.
2. Empirical Validation: For empirical support, experiments are conducted across five modalities and a range of models, showing significant improvements when LLF is applied alongside DPO and PPO. Notably, it achieves an average of 5.83 times performance enhancement over standard RLHF.
3. Comparative Performance: The integration of language feedback has demonstrated superior results in aligning multimodal models compared to traditional binary annotation techniques, particularly for the subtasks that require nuanced and composite preferences.

Prospective Impacts and Future Research

The implications of this research extend both practically and theoretically. Practically, the open-sourcing of the alignment framework, datasets, and trained models widens accessibility, inviting further research to develop and evaluate multimodal models more holistically. Theoretically, it paves the way for future interdisciplinary studies, potentially integrating more nuanced aspects of human communication beyond those explored in current models.

Exploring further into all-modality model training, the suggestions for future works include scaling datasets to millions and expanding evaluation metrics for even more intricate multimodal interactions. Thus, this paper sets a precedent for dialog concerning artificial intelligence models, which are increasingly intertwined with complex human-centric communications across various information modalities.

Overall, this paper advances the methodology for engaging AI systems in a more robust alignment with human values and interactions, offering a critical baseline upon which more comprehensive multimodal approaches can be structured.