Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey (2409.11564v2)

Abstract: Preference tuning is a crucial process for aligning deep generative models with human preferences. This survey offers a thorough overview of recent advancements in preference tuning and the integration of human feedback. The paper is organized into three main sections: 1) introduction and preliminaries: an introduction to reinforcement learning frameworks, preference tuning tasks, models, and datasets across various modalities: language, speech, and vision, as well as different policy approaches, 2) in-depth exploration of each preference tuning approach: a detailed analysis of the methods used in preference tuning, and 3) applications, discussion, and future directions: an exploration of the applications of preference tuning in downstream tasks, including evaluation methods for different modalities, and an outlook on future research directions. Our objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners. We hope to encourage further engagement and innovation in this area.

• The paper presents an in-depth taxonomy of reinforcement learning-based preference tuning methods across language, speech, and vision tasks.
• It details a structured training process comprising supervised fine-tuning, reward modeling, and reinforcement learning with human feedback.
• The study highlights robust evaluation metrics and future research directions for enhancing model safety, multilingual, and multimodal alignment.

Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: An Analytical Survey

Abstract Overview

The paper under review, titled "Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey," dives into the intricate domain of aligning deep generative models with human preferences through preference tuning. This extensive survey integrates research and methods across multiple modalities—language, speech, and vision—providing in-depth insights into the reinforcement learning frameworks, policy approaches, models, and datasets used in preference tuning. The structured approach comprises an introduction to reinforcement learning frameworks, a granular analysis of preference tuning methods, and a detailed discussion on the applications and future research directions, highlighting the diverse methodologies and their implications.

Reinforcement Learning and Preference Tuning

Reinforcement Learning Frameworks

The exploration begins with an overview of reinforcement learning (RL) frameworks, emphasizing their critical role in preference tuning. The paper details the RL problem's formulation for generative models, defining key components such as the policy model ($\pi_\theta$), reward model ($r_\theta$), action space, and environment. These components form the backbone of the RL framework, guiding the training process to align generative models with human preferences.

Preference Data and Taxonomy

The authors introduce and taxonomize the components of preference tuning methods, categorizing them based on sampling methods (offline, online), modality (text, speech, vision), and reward granularity (sample-level, token-level). This systematic classification enhances the understanding of preference tuning’s diverse applications and modalities.

Training Phases in Preference Tuning

Supervised Fine-Tuning (SFT)

The training begins with supervised fine-tuning (SFT), where generative models are trained on large datasets through maximum likelihood estimation (MLE). This stage ensures that models grasp the fundamental capability to generate coherent text sequences before proceeding to more complex alignment stages.

Reward Modeling

Reward models are essential for shaping the policy models' outputs and are trained either separately (offline) or jointly (online). Models like the Bradley-Terry Reward Model and Absolute-Rating Multi-Objective Reward Model (ArmoRM) are crucial in this phase, translating human preferences into quantifiable rewards.

Reinforcement Learning with Human Feedback (RLHF)

RLHF, utilizing algorithms like Proximal Policy Optimization (PPO) and REINFORCE, further aligns generative models with human preferences. The paper extensively discusses online and offline alignment methods, highlighting their applicability in improving model performance and safety.

Applications Across Modalities

Language Tasks

In language tasks, models such as LLaMA, Phi, and Mistral benefit from preference tuning by aligning with human feedback to improve task-specific skills, coherence, and fluency. The reinforcement learning techniques enable these models to avoid undesired outputs and generate more human-like text.

Vision and Text Alignment

For vision-text tasks, alignment methods such as RLHF-V and DPOK enhance the representation of both modalities, using pre-trained models like CLIP and CoCa. Techniques like Reward Feedback Learning (ReFL) and Direct Reward Fine-Tuning (DRaFT) are particularly effective in bridging the gap between textual descriptions and visual outputs.

Speech Alignment

Speech tasks, though less explored, show promise through models integrating subjective human evaluation into the training loop. Models like BAT and SpeechGPT, with techniques like DLPO, demonstrate the potential for better aligning synthetic speech with human preferences.

Evaluation and Future Directions

Evaluation Metrics

The survey emphasizes the significance of robust evaluation methodologies, including LLM judges and benchmarks like AlpacaEval, ChatbotArena, and MT-Bench. These metrics offer a scalable approach to assess model alignment with human expectations.

Multilingual and Multimodal Extensions

Future research directions suggest expanding preference tuning to multilingual and multimodal contexts. This involves addressing cultural nuances in multilingual settings and improving alignment techniques for complex, multi-domain tasks.

Unlearning and Mechanistic Understanding

Unlearning techniques for removing harmful responses and a deeper mechanistic understanding of preference tuning methods are highlighted as promising research areas. These approaches aim to enhance model safety and reliability, ensuring they align better with user expectations over time.

Conclusion

This exhaustive survey provides a critical foundation for understanding the current landscape and future potentials of preference tuning with human feedback. It presents a comprehensive taxonomy, robust evaluation techniques, and identifies emerging research areas, providing valuable insights and directions for both academic researchers and practitioners aiming to innovate in the field of AI model alignment.