Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback (2410.19133v5)

Abstract: Learning from human feedback has enabled the alignment of LLMs (LMs) with human preferences. However, collecting human preferences is expensive and time-consuming, with highly variable annotation quality. An appealing alternative is to distill preferences from LMs as a source of synthetic annotations, offering a cost-effective and scalable alternative, albeit susceptible to other biases and errors. In this work, we introduce HyPER, a Hybrid Preference routER that defers an annotation to either humans or LMs, achieving better annotation quality while reducing the cost of human-only annotation. We formulate this as an optimization problem: given a preference dataset and an evaluation metric, we (1) train a performance prediction model (PPM) to predict a reward model's (RM) performance on an arbitrary combination of human and LM annotations and (2) employ a routing strategy that selects a combination that maximizes the predicted performance. We train the PPM on MultiPref, a new preference dataset with 10k instances paired with humans and LM labels. We show that the selected hybrid mixture of synthetic and direct human preferences using HyPER achieves better RM performance compared to using either one exclusively by 7-13% on RewardBench and generalizes across unseen preference datasets and other base models. We also observe the same trend in other benchmarks using Best-of-N reranking, where the hybrid mix has 2-3% better performance. Finally, we analyze features from HyPER and find that prompts with moderate safety concerns or complexity benefit the most from human feedback.

• The paper introduces an optimization-based routing framework that judiciously allocates annotation tasks between human annotators and language models.
• It achieves 7–13% improvement in RewardBench accuracy and up to 3% boosts in downstream evaluations by balancing input sources.
• This strategic hybrid approach reduces annotation costs while enhancing overall language model performance.

Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback

The paper presents an innovative exploration into hybrid preference annotation by introducing a routing framework that synergizes human and AI feedback for LLM (LM) training. With the rising utility of reinforcement learning from human feedback (RLHF) in aligning LMs with human preferences, this research addresses the challenges posed by the traditional methods of collecting human annotations, which are often expensive, time-consuming, and susceptible to high variance.

Core Methodology

Central to this paper is an optimization-based routing framework designed to intelligently partition preference instances between human annotators and LMs. This framework aims to enhance annotation quality while minimizing human annotation costs. The crux of this approach is a performance prediction model (PPM) that predicts the efficacy of a reward model trained on a hybrid dataset combining human and LM annotations. By employing a strategic routing system, the framework selects an optimal annotation mix that maximizes the predicted performance of the reward model.

The researchers trained their performance prediction model using "MultiPref," a diverse dataset comprising 10,000 instances annotated by both humans and LMs. Through this hybrid methodology, the paper demonstrates superior reward model performance compared to scenarios where only human or only LM annotations are utilized.

Numerical Results and Generalizability

The empirical results reveal that hybrid annotations consistently outperform both exclusive human and AI annotations on various evaluation metrics, including RewardBench and common LM benchmarks. Importantly, the framework achieves this with only a fraction of the necessary human input compared to purely human-annotated datasets.

The paper reports 7–13% improvements in RewardBench accuracy and up to 3% enhancements in downstream evaluations. These results highlight the robustness and efficacy of the routing framework across different datasets, underscoring its potential generalizability and adaptation to diverse annotation needs.

Implications and Future Directions

The implications of this research extend well beyond immediate annotations and into broader AI training methodologies. By optimizing the balance between costly human input and scalable AI annotations, the paper marks a pivotal shift in preference learning strategies. It opens up avenues for developing more cost-effective, scalable, and nuanced LM training paradigms leveraging the strengths of both human insight and machine learning consistency.

Moreover, the insights gained from understanding which instances benefit more from human feedback—such as those with moderate safety concerns or specific domain complexities—can refine future AI development and deployment strategies.

Conclusion

The findings outlined in the paper provide a compelling case for integrating routed hybrid annotations in LLM training, offering new avenues for enhancing model alignment with human-like judgments. By offering a practical, optimized approach to balancing human and AI feedback, this research contributes significantly to the ongoing discourse in AI preference learning and lays the groundwork for subsequent advancements in the intelligent deployment of hybrid systems. Future research could explore expanding this methodology to different types of AI models and further refine the routing strategy to include more nuanced decision metrics.

In conclusion, the paper effectively demonstrates that a strategic combination of human and AI annotations can yield superior results in LLM training, setting a benchmark for the efficient alignment of AI systems with human values and preferences.