Model Extrapolation Expedites Alignment

Abstract: Given the high computational cost of preference alignment training of LLMs, exploring efficient methods to reduce the training overhead remains an important and compelling research problem. Motivated by the observation that alignment training typically involves only small parameter changes without injecting new knowledge into models, we propose a straightforward method called ExPO (model extrapolation) to expedite LLMs' alignment with human preferences. Given a partially-trained model and its initial SFT checkpoint, ExPO improves the implicit optimization objective of alignment training by simply amplifying the parameter change based on a first-order approximation, without any additional training overhead. Through controlled experiments, we demonstrate that ExPO boosts a DPO model trained with only 20% steps to outperform the fully-trained one. Moreover, we show that ExPO notably improves existing open-source LLMs (ranging from 1.8B to 70B parameters) on the leading AlpacaEval 2.0 and MT-Bench benchmarks, which highlights ExPO's broader utility in efficiently enhancing LLM alignment.

• The paper introduces ExPO, a novel extrapolation approach that enhances LLM alignment by interpolating between suboptimal and medium-aligned models.
• It demonstrates that limited training data, when processed through ExPO, can yield performance on par with or exceeding full-data models.
• Experimental results using AlpacaEval 2.0 indicate that larger models gain the most from this scalable and economical alignment method.

ExPO: A Method for Enhancing LLMs’ Alignment with Human Preferences via Model Extrapolation

Introduction and Motivation

The development of LLMs like GPT-4 has included efforts to enhance their alignment with human preferences through stages of Supervised Fine-Tuning (SFT) and subsequent reinforcement learning (RL) or direct preference optimization (DPO). While these alignment strides are substantial, they are often restricted by available resources. This paper introduces ExPO (model extrapolation), a novel technique devised to extrapolate from existing less-aligned and medium-aligned models to generate a superior model exhibiting higher alignment with human preferences, without additional costly training processes. This method leverages insights from model interpolation literature and provides a practical approach to potentially bypass some of the resource-intensive stages of model training.

Methodology

Assumptions and Theoretical Foundations

At its core, ExPO is predicated on the assumption that a medium-aligned model (M) can be interpolated from a less-aligned model (M_w) and a hypothetical better-aligned model (M_s). The model M is understood to be an output of initial alignment processes such as SFT, while M_s represents an achievable, yet not directly trained, superior state of alignment. By manipulating the interpolation coefficients, ExPO aims to reverse-engineer M_s from M and M_w.

Practical Implementation

The extrapolation in ExPO follows a straightforward formulaic adjustment, where the coefficient α (alpha) adjusts the influence of changes derived from model M relative to M_w. This coefficient can be fine-tuned efficiently like a decoding hyperparameter, rendering the process computationally economical and feasible without further extensive training.

Experiments and Results

Overview of Experimental Setup

The paper outlines experiments on varying scales of preference data (10%, 20%, and 100%), using models trained on these datasets, and then applying ExPO to evaluate performance enhancements. Experiments were predominantly evaluated using the AlpacaEval 2.0 benchmark, focusing on comparing length-controlled win rates over a GPT-4 baseline.

Key Findings

Models trained with reduced data sets (10% & 20%), when treated with ExPO, not only reached but occasionally surpassed the performance of models trained with complete data sets (100%). This indicates that ExPO can effectively leverage learned alignment from suboptimal models to approximate and even exceed fully optimized models. Additionally, results showed that larger models exhibited more significant improvements, highlighting ExPO’s scalability.

Theoretical and Practical Implications

ExPO represents an economical and scalable method to enhance LLMs' alignment with human preferences beyond initial training limitations. It suggests that previously underutilized model states, typically considered suboptimal, can serve as fundamental components in developing stronger models. For ongoing and future implementations of LLMs, ExPO offers a pragmatic approach to continual improvement of models in alignment-focused applications.

Future Directions

While the current methodology provides a robust foundation, future work could explore adaptive module-specific extrapolation coefficients, eliminate dependencies on external reward models, and theoretically encapsulate the mechanistic underpinnings of ExPO’s effectiveness. Additionally, exploring the applicability of ExPO across diverse model architectures and multimodal LLMs could broaden its utility in the AI field.

Conclusion

ExPO provides a promising avenue for improving the capabilities of LLMs concerning human preference alignment, utilizing an efficient, straightforward computational approach. This method, by enabling superior model performance without additional data or extensive training, aligns with the economical and practical demands of modern AI research and applications, warranting further exploration and development.