Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective (2410.16586v1)

Abstract: Aligning the output of LLMs with human preferences (e.g., by means of reinforcement learning with human feedback, or RLHF) is essential for ensuring their effectiveness in real-world scenarios. Despite significant advancements in LLM alignment techniques, the impact of different type of preference data on model performance has yet to be systematically explored. In this study, we investigate the scalability, data efficiency, and effectiveness of Direct Preference Optimization (DPO) in fine-tuning pre-trained LLMs, aiming to reduce their dependency on extensive amounts of preference data, which is expensive to collect. We (1) systematically compare the performance of models fine-tuned with varying percentages of a combined preference judgement dataset to define the improvement curve of DPO and assess its effectiveness in data-constrained environments; and (2) provide insights for the development of an optimal approach for selective preference data usage. Our study reveals that increasing the amount of data used for training generally enhances and stabilizes model performance. Moreover, the use of a combination of diverse datasets significantly improves model effectiveness. Furthermore, when models are trained separately using different types of prompts, models trained with conversational prompts outperformed those trained with question answering prompts.

Aligning LLMs with human preferences is a critical step for their effective deployment in real-world applications. Traditional methods like Reinforcement Learning from Human Feedback (RLHF) are effective but often require significant amounts of data and involve complex multi-stage training processes, including training a separate reward model. Direct Preference Optimization (DPO) [2023.05.18260] offers a simpler alternative by directly optimizing the LLM based on human preference data through a specific loss function, bypassing the need for an explicit reward model and reinforcement learning.

The paper "Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective" (Bernardelle et al., 22 Oct 2024 ) investigates the data efficiency and effectiveness of DPO, aiming to understand its performance with varying amounts and types of preference data. The paper addresses two main research questions:

1. How does the performance of LLMs fine-tuned with DPO change as the amount of preference data increases?
2. How does the type of training data (specifically conversational versus question answering) affect DPO performance?

To explore these questions, the authors used OpenHermes-2.5-Mistral-7B as the base model, a high-performing open-source LLM suitable for reproduction. Three open-source preference datasets from Hugging Face/Argilla were utilized:

• Dataset A (distilabel-capybara-dpo-7k-binarized): Primarily conversational prompts.
• Dataset B (distilabel-intel-orca-dpo-pairs): Question-answering prompts.
• Dataset C (ultrafeedback-binarized-preferences): Question-answering prompts, the largest dataset.

The total size of the combined dataset was over 84,000 preference pairs.

Experiment 1: Data Volume Impact (RQ1)

For the first experiment, the three datasets were combined into a single pool. This pool was split into training (80%), evaluation (10%), and testing (10%) sets. Random subsets corresponding to 20%, 40%, 60%, 80%, and 100% of the training data were sampled. Separate instances of the base model were fine-tuned using DPO on each data subset percentage. This process was repeated three times with different random seeds to account for variability, resulting in 15 models.

Models were evaluated using MT-Bench [2024.03.03182], a standard benchmark that pits the fine-tuned model against a base model on a set of questions and uses an LLM-as-a-judge to determine wins, losses, and ties. Performance was measured as the percentage improvement in wins over the base model and the tie rate (percentage of questions where the fine-tuned model's response was indistinguishable from the base model's response).

Results showed that increasing the data volume generally led to enhanced performance improvements and increased stability across runs, supporting the idea that more data helps DPO alignment. However, the trend was not perfectly smooth, with noticeable performance dips observed at intermediate data percentages (e.g., around 60%). This variability highlights that the specific data points included in a subset can significantly impact training outcomes.

Furthermore, the tie rate between the DPO-aligned model and the base model consistently decreased as data volume increased, indicating that the DPO training made the model's responses more clearly distinguishable and preferred by the judge model. An exception was observed at 100% data usage, where the tie rate slightly increased, suggesting a potential plateau in performance gains where additional data adds less value in differentiating the model.

Experiment 2: Data Type Impact (RQ2)

The second experiment mirrored the first but applied the DPO fine-tuning process to each individual dataset (A, B, and C) separately, again sampling subsets from 20% to 100% of each dataset's training split and repeating three times (45 models total).

This experiment revealed the distinct impact of data types:

• Dataset A (Conversational): Despite being the smallest, it showed a steady positive trend in performance improvement with increased data usage. The conversational nature of the prompts appeared particularly effective in improving model performance, potentially by providing richer context and dynamics.
• Dataset B (Question-Answering): This dataset also showed potential for significant improvements but exhibited more variability and non-linear trends, including unexpected performance dips, similar to the combined dataset results at intermediate volumes. This suggests the importance of careful data curation even within a specific type.
• Dataset C (Question-Answering): As the largest dataset, it resulted in the smoothest and most consistent improvement curve. The larger volume seemingly helped average out noise and allowed the model to leverage the data more effectively for alignment.

Practical Implications for Implementation

The paper offers several key practical implications for developers and practitioners applying DPO:

1. Data Volume Matters, But Quality/Composition is Crucial: While generally, more data improves DPO performance and stability, the paper shows the relationship is not linear and is heavily influenced by the specific data used. Simply collecting more data without considering its composition may not yield optimal results.
2. Data Diversity is Beneficial: Combining diverse datasets (conversational and question-answering) consistently resulted in the highest peak and average performance improvements compared to using individual datasets (Table 2 in the paper). This suggests that for practical applications, curating a varied preference dataset covering different interaction styles and topics is highly valuable.
3. Conversational Data Efficiency: Conversational prompts (like in Dataset A) appear particularly effective for DPO alignment, leading to steady improvements even with smaller data volumes. Prioritizing or ensuring representation of such data types in collection efforts could be a data-efficient strategy.
4. Potential for Data Curation Strategies: The observed performance dips across different data volumes highlight the need for more sophisticated data selection or sampling methods within DPO. Random sampling might introduce subsets that are less effective for optimization. Future implementations could explore weighted sampling or active learning-like approaches to select the most informative preference pairs.
5. Computational Resources: The experiments were feasible on a single H100 GPU within reasonable timeframes (hours to a day depending on data size), providing a baseline estimate for computational requirements for fine-tuning 7B parameter models with DPO.

For implementing DPO, practitioners can leverage open-source libraries like trl (Transformer Reinforcement Learning) from Hugging Face, which provides implementations of DPO. The process involves:

1. Loading a pre-trained LLM (e.g., OpenHermes-2.5-Mistral-7B).
2. Loading or preparing a preference dataset. This dataset should contain pairs of responses (one preferred, one rejected) for given prompts.
3. Configuring the DPO trainer with hyperparameters (learning rate, batch size, number of epochs, $\beta$ parameter for the DPO loss).
4. Training the model using the DPO trainer on the preference data.

An example using the trl library might look like this:

from trl import DPOConfig, DPOTrainer from transformers import AutoModelForCausaLLM, AutoTokenizer from datasets import load_dataset model_name = "teknium/OpenHermes-2.5-Mistral-7B" model = AutoModelForCausaLLM.from_pretrained(model_name) tokenizer = AutoTokenizer.from_pretrained(model_name) tokenizer.pad_token = tokenizer.eos_token # Set pad token dataset = load_dataset("argilla/distilabel-capybara-dpo-7k-binarized", split="train") training_args = DPOConfig( per_device_train_batch_size=4, gradient_accumulation_steps=1, learning_rate=5e-5, # Example learning rate num_train_epochs=1, # Example number of epochs output_dir="./dpo_results", logging_steps=100, save_steps=1000, # Add other relevant arguments beta=0.1 # Beta parameter for DPO loss - controls strength of preference penalty ) dpo_trainer = DPOTrainer( model, None, # Reference model - if None, uses the training model itself as ref args=training_args, train_dataset=dataset, tokenizer=tokenizer, # Add peft_config if using LoRA ) dpo_trainer.train() dpo_trainer.save_model("./fine_tuned_dpo_model")

Note: The reference model (model_ref) in DPOTrainer is optional. If None, the training model is used as its own reference, which is a common practice in DPO implementations like the original paper and subsequent work.

The paper concludes that while DPO is promising, the efficiency of alignment is not solely dependent on data volume but significantly on the specific data instances used. Future work should focus on developing strategies for selecting the most impactful preference data, potentially leading to more efficient and cost-effective LLM alignment even with limited datasets.