Understanding the performance gap between online and offline alignment algorithms (2405.08448v1)

Abstract: Reinforcement learning from human feedback (RLHF) is the canonical framework for LLM alignment. However, rising popularity in offline alignment algorithms challenge the need for on-policy sampling in RLHF. Within the context of reward over-optimization, we start with an opening set of experiments that demonstrate the clear advantage of online methods over offline methods. This prompts us to investigate the causes to the performance discrepancy through a series of carefully designed experimental ablations. We show empirically that hypotheses such as offline data coverage and data quality by itself cannot convincingly explain the performance difference. We also find that while offline algorithms train policy to become good at pairwise classification, it is worse at generations; in the meantime the policies trained by online algorithms are good at generations while worse at pairwise classification. This hints at a unique interplay between discriminative and generative capabilities, which is greatly impacted by the sampling process. Lastly, we observe that the performance discrepancy persists for both contrastive and non-contrastive loss functions, and appears not to be addressed by simply scaling up policy networks. Taken together, our study sheds light on the pivotal role of on-policy sampling in AI alignment, and hints at certain fundamental challenges of offline alignment algorithms.

Is Online Reinforcement Learning Really Necessary for AI Alignment?

Background

The debate between online and offline reinforcement learning (RL) methods has been ongoing in the AI community. With the rise of LLMs, alignments methods like Reinforcement Learning from Human Feedback (RLHF) have become a staple for improving these models. However, offline alignment methods, which avoid the need for active online interactions, have shown impressive empirical performance. This research dives into a core question: is online RL essential for AI alignment, or can offline methods do the job just as well?

Comparing Online and Offline Algorithms

Over-Optimization and Goodhart's Law

When working with both online and offline RL methods, a concept called Goodhart's law comes into play. Essentially, it states that when a measure becomes a target, it loses its efficacy as a measure.

The researchers observed that both online and offline algorithms exhibit over-optimization behavior, where performance initially improves but then deteriorates due to becoming overly specialized to a specific target. The paper showed that online algorithms generally outperformed offline ones for the same optimization budget, measured as KL divergence from the starting supervised model.

Hypotheses for Performance Differences

To understand why online algorithms seem to have the edge, the researchers tested several hypotheses:

1. Data Coverage Hypothesis: The online methods might benefit from a more diverse range of data.
2. Sub-optimal Offline Dataset Hypothesis: Offline algorithms might be limited by the quality of the initial dataset.
3. Classification Accuracy Hypothesis: Better classification accuracy should lead to better performance.
4. Non-Contrastive Loss Function Hypothesis: The performance gap might be due to the types of loss functions used, rather than the online/offline nature of the algorithm.
5. Scaling Hypothesis: Scaling up model sizes might eliminate the performance gap between online and offline methods.

Experimental Setup

The paper used various open-source LLMs and focused on tasks like text summarization and helpfulness. To ensure comprehensive results, the experiments involved:

• Accurate measurement of KL divergence to gauge optimization budgets.
• Fair comparison of online and offline algorithms by standardizing the data and initial conditions.

Key Findings

Hypothesis Testing and Ablations

1. Data Coverage: Offline algorithms, even when given the same diverse data coverage as online algorithms (but shuffled randomly), did not achieve the same performance levels.
2. Sub-optimal Offline Dataset: Offline algorithms trained on datasets generated by high-performance policies still did not bridge the gap, negating the hypothesis that the initial data's quality was the primary issue.
3. Classification Accuracy: Surprisingly, improving classification accuracy did not translate to better performance. The policies did not boost the probability of winning responses as much as expected.
4. Non-Contrastive Loss Function: Even with simpler, non-contrastive loss functions like Best-of-2 (Bo2), the performance gap between online and offline methods persisted.
5. Scaling Policy Size: Scaling up the model size helped improve peak performance but did not completely eliminate the gap. Online methods still held a slight advantage, suggesting that data diversity and evolving policies provide unique benefits that scaling alone cannot offer.

Crucial Role of On-Policy Sampling

One major takeaway is that merely scaling models or improving dataset quality is not enough to replicate the benefits of online methods. The constantly evolving policy and on-policy sampling unique to online methods are fundamental for optimal alignment performance.

Implications and Future Directions

This research offers significant insights into the practicalities of AI alignment:

• Practical Applications: Online methods, while computationally more intensive, may still be necessary for the highest levels of model alignment.
• Theoretical Insights: The paper challenges some existing theoretical assumptions and highlights the need for more nuanced models that account for practical complexities in data and training dynamics.
• Future Developments: There is potential in exploring hybrid methods that blend online and offline approaches to balance efficiency and performance. Additionally, as models continue to scale, new techniques to manage and utilize diverse data more effectively will likely emerge.

In conclusion, the paper sheds light on the intrinsic strengths of online RL methods and why they currently outperform offline methods despite the latter's practical appeal. The findings reinforce the importance of dynamic, on-policy training in achieving robust AI alignment.