Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm (2505.01706v1)

Abstract: Direct Preference Optimisation (DPO) has emerged as a powerful method for aligning LLMs with human preferences, offering a stable and efficient alternative to approaches that use Reinforcement learning via Human Feedback. In this work, we investigate the performance of DPO using open-source preference datasets. One of the major drawbacks of DPO is that it doesn't induce granular scoring and treats all the segments of the responses with equal propensity. However, this is not practically true for human preferences since even "good" responses have segments that may not be preferred by the annotator. To resolve this, a 2-dimensional scoring for DPO alignment called 2D-DPO was proposed. We explore the 2D-DPO alignment paradigm and the advantages it provides over the standard DPO by comparing their win rates. It is observed that these methods, even though effective, are not robust to label/score noise. To counter this, we propose an approach of incorporating segment-level score noise robustness to the 2D-DPO algorithm. Along with theoretical backing, we also provide empirical verification in favour of the algorithm and introduce other noise models that can be present.

Overview of Inducing Robustness in a 2-dimensional Direct Preference Optimisation Paradigm

The paper "Inducing Robustness in a 2-dimensional Direct Preference Optimisation Paradigm" explores enhancements to the Direct Preference Optimization (DPO) method used for aligning LLMs with human preferences. With the increasing importance of generating responses that adhere to human preferences, this paper contributes to developing more nuanced and robust models capable of handling uncertainties and inaccuracies inherent in human-annotated data.

Direct Preference Optimization has traditionally offered an efficient alternative to Reinforcement Learning from Human Feedback (RLHF) by addressing some of its limitations, specifically the computational intensity and the instability associated with RLHF's two-step optimization process. The DPO aligns model outputs directly to a preference-based dataset, streamlining the process and achieving comparable performances to RLHF. However, it has been identified to lack granularity in evaluating responses since it treats all segments within a response uniformly, which diverges from realistic human judgment often influenced by particular segments within responses.

To overcome this limitation, the paper builds upon the concept of 2-dimensional DPO (2D-DPO) introduced by Li et al., which brings a multi-aspect evaluation of response segments. Each segment within a response is evaluated across five dimensions: Completeness, Clarity, Correctness, Safety, and Helpfulness. This multi-faceted approach aims to provide a more accurate reflection of human preference structures by aggregating the segment scores to a composite response score. Segments are assessed individually and contribute variably to the overall preference alignment based on these scores.

Despite the advances in modeling, the paper identifies robustness as a critical shortfall in existing methodologies under segment-wise and overall preference evaluation models. The authors propose a robust extension to the 2D-DPO, which addresses the sensitivity of the algorithm to noise in segment-level scores, a common occurrence given subjective variabilities in human annotations. The proposed framework introduces noise robustness to the segment-level scores, applying uniform perturbations and mitigating the effects through novel optimization processes.

Empirical analyses in the paper substantiate the theoretical claims through experiments on modified datasets accounting for noise. The evaluation involved comparing win rates in response preference alignment under different models, including the vanilla DPO, vanilla 2D-DPO, and the newly introduced robust 2D-DPO. Results indicate a decline in performance for non-robust models subjected to noise, while the robust variant maintains satisfactory performance levels, presenting a promising advancement in preference optimization strategies.

The broader implications of these findings suggest an improved capacity for LLMs to handle nuance and noise in user preferences, enhancing the reliability and adoption potential of AI models in real-world applications involving subjective human feedback. The authors recognize the potential developments in aligning AI interactions with human expectations more closely through robust, multi-dimensional preference tailoring. Future investigative directions are proposed, including modeling aspect-level noise robustness and synthesizing comprehensive noise frameworks involving complex deformation of human annotations.

By presenting the robust 2D-DPO approach, the paper extends the capabilities of human preference alignment techniques, essential for creating more adaptable and responsive LLM systems. As the landscape of AI continues to emphasize ethical and practical alignment with human values, such contributions are poised to facilitate responsible and impactful AI integrations.