--- title: 'LiPO-λ: Listwise Preference Optimization' url: https://www.emergentmind.com/topics/lipo type: topic --- # LiPO-λ: Listwise Preference Optimization LiPO-λ (Lambda-Loss Listwise Preference Optimization) is a listwise policy optimization algorithm developed to align language models with rankwise human or AI-generated preference data. Building on the observation that preference feedback in practical LM alignment often consists of ranked lists rather than binary comparisons, LiPO-λ leverages a listwise learning-to-rank (LTR) objective incorporating LambdaLoss weighting, producing listwise- and label-sensitive updates. It generalizes several prominent preference optimization objectives, including DPO and SLiC, and empirically outperforms these on canonical language model alignment tasks [2402.01878]. ## 1. Listwise Objective and LambdaLoss Formulation LiPO-λ treats each datapoint as a prompt $x$ paired with a list of $K$ responses $\mathbf{y} = (y_1, \ldots, y_K)$ and their corresponding scalar preference scores $\boldsymbol\psi = (\psi_1,\ldots,\psi_K)$. For each response, a score is computed: \[ s_i = \beta \log\frac{\pi_\theta(y_i\mid x)}{\pi_{\rm ref}(y_i\mid x)} \] where $\pi_\theta$ is the trainable policy, $\pi_{\rm ref}$ is the fixed reference (SFT) policy, and $\beta>0$ is a KL-control coefficient. The core per-example loss is: \[ \ell_{\lambda}(\boldsymbol\psi, \mathbf{s}) = -\sum_{i,j:\,\psi_i>\psi_j} \Delta_{i,j}\log\left(1 + e^{-(s_i - s_j)}\right) \] with Lambda weight \[ \Delta_{i,j} = |G(\psi_i) - G(\psi_j)| \cdot \left| D(\tau(i))^{-1} - D(\tau(j))^{-1} \right| \] where - $G(\psi)$ (gain function), commonly $G(\psi)=2^\psi-1$ - $D(r)$ (discount function), typically $D(r)=\log(1 + r)$ - $\tau(i)$ is the predicted rank of item $i$ under model scores $\mathbf{s}$ (sorted descending). The full objective averages this loss over all prompt–response lists in the dataset $\mathcal D$: \[ \mathcal L_{\lambda}(\theta) = \mathbb{E}_{(x,\mathbf{y}, \boldsymbol\psi)\sim \mathcal D} \left[ \ell_\lambda(\boldsymbol\psi, \mathbf{s}(\theta)) \right] \] This design leverages all $\binom{K}{2}$ response pairs and adapts their contribution via LambdaLoss scaling. ## 2. LambdaLoss Weighting: Listwise and Label Sensitivity The “λ” in LiPO-λ specifically refers to the LambdaLoss weighting scheme. Every pair $(i,j)$ with $\psi_i>\psi_j$ is weighted not uniformly, but by $\Delta_{i,j}$, incorporating: - **Gain sensitivity:** $|G(\psi_i)-G(\psi_j)|$ incorporates the *magnitude* of preference between responses, in contrast to merely using their ordering. - **Listwise sensitivity:** $|D(\tau(i))^{-1}-D(\tau(j))^{-1}|$ introduces dependence on the full ranking of items as predicted by the current model policy. Omitting these weights ($\Delta_{i,j}=1$) reduces the objective to the plain pairwise logistic (Bradley–Terry) loss. With $K=2$, this yields the DPO$_{\rm BT}$ loss. The label- and listwise-sensitivity are essential for exploiting the structure of $K>2$ preference lists and for more faithful alignment to ranking metrics such as discounted cumulative gain (DCG). ## 3. Gradient Computation and Optimization LiPO-λ's pairwise logistic kernel yields nearly closed-form gradients with respect to model scores. For each $s_i$, \[ \frac{\partial \ell_\lambda}{\partial s_i} = - \sum_{j:\psi_i > \psi_j} \Delta_{i,j} \sigma(s_i - s_j) + \sum_{k:\psi_k > \psi_i} \Delta_{k,i} \sigma(s_k - s_i) \] where $\sigma(z) = 1/(1 + e^{-z})$. The policy parameter gradients follow by the chain rule: \[ \nabla_\theta \ell_\lambda = \beta \sum_{i=1}^K \frac{\partial \ell_\lambda}{\partial s_i} \nabla_\theta \log \pi_\theta(y_i | x) \] Optimization proceeds via stochastic gradient descent (e.g., Adam or Adafactor), with policy updates: \[ \theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}_\lambda \] where $\eta$ is the learning rate. ## 4. Comparative Analysis with DPO and SLiC LiPO-λ subsumes earlier objectives as limiting cases: - **DPO$_{\rm BT}$**: Set $\Delta_{i,j}=1$, $K=2$ to recover a pairwise logistic loss. - **SLiC$_{\rm norm}$**: For $K=2$ and a hinge kernel, recovers normalized hinge loss. - **DPO$_{\rm PL}$** (Plackett–Luce list-MLE): Optimizes \[ \ell_{\rm PL} = -\log \prod_{k=1}^K \frac{e^{s_{\tau(k)}}}{\sum_{j \geq k} e^{s_{\tau(j)}}} \] but only respects the static label permutation, not label magnitudes or predicted permutation. By contrast, LiPO-λ: - Uses all $\binom{K}{2}$ pairs with non-uniform, listwise $\Delta_{i,j}$. - Encodes both label-magnitude and predicted-rank (permutation) sensitivity. - Retains a smooth logistic kernel, in contrast to the listwise hinge in SLiC. The following table summarizes these distinctions: | Method | Pairwise/Listwise | Label Sensitivity | Kernel Type | |-----------------|-------------------|-------------------|---------------------| | DPO$_{\rm BT}$ | Pairwise ($K=2$) | No | Logistic (BT) | | SLiC$_{\rm norm}$ | Pairwise ($K=2$) | No | Hinge | | DPO$_{\rm PL}$ | Listwise | Ordering only | List-MLE (PL) | | **LiPO-λ** | Listwise ($K>2$) | Yes (magnitude) | Logistic (RankNet) | ## 5. Empirical Performance LiPO-λ was evaluated on two public LM alignment tasks: - **Reddit TL;DR summarization** - **AnthropicHH dialogue** Models were fine-tuned from a T5-large (770M) SFT baseline. For each prompt, $K=8$ response candidates were sampled ($\mathrm{temp}=0.7$, top-$k=40$), and all $\binom{8}{2}=28$ pairs were labeled using a T5-XXL reward model. Training was conducted with batch size $32$, learning rate $2\times10^{-5}$, and $\beta=0.05$. Automatic evaluation against the reward model (“proxy reward”) and via PaLM 2–IT side-by-side (“AutoSxS”) was complemented by human side-by-side and pointwise quality assessments. ### Reported Automatic Metrics (Proxy Reward and AutoSxS, Table 1; T5-large policy) | Method | TL;DR (Proxy) | HH (Proxy) | TL;DR (AutoSxS) | HH (AutoSxS) | |----------------|---------------|------------|-----------------|--------------| | DPO$_{\rm BT}$ | 88.52% | 91.11% | 67.09% | 44.80% | | DPO$_{\rm PL}$ | 88.27% | 90.61% | 67.23% | 43.25% | | LiPO-λ | **90.60%** | **92.60%** | **68.06%** | **47.90%** | With a T5-XXL policy, LiPO-λ led by approximately 1 percentage point on both metrics (Table 2). ### Human Side-by-Side (Table 3) For TL;DR, LiPO-λ was preferred 40% of the time (compared to 19%/16% for baselines); for HH, LiPO-λ attained 27% preference (20%/20% for baselines), with higher mean pointwise quality ratings. *This suggests that listwise and label-sensitive objectives enable more effective use of listwise preference feedback, particularly as $K$ increases.* ## 6. Implementation and Training Schema Minimal pseudocode expressing the LiPO-λ pipeline follows the procedure outlined in Algorithm 1: ```python initialize θ ← SFT-checkpoint for each training step do batch ← sample B prompts x from D for each x in batch do {y₁...yₖ} ← sample K responses from π_ref for all iψ_j} Δ_{i,j}·log(1+exp(–(s_i–s_j))) θ ← θ – η·Adam(∇_θ loss) end for ``` Key hyperparameters are: optimizer = Adafactor/Adam, learning rate $2\times10^{-5}$, batch size $32$, $K=8$, $\beta=0.05$, sampling temperature $0.7$, and top\_k $=40$. ## 7. Significance and Utilization LiPO-λ augments standard pairwise preference optimization with listwise-aware LambdaLoss weights, enabling smooth optimization and effective learning from $K>2$ preference lists. It provides a principled framework for mapping LM alignment to LTR objectives, formally subsumes important special cases (DPO, SLiC), and empirically delivers consistent gains across alignment benchmarks [2402.01878]. A plausible implication is that as ranked preference data becomes more prevalent, listwise objectives such as LiPO-λ represent a robust methodological direction for preference-based LM alignment. Source: https://www.emergentmind.com/topics/lipo