Which Pairs to Compare for LLM Post-Training?

Abstract: Preference-based post-training has become a central paradigm for aligning LLMs. A common data-collection strategy is to generate a small set of completions for each prompt and label the resulting comparison pairs. However, human preference labels are often much more expensive than generating additional completions, suggesting a different use of the same labeling budget: generate a larger pool of completions, but label only the most informative comparison pairs. This paper studies which pairs should be compared in preference-based post-training. We formulate comparison curation as a sampling-design problem and evaluate designs by the quality of the final policy under the preference-based post-training objective. We instantiate this framework for Direct Preference Optimization (DPO), analyzing how the choice of labeled pairs propagates through DPO training to downstream policy performance. Our main results provide matching upper and lower bounds on the post-training optimality gap of the DPO-trained policy. The bounds show that comparison selection affects downstream performance through a single design-dependent information matrix, which links label allocation to parameter estimation error and policy suboptimality. This yields an explicit optimization criterion for budgeted comparison curation and motivates practical sampling designs for selecting informative pairs from large generated completion pools. Experiments on synthetic settings and language-model post-training benchmarks show that the proposed designs consistently improve sample efficiency over common comparison-selection heuristics.

• The paper's main contribution is establishing an information-theoretic framework that connects optimal comparison pair selection with improved RLHF sample efficiency.
• It introduces a plug-in rule using reference parameters as a practical approximation to the optimal design, validated through both synthetic and experimental settings.
• Experimental results show that trace-optimal designs yield lower RLHF suboptimality gaps compared to uniform or proportional selection methods.

Optimal Comparison Selection for Preference-Based LLM Post-Training

Introduction

This paper addresses the critical problem of selecting pairs for human preference labeling during preference-based post-training of LLMs, with an emphasis on sample efficiency under labeling cost constraints. It examines the interplay between comparison curation and the downstream policy performance for methods such as Direct Preference Optimization (DPO). The work reformulates dataset curation as a sampling-design problem, focusing on how to allocate a fixed preference-annotation budget across the possible comparison pairs generated for each prompt.

Theoretical Framework

The central analytical contribution is the connection of comparison selection to statistical information theory. The effect of pair selection propagates through a design-dependent information matrix, which captures the parameter directions most relevant for downstream RLHF (Reinforcement Learning from Human Feedback) performance. In this setting, the labeling budget is inherently limited, and the challenge lies in selecting the most informative comparisons from a larger, cheaply-generated candidate set per prompt.

The main result derives both matching upper and lower finite-sample bounds for the optimality gap of the final DPO-trained policy. Both bounds are controlled by a trace functional involving the Fisher information matrix of the policy and the design covariance matrix induced by the chosen comparison pairs: $$\operatorname{tr}\Big( I(\theta^\star)\Sigma_D^\dagger(\theta^\star) \Big)$$ where $I(\theta^\star)$ is the policy Fisher information at the RLHF optimum, and $\Sigma_D(\theta^\star)$ is the design covariance for the sampling distribution $D$. This trace serves as the explicit sample-efficiency criterion for curation.

The information matrix $\Sigma_D$ is the second-moment matrix of the pairwise sensitivity vectors—gradients of the model's logit difference with respect to parameters—for the policy under the comparison-sampling distribution $D$. This criterion elegantly connects the design of preference-label queries directly to post-training policy optimality, rather than mere parameter estimation accuracy or reward-model performance.

Practical Design Criterion

Since the optimal design depends on the unknown RLHF optimum parameter $\theta^\star$, the authors propose a plug-in rule: use the reference policy parameter $\theta_0$ (from SFT) as a proxy. They show trace-equivalence bounds between the oracle information criterion and the plug-in variant, ensuring that the proposed practical policy is near-optimal, up to constants depending on the proximity of $\theta_0$ to $\theta^\star$.

Experimental Results

Three types of experiments corroborate the theory:

• Synthetic Settings (Tabular & Contextual Models): The authors validate their theoretical predictions in fully-controlled settings where the ground truth reward function and optimal policy are known.

Figure 1: In the tabular setting, the D

and plug-in designs significantly outperform uniform and policy-proportional heuristics in terms of RLHF optimality gap.*

Figure 2: In a response selection task on IMDb with GPT-2-Large, the information-theoretic D

design achieves strictly better reward–KL trade-offs compared to common selection rules.*

• IMDb Experiments (LLM Fine-Tuning): Using GPT-2-Large, they demonstrate that response- and prompt-selection based on the trace-optimal design criteria consistently yield improved reward--KL frontiers versus baselines.
• Anthropic-HH Benchmark: With Pythia-2.8B as the base model, using GPT-4.1 as an automatic judge, curated datasets selected via the proposed trace-based method yield higher win rates than arbitrary selection, across sampling temperatures and a range of annotation budgets.

Figure 3: On Anthropic-HH, the D

design gives strictly higher GPT-4.1 win rates at fixed sample budgets ($I(\theta^\star)$0), indicating more effective dataset curation.*

The empirical results robustly support the claim that optimizing for the design-dependent trace criterion leads to consistently superior sample efficiency over heuristics such as within-prompt uniform or SFT-proportional selection.

Implications for Preference-Based LLM Alignment

The findings have direct implications for RLHF pipelines in industrial LLM alignment, where human labeling cost is a critical bottleneck. The analysis demonstrates that, for a fixed labeling budget, considerable efficiency can be gained by offline curation with theoretically justified sampling distributions—a process that can be integrated into policy training with simple plug-in approximations. The approach tightens the gap between practical labeling strategies and the information-theoretic lower bounds for RLHF sample complexity.

On a theoretical level, the result generalizes optimal design principles for the Bradley–Terry model from classical statistics to the high-dimensional, contextualized, and regularized setting of LLM RLHF, explicitly accounting for the RL objective's KL regularization and policy parameterization.

Limitations and Future Directions

While the analysis assumes realizability, regularity, and offline curation (rather than adaptive/interleaved designs), it sets the groundwork for several future extensions:

• Adaptive or online curation policies that utilize feedback during training
• Integration with higher-capacity neural policies and feature representations
• Robustness to label noise or quality variations
• Generalization to other preference-based objectives beyond DPO and to settings with complex sample dependencies

Potential future advancements include adapting the trace criterion for iterative or minibatch designs and extending information-theoretic lower bounds to hierarchical and active RLHF paradigms.

Conclusion

This paper offers a principled, information-theoretic foundation for selecting comparison pairs in LLM preference-based post-training under budget constraints. By showing that a trace-based criterion governs both upper and lower bounds for RLHF policy suboptimality, and by validating empirical improvements in LLM alignment benchmarks, it supplies both a practical methodology for preference-data curation and a theoretical benchmark for sample efficiency in RLHF pipelines.