DeltaUCB: Active Learning for Efficient RLHF

• DeltaUCB is an active learning acquisition function that selects informative preference pairs for RLHF by leveraging uncertainty estimates and the Bradley–Terry model.
• It computes upper and lower confidence bounds on reward estimates to deterministically identify pairs with the largest plausible quality gaps, reducing annotation requirements.
• Empirical results show DeltaUCB achieves up to a six-fold increase in sample efficiency and enhanced downstream performance across benchmarks like GSM8K and TruthfulQA.

DeltaUCB is an active learning acquisition function designed to efficiently select informative preference pairs for Reinforcement Learning from Human Feedback (RLHF), with the objective of maximizing the sample efficiency of reward modeling and downstream LLM fine-tuning. Arising from the ActiveUltraFeedback pipeline, DeltaUCB prioritizes response pairs exhibiting large predicted quality gaps under uncertainty, operationalizing the “Delta Learning Hypothesis” that such pairs provide the most potent fine-tuning signals. By leveraging a principled upper confidence bound framework rooted in distributions over reward estimates and the Bradley–Terry model for preference probabilities, DeltaUCB enables deterministic and cost-effective query selection, significantly reducing annotation requirements while maintaining or improving model performance (Melikidze et al., 10 Mar 2026).

1. Motivation and Conceptual Foundations

RLHF workflows for LLMs typically require acquiring pairwise human preference labels (e.g., A ≻ B) to train a reward model, which then steers further model optimization. The high annotation cost—particularly acute in specialist domains—necessitates methods that maximize the informational content of each labeled comparison. Many dueling bandit approaches aim to minimize cumulative regret or globally reduce uncertainty, but these do not align directly with the RLHF objective of learning a robust reward model for fine-tuning.

DeltaUCB is motivated by the Delta Learning Hypothesis (DLH): preference comparisons with the largest true quality gaps convey the strongest fine-tuning gradient signals. Consequently, DeltaUCB targets, for each prompt, the response pair that could plausibly differ most in true reward, given the model's current uncertainty estimates. This direct focus on large quality gaps distinguishes DeltaUCB from classical acquisition strategies that emphasize exploration or regret minimization.

2. Formal Mathematical Formulation

Given a prompt $x$ and $m$ candidate responses $y_1, ..., y_m$, a reward model predicts, for each response $y_j$:

• Point estimate: $\hat{r}_j = r_\phi(x, y_j)$
• Uncertainty: $\sigma_j = \sigma_\phi(x, y_j)$

DeltaUCB computes (for fixed uncertainty scaling $\beta$):

• Upper and lower confidence bounds: $\overline{r}_j = \hat{r}_j + \beta \sigma_j$, $\underline{r}_j = \hat{r}_j - \beta \sigma_j$
• The win probability under the Bradley–Terry model:

$$p_\phi(y_j \succ y_k \mid x) = s(r_\phi(x, y_j) - r_\phi(x, y_k)), \qquad s(z) = \frac{1}{1 + e^{-z}}$$

• Upper confidence bound on win probability:

$m$0

DeltaUCB selects the ordered pair $m$1 that maximizes this bound:

$m$2

An equivalent gap-based form is $m$3, but in practice the algorithm uses the more computationally efficient $m$4 construction (Melikidze et al., 10 Mar 2026).

3. Statistical Interpretation and Theoretical Basis

Reward uncertainty is modeled via an Ensemble Neural Network (ENN), such that $m$5 is approximately Gaussian with mean $m$6 and standard deviation $m$7. The quality difference $m$8 thus has mean $m$9 and variance $y_1, ..., y_m$0.

A one-sided upper confidence bound on the gap is $y_1, ..., y_m$1, where $y_1, ..., y_m$2. Under the Bradley–Terry model, substituting this "optimistic" gap into the sigmoidal preference function produces an upper bound on the probability that $y_1, ..., y_m$3 is preferred over $y_1, ..., y_m$4.

The practical DeltaUCB algorithm instead substitutes $y_1, ..., y_m$5, which is a deterministic (though slightly looser) proxy for the full Gaussian upper confidence bound. This construction ensures that the selected pairs are those for which the model is both uncertain and permits the largest plausible quality difference, thus operationalizing the information-seeking principle underlying DLH.

4. Algorithmic Implementation

DeltaUCB is deployed within the selection loop for each prompt in the ActiveUltraFeedback pipeline. The operational steps, with variables as defined above, are:

$\hat{r}_j = r_\phi(x, y_j)$5

For each prompt, $y_1, ..., y_m$6 diverse candidate responses are generated. Their reward point estimates and uncertainties are computed by an ENN with 20 shallow MLP heads attached to a frozen LLM backbone, each using a hidden size of 128 and two layers. The selected pair $y_1, ..., y_m$7 is then annotated, and the data is aggregated into a replay buffer used to iteratively update the reward model (Melikidze et al., 10 Mar 2026).

5. Hyperparameters and Practical Settings

Parameter	Default/Typical Value	Functional Role
$y_1, ..., y_m$8 (candidate pool)	30, drawn from three prompting "principles"	Diversity and informativeness per prompt
ENN heads	$y_1, ..., y_m$9 shallow MLP heads	Reward/uncertainty estimation ensemble
Hidden size, layers	128, 2	MLP architecture for ENN heads
Uncertainty scaling $y_j$0	$y_j$1 (e.g., $y_j$2 DPO, $y_j$3 reward modeling)	Scales width of acquisition bound
Replay buffer factor $y_j$4	1000	Controls mixing of past comparisons in updates
Annotation granularity	1–5 on four aspects, combined by softmax over logit probabilities	LLM judge supervision mechanism

DeltaUCB requires only two annotations per prompt to maximize information gain, in contrast to methods such as MaxMin that may need up to $y_j$5 annotations per prompt, enabling substantial annotation cost savings.

6. Empirical Performance and Benchmarking

DeltaUCB consistently delivers high sample efficiency and robust downstream accuracy across a range of RLHF settings. On four benchmarks (GSM8K, IFEval, TruthfulQA, AlpacaEval-2), it yields a mean relative improvement of $y_j$6 over the base LLM (with DRTS at $y_j$7, DeltaQwen at $y_j$8, and Random at $y_j$9). In reward model evaluation on RewardBench 2, DeltaUCB achieves $\hat{r}_j = r_\phi(x, y_j)$0 improvement, outperforming all other active and static baselines.

Notably, fine-tuning on only $\hat{r}_j = r_\phi(x, y_j)$1–$\hat{r}_j = r_\phi(x, y_j)$2 preference pairs from DeltaUCB matches or exceeds the performance of models trained on $\hat{r}_j = r_\phi(x, y_j)$3–$\hat{r}_j = r_\phi(x, y_j)$4 pairs drawn from Random or static heuristics, reflecting a roughly six-fold increase in sample efficiency. This efficiency holds across multiple prompt sets (UltraFeedback, Skywork, Tulu 3) and diverse RLHF objectives (DPO, IPO, SimPO), and positions DeltaUCB among the top two active selection methods, outperforming all passive strategies.

DeltaUCB’s focus on preference pairs with large, plausible quality gaps yields stronger fine-tuning gradients than classic dueling bandit approaches (e.g., DTS, InfoMax, MaxMinLCB), which often prioritize alternative exploration-exploitation balances and thus may underexploit large gaps (Melikidze et al., 10 Mar 2026).

7. Comparative Analysis and Implications

DeltaUCB’s distinctive feature is its integration of Gaussian-style uncertainty bounds with the Bradley–Terry preference model, enabling optimistic acquisition of highly informative pairs. While previous dueling bandit approaches have emphasized regret minimization or reducing global model uncertainty, DeltaUCB instead targets maximal training signal for reward learning. This distinction underlies its superior annotation efficiency and fine-tuning efficacy.

A plausible implication is that RLHF pipelines seeking maximal downstream generalization should prioritize acquisition objectives that directly encode the informativeness of large-gap pairs, as instantiated by DeltaUCB. Its deterministic nature and low annotation requirements further recommend it for practical scale-up in both general and domain-specialist RLHF settings (Melikidze et al., 10 Mar 2026).