ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Abstract: Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning LLMs, yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains. To address this, we introduce ACTIVEULTRAFEEDBACK, a modular active learning pipeline that leverages uncertainty estimates to dynamically identify the most informative responses for annotation. Our pipeline facilitates the systematic evaluation of standard response selection methods alongside DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB, two novel methods prioritizing response pairs with large predicted quality gaps, leveraging recent results showing that such pairs provide good signals for fine-tuning. Our experiments demonstrate that ACTIVEULTRAFEEDBACK yields high-quality datasets that lead to significant improvements in downstream performance, notably achieving comparable or superior results with as little as one-sixth of the annotated data relative to static baselines. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback.

• The paper introduces an active learning pipeline that treats preference collection as a contextual dueling bandit problem and uses uncertainty-aware reward models to select two informative responses per prompt.
• DRTS and DeltaUCB target large predicted quality gaps, enabling models trained on 5,000–10,000 actively selected samples to outperform systems trained on 60,000 statically or poorly selected comparisons.
• The results show that contrastive data quality matters more than regret minimization for DPO-style training, while computational cost, reliance on LLM judges, and evaluation variance remain important limitations.

ActiveUltraFeedback addresses a persistent bottleneck in RLHF: preference annotation is expensive, yet standard synthetic data pipelines such as UltraFeedback, Magpie, and Nectar select response pairs with static heuristics that ignore model uncertainty and consume budget on uninformative comparisons (2603.09692). The paper frames preference data collection as a contextual dueling bandit problem—prompts are contexts, candidate responses are arms—and introduces a modular active learning pipeline that iteratively trains an uncertainty-aware reward model to decide which two responses per prompt should be annotated.

Pipeline design

The pipeline operates in five repeated steps per batch of prompts: (1) generate $m$ candidate responses per prompt from a pool of 30 open-weight LLMs spanning 12 families (0.5B–671B parameters), with randomly sampled guiding principles for diversity; (2) predict rewards with confidence bounds using an Epistemic Neural Network (ENN)—an ensemble of shallow MLP heads over a frozen Skywork Reward V2 Qwen3 4B backbone; (3) select a response pair via an acquisition function; (4) annotate the pair with an LLM judge; and (5) retrain the ENN on all collected preferences. The judge scores each response on four aspects (truthfulness, instruction following, honesty, helpfulness) on a 1–5 scale, extracting continuous scores by softmax over digit-token logits rather than discrete generation. This probabilistic scoring achieves a 76.70% win rate on UltraFeedback prompts versus 75.36% for discrete generation, eliminates ties entirely, produces zero parse errors, and runs roughly 15× faster than reasoning-enabled configurations—an important ablation showing that enabling chain-of-thought reasoning in the judge actually degrades label quality through score saturation.

Acquisition functions

The paper benchmarks four passive heuristics (Random, MaxMin, UltraFeedback's best-of-4 heuristic, and DeltaQwen from the Delta Learning Hypothesis), three dueling bandit methods (InfoMax, Double Thompson Sampling, MaxMinLCB), and two novel "active delta learning" methods motivated by the Delta Learning Hypothesis:

• DRTS draws two independent Thompson samples from the reward posterior and selects the responses maximizing and minimizing their samples, explicitly targeting large predicted quality gaps while retaining stochastic exploration.
• DeltaUCB deterministically selects the pair maximizing the upper confidence bound on the pairwise preference probability $\overline{p}_\phi(y_j \succ y_{j'})$, steering toward optimistically large quality gaps.

Both require only two annotations per prompt, unlike MaxMin (which annotates the full candidate set) or UltraFeedback's heuristic (four annotations).

Main results

The central empirical finding is a sharp misalignment between classical dueling bandit objectives and preference data generation. DTS and MaxMinLCB succeed at their theoretical goal—identifying high-quality responses—but produce datasets whose chosen and rejected responses have nearly identical mean judge scores (e.g., 4.855 vs. 4.584 for DTS), yielding datasets that underperform even random sampling on downstream DPO evaluation. Regret minimization is simply the wrong objective when the goal is contrastive training signal. By contrast, DRTS and DeltaUCB mirror the high-gap distributions of the expensive MaxMin baseline at a fraction of its annotation cost.

On UltraFeedback prompts with DPO fine-tuning from Tulu 3 8B SFT, the headline numbers are substantial:

Method	Mean downstream delta	RewardBench 2 delta
Original UltraFeedback	+0.037	+0.295
Random	+0.046	+0.278
DeltaQwen	+0.137	+0.100
DRTS	+0.127	+0.312
DeltaUCB	+0.120	+0.339

DRTS and DeltaUCB are the only practical methods that improve both reward modeling and fine-tuning simultaneously. The sample-efficiency results are the strongest claim in the paper: models fine-tuned on only 5,000–10,000 actively selected samples outperform those trained on 60,000 samples from static or dueling-bandit datasets, and surpass training on the original UltraFeedback dataset—consistent with the abstract's claim of comparable performance with as little as one-sixth of the annotated data. Reward modeling saturates more gradually, requiring roughly 40,000 samples to match full-dataset performance, and Random performs surprisingly well there, suggesting diversity matters more than quality deltas for reward modeling specifically.

Generalization checks support these conclusions across three additional prompt sources (Skywork 80k, a combined 140k set, and the Tulu 3 272k mixture) and across IPO and SimPO optimization algorithms. Notably, DeltaQwen—which slightly edges out DRTS on mean downstream score under DPO—collapses under IPO and SimPO and fails badly on RewardBench 2 (+0.085 to +0.153), confirming that intra-family size pairing is confined to specific setups and training distributions. One caveat worth noting: AlpacaEval 2 deltas dominate the reported means (deltas an order of magnitude larger than other benchmarks), so the aggregate rankings are disproportionately driven by one benchmark; the appendix provides trajectories without it, where trends remain favorable but less dramatic.

Limitations and open questions

The paper concedes several limitations directly. First, the pipeline incurs substantial computational cost—roughly 1,200 GPU hours for generation and pre-computed annotations on UltraFeedback alone, and approximately 200,000 GPU hours for all experiments—because every prompt requires responses from many LLMs; selecting which models to query, rather than selecting among already-generated responses, is left open. Second, all experiments use an LLM-as-a-judge (Qwen 3 235B A22B) rather than human annotators, so the reported sample-efficiency gains concern AI feedback; extension to human annotation loops remains untested. Third, the judge itself lags dedicated reward models on RewardBench 2 (mean 0.744 vs. 0.839 for Skywork-Reward-V2-Llama-3.1-8B), yet early experiments showed that substituting the stronger reward model's scores degraded fine-tuned models—a tension the authors resolve empirically but do not fully explain. Fourth, prompt selection is not part of the active learning loop, and no explicit diversity constraints are imposed beyond the model pool composition. Finally, the stability analysis shows sampling-based methods (DRTS) have higher variance on RewardBench 2 (std. dev. 0.025) than deterministic ones (0.011), and the RewardBench 2 "Ties" metric is highly sensitive to initialization (std. dev. 0.072), indicating some fragility in reward-model evaluations.

Conclusion

ActiveUltraFeedback contributes a unified, open-source platform for systematically comparing response pair selection strategies across reward modeling and preference fine-tuning, and demonstrates that prioritizing large predicted quality deltas—via DRTS and DeltaUCB—yields datasets that match or exceed static baselines with a small fraction of the annotations, while remaining agnostic to the downstream optimization algorithm. Its most consequential negative result is equally clear: regret-minimizing dueling bandit methods, despite their theoretical guarantees, are poorly suited to preference data generation because they suppress the quality contrasts that DPO-style objectives require.