PrefCleanBench: Data Cleaning for LLM Alignment

• PrefCleanBench is a benchmark suite and evaluation protocol for cleaning noisy human preference data to improve large language model alignment outcomes.
• It distinguishes high-quality from low-quality data using LLM judgments, reward-model scores, and heuristic filtering methods.
• Empirical findings show that data removal outperforms label flipping and committee-based voting enhances robustness across datasets and models.

Searching arXiv for the benchmark paper and closely related alignment papers for citation support. arxiv_search.query({"search_query":"id:(Yeh et al., 28 Sep 2025)", "max_results": 5}) Searching for the PrefCleanBench paper by arXiv identifier. PrefCleanBench is a benchmark suite and evaluation protocol for preference data cleaning in the context of LLM alignment. Introduced in "Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment" (Yeh et al., 28 Sep 2025), it treats data cleaning as a first-class component of the alignment pipeline rather than as an informal preprocessing step. The benchmark evaluates whether automated cleaning of noisy human preference data improves downstream alignment, and whether any such improvement generalizes across datasets, model architectures, and preference optimization algorithms. Its core object is the pairwise preference datum $(x, y_c, y_r)$, where $y_c$ is the chosen response and $y_r$ is the rejected response, and its central concern is the effect of cleaning these data before reward modeling or preference optimization.

1. Conceptual framing and problem formulation

PrefCleanBench is motivated by the observation that modern alignment pipelines rely heavily on preference-based learning, yet the underlying preference data are often noisy, inconsistent, or low quality (Yeh et al., 28 Sep 2025). In the benchmark’s formulation, a preference dataset is

$$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$

with each triplet encoding a comparison in which the chosen response is preferred to the rejected response. Such data are used to train reward models $r_\theta(x,y)$ and to optimize policies through RLHF or offline preference optimization methods including DPO, IPO, SLiC, KTO, AOT, ORPO, and rDPO (Yeh et al., 28 Sep 2025).

The paper formalizes noise by distinguishing high-quality and low-quality preference distributions. It defines $\mathbb{P}_+$ as a distribution over reliable preference data, where prompts, responses, and labels are all of acceptable quality, and $\mathbb{P}_-$ as a distribution over low-quality data, where any of these components may be deficient. The observed data are modeled as a mixture,

$$\mathbb{P} = (1-\alpha)\,\mathbb{P}_+ + \alpha\,\mathbb{P}_-,$$

and preference data cleaning is the task of transforming a noisy dataset $\mathcal{D} \sim \mathbb{P}$ into a cleaned dataset $\mathcal{D}'$ that contains only data from $y_c$0, either by removing unreliable examples or by correcting their labels (Yeh et al., 28 Sep 2025).

This formulation places label noise, content noise, and ambiguity under a common framework. PrefCleanBench therefore does not define cleaning narrowly as mislabel correction. Instead, it evaluates cleaning methods by their downstream effect on aligned models, reflecting the fact that vague prompts, harmful or low-quality responses, and meaningless comparisons can all degrade reward models and preference-optimized policies even when a literal label flip would be insufficient.

2. Benchmark scope, datasets, and notation

PrefCleanBench consists of four preference datasets, a taxonomy of 13 cleaning methods, a standardized training pipeline, and a metric suite for assessing both alignment performance and generalizability (Yeh et al., 28 Sep 2025). All datasets are cast into pairwise preference form $y_c$1, with train and test splits.

Dataset	Train	Test
Anthropic-HH	160,800	8,552
UltraFeedback	54,825	6,092
PKU-SafeRLHF	72,996	8,109
HelpSteer2	8,677	448

Anthropic-HH is described as containing human helpfulness/harmlessness preferences; UltraFeedback as LLM-scored question-answering and instruction preferences; PKU-SafeRLHF as safety-focused preferences; and HelpSteer2 as multi-attribute helpfulness data converted into binarized pairs (Yeh et al., 28 Sep 2025). UltraFeedback and HelpSteer2 originally involve Likert-style ratings, but the benchmark uses processed, binarized variants.

For reward modeling and preference optimization, PrefCleanBench adopts a Bradley–Terry-style representation of pairwise preference probabilities:

$y_c$2

where $y_c$3 is a reward model and $y_c$4 is the logistic function (Yeh et al., 28 Sep 2025). Cleaning methods are designed to identify cases in which this relationship is unreliable, whether because the nominally chosen response is not actually better or because the comparison itself is uninformative.

A notable design choice is that the benchmark evaluates cleaning exclusively on training data; test splits are never cleaned. This preserves a common evaluation ground across methods and ensures that differences in alignment performance arise from the treatment of the training preferences rather than from selective manipulation of the evaluation distribution (Yeh et al., 28 Sep 2025).

3. Taxonomy of cleaning methods

PrefCleanBench evaluates 13 cleaning methods grouped into three families: LLM-as-a-Judge methods, reward-model-based methods, and heuristic-based methods (Yeh et al., 28 Sep 2025). Each method applies one of two treatments to flagged examples: removal, indicated by the suffix -R, or label flipping, indicated by -F.

3.1 LLM-as-a-Judge methods

The benchmark uses GPT-4o-2024-05-13 as a data quality judge. Given a prompt $y_c$5, a chosen response $y_c$6, and a rejected response $y_c$7, the judge scores the responses on criteria such as helpfulness, safety, and detail. A pair is flagged as unreliable if the judge assigns a higher score to the nominally rejected response (Yeh et al., 28 Sep 2025). To mitigate positional bias, the responses are presented in random order.

Two variants are defined. LLM-Judge-R removes the flagged pairs. LLM-Judge-F flips their labels, so that the originally rejected response becomes the chosen one (Yeh et al., 28 Sep 2025).

3.2 Reward-model-based methods

The first reward-model-based family is reward gap, denoted RwGap. For a preference datum $y_c$8, the benchmark defines

$y_c$9

If this quantity is small or negative, the label may be wrong or uninformative. The protocol trains eight reward models on the target dataset using DPO-style training with different random seeds, computes and averages the reward gaps, and treats the bottom $y_r$0 of examples as unreliable for $y_r$1 (Yeh et al., 28 Sep 2025). RwGap-R removes these pairs, and RwGap-F flips their labels.

The second family is reward-model voting, or RM voting. Instead of training reward models on the target dataset, this method uses a committee of six strong pre-trained reward models chosen from the RewardBench leaderboard, spanning architectures such as InfoRM, QRM, GRM, and ArmoRM (Yeh et al., 28 Sep 2025). Each model votes that a label is incorrect if it assigns higher reward to $y_r$2 than to $y_r$3. Two aggregation rules are used: VoteAll, which flags a pair only if all models vote incorrect, and VoteMaj, which flags a pair if a majority vote incorrect. Each rule has removal and flipping variants: VoteAll-R, VoteAll-F, VoteMaj-R, and VoteMaj-F (Yeh et al., 28 Sep 2025).

3.3 Heuristic-based methods

The heuristic family includes prompt tagging and instruction-following difficulty. For prompt tagging, PrefCleanBench uses InsTagger to assign semantic tags to prompts and adopts two strategies from InsTag (Yeh et al., 28 Sep 2025). Tag-Cmp treats prompts with more tags as more complex and keeps the top 6k prompts with highest complexity scores, discarding the rest. Tag-Div selects prompts to maximize tag diversity in the retained subset and filters prompts whose tags are already covered.

Instruction Following Difficulty is defined for a prompt–response pair $y_r$4 as

$y_r$5

where perplexity is measured under a base LLM, specified here as Llama3-8B (Yeh et al., 28 Sep 2025). If $y_r$6, the prompt is treated as providing no useful context for predicting the response; if the value is very small, the instruction may be too easy and therefore of limited learning value. IFD-R first removes all data with $y_r$7 and then removes an additional $y_r$8 of data with the lowest IFD scores for $y_r$9. The benchmark also defines

$$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$0

and uses it to construct IFD-Gap-R and IFD-Gap-F, which remove or flip the bottom $$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$1 of pairs with the smallest gaps (Yeh et al., 28 Sep 2025).

Taken together, these families operationalize different conceptions of unreliability: explicit reevaluation by a strong LLM, inconsistency with reward-model scores, and proxy filtering based on prompt complexity, diversity, or conditional predictability.

4. Training pipeline and evaluation protocol

PrefCleanBench implements an end-to-end alignment pipeline consisting of cleaning, model training, and evaluation (Yeh et al., 28 Sep 2025). For each dataset $$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$2 and cleaning method $$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$3, the method is applied to the training split to mark unreliable pairs and produce a cleaned dataset $$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$4 through removal or label flipping. The test set remains untouched.

Training follows a two-stage procedure. In the supervised fine-tuning stage, a base model such as Llama3-8B is fine-tuned on prompt–chosen-response pairs $$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$5 for both the original and cleaned datasets. The reported hyperparameters are 1 epoch, learning rate $$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$6, batch size 96, sequence length 512, and LoRA rank 0 (Yeh et al., 28 Sep 2025). In the preference optimization stage, the SFT model is further fine-tuned on preference pairs $$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$7. The main experiments use DPO with 1 epoch, learning rate $$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$8, batch size 64, and LoRA rank 16. Additional ablations use CPO, SLiC, KTO, AOT, IPO, rDPO, and ORPO (Yeh et al., 28 Sep 2025).

The benchmark compares a clean-trained policy $$\mathcal{D} = \{ (x^{(i)}, y_c^{(i)}, y_r^{(i)}) \}_{i=1}^n,$$9 against a baseline policy $r_\theta(x,y)$0 trained on the uncleaned data. It uses two primary alignment metrics. The first is Win-tie rate (WinTie). For shared test prompts, both policies generate responses; GPT-4o judges which response is better or whether they are tied, with both response orderings presented and averaged to reduce positional bias. The metric is defined as

$r_\theta(x,y)$1

A value greater than $r_\theta(x,y)$2 indicates that the clean-trained model is at least not worse than the original on most prompts. Because of cost, the paper estimates this metric on 200 randomly sampled test prompts per setting (Yeh et al., 28 Sep 2025).

The second metric is Average gold reward (Avg. Rwd). For each model, responses on the test set are scored by a held-out high-quality reward model $r_\theta(x,y)$3, with LxzGordon/URM-LLaMa-3.1-8B used by default and QRM and OffsetBias used in robustness checks. The metric is

$r_\theta(x,y)$4

The paper additionally reports that GPT-4o judgments were validated against human annotators using Cohen’s kappa and showed substantial agreement, supporting the use of LLM-as-a-judge for WinTie (Yeh et al., 28 Sep 2025).

A defining feature of the protocol is its emphasis on generalizability. All 13 cleaning methods are evaluated across four datasets, across multiple model backbones—Qwen2.5-7B, Mistral-7B, phi-2, and Llama3.2-1B in addition to Llama3-8B—and across multiple optimization algorithms (Yeh et al., 28 Sep 2025). This design makes the benchmark a stress test of whether a cleaning strategy is robust beyond a single experimental configuration.

5. Empirical findings

The paper reports several consistent findings from the benchmark evaluation (Yeh et al., 28 Sep 2025). The most prominent is that removal usually outperforms label flipping. Across almost every identification strategy, the -R variants achieve better WinTie and Avg. Rwd than the corresponding -F variants. The benchmark gives specific examples for Llama3-8B with DPO on Anthropic-HH: VoteMaj-R reaches WinTie $r_\theta(x,y)$5 and Avg. Rwd $r_\theta(x,y)$6, whereas VoteMaj-F reaches $r_\theta(x,y)$7 and $r_\theta(x,y)$8. LLM-Judge-R also exceeds LLM-Judge-F, and RwGap-R outperforms RwGap-F on multiple datasets (Yeh et al., 28 Sep 2025).

The interpretation offered in the paper is that label flipping addresses only one failure mode, namely pure mislabeling where the better response is marked as rejected. Real low-quality data often involve poor or vague prompts, both responses being harmful or unhelpful, or comparisons with no meaningful preference signal. In such cases, flipping may preserve or worsen noise rather than resolve it. Manual inspection on HelpSteer2 supports this account: many pairs flagged by VoteMaj contain low-quality prompts such as greetings, vague requests, or underspecified tasks that induce generic or hallucinated responses (Yeh et al., 28 Sep 2025).

A second major finding is that committee-based identification tends to outperform single-judge approaches. VoteMaj-R consistently ranks near the top or best across all four datasets in both Avg. Rwd and WinTie, whereas LLM-Judge-R is less consistent (Yeh et al., 28 Sep 2025). The benchmark attributes this to the bias-reduction effect of diverse judges: when both responses are similar in quality, a single evaluator may make unstable preferences, while a committee is less likely to misclassify borderline examples systematically.

Heuristic methods can also be strong. In the main Llama3-8B plus DPO experiments, Tag-Cmp obtains the best WinTie on Anthropic-HH at $r_\theta(x,y)$9 and on PKU-SafeRLHF at $\mathbb{P}_+$0, while Tag-Div achieves strong Avg. Rwd on PKU-SafeRLHF and HelpSteer2 (Yeh et al., 28 Sep 2025). However, these benefits appear less robust across backbones. On smaller models such as phi-2 and Llama3.2-1B, Tag-Cmp sometimes fails to achieve WinTie above $\mathbb{P}_+$1 or yields lower Avg. Rwd than the uncleaned baseline, whereas VoteMaj-R consistently improves or maintains both metrics (Yeh et al., 28 Sep 2025).

The benchmark also studies interactions with optimization algorithms. VoteMaj-R and Tag-Cmp generally improve alignment relative to vanilla training across DPO, CPO, SLiC, KTO, AOT, IPO, rDPO, and ORPO, but their relative ranking varies (Yeh et al., 28 Sep 2025). VoteMaj-R tends to outperform Tag-Cmp for AOT and ORPO, which the paper describes as more distribution-aware and sensitive to noisy preference signals. Tag-Cmp sometimes does better for KTO and rDPO, which the paper describes as more robust to noise in the objective. This suggests that cleaning efficacy is partly conditioned by the inductive biases of the optimization algorithm.

Finally, PrefCleanBench quantifies a trade-off between data quantity and data quality. For threshold-based methods such as RwGap-R and IFD-Gap-R, filtering rates between 20% and 30% often yield the best balance, whereas 40% filtering frequently hurts or plateaus (Yeh et al., 28 Sep 2025). On Anthropic-HH, for example, RwGap-R reaches its highest Avg. Rwd at 30% filtering; on HelpSteer2, IFD-Gap-R reaches its highest Avg. Rwd and strong WinTie around 30%. This suggests that smaller, cleaner datasets can outperform larger, noisier ones, but only up to a point.

6. Technical interpretation and relation to preference optimization

Although PrefCleanBench does not introduce a new alignment objective, it situates cleaning within the mechanics of reward modeling and preference optimization (Yeh et al., 28 Sep 2025). Under the benchmark’s preference modeling view, reward models estimate

$\mathbb{P}_+$2

and are trained with a cross-entropy loss,

$\mathbb{P}_+$3

Reward-gap methods reuse this structure at inference time: if $\mathbb{P}_+$4 is strongly negative, the corresponding preference is treated as suspect (Yeh et al., 28 Sep 2025).

For DPO-style alignment, the benchmark presents the objective as

$\mathbb{P}_+$5

In this view, noisy preferences distort gradients by rewarding undesirable responses or by amplifying distinctions that are semantically vacuous (Yeh et al., 28 Sep 2025). Cleaning is therefore interpreted as a mechanism for reducing gradient contamination before optimization, rather than as a post hoc correction after the policy has already internalized noise.

The empirical preference for removal over flipping is consistent with this interpretation. If the latent problem is ambiguity, underspecification, or content degradation rather than a simple sign error, then removal decreases the effective noise rate whereas flipping may inject fresh errors. Likewise, the strong performance of committee methods suggests that detecting latent noise scores through multiple reward models is more stable than relying on a single evaluator.

A plausible implication is that PrefCleanBench shifts attention from objective-level robustness alone to the interaction between objective design and data curation. The paper explicitly notes that robust optimization alone is not enough and that data cleaning can materially change alignment outcomes (Yeh et al., 28 Sep 2025).

7. Modularity, limitations, and broader implications

PrefCleanBench is released as an open-source, modular toolkit at the repository specified in the paper, with implementations of all benchmarked cleaning methods (Yeh et al., 28 Sep 2025). Its architecture includes a modular cleaning API, support for plug-and-play extension with new scoring or identification functions, and a common training harness built on standard libraries including Transformers, TRL, and DeepSpeed. The implementation also documents fixed seeds for reward-model training, prompts used for LLM-as-a-judge evaluation, and hardware requirements and approximate compute or API costs, including examples such as cleaning Anthropic-HH with VoteMaj-R or RwGap-R on 4×H200 GPUs and GPT-4o cost estimates for LLM-Judge (Yeh et al., 28 Sep 2025).

The benchmark is framed as part of a broader data-centric perspective on alignment. Rather than focusing exclusively on PPO versus DPO or on model architecture, it foregrounds the quality, consistency, and preprocessing of human feedback as a major determinant of alignment outcomes (Yeh et al., 28 Sep 2025). The slogan “clean first, align later,” which appears in the title and conclusion, captures this orientation.

The paper also identifies several limitations. PrefCleanBench evaluates cleaning indirectly through downstream alignment performance because there is no ground-truth label for which preference pairs are truly noisy or low quality (Yeh et al., 28 Sep 2025). As a result, it cannot directly measure the precision or recall of noisy-example identification, and its alignment metrics inevitably conflate cleaning quality with the inductive biases of both the training algorithms and the evaluation models. The authors suggest curated preference benchmarks with partially verified labels and human-in-the-loop auditing of suspected noisy examples as future directions.

The broader impacts discussion emphasizes possible bias amplification. Automated cleaning methods may disproportionately remove minority or non-majority viewpoints, and heavy reliance on LLM judges or existing reward models may entrench their biases into both datasets and downstream policies (Yeh et al., 28 Sep 2025). This suggests that transparent, measurable cleaning is not only a performance issue but also a governance issue in responsible AI development.

Within the benchmark’s own results, VoteMaj-R emerges as the most consistently strong method, especially across datasets, architectures, and algorithms, while Tag-Cmp appears as a competitive but less robust heuristic (Yeh et al., 28 Sep 2025). More generally, PrefCleanBench establishes preference data cleaning as a distinct research object in LLM alignment: one that can be benchmarked, ablated, and generalized in its own right, rather than treated as an opaque preprocessing choice.