RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences

Abstract: Reinforcement Learning from Human Feedback has become the standard paradigm for LLM alignment, where reward models directly determine alignment effectiveness. In this work, we focus on how to evaluate the generalizability of reward models. By "generalizability", we mean the ability of RMs to correctly rank responses to align with diverse user preferences. However, existing reward model benchmarks are typically designed around a universal preference, failing to assess this generalization. To address this critical gap, we introduce RMGAP, a benchmark comprising 1,097 instances across Chat, Writing, Reasoning, and Safety domains. Since different users exhibit diverse preferences for the same task, we first generate four distinct responses with different linguistic profiles for each collected prompt. However, the original prompt set lacks the specificity to convey different preferences. We therefore construct tailored prompts by contrasting these candidates and designing scenarios in which one response becomes the uniquely appropriate choice. Moreover, we observe that users often express the same preference using different phrasings, and thus extend each prompt with two paraphrased variants. Our evaluation of 24 state-of-the-art RMs reveals their substantial limitations: even the best RM achieves only 49.27% Best-of-N accuracy, highlighting considerable room for improvement in reward model generalization. Related data and code are available at https://github.com/nanzhi84/RMGAP.

• The paper introduces RMGAP, a benchmark designed to evaluate reward models' ability to generalize across diverse user preferences and paraphrasing variations.
• It employs a four-stage pipeline—including prompt selection, stylistic response generation, tailored prompts, and paraphrase extension—to create a robust evaluation dataset.
• Experimental results highlight trade-offs between pairwise accuracy and consistency, emphasizing limitations in current RLHF alignment methods and the need for robust solutions.

Benchmarking Reward Model Generalization across Diverse Preferences with RMGAP

Motivation and Background

As LLMs are increasingly deployed in practical applications, the assumption of a universal user preference in RLHF alignment is untenable. Users exhibit heterogeneous and context-dependent preferences for response style, which necessitates reward models (RMs) that can generalize across this diversity. Existing benchmarks focus on ranking "good" versus "bad" responses according to fixed criteria, neglecting the challenge of adapting to varied user demands. The RMGAP benchmark directly addresses this gap by systematically probing the generalizability of RMs along two axes: sensitivity to explicit diverse preference instructions and consistency across paraphrased expressions of those instructions.

Figure 1: RMGAP contrasts universal preference benchmarks with its own paradigm, which evaluates generalization to specific, distinct user preferences.

RMGAP Construction Pipeline

RMGAP comprises 1,097 instances spanning Chat, Writing, Reasoning, and Safety. Each instance is built via a structured four-stage pipeline:

1. Prompt Selection and Filtering using open-source datasets, removing persona instructions and non-English prompts through regex and embedding-based deduplication.
2. Stylistic Response Generation: For each prompt, four responses are generated along five orthogonal linguistic axes (Formality, Conciseness, Technicality, Objectivity, Structural Coherence). The stylistic combinations are selected with minimum Manhattan distance constraints to guarantee diversity.
3. Tailored Prompt Construction: For each response, a bespoke prompt is synthesized that implicitly favors the corresponding stylistic profile, avoiding explicit style labels and relying on realistic scenario-based bias.
4. Paraphrase Extension: Each tailored prompt is augmented with two paraphrased variants to probe the ranking stability of RMs under natural linguistic variation.

This yields, per instance, 4 responses and 12 prompts, facilitating a comprehensive and robust evaluation of reward model generalization.

Figure 2: RMGAP's construction pipeline leverages seed prompt selection, response generation across style dimensions, tailored prompting, and paraphrase variation.

Evaluation Paradigms and Metrics

The benchmark evaluates 24 state-of-the-art RMs, including Scalar, Generative, and Implicit (DPO-trained) models, spanning 7B to 235B parameters. Evaluation metrics are strategically chosen:

• Pairwise Accuracy: Measures the RM's discrimination between a preferred (prompt-aligned) and a rejected response.
• Best-of-N (BoN) Accuracy: Measures RM performance in selecting the optimal response from four candidates, reflecting practical use cases in inference-time selection.
• Ranking Consistency: Quantifies robustness to paraphrased instruction variants, detecting brittleness to surface linguistic cues.

These metrics are orthogonal, with BoN being strictly more difficult than pairwise, and consistency capturing stability rather than correctness.

Experimental Results

Generalization Performance

Scalar reward models dominate across metrics, with Skywork-Reward-V2-Qwen3-8B achieving 69.97% pairwise accuracy and 49.16% BoN accuracy—modest but superior to all generative and implicit baselines. DPO-trained policies, even with identical model backbones, underperform explicit RMs across all metrics. Notably, BoN accuracy remains low for all models, indicating a fundamental challenge: even state-of-the-art RMs reliably select the best among multiple candidates only about half the time.

Trade-offs and Failure Modes

A critical finding is the trade-off between accuracy and consistency, especially for generative listwise ranking approaches, which exhibit high pairwise accuracy but extreme instability across paraphrases (e.g., Listwise scoring drops to below 2% consistency). Scalar RMs maintain the best balance, but even they exhibit only moderate consistency (58%).

Figure 3: Empirical relationship between accuracy and consistency for reward models on RMGAP, demonstrating distinct failure modes and trade-offs.

Scaling and Model Capacity

Scaling generative reward models (e.g., Qwen3-235B-A22B-2507) does not substantially resolve brittleness, as improvements in accuracy do not yield commensurate gains in consistency. The strongest scalar RM (Skywork-Reward-V2-Qwen3-8B) matches or exceeds performance of much larger generative RMs, suggesting current limitations are not simply a matter of insufficient model capacity, but rather of intrinsic to the scoring mechanisms.

Domain-specific Challenges

Chat and Safety domains yield the highest RM performance, while Writing introduces severe degradation for generative RMs—likely due to longer, more varied generations. Reasoning remains challenging across architectures, reflecting the constrained stylistic space and subtle variation in mathematical and coding responses.

Implications and Future Directions

The results expose fundamental limits of current reward model training and evaluation paradigms. Even best models fail to generalize robustly across explicitly different user preferences, and ranking stability across paraphrases remains elusive. This highlights several implications:

• RLHF policy training pipelines should incorporate explicit preference generalization data and metrics to avoid overfitting to universal preference distributions.
• Model and loss architecture: There is evidence favoring scalar reward heads for evaluator reliability; generative approaches (especially listwise) are vulnerable to prompt sensitivity and scaling does not resolve brittleness.
• Benchmark design: Robust alignment requires datasets and metrics that cover real-world user diversity and surface variation, as typified by RMGAP.
• Future developments: Research should prioritize scalable, robust reward modeling protocols that optimize for both accuracy and consistency on benchmarks like RMGAP, potentially integrating richer human-authored data, cross-lingual generalization, and continuous preference spectra.

Conclusion

RMGAP fills a critical benchmarking gap by providing a systematic, large-scale probe of reward model generalization across diverse preferences and linguistic expressions. Evaluations reveal substantial limitations in current RMs, indicating urgent need for research into more generalizable, robust reward modeling strategies for LLM alignment. The benchmark advances the field's understanding of preference sensitivity, prompt robustness, and architecture trade-offs, and offers practical guidance for improving evaluator reliability in RLHF workflows.