RoleRMBench: Evaluating Role-Play Dialogue Models

• RoleRMBench is a benchmark that systematically evaluates reward models in persona-grounded role-play dialogue systems using a multi-dimensional framework.
• It employs rigorous pairwise annotation and a fine-grained capability taxonomy—including narrative, scene transition, and role consistency—to capture expert human judgment on dialogue quality.
• Empirical findings show that RoleRM, trained with a FULL-pair strategy on RoleRMBench, achieves over 25% relative improvement in pairwise accuracy compared to baselines.

RoleRMBench defines a systematic evaluation suite and methodology for assessing reward models in profile-based, role-playing dialogue systems. Its design addresses the particular challenges of aligning LLMs to nuanced, persona-grounded human preferences in open-ended conversational settings, distinguishing itself through multi-dimensional coverage, rigorous annotation, and fine-grained comparative protocols (Ding et al., 11 Dec 2025).

1. Motivation and Scope

RoleRMBench was introduced to close a persistent gap in reward model (RM) evaluation for dialogue: traditional reward modeling excels in objective or safety-driven domains, but fails to capture subtle, subjective qualities essential to immersive, in-character role play. While many datasets for role play emphasize persona maintenance or basic instruction-following, RoleRMBench treats the full subjective spectrum—narrative flow, tone, engagement, coherence, and more—as first-class targets for model alignment. The benchmark is public, multi-dimensional, and specifically targeted for systematically measuring how well an RM replicates expert human judgment regarding what constitutes “a better” continuation in persona-grounded conversation (Ding et al., 11 Dec 2025).

2. Fine-Grained Capability Taxonomy

RoleRMBench evaluates seven fundamental capabilities in profile-based role-play, split into a comprehensive narrative cluster and four additional axes rooted in interactive quality and safety:

Cluster/Dimension	Focus Area	Description
Narrative (Nar)	Story management	Introduction, Progression, Stitching of new/ongoing plot lines
Scene Transition (Scn)	Temporal/spatial/causal flow	Seamless logical transitions between scenes
Role Consistency (Con)	Persona fidelity	Maintenance of character identity, mannerisms, tone
Instruction Following (IF)	Command execution	In-character response to valid/invalid user commands
Safety (Saf)	Safe boundaries	Policy- and character-consistent adherence to safety rules
Multi-turn Coherence (MT)	Contextual continuity	Preservation of context/memory across dialogue turns
Attractiveness (Att)	User engagement	Vivid, emotionally rich, and dynamic language usage

Each benchmark instance consists of a prompt, persona specification, and two conversational continuations from different systems (or system/human), provided in identical context with only the assistant’s latest turn differing. Expert annotators indicate which continuation is strictly superior along at least one dimension, ensuring a clear, unambiguous ground-truth signal for each comparative pair (Ding et al., 11 Dec 2025).

3. Dataset Construction and Annotation Protocol

RoleRMBench integrates and standardizes data from several extensive human-annotated corpora: CoSER (17,966 characters), RoleMRC (10.2K roles), CharacterBench, and CharacterEval. Only their test and validation splits are used. Annotation involves three independent NLP-expert annotators per instance, applying precise operational criteria for each dimension. Final selection mandates that the “better” candidate outperforms the “worse” in at least one (never inferior in any) of the benchmark’s target capabilities.

All pairwise comparisons use a strict protocol:

• Each prompt–persona–context combination yields five independently sampled responses.
• Annotators produce a continuous “better–worse” ordering, not just discrete pass/fail or binary signals.
• Pairs for reward model supervision are constructed using three strategies: Neighbor (NEB), Best/Worst (BW), and Full Permutation (FULL), with the latter supporting densest preference extraction (Ding et al., 11 Dec 2025).

4. Evaluation Metrics and Modeling Framework

The benchmark defines reward modeling success as pairwise accuracy: for each triplet (x, y₊, y₋), the RM rθ “wins” if rθ(x, y₊) > rθ(x, y₋). Aggregate accuracy across all tasks is reported as:

$$Acc = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[ r_\theta(x_i, y_{i,+}) > r_\theta(x_i, y_{i,-}) ]$$

Training is governed by the standard Bradley–Terry loss using the model’s scalar scores:

$$P(y_0 \succ y_1 \mid x) = \sigma \left( r_\theta(x, y_0) - r_\theta(x, y_1) \right)$$

$$\mathcal{L}_{BT} = -\mathbb{E}_{(x, y_0, y_1, Y) \sim D}[ \log P(Y \mid x, y_0, y_1) ]$$

RoleRMBench further formalizes “continuous implicit preference” (CIP), where annotation seeks to yield a consistent, dense preference landscape by leveraging all possible pairs from in-context expert rankings. This methodology is essential for capturing the granularity of subjective conversational quality (Ding et al., 11 Dec 2025).

5. Model Training, Structuring Strategies, and Main Results

RoleRM, the baseline model evaluated and optimized on RoleRMBench, utilizes Llama-3.1-8B-Instruct as a backbone, with a scalar reward head atop the reply’s final hidden state. The training corpus comprises 205K human-labeled dialogue pairs (from 35K newly annotated multi-turn sessions), balanced and augmented with further open-domain preference data, while ensuring test/validation disjointness.

Empirical results demonstrate:

• RoleRM achieves 88.3% average pairwise accuracy across all sub-tasks, outperforming the best open-source baseline (70.6%) and proprietary models (GPT-4o at 69.1%).
• On narrative dimensions, RoleRM’s gain exceeds +28.8% over open-source RMs.
• Instruction following and scene transition accuracies reach 94.0% and 90.9%, respectively, with consistent improvements across multi-turn coherence and attractiveness.
• Training with FULL-pair strategy is crucial—NEB yields weak gradients and poor convergence, BW improves, but FULL delivers the most stable and effective optimization, especially with filtering for high-consistency pairs.

Capability	Baseline	RoleRM	Δ rel. %
Narrative	70.4	90.7	+28.8%
MT Coherence	71.4	82.5	+15.6%
Consistency	70.4	80.3	+14.0%
IF	76.0	94.0	+23.7%
Scn	75.8	90.9	+19.9%
Safety	76.3	91.5	+19.9%
Attractiveness	77.9	88.2	+13.3%
Average	70.6	88.3	+25.1%

(Ding et al., 11 Dec 2025)

6. Significance, Implications, and Limitations

The central findings of RoleRMBench are:

• General-purpose RMs exhibit persistent, often severe misalignment with human evaluators on narrative, style, and engagement—even when excelling on factual or safety tests.
• Continuous, consistent pairwise annotation is essential for capturing the full spectrum of subjective conversational quality. Simple binary distinctions are insufficient.
• High annotation consistency, enforced via multi-expert voting and style normalization, dramatically reduces label noise (from 14.2% disagreement to 2.8%).
• The effectiveness of RoleRM, when trained on RoleRMBench with FULL-pair/CIP supervision, establishes a new standard for subjective alignment in human-centered dialogue systems.

Known limitations are primarily scale-related: RoleRM uses an 8B backbone, and benchmark growth is ongoing to include more diverse and emergent role-play scenarios. A plausible implication is that scaling to larger architectures and broader annotation could further increase alignment fidelity and generalization (Ding et al., 11 Dec 2025).

7. Position Within the Role-Play Dialogue Benchmark Ecosystem

RoleRMBench directly addresses the reward modeling gap left by benchmarks centered on persona maintenance, instruction following, or scenario fidelity alone (e.g., RoleMRC (Lu et al., 17 Feb 2025), RMTBench (Xiang et al., 27 Jul 2025), FURINA-Bench (Wu et al., 8 Oct 2025)). Its unique multi-dimensional, pairwise-judgment structure provides a higher-resolution measure of alignment with nuanced, persona-driven user preferences. As other benchmarks continue to expand coverage (e.g., FURINA’s trade-off between RP performance and hallucination), RoleRMBench offers the crucial evaluation infrastructure for optimizing RMs on dimensions that underpin subjective user satisfaction and engagement in LLM-driven conversational agents (Wu et al., 8 Oct 2025, Lu et al., 17 Feb 2025, Xiang et al., 27 Jul 2025).