OWL-RolePlay: Reward Modeling in Dialogue

• OWL-RolePlay is a reward modeling framework that defines and quantifies role-play capabilities using continuous pairwise supervision.
• It decomposes dialogue into measurable dimensions such as narrative coherence, role consistency, and safety to enable fine-grained evaluation.
• Empirical results demonstrate significant improvements over traditional models, aligning role-playing outputs with expert human preferences.

OWL-RolePlay is a reward modeling and evaluation framework for profile-based role-playing dialogue systems, designed to address the challenges posed by subjective and persona-grounded assessment of LLM outputs. It systematically defines, annotates, and quantifies the fine-grained dimensions of effective role play, leveraging structured continuous preference signals and dense pairwise supervision to align model behavior with nuanced human expectations.

1. Foundations of Reward Modeling in Role-Play Dialogue

Profile-based role play in dialogue systems requires the model to impersonate a character, maintain consistent persona, and produce engaging narrative. Traditional reward models (RMs) exhibit substantial performance degradation in such settings, especially concerning narrative management and stylistic fidelity. The inability of generic RMs to capture the subtle, subjective, and dynamic attributes of role play necessitated the development of the RoleRMBench benchmark and RoleRM reward modeling method. OWL-RolePlay adopts and generalizes this approach to provide an actionable evaluation and training pipeline for subjective, human-centered conversational agents (Ding et al., 11 Dec 2025).

2. Decomposition of Role-Playing Capabilities

The measurement of "good role play" is operationalized via seven capabilities, with the first three forming a narrative cluster and the remaining four as standalone dimensions. Each capability is annotated and evaluated independently, facilitating focused model improvement and fine-grained error analysis. The dimensions are as follows:

Capability	Description
Introduction (Nar)	Setting scene, character, and tone
Progression (Nar)	Plot development with conflict, twists, suspense
Stitching (Nar)	Integrating user actions/ideas into a coherent storyline
Scene Transition (Scn)	Smooth temporal/spatial shifts without incoherence
Role Consistency (Con)	Persona fidelity (identity, tone, worldview) over time
Instruction Following (IF)	In-character command execution or graceful refusal for inappropriate instructions
Safety (Saf)	Avoiding unsafe or policy-violating content while maintaining character
Multi-turn Coherence (MT)	Sustaining logical consistency across dialogue turns
Attractiveness (Att)	Engaging, expressive, emotionally rich, and vivid language

Each evaluation instance consists of a context $x$, two candidate responses $(y_0, y_1)$, and a system persona prompt. Pairwise accuracy is the key metric, where the preference probability is modeled as: $$P(y_0 \succ y_1 \mid x) = \sigma(r_\theta(x, y_0) - r_\theta(x, y_1))$$ with $\sigma$ denoting the sigmoid function and $r_\theta$ the scalar reward head (Ding et al., 11 Dec 2025).

3. Continuous Implicit Preferences (CIP) and Annotation Protocol

In OWL-RolePlay’s framework, subjective evaluation is recast into robust, high-agreement continuous pairwise supervision. Rather than static numeric scores, annotators are tasked with ranking $N \geq 5$ response candidates for each context-persona pair, generating a local ordering (e.g., $A > B > C > D > E$). This yields a continuous preference spectrum $[0,1]$, enabling extraction of dense pairwise signals via three structuring strategies:

• Neighbor Pairs (NEB): Only adjacent comparisons, capturing incremental distinctions.
• Best/Worst Pairs (BW): Comparisons between best and others, or worst and others, prioritizing sharp differences.
• Full Permutation (FULL): All $N\cdot(N-1)/2$ pairwise orderings for maximum granularity.

The annotation pipeline entails automated candidate generation via persona-aware LLMs, expert ranking by unified guidelines, majority-vote adjudication, and exclusion of low-agreement items. Pairwise data is subsequently extracted through the chosen structuring strategy (Ding et al., 11 Dec 2025).

4. Training Objectives and Optimization

The reward model is instantiated by prefix-tuning an LLM backbone (e.g., Llama-3.1-8B-Instruct) with a scalar evaluation head and trained on curated CIP pairs. The objective follows the Bradley-Terry loss:

$$\mathcal{L}_{CIP} = -\sum_{(x, y_i, y_j) \in \mathcal{D}_{pairs}} \log \sigma(r_\theta(x, y_i) - r_\theta(x, y_j))$$

Core hyperparameter settings include batch size 256 (micro 8), learning rate $(y_0, y_1)$0, AdamW optimization, two epochs, and sequence truncation to 1024 tokens. The training corpus interleaves $(y_0, y_1)$1K human CIP pairs, $(y_0, y_1)$2K open-source role-play preferences, and $(y_0, y_1)$3K general dialogue preference pairs from curated sources (HelpSteer2, Skywork, Tulu). Majority-vote filtering and edge-case re-annotation ensure high consistency (Ding et al., 11 Dec 2025).

5. Quantitative Performance and Comparative Analysis

OWL-RolePlay, via the RoleRM model, demonstrates significant improvements over open- and closed-source baselines across all RoleRMBench dimensions, particularly in narrative and stylistic metrics. Performance is quantified as follows:

Model	Avg	Nar	MT	Con	IF	Scn	Saf	Att
Best open-source baseline	70.6%	70.4%	68.3%	67.6%	76.0%	72.7%	66.1%	75.0%
Best closed-source (GPT-5)	69.3%	68.5%	73.0%	59.9%	83.0%	68.9%	70.3%	65.4%
RoleRM (FULL, re-anno)	88.3%	90.7%	82.5%	80.3%	94.0%	90.9%	91.5%	88.2%

RoleRM achieves +17.7 percentage points over the strongest open-source model (average), and >24% relative improvement on narrative, scene transition, and attractiveness dimensions. Closed-source commercial models do not exceed 70% average, with the weakest performance on nuanced narrative subtasks ($(y_0, y_1)$4). This illustrates the inability of general-purpose RMs to track evolving storylines, attribute engagement, and maintain character consistency in dynamic contexts (Ding et al., 11 Dec 2025).

6. Failure Modes of Generic Reward Models and Remediation Strategies

Generic reward models are deficient in three critical areas:

• Narrative Coherence: Unable to capture storyline evolution, with open-source baselines often scoring below 65%.
• Role Consistency: Confusion of persona under context shifts leads to degraded identity fidelity (Con $(y_0, y_1)$5 60%).
• Attractiveness/Engagement: Emotional tone and stylistic expressiveness are poorly quantified, with observed variances up to 10–15 points.

RoleRM and OWL-RolePlay overcome these deficits by leveraging explicit multi-dimensional annotation, continuous pairwise supervision (CIP), majority-vote filtering, targeted re-annotation, and dense permutation-based supervision (FULL plus BW-augmentation), yielding more stable and interpretable reward gradients (Ding et al., 11 Dec 2025).

7. Implications and Recommendations for Role-Play Alignment

RoleRMBench and RoleRM establish that subjective, persona-grounded evaluation in dialogue cannot be delegated to generic, single-dimensional reward models. Instead, CIP-based continuous preference modeling and rigorous multi-attribute annotation are necessary for high-fidelity alignment. For effective OWL-RolePlay reward modeling:

• Define a set of orthogonal, operational role-play capabilities.
• For each context, collect $(y_0, y_1)$6 candidates and annotate via expert end-to-end ranking.
• Employ FULL and BW structuring to extract dense, reliable pairs.
• Train a prefix-tuned LLM reward head on the Bradley-Terry objective.
• Monitor agreement rates and re-annotate ambiguous cases.

This protocol produces reward models that achieve approximately 90% pairwise alignment with expert judgments, generalize across LLM architectures, and support actionable, fine-grained RL optimization of role-playing agents (Ding et al., 11 Dec 2025). This suggests that future OWL-RolePlay research will benefit from scalable, high-consistency annotation and the continued refinement of multi-dimensional, persona-anchored reward modeling.