Verifiable Role-Awareness Reward (VRAR)

• The paper presents VRAR as a deterministic reward mechanism that enforces role-specific key attributes and structured response formats.
• It introduces two mining strategies—STV for exact keyword matching and MTDP for dynamic parsing of role-relevant features.
• Empirical studies show VRAR significantly improves metrics like role consistency (SBK) and conversational memory (CM) in RPCAs.

A Verifiable Role-Awareness Reward (VRAR) is a class of deterministic, transparent reward signals designed for reinforcement learning (RL) in role-playing conversational agents (RPCAs), enabling quantitative, interpretable, and reproducible optimization of role-consistency, role-specific knowledge, and character immersion. Distinct from subjective or LLM-based reward models, VRAR frameworks employ explicit, checkable mechanisms for linking rewards to role-specific response attributes—combining accuracy-oriented signals with structured format constraints. Two influential implementations of VRAR have been formalized in recent research: RAIDEN-R1 (Wang et al., 15 May 2025) and Character-R1 (Tang et al., 8 Jan 2026), both leveraging Group Relative Policy Optimization (GRPO).

1. Formal Definitions of VRAR

In the RAIDEN-R1 framework, VRAR is defined over a collection $\mathcal{D}=\{(h_i, q_i, L_i, X_i)\}_i$ of samples, where $h_i$ is dialogue history, $q_i$ is query, $L_i\in\{$STV, MTDP$\}$ encodes the mining strategy, and $X_i$ is either a single keyword or a parsing function. Given a generative model $M$ yielding response $R_i=M(h_i,q_i)$, the reward is a tuple $$r_i = [R_\mathrm{acc}(L_i,X_i,R_i), R_\mathrm{fmt}(R_i)]^\top$$, each term binary. $R_\mathrm{acc}$ enforces role-specific key or function match, and $h_i$0 enforces CoT style and composition. The total per-example reward is $h_i$1, taking values in $h_i$2, and supports maximization via policy gradient.

In Character-R1, for each sample $h_i$3, VRAR is a multi-component, normalized reward comprising: (1) Cognitive Focus Reward ($h_i$4) that incentivizes correct selection and description of role core attributes, (2) Reference-Guided Reward ($h_i$5) measuring answer-reference overlap, and (3) Character-Conditioned Normalization. Each component is explicit: $h_i$6 combines exact-match for focus label and BLEU-1 for attribute, $h_i$7 is BLEU-4 for final answer, and normalization is computed per character or character-cluster, supporting robust RL (Tang et al., 8 Jan 2026).

2. Mining Strategies for Role-Specific Keys

VRAR frameworks require automatic and verifiable extraction of role-relevant features from training data. RAIDEN-R1 provides two strategies:

A. Single-Term Validation (STV) precisely applies when a single unambiguous keyword is necessary:

1. Question-Type Filtering: Only WH-questions are retained.
2. Entity-Type Validation: Claude 3.5 extracts nominal candidates; non-nominal or absent candidates are discarded.
3. Cardinality Constraint: Retain samples with exactly one keyword.
4. Multi-Reference Verification: The candidate keyword is accepted if present in reference responses from GPT-4, MiniMax-abab6-chat, Baichuan-NPC, and GPT-3.5.

Pseudocode:

$L_i\in\{$6

B. Multi-Term Dynamic Parsing (MTDP) addresses free-form, variant-rich answers:

1. Keyword Expansion: QwQ-32B proposes a set of semantically aligned variants.
2. Legitimacy Verification: Qwen-72B filters for relevance.
3. Python Code Generation: QwQ-32B synthesizes a function $h_i$8 to match any valid variant in output $h_i$9.
4. Validation: Accepted only if $q_i$0 matches QwQ-32B's judgment on 10 LLMs $q_i$170% agreement.

The function $q_i$2 becomes the reference check in reward calculation.

3. Mathematical Derivation and Quantification of VRAR

The VRAR score for a model $q_i$3 over $q_i$4 evaluation instances is:

$q_i$5

Both $q_i$6 and $q_i$7 are binary, the maximum per-sample reward being 2. Components can be reported separately:

• $q_i$8
• $q_i$9

In Character-R1 (Tang et al., 8 Jan 2026), VRAR components undergo normalization:

$L_i\in\{$0

A final scalar reward combines normalized Cognitive Focus and Reference-Guided components with tunable weights as $L_i\in\{$1.

4. Role-Aware Chain-of-Thought Dataset Construction

RAIDEN-R1 constructs a synthetic first-person chain-of-thought corpus to enable CoT reasoning and explicit internal monologue for RPCAs:

• Raw Generation: DeepSeek-R1-671B generates long-form ($L_i\in\{$2 tokens) CoT and answer, conditioned on profile/history.
• Content Compression: Low-information meta-instructions are removed, preserving logic and character traits.
• Style Adaptation: Internal monologue is recast to match persona voice.
• Answer Generation: Claude 3.5 derives the final in-character response.

Key statistics:

Dataset Type	Pairs	Original CoT Length	Compressed CoT Length	Post-GRPO Generated CoT
Cold-start corpus	10,000	$L_i\in\{$3500	$L_i\in\{$460	30.1 (mean) tokens

This resource primes models for explicit “> …</think>” steps, supporting structure-enforced reasoning (Wang et al., 15 May 2025).

5. Experimental Setups and Results

RAIDEN-R1 evaluates five models on the RAIDEN test set, comparing GRPO with VRAR to strong SFT and CoT baselines using SBK (Script-Based Knowledge), CM (Conversation Memory), and other metrics. All experiments leverage Open-R1 (8×H800, bf16, LR=3e-6, cosine schedule, batch 4, 1 epoch) and use 1,000 SBK, 1,000 CM, and 1,000 challenging role-play samples for training.

$L_i\in\{$5

Key outcomes: GRPO with VRAR attains the highest SBK (88.04%) and CM (88.65%) scores. Pure CoT SFT and even SFT on VRAR-augmented data are less effective than GRPO with VRAR (Wang et al., 15 May 2025).

Character-R1, evaluated on CharacterBench and SocialBench, demonstrates improvements in Fact Accuracy (+4.0%), Memory Consistency (+9.5%), and outperforms all tested baselines in both quantitative and human assessments. Ablations confirm that all reward components are required for optimal performance (Tang et al., 8 Jan 2026).

6. Addressing Non-Quantifiability in RPCAs: Advantages of VRAR

Traditional approaches to RPCA reward design are hampered by the non-quantifiability problem: the absence of singularly correct responses and the subjectivity of LLM-judged preference modeling. VRAR introduces:

• Deterministic reward signals: Binary checks for keywords or parsing functions (RAIDEN-R1), or EM/BLEU-based scoring (Character-R1), replacing high-variance black-box proxies.
• Task-driven, role-key-centric mining: Automatic pipelines ensure only role-relevant correctness is measured.
• Structured format enforcement: Format rewards guarantee interpretable, in-character reasoning outputs (“<think>…” block), promoting consistency.

• Extensional verifiability: Rules and reward statistics are fully auditable; all matches can be exhaustively traced.

Empirical results demonstrate increased metric stability, resistance to RL collapse, and emergence of first-person reasoning stages, all quantified under transparent reward definitions. This provides a reproducible path for RL-driven improvement in complex conversational behaviors.

7. Comparative Frameworks and Further Directions

The emergence of VRAR coincides with broader efforts to enhance role-awareness in LLMs, as seen in Character-R1’s three-pronged reward scheme (Cognitive Focus, Reference-Guided, Normalization) (Tang et al., 8 Jan 2026) and RAIDEN-R1’s binary, rule-enforced rewards (Wang et al., 15 May 2025). Both frameworks integrate VRAR into the GRPO paradigm, resulting in significant improvements to knowledge, memory, and style immersion benchmarks. A plausible implication is that VRAR frameworks can be extended to even richer, context-sensitive skill sets provided suitable mining functions and transparent reward audits can be established. This suggests a general methodology for scaling deterministic RL reward design to other LLM-fine-tuning use cases beyond role-play.