Role-playing Fidelity Score (RFS)

• Role-playing Fidelity Score (RFS) is a multidimensional family of metrics designed to quantify how faithfully systems adopt and maintain assigned personas, including consistency in knowledge and behavior.
• Evaluations under RFS span diverse approaches, from exact answer-set matching in professional simulations to speech-based measures of personality and prosodic fidelity.
• Recent research emphasizes that effective RFS assessment requires aligning semantic content with acoustic or narrative elements, while addressing methodological challenges in evaluation.

Role-playing Fidelity Score (RFS) denotes the measurement of how faithfully a system embodies an assigned character, profession, or persona, but the term is not standardized across recent arXiv literature. An explicit metric named Role-playing Fidelity Score (RFS) appears in EduGuardBench, where it measures professional fidelity for simulated teachers, while most other works use adjacent constructs such as role-playing fidelity, role consistency, Character Fidelity, Knowledge Boundary Fidelity, or checklist-based Quality Scores (Jiang et al., 10 Nov 2025, Jiang et al., 4 Aug 2025, Ding et al., 11 Dec 2025, Tang et al., 24 Jun 2026, Rosati et al., 13 Apr 2026). Interpreted across these benchmarks, RFS is best understood as a family of evaluation schemes for persona adherence, knowledge and value consistency, behavioral realism, and, in speech settings, vocal embodiment.

1. Terminological status and formal definitions

Only EduGuardBench defines a literal metric called Role-playing Fidelity Score. In that benchmark, RFS is computed over a 2636-entry Select-All-That-Apply question set for simulated teachers as

$$\text{RFS} = \frac{1}{|Q|} \sum_{q \in Q} s(P_q, C_q),$$

where $P_q$ is the model-selected option set and $C_q$ is the ideal option set (Jiang et al., 10 Nov 2025). The per-question score is 1.0 for a perfect match, 0.5 for a non-empty strict subset of the correct answers with no wrong selections, and 0.0 if the model selects at least one incorrect answer; the resulting RFS therefore ranges from 0 to 1 (Jiang et al., 10 Nov 2025).

Most other works do not define RFS by name, but they define close equivalents. In SpeechRole, “role-playing fidelity” is one of three top-level benchmark dimensions and is operationalized by Personality Consistency (PeC) and Knowledge Consistency (KC); each metric is scored as the ratio of a model-generated speech score to a paired reference score, with Gemini-based judgments on a 1–10 scale (Jiang et al., 4 Aug 2025). RoleRMBench does not define a single fidelity scalar for one response, but its official overall score is the average pairwise accuracy across seven role-play sub-datasets: Narrative Cluster, Scene Transition, Role Consistency, Instruction Following, Safety, Multi-turn Coherence, and Attractiveness (Ding et al., 11 Dec 2025). RoleCDE centers deep fidelity on decision behavior under value conflict and uses DBR as its closest scalar, namely the proportion of instances in which the model chooses the role-consistent side through Role-Following or Role-Compromise (Lai et al., 1 Jun 2026). RPA-Check, similarly, does not use the term RFS, but its Quality Score is the proportion of passed checklist items within dimensions such as Behavioral Role Fidelity and Procedural Convergence and Stability (Rosati et al., 13 Apr 2026).

Work	Native term	Operational core
EduGuardBench (Jiang et al., 10 Nov 2025)	RFS	Average SATA score $s(P_q,C_q)$
SpeechRole (Jiang et al., 4 Aug 2025)	Role-playing fidelity	PeC and KC
RoleRMBench (Ding et al., 11 Dec 2025)	Avg pairwise accuracy	Seven role-play capabilities
RoleCDE (Lai et al., 1 Jun 2026)	DBR	Proportion of RF+RC decisions
RPA-Check (Rosati et al., 13 Apr 2026)	QS	Passed-checklist ratio

This distribution of terms shows that “RFS” in current usage is usually either a benchmark-specific scalar, as in EduGuardBench, or an inferred shorthand for a broader role-fidelity dimension.

2. Recurrent dimensions of role-playing fidelity

Across benchmarks, the most stable fidelity dimensions are persona consistency, knowledge or boundary consistency, and interactional appropriateness. SpeechRole makes this explicit through PeC and KC, where PeC asks whether responses consistently reflect personality traits such as optimism, sarcasm, or authority, and KC asks whether responses remain grounded in background, knowledge, and relationships without fabricating out-of-character facts (Jiang et al., 4 Aug 2025). RoleLLM uses a parallel but text-centric decomposition: CUS for speaking style imitation, RAW for instruction-answer accuracy, and SPE for role-specific knowledge and memory, grounded conceptually in Lexical Consistency, Dialogic Fidelity, Script-Based Knowledge, and Script-Agnostic Knowledge (Wang et al., 2023). “Fame Fades, Nature Remains” further sharpens this decomposition by separating Parametric Identity from Attributive Identity, arguing that fidelity should distinguish pretraining-based character familiarity from profile-grounded behavioral properties such as morality, worldview, and interpersonal style (Jun et al., 8 Jan 2026).

A second recurring dimension is knowledge access and boundary control rather than knowledge volume alone. RoleMRC operationalizes role-playing partly as staying within a role’s predefined Ability and Knowledge Boundaries, including correct transitions between answer, refusal, no-answer, and attempt behaviors under passage answerability and role ability constraints (Lu et al., 17 Feb 2025). REVERIEMEM makes the same issue explicit as Factual Overreach and measures it with Knowledge Boundary Fidelity, defined as the sample-weighted harmonic mean of visible-item accuracy and invisible-item refusal accuracy in KBF-QA (Tang et al., 24 Jun 2026). RoleBreak reframes this failure as character hallucination, then evaluates not only Hallucination Rate, but also Role Fidelity, Query Fidelity, and Story Coherence, precisely because low hallucination can otherwise be achieved by over-reliance on refusal strategies (Tang et al., 2024).

A third recurrent dimension is decision fidelity under conflict. RoleRMBench treats role-play quality as a multidimensional subjective preference problem involving narrative management, role consistency, multi-turn coherence, and attractiveness (Ding et al., 11 Dec 2025). Beyond One World extends fidelity to canon-specific superhero versions and separates canonical accuracy, reasoning fidelity, action fidelity, and Think–Act Matching as a proxy for trustworthiness (Ngokpol et al., 16 Oct 2025). RoleCDE pushes this further by arguing that surface role fidelity is insufficient when role-specific values conflict with alignment-oriented constraints; its categories RF, RC, AC, and AF make fidelity partly a matter of how an agent resolves role-versus-alignment dilemmas (Lai et al., 1 Jun 2026).

Taken together, these works suggest that RFS is best treated as a multidimensional construct rather than a single proxy for “sounding in character.”

3. Speech and multimodal formulations

Speech benchmarks make the role-fidelity problem explicitly bimodal. SpeechRole defines Speech Role-Playing Agents as systems that should exhibit “consistent vocal traits, expressive prosody, and coherent persona behavior across interactions,” and it evaluates both single-turn and multi-turn speech dialogues across 98 roles and 112k conversations (Jiang et al., 4 Aug 2025). Its benchmark separates fundamental interaction ability, speech expressiveness, and role-playing fidelity, but the paper repeatedly stresses that realistic spoken role-play depends on maintaining both persona and vocal style (Jiang et al., 4 Aug 2025).

AudioRole makes the same split more formal. Its ARP-Eval framework distinguishes response quality from role fidelity, with fidelity operationalized as Acoustic Personalization (AP) and Content Personalization (CP), while Acoustic Quality (AQ) and Content Quality (CQ) serve as supporting quality measures (Li et al., 27 Sep 2025). AP is intended to measure preservation of the target character’s acoustic identity through speaker embeddings, whereas CP uses GPT-4o-audio to judge whether generated audio and reference audio exhibit the same character style (Li et al., 27 Sep 2025). A plausible implication is that an audio RFS should center on AP and CP, with AQ and CQ reported separately or used as constraints rather than as fidelity proper.

DeSRPA generalizes this bimodal view into a “mind” and “voice” decomposition. It evaluates speech role-play through multimodal judged dimensions including Personality Consistency, Knowledge Consistency, Prosodic Consistency, and Emotion Appropriateness, together with objective measures such as SIM for speaker similarity and EEA for emotion execution accuracy (Tang et al., 16 Jun 2026). On this view, a speech-oriented RFS is not reducible to textual persona preservation, because character fidelity depends on synchronized internal cognition and external expressive rendering (Tang et al., 16 Jun 2026).

The speech literature therefore supports a strong general claim: in multimodal role-play, fidelity requires alignment between semantic content and vocal realization, not merely one or the other.

4. Narrative, memory, and canonical fidelity

Several benchmarks shift fidelity away from surface style and toward epistemic perspective, canon specificity, and long-horizon narrative grounding. REVERIEMEM is exemplary in this regard. It identifies two out-of-character failures—Factual Overreach and Stylistic Monotony—and addresses them with a three-layer memory architecture consisting of episodic, semantic, and personality memory (Tang et al., 24 Jun 2026). Its benchmark, KBF-QA, contains 4,386 questions over eight novels, and its main fidelity metric, Knowledge Boundary Fidelity, evaluates whether a character answers only what that character could plausibly know from its narrative perspective (Tang et al., 24 Jun 2026). Open-ended generation is then assessed with the BOOKWORLD five-dimension pairwise narrative protocol, including Anthropomorphism, Character Fidelity, Immersion & Setting, Writing Quality, and Storyline Quality or Creativity (Tang et al., 24 Jun 2026). This implies that long-form role fidelity is partly epistemic: the response must arise from the character’s own access to events, not merely from globally correct facts.

“Codifying Character Logic in Role-Playing” approaches the same problem through executable behavioral rules. It treats fidelity as scene-conditioned behavioral correctness, persistence, updatability, and controllable randomness, and evaluates generated actions against ground truth with an NLI-based scoring system of 100 for entailed, 50 for neutral, and 0 for contradicted (Peng et al., 12 May 2025). Because errors can be traced to specific triggered statements and revised, this work makes fidelity not only measurable but also diagnosable (Peng et al., 12 May 2025).

Beyond One World extends canonical fidelity to multiversal superhero settings. Its benchmark spans 30 iconic heroes and 90 canon-specific versions, and it argues that role-play requires not generic superhero imitation but faithful portrayal of a specific version at a specific life phase with the right history, values, and moral code (Ngokpol et al., 16 Oct 2025). Its measurements separate canonical correctness from reasoning and action fidelity, while Think–Act Matching estimates alignment between a model’s internal deliberation and outward behavior (Ngokpol et al., 16 Oct 2025). A plausible implication is that canon-sensitive RFS formulations should preserve separate subscores for knowledge, behavior, and reasoning-action agreement rather than collapsing them immediately.

“Fame Fades, Nature Remains” adds a further long-horizon result: famous characters enjoy a significant initial advantage, but this advantage rapidly weakens in multi-turn settings, while negative morality and negative interpersonal relationships remain hard regardless of fame (Jun et al., 8 Jan 2026). For role-fidelity evaluation, this indicates that persistence over turns and sensitivity to socially negative roles are not secondary issues but central fault lines.

5. Instruction following, professional simulation, and institutional roles

A distinct strand of work treats role fidelity as correct behavior under explicit constraints, not merely persona resemblance. RoleMRC is explicit that good role-playing means maintaining role identity, respecting predefined ability limits, expressing role style, and following layered instructions simultaneously (Lu et al., 17 Feb 2025). Its five judge dimensions—Knowledge Boundary, Role Style, Multi-turn Instruction-following, Nested Instruction-following, and Prioritized Instruction-following—make fidelity partly a matter of producing the correct response type under constraints, including refusal when system-level instruction hierarchy requires it (Lu et al., 17 Feb 2025).

EduGuardBench brings this logic into profession simulation. Its RFS evaluates whether a model behaves like a competent teacher across Problem Solving, Error Correction, Idea Provision, Personalized Learning Support, and Emotional Support, while avoiding Incompetence (S1), Offensiveness (S2), and Indolence (S3) (Jiang et al., 10 Nov 2025). Because the metric is deterministic answer-set matching rather than LLM judging, it is one of the few role-fidelity formulations that is fully scriptable and directly reproducible (Jiang et al., 10 Nov 2025). The benchmark’s own interpretation is that low teacher-role fidelity is driven mainly by incompetence rather than by overtly abusive behavior (Jiang et al., 10 Nov 2025).

RPA-Check extends professional and institutional fidelity into long, constrained simulations. In legal scenarios, it decomposes evaluation into Behavioral Role Fidelity, Procedural Convergence and Stability, and Linguistic Formalism and Orthography, then scores transcripts via filtered boolean checklists and an LLM judge (Rosati et al., 13 Apr 2026). Its Quality Score is the proportion of passed checklist items, while Retry Rate counts catastrophic failures such as loops or freezes (Rosati et al., 13 Apr 2026). This framework broadens the concept of RFS beyond character resemblance: an agent can be linguistically fluent yet unfaithful if it violates adversarial duties, procedural neutrality, or institutional role boundaries (Rosati et al., 13 Apr 2026).

Identity-Driven Hierarchical Role-Playing Agents contributes a different professional and social perspective by evaluating identity-level fidelity through scale evaluation and open situation evaluation (Sun et al., 2024). Personality fit is measured with Big Five scales, profession fit with a profession scale that rewards target-domain competence and non-target ignorance, and open situations test whether a judge can infer the assigned identities from anonymous dialogue (Sun et al., 2024). This suggests that identity fidelity may also be framed as recoverability of the intended role from situated behavior.

6. Empirical patterns, controversies, and limitations

Several empirical regularities recur across the literature. First, generic evaluators and generic role-play prompting are usually insufficient for high-fidelity behavior. RoleRMBench finds large gaps between general-purpose reward models and human judgment, and reports that RoleRM surpasses strong open- and closed-source reward models by over 24% on average, with especially strong gains in narrative coherence and stylistic fidelity (Ding et al., 11 Dec 2025). SpeechRole likewise reports that cascaded agents generally outperform end-to-end systems across most evaluation dimensions, though task-specific fine-tuning of end-to-end systems can substantially improve Personality Consistency and Knowledge Consistency (Jiang et al., 4 Aug 2025).

Second, visible identity cues can inflate apparent fidelity. “Rethinking Role-Playing Evaluation” shows that anonymization significantly degrades performance, indicating that famous names carry implicit information, and it further finds that self-generated personality traits achieve performance comparable to human-annotated personality information under anonymous evaluation (Peng et al., 4 Mar 2026). “Fame Fades, Nature Remains” complements this by showing that famous characters have a significant single-turn advantage, but that this edge weakens as dialogue history grows and models rely more on accumulated context and self-conditioning (Jun et al., 8 Jan 2026).

Third, the hardest failures are often not stylistic but social, moral, or procedural. RoleCDE identifies a Role Value Decoupling phenomenon in which agents default to alignment- and morality-consistent decisions rather than role-specific values when the two conflict (Lai et al., 1 Jun 2026). EduGuardBench reports that reasoning-oriented models generally achieve higher mean RFS than non-reasoning models, but that the strongest statistically robust group difference is in lower incorrect-inclusion rates rather than in the aggregate RFS after correction, and that Incompetence remains the dominant failure mode across most models (Jiang et al., 10 Nov 2025). RoleBreak likewise argues that Hallucination Rate alone is inadequate, because it encourages refusal-heavy strategies that reduce narrative development; its Narrator Mode improves hallucination control while also improving role fidelity, query fidelity, and story coherence on GPT-3.5 (Tang et al., 2024).

Fourth, the literature remains methodologically heterogeneous. Several papers rely heavily on LLM-as-a-judge rather than human calibration, and some metrics are explicitly described as proxies rather than validated constructs. Beyond One World presents Think–Act Matching as a proxy for trustworthiness rather than a validated trust metric (Ngokpol et al., 16 Oct 2025). AudioRole leaves an important ambiguity in Acoustic Personalization, describing it in terms of cosine distance while reporting it as a higher-is-better score (Li et al., 27 Sep 2025). RPA-Check depends on generated and filtered checklists whose content partly determines the resulting score (Rosati et al., 13 Apr 2026). These issues do not invalidate the benchmarks, but they show that RFS is still a developing evaluation family rather than a settled measurement standard.

The literature therefore converges on a negative conclusion and a constructive one. The negative conclusion is that no single scalar currently captures all aspects of role fidelity across text, speech, narrative, profession simulation, and value conflict. The constructive conclusion is that recent benchmarks collectively define a stable evaluation agenda: fidelity should separate persona consistency, knowledge or boundary control, behavior under instruction and value conflict, long-horizon persistence, and, when relevant, acoustic embodiment (Jiang et al., 4 Aug 2025, Jiang et al., 10 Nov 2025, Ding et al., 11 Dec 2025, Tang et al., 24 Jun 2026, Lai et al., 1 Jun 2026).