SpeechRole-Eval Benchmark
- SpeechRole-Eval is a multidimensional benchmark that evaluates speech role-playing agents by examining role coherence, vocal naturalness, prosody, and emotional expression.
- It leverages a large-scale corpus with 98 distinct roles and approximately 112k dialogues, employing automated metrics to measure interaction quality, speech expressiveness, and role fidelity.
- The evaluation protocol combines reference-normalized scoring with human validation, highlighting performance differences between cascaded and end-to-end SRPA systems.
SpeechRole-Eval is a multidimensional benchmark for evaluating Speech Role-Playing Agents (SRPAs): systems that take spoken user input and generate spoken responses while remaining in character and preserving role-specific personality, background, timbre, prosody, and emotion. Introduced together with SpeechRole-Data, it was proposed to address the lack of systematic evaluation for speech-based role-playing, especially in settings where role coherence depends not only on textual content but also on vocal style consistency and paralinguistic realization. The benchmark is built around 98 distinct roles, approximately 112k speech dialogues, and an automated evaluation protocol spanning fundamental interaction ability, speech expressiveness, and role-playing fidelity (Jiang et al., 4 Aug 2025).
1. Conceptual background and scope
SpeechRole-Eval emerged from two converging research lines. The first is explicit speaker-role modeling in dialogue systems: earlier work on task-oriented dialogue showed that separating dialogue history by speaker role, rather than collapsing all turns into one context encoder, improved both language understanding and dialogue policy learning on DSTC4, supporting the claim that different roles contribute context differently (Chi et al., 2017). The second is the rapid expansion of text-based role-playing benchmarks, which formalized character style, role-specific knowledge, and social behavior, but remained fundamentally transcript-centric rather than speech-native (Wang et al., 2023, Chen et al., 2024).
Within that landscape, SpeechRole-Eval defines SRPA evaluation as a three-way problem. A system must remain in character, produce coherent spoken dialogue, and render that dialogue with role-appropriate speech naturalness, prosody, and emotion. This scope is narrower than general spoken dialogue understanding, because the target is not merely correct response generation, but character-conditioned response generation. It is also broader than text-only persona evaluation, because the benchmark treats timbre and prosody as constitutive parts of role identity rather than optional stylistic decoration (Jiang et al., 4 Aug 2025).
This positioning suggests that SpeechRole-Eval is not simply a dataset-derived leaderboard. It operationalizes speech role-play as the joint preservation of interaction quality, vocal expression, and canon- or persona-level fidelity under both single-turn and multi-turn conditions.
2. SpeechRole-Data and benchmark construction
SpeechRole-Eval is inseparable from SpeechRole-Data, the corpus that provides role metadata, reference voices, and evaluation dialogues. The dataset contains 98 distinct roles and 111,846 post-filtering dialogues, split into 53,902 single-turn and 57,944 multi-turn instances. The benchmark test split uses 392 dialogues in total: 196 single-turn and 196 multi-turn, with two single-turn and two multi-turn dialogues per role, so that all 98 roles appear in test (Jiang et al., 4 Aug 2025).
The roles come from two prior role-playing resources: 18 roles from ChatHaruhi and 80 roles from RoleLLM. The selection was balanced by language, gender, and personality traits, with emphasis on roles having distinctive vocal characteristics. Role metadata were organized into Role Profile, Background, and Character Lines, and then gpt-4.1-2025-04-14 was used to generate approximately 800 single-turn dialogues per role and approximately 800 multi-turn dialogues per role. English relevance filtering used all_MiniLM_L6_v2 with threshold 0.4; Chinese relevance filtering used text2vec-bge-large-chinese with threshold 0.45. Near-duplicate removal used threshold 0.9 for English and 0.85 for Chinese (Jiang et al., 4 Aug 2025).
Speech references were collected from Genshin Impact audio and other films and television works. The appendix reports 105 video files and approximately 181 hours of source material. Audio was extracted with ffmpeg into mono, 16 kHz, WAV format, then processed with Emilia for source separation, speaker separation or diarization, and VAD. The resulting 3–10 second single-speaker clips were transcribed with whisper-medium-v3 and filtered using DNSMOS P.835 OVRL, with speech removed when DNMOS < 3. Because diarization only supplies anonymous speaker IDs, character assignment used gpt-4.1-2025-04-14 and DeepSeek-V3-0324, followed by manual verification. For each role, CAM++ embeddings were used to select the canonical reference clip with the highest average cosine similarity to other clips from the same role, making that clip the benchmark’s default proxy for timbre (Jiang et al., 4 Aug 2025).
| Split | Characters | Samples | Speech hours |
|---|---|---|---|
| Train | 78 | 89,461 | 875.25 |
| Dev | 20 | 21,993 | 203.60 |
| Test | 98 | 392 | 4.20 |
The construction pipeline is notable because it couples role-grounded text generation with real-performance speech extraction. This suggests that SpeechRole-Eval was designed less as a pure synthetic benchmark than as a hybrid resource in which textual role descriptions, canonical lines, and authentic voice evidence are jointly available.
3. Task taxonomy and metric design
SpeechRole-Eval organizes prompts into three major task categories: Internal Reasoning, Experiential Narration, and Social Communication. These are further subdivided into eight subtasks: Opinion and Emotion Inquiry (14.73%), Reasoning and Motivation (14.73%), Reflection and Change (9.55%), Action Description and Retrospection (14.72%), Skills and Abilities (6.38%), Historical and Background Description (11.55%), Judgment of Others and Events (14.73%), and Advice and Life Experience (13.61%) (Jiang et al., 4 Aug 2025).
The benchmark’s core evaluation apparatus comprises 8 metrics grouped into 3 dimensions.
| Dimension | Metric | Function |
|---|---|---|
| Fundamental interaction ability | Instruction Adherence (IA) | Follows task instruction and remains fully in character |
| Fundamental interaction ability | Speech Fluency (SF) | Smooth articulation and pacing |
| Fundamental interaction ability | Conversational Coherence (CC) | Logical continuity and cross-turn consistency |
| Speech quality and expressiveness | Speech Naturalness (SN) | Human-like speech without obvious synthetic artifacts |
| Speech quality and expressiveness | Prosodic Consistency (PC) | Pitch, stress, and intonation aligned with character style |
| Speech quality and expressiveness | Emotion Appropriateness (EA) | Emotional cues fit dialogue context and character state |
| Role-playing fidelity | Personality Consistency (PeC) | Stability of traits such as optimism, sarcasm, or authority |
| Role-playing fidelity | Knowledge Consistency (KC) | Alignment with character background, knowledge, and relationships |
This metric design makes a strong separation between three failure types. A system may be conversationally competent but not vocally convincing; vocally expressive but not canon-faithful; or faithful in knowledge while weak in spoken delivery. The benchmark therefore refuses to collapse speech role-play into a single surface notion of “sounding like the character.” Instead, it treats role-play as the conjunction of content-level and speech-level consistency (Jiang et al., 4 Aug 2025).
4. Scoring protocol and human validation
SpeechRole-Eval uses a fully automated judge, gemini-2.5-pro-preview-06-05, but the scoring protocol is more structured than direct single-score prompting. The judge must first produce a brief rationale and then assign a 1–10 score to both the test response and a high-quality reference answer on each of the eight metrics. The final metric value is the ratio of the test score to the reference score:
The overall benchmark score is the average across the eight metric ratios:
In addition to judge-based metrics, the benchmark includes speaker similarity for voice-controllable systems, computed by extracting speaker embeddings with WavLM-large and applying cosine similarity. This supplementary objective metric is restricted to systems with explicit voice-cloning capability, so it does not apply symmetrically across all SRPAs (Jiang et al., 4 Aug 2025).
Human validation was performed on 20 dialogue samples—5 single-turn and 15 multi-turn—with 2 model-generated responses per sample and 4 independent annotators comparing the pair on each metric. Pairwise human preference used the scoring rule 1 for better, 0 for worse, and 0.5 for ties. The automated metric differences were then compared against human differences. The benchmark reports Pearson = 0.7162 and MSE = 0.1260 overall, with especially strong human agreement for Emotion Appropriateness (0.8881) and Speech Naturalness (0.8093), and weaker but still positive correlation for Conversational Coherence (0.6076) and Prosodic Consistency (0.6316) (Jiang et al., 4 Aug 2025).
This validation does not eliminate judge-model dependence, but it does suggest that the reference-normalized Gemini protocol is substantially aligned with human pairwise judgments on the benchmark’s intended dimensions.
5. Baseline systems and empirical findings
SpeechRole-Eval compares 11 SRPAs spanning cascaded and end-to-end architectures. The cascaded group includes one proprietary pipeline—Alibaba Cloud API, composed of Paraformer Realtime ASR v2, Qwen-Plus-Character, and CosyVoice-v2—and open-source pipelines using whisper-large-v3-turbo for ASR, LLaMA3.1-8B for the text LLM, and one of several zero-shot timbre-cloning TTS models: XTTS, CosyVoice2, E2 TTS, F5-TTS, FireRedTTS, GPT-SoVITS, Spark-TTS, and MaskGCT. The end-to-end group contains GPT-4o Audio and Qwen2.5-Omni-7B (Jiang et al., 4 Aug 2025).
The main result is that cascaded systems generally outperform end-to-end systems across most evaluation dimensions, especially on role fidelity and voice control. On the English benchmark, the top overall systems are E2 TTS (1.0417) and F5-TTS (1.0397), followed by Spark-TTS (0.9970), CosyVoice2 (0.9732), and GPT-4o Audio (0.9508). On the Chinese benchmark, the strongest overall system is Alibaba Cloud API (0.9301), followed by GPT-4o Audio (0.9047), Spark-TTS (0.7690), and F5-TTS (0.7680) (Jiang et al., 4 Aug 2025).
The exception within end-to-end systems is GPT-4o Audio, which is especially strong in Conversational Coherence and Speech Naturalness. The paper identifies it as the strongest end-to-end model in overall coherence and naturalness, even though it still trails the best cascaded systems in style and identity preservation. By contrast, Qwen2.5-Omni-7B is weak across both languages, scoring 0.4432 overall in English and 0.5263 in Chinese (Jiang et al., 4 Aug 2025).
The fine-tuning experiment further shows that the benchmark is sensitive to role-play specialization. Fine-tuning Qwen2.5-Omni-7B on the SpeechRole training set yields SpeechRole-Agent, which improves the base model from 0.4340 → 0.7098 on in-domain English, 0.4786 → 0.6795 on out-of-domain English, 0.5296 → 0.8468 on in-domain Chinese, and 0.5117 → 0.8028 on out-of-domain Chinese. The largest gains are reported in Instruction Adherence, Emotion Appropriateness, and Personality Consistency (Jiang et al., 4 Aug 2025).
The speaker-similarity results expose a separate trade-off. Among voice-cloning systems, E2 TTS and F5-TTS are strongest in English, MaskGCT is strongest in Chinese, and GPT-SoVITS is weakest. This indicates that SRPA performance cannot be reduced to a single generation architecture; the ASR–LLM–TTS decomposition remains competitive precisely because voice identity control is still easier to optimize in modular form (Jiang et al., 4 Aug 2025).
6. Position in the broader speech role-play evaluation ecosystem
SpeechRole-Eval occupies an early and specific place in the evolution of speech role-play benchmarking. Its defining contribution is to formulate SRPA evaluation as a joint problem of interaction quality, speech expressiveness, and role fidelity, using a role-rich speech corpus and a reference-normalized multimodal judge (Jiang et al., 4 Aug 2025). Later or adjacent work extended this in several directions.
One line expanded dataset scale and character coverage. VoxRole introduced 13,335 multi-turn dialogues, 65.6 hours of speech, and 1,228 unique characters across 261 movies, with persona profiles spanning personality, linguistic style, interpersonal relationships, and acoustic characteristics, plus an acoustically aware judge that scores Human-Likeness, Personality Consistency, Linguistic Fidelity, Relational Coherence, Contextual Coherence, and Paralinguistic Appropriateness (Wu et al., 4 Sep 2025). A second line emphasized human-aligned evaluators rather than zero-shot ALLM judging. Speech-DRAME split evaluation into Archetype Evaluation and Realism Evaluation, trained DRAME-Eval, and reported Pearson improvements from 0.480 to 0.629 on archetypes and from 0.390 to 0.625 on realism relative to zero-shot ALLM judges (Shi et al., 3 Nov 2025). A third line focused on audio-native role judges and multidimensional reasoning over character profile, dialogue history, and response audio. RoleJudge defined the five dimensions Logical Coherence, Content Relevance, Context Consistency, Emotional Appropriateness, and Style Alignment, and reached 86.00 overall accuracy on RoleChat (Fu et al., 15 Apr 2026).
These later systems do not supersede SpeechRole-Eval so much as clarify its status. SpeechRole-Eval established the benchmark problem and a practical automated protocol; later work increased corpus size, strengthened human alignment, or moved from benchmark-level scoring to trained evaluators and richer multimodal rubrics. This suggests that SpeechRole-Eval’s historical significance lies less in maximal evaluator sophistication than in making speech role-playing a standardizable evaluation target.
The benchmark’s limitations are correspondingly clear. It depends heavily on a single judge model, Gemini-2.5, for most scores; prosody is evaluated implicitly rather than through explicit structured annotations; performance is stronger in English than Chinese; the role inventory is drawn largely from fictional, media-derived characters; and the test set remains relatively small at 392 dialogues. In addition, speaker similarity is only available for voice-controllable systems, which makes some comparisons asymmetric. These constraints do not negate the benchmark’s utility, but they delimit what SpeechRole-Eval measures best: not full human theatrical realism, but systematic SRPA competence under a role-aware, speech-aware, and reference-normalized evaluation regime (Jiang et al., 4 Aug 2025).