---
title: SpeechRole-Eval Benchmark
url: https://www.emergentmind.com/topics/speechrole-eval
type: topic
---

# SpeechRole-Eval Benchmark

SpeechRole-Eval is a multidimensional benchmark for evaluating **Speech Role-Playing Agents (SRPAs)**: systems that take **spoken user input** and generate **spoken responses** while remaining in character and preserving role-specific personality, background, **timbre**, **prosody**, and emotion. Introduced together with **SpeechRole-Data**, it was proposed to address the lack of systematic evaluation for speech-based role-playing, especially in settings where role coherence depends not only on textual content but also on vocal style consistency and paralinguistic realization. The benchmark is built around **98 distinct roles**, approximately **112k** speech dialogues, and an automated evaluation protocol spanning **fundamental interaction ability**, **speech expressiveness**, and **role-playing fidelity** [2508.02013].

## 1. Conceptual background and scope

SpeechRole-Eval emerged from two converging research lines. The first is explicit **speaker-role modeling** in dialogue systems: earlier work on task-oriented dialogue showed that separating dialogue history by speaker role, rather than collapsing all turns into one context encoder, improved both **language understanding** and **dialogue policy learning** on DSTC4, supporting the claim that different roles contribute context differently [1710.00164]. The second is the rapid expansion of **text-based role-playing benchmarks**, which formalized character style, role-specific knowledge, and social behavior, but remained fundamentally transcript-centric rather than speech-native [2310.00746][2403.13679].

Within that landscape, SpeechRole-Eval defines SRPA evaluation as a three-way problem. A system must remain **in character**, produce **coherent spoken dialogue**, and render that dialogue with role-appropriate **speech naturalness**, **prosody**, and **emotion**. This scope is narrower than general spoken dialogue understanding, because the target is not merely correct response generation, but **character-conditioned** response generation. It is also broader than text-only persona evaluation, because the benchmark treats **timbre** and **prosody** as constitutive parts of role identity rather than optional stylistic decoration [2508.02013].

This positioning suggests that SpeechRole-Eval is not simply a dataset-derived leaderboard. It operationalizes speech role-play as the joint preservation of **interaction quality**, **vocal expression**, and **canon- or persona-level fidelity** under both **single-turn** and **multi-turn** conditions.

## 2. SpeechRole-Data and benchmark construction

SpeechRole-Eval is inseparable from **SpeechRole-Data**, the corpus that provides role metadata, reference voices, and evaluation dialogues. The dataset contains **98 distinct roles** and **111,846** post-filtering dialogues, split into **53,902 single-turn** and **57,944 multi-turn** instances. The benchmark test split uses **392** dialogues in total: **196 single-turn** and **196 multi-turn**, with **two single-turn and two multi-turn dialogues per role**, so that all **98 roles** appear in test [2508.02013].

The roles come from two prior role-playing resources: **18 roles from ChatHaruhi** and **80 roles from RoleLLM**. The selection was balanced by **language**, **gender**, and **personality traits**, with emphasis on roles having **distinctive vocal characteristics**. Role metadata were organized into **Role Profile**, **Background**, and **Character Lines**, and then **gpt-4.1-2025-04-14** was used to generate approximately **800 single-turn dialogues per role** and approximately **800 multi-turn dialogues per role**. English relevance filtering used **all_MiniLM_L6_v2** with threshold **0.4**; Chinese relevance filtering used **text2vec-bge-large-chinese** with threshold **0.45**. Near-duplicate removal used threshold **0.9** for English and **0.85** for Chinese [2508.02013].

Speech references were collected from **Genshin Impact** audio and other **films and television works**. The appendix reports **105 video files** and approximately **181 hours** of source material. Audio was extracted with **ffmpeg** into **mono**, **16 kHz**, **WAV** format, then processed with **Emilia** for source separation, speaker separation or diarization, and VAD. The resulting **3–10 second** single-speaker clips were transcribed with **whisper-medium-v3** and filtered using **DNSMOS P.835 OVRL**, with speech removed when **DNMOS < 3**. Because diarization only supplies anonymous speaker IDs, character assignment used **gpt-4.1-2025-04-14** and **DeepSeek-V3-0324**, followed by manual verification. For each role, **CAM++** embeddings were used to select the canonical reference clip with the highest average cosine similarity to other clips from the same role, making that clip the benchmark’s default proxy for **timbre** [2508.02013].

| Split | Characters | Samples | Speech hours |
|---|---:|---:|---:|
| Train | 78 | 89,461 | 875.25 |
| Dev | 20 | 21,993 | 203.60 |
| Test | 98 | 392 | 4.20 |

The construction pipeline is notable because it couples **role-grounded text generation** with **real-performance speech extraction**. This suggests that SpeechRole-Eval was designed less as a pure synthetic benchmark than as a hybrid resource in which textual role descriptions, canonical lines, and authentic voice evidence are jointly available.

## 3. Task taxonomy and metric design

SpeechRole-Eval organizes prompts into **three major task categories**: **Internal Reasoning**, **Experiential Narration**, and **Social Communication**. These are further subdivided into **eight subtasks**: **Opinion and Emotion Inquiry** (**14.73%**), **Reasoning and Motivation** (**14.73%**), **Reflection and Change** (**9.55%**), **Action Description and Retrospection** (**14.72%**), **Skills and Abilities** (**6.38%**), **Historical and Background Description** (**11.55%**), **Judgment of Others and Events** (**14.73%**), and **Advice and Life Experience** (**13.61%**) [2508.02013].

The benchmark’s core evaluation apparatus comprises **8 metrics** grouped into **3 dimensions**.

| Dimension | Metric | Function |
|---|---|---|
| Fundamental interaction ability | Instruction Adherence (IA) | Follows task instruction and remains fully in character |
| Fundamental interaction ability | Speech Fluency (SF) | Smooth articulation and pacing |
| Fundamental interaction ability | Conversational Coherence (CC) | Logical continuity and cross-turn consistency |
| Speech quality and expressiveness | Speech Naturalness (SN) | Human-like speech without obvious synthetic artifacts |
| Speech quality and expressiveness | Prosodic Consistency (PC) | Pitch, stress, and intonation aligned with character style |
| Speech quality and expressiveness | Emotion Appropriateness (EA) | Emotional cues fit dialogue context and character state |
| Role-playing fidelity | Personality Consistency (PeC) | Stability of traits such as optimism, sarcasm, or authority |
| Role-playing fidelity | Knowledge Consistency (KC) | Alignment with character background, knowledge, and relationships |

This metric design makes a strong separation between three failure types. A system may be conversationally competent but not vocally convincing; vocally expressive but not canon-faithful; or faithful in knowledge while weak in spoken delivery. The benchmark therefore refuses to collapse speech role-play into a single surface notion of “sounding like the character.” Instead, it treats role-play as the conjunction of **content-level** and **speech-level** consistency [2508.02013].

## 4. Scoring protocol and human validation

SpeechRole-Eval uses a fully automated judge, **gemini-2.5-pro-preview-06-05**, but the scoring protocol is more structured than direct single-score prompting. The judge must first produce a **brief rationale** and then assign a **1–10** score to both the **test response** and a **high-quality reference answer** on each of the eight metrics. The final metric value is the **ratio of the test score to the reference score**:

$$
\text{MetricScore} = \frac{\text{score of test response}}{\text{score of reference response}}
$$

The overall benchmark score is the average across the eight metric ratios:

$$
\text{Overall} = \frac{1}{8}\sum_{m \in \{\text{IA,SF,CC,SN,PC,EA,PeC,KC}\}} \text{MetricScore}_m
$$

In addition to judge-based metrics, the benchmark includes **speaker similarity** for voice-controllable systems, computed by extracting speaker embeddings with **WavLM-large** and applying **cosine similarity**. This supplementary objective metric is restricted to systems with explicit voice-cloning capability, so it does not apply symmetrically across all SRPAs [2508.02013].

Human validation was performed on **20 dialogue samples**—**5 single-turn** and **15 multi-turn**—with **2 model-generated responses** per sample and **4 independent annotators** comparing the pair on each metric. Pairwise human preference used the scoring rule **1** for better, **0** for worse, and **0.5** for ties. The automated metric differences were then compared against human differences. The benchmark reports **Pearson = 0.7162** and **MSE = 0.1260** overall, with especially strong human agreement for **Emotion Appropriateness** (**0.8881**) and **Speech Naturalness** (**0.8093**), and weaker but still positive correlation for **Conversational Coherence** (**0.6076**) and **Prosodic Consistency** (**0.6316**) [2508.02013].

This validation does not eliminate judge-model dependence, but it does suggest that the reference-normalized Gemini protocol is substantially aligned with human pairwise judgments on the benchmark’s intended dimensions.

## 5. Baseline systems and empirical findings

SpeechRole-Eval compares **11 SRPAs** spanning **cascaded** and **end-to-end** architectures. The cascaded group includes one proprietary pipeline—**Alibaba Cloud API**, composed of **Paraformer Realtime ASR v2**, **Qwen-Plus-Character**, and **CosyVoice-v2**—and open-source pipelines using **whisper-large-v3-turbo** for ASR, **LLaMA3.1-8B** for the text LLM, and one of several zero-shot timbre-cloning TTS models: **XTTS**, **CosyVoice2**, **E2 TTS**, **F5-TTS**, **FireRedTTS**, **GPT-SoVITS**, **Spark-TTS**, and **MaskGCT**. The end-to-end group contains **GPT-4o Audio** and **Qwen2.5-Omni-7B** [2508.02013].

The main result is that **cascaded systems generally outperform end-to-end systems across most evaluation dimensions**, especially on role fidelity and voice control. On the **English** benchmark, the top overall systems are **E2 TTS** (**1.0417**) and **F5-TTS** (**1.0397**), followed by **Spark-TTS** (**0.9970**), **CosyVoice2** (**0.9732**), and **GPT-4o Audio** (**0.9508**). On the **Chinese** benchmark, the strongest overall system is **Alibaba Cloud API** (**0.9301**), followed by **GPT-4o Audio** (**0.9047**), **Spark-TTS** (**0.7690**), and **F5-TTS** (**0.7680**) [2508.02013].

The exception within end-to-end systems is **GPT-4o Audio**, which is especially strong in **Conversational Coherence** and **Speech Naturalness**. The paper identifies it as the strongest end-to-end model in overall coherence and naturalness, even though it still trails the best cascaded systems in style and identity preservation. By contrast, **Qwen2.5-Omni-7B** is weak across both languages, scoring **0.4432** overall in English and **0.5263** in Chinese [2508.02013].

The fine-tuning experiment further shows that the benchmark is sensitive to role-play specialization. Fine-tuning **Qwen2.5-Omni-7B** on the SpeechRole training set yields **SpeechRole-Agent**, which improves the base model from **0.4340 → 0.7098** on **in-domain English**, **0.4786 → 0.6795** on **out-of-domain English**, **0.5296 → 0.8468** on **in-domain Chinese**, and **0.5117 → 0.8028** on **out-of-domain Chinese**. The largest gains are reported in **Instruction Adherence**, **Emotion Appropriateness**, and **Personality Consistency** [2508.02013].

The speaker-similarity results expose a separate trade-off. Among voice-cloning systems, **E2 TTS** and **F5-TTS** are strongest in English, **MaskGCT** is strongest in Chinese, and **GPT-SoVITS** is weakest. This indicates that SRPA performance cannot be reduced to a single generation architecture; the ASR–LLM–TTS decomposition remains competitive precisely because voice identity control is still easier to optimize in modular form [2508.02013].

## 6. Position in the broader speech role-play evaluation ecosystem

SpeechRole-Eval occupies an early and specific place in the evolution of speech role-play benchmarking. Its defining contribution is to formulate SRPA evaluation as a joint problem of **interaction quality**, **speech expressiveness**, and **role fidelity**, using a role-rich speech corpus and a reference-normalized multimodal judge [2508.02013]. Later or adjacent work extended this in several directions.

One line expanded **dataset scale and character coverage**. **VoxRole** introduced **13,335 multi-turn dialogues**, **65.6 hours of speech**, and **1,228 unique characters** across **261 movies**, with persona profiles spanning **personality**, **linguistic style**, **interpersonal relationships**, and **acoustic characteristics**, plus an acoustically aware judge that scores **Human-Likeness**, **Personality Consistency**, **Linguistic Fidelity**, **Relational Coherence**, **Contextual Coherence**, and **Paralinguistic Appropriateness** [2509.03940]. A second line emphasized **human-aligned evaluators** rather than zero-shot ALLM judging. **Speech-DRAME** split evaluation into **Archetype Evaluation** and **Realism Evaluation**, trained **DRAME-Eval**, and reported Pearson improvements from **0.480 to 0.629** on archetypes and from **0.390 to 0.625** on realism relative to zero-shot ALLM judges [2511.01261]. A third line focused on **audio-native role judges** and multidimensional reasoning over character profile, dialogue history, and response audio. **RoleJudge** defined the five dimensions **Logical Coherence**, **Content Relevance**, **Context Consistency**, **Emotional Appropriateness**, and **Style Alignment**, and reached **86.00** overall accuracy on **RoleChat** [2604.13804].

These later systems do not supersede SpeechRole-Eval so much as clarify its status. SpeechRole-Eval established the benchmark problem and a practical automated protocol; later work increased corpus size, strengthened human alignment, or moved from benchmark-level scoring to trained evaluators and richer multimodal rubrics. This suggests that SpeechRole-Eval’s historical significance lies less in maximal evaluator sophistication than in making **speech role-playing** a standardizable evaluation target.

The benchmark’s limitations are correspondingly clear. It depends heavily on a single judge model, **Gemini-2.5**, for most scores; prosody is evaluated implicitly rather than through explicit structured annotations; performance is stronger in **English** than **Chinese**; the role inventory is drawn largely from **fictional, media-derived characters**; and the test set remains relatively small at **392 dialogues**. In addition, speaker similarity is only available for voice-controllable systems, which makes some comparisons asymmetric. These constraints do not negate the benchmark’s utility, but they delimit what SpeechRole-Eval measures best: not full human theatrical realism, but systematic SRPA competence under a role-aware, speech-aware, and reference-normalized evaluation regime [2508.02013].

Source: https://www.emergentmind.com/topics/speechrole-eval