Speech-DRAME-EvalBench Evaluation

• Speech-DRAME-EvalBench is an evaluation paradigm that integrates bilingual, human-annotated corpora with metric and rubric-based protocols for speech role-play benchmarking.
• It employs two distinct strategies—Archetype and Realism—to assess model performance in terms of expressiveness, quality, and adherence to sociocultural archetypes.
• The framework supports diverse training regimes and provides granular, reproducible metrics that are essential for fine-tuning and comparing speech foundation models.

Speech-DRAME-EvalBench is an evaluation paradigm, data resource, and methodological framework developed for comprehensive, human-aligned benchmarking of generative models in speech role-play contexts. Originating from the Speech-DRAME initiative, it integrates metric-based and rubric-based protocols, bilingual (Mandarin/English) human-annotated corpora, and training-testing regimens for speech evaluation models (SEMs). The benchmark formalizes two distinct strategies—Archetype Evaluation and Realism Evaluation—enabling reproducible, multi-level assessments of spoken role-play quality, expressiveness, and adherence to both sociocultural archetypes and authentic human behavior. Speech-DRAME-EvalBench sets a methodological foundation for the training and large-scale comparison of SEMs and serves as a substrate for evaluating and advancing speech foundation models (SFMs), especially within the context of complex, multimodal generation tasks (Shi et al., 3 Nov 2025).

1. Data Composition and Structure

Speech-DRAME-EvalBench is organized into two principal corpora, each addressing complementary evaluation paradigms:

• Archetype Evaluation Corpus: Composed of 8,280 utterances (evenly split between Mandarin and English), covering 552 unique role/scene prompts (e.g., “You are a firefighter… smoke fills the building… you say:”). Each prompt is synthesized via eight speech foundation models (end-to-end and cascaded), with a total of 6,780 samples for training and 1,500 for testing. An additional 1,250 scenario–model pairs are held out for system-level benchmarking. Utterances are truncated or filtered to a 30-seconds maximum, and invalid clips are excluded via a binary “Content Pass” phase.
• Realism Evaluation Corpus: Contains 15,000 utterances—9,000 in Mandarin, 5,000 in English—spanning 12,000 train, 2,000 mixed (base) test, and 1,000 human-recorded test files. Data sources include authentic media (TV/film), public corpora (e.g., NCSSD), LLM-generated character profiles and scenes, plus contrastive negatives synthesized by content/prompt mismatches or diverse TTS resyntheses. Rigorous semantic filtering enforces relevance and introduces negative samples across varying mismatch levels.

The benchmark ensures distributional rigor by reserving unique scenarios for each split, enabling clean regime-separation in both zeroshot/fewshot and supervised evaluations.

2. Annotation Procedures and Rubric Systems

Speech-DRAME-EvalBench implements high-resolution, rubric-driven human annotation protocols in two stages:

• Content Pass Screening: Rejects files that are too short (<2s), non-compliant with language or prompt, or exhibit severe synthesis or audio breakdowns. Rejected items are assigned minimum rubric values.
• Archetype Rubrics: Each utterance receives three ratings on 1–5 scales:
  • Audio Quality (artifact-focused, content-free)
  • Human Likeness (perceived human similarity independent of artifacts)
  • Appropriateness (context-dependent, measures tonal and emotional fit to the stated role/scene).

Annotators supply comments for any score ≤3 (Audio Quality, Human Likeness) or always for Appropriateness, and indicate confidence (high/moderate/low) for Appropriateness.

• Realism Rubrics: A hierarchical, gated scoring system with up to ten dimensions:
  • Low-level: Pitch Variation, Rhythmic Naturalness, Stress & Emphasis.
  • Emotional: Emotion Accuracy (always), then—if the previous ≥3—Intensity and Dynamic Range.
  • Character: Voice Identity Matching, Trait Embodiment.
  • Contextual Relevance: Local Scene Fit, Global Story Coherence, Semantic Match.

Annotators review the Character Profile and Scene, with hard gates on dimensions if the voice or semantic context misaligns. Confidence (1–5) and optional comments are recorded.

3. Evaluation Criteria, Protocols, and Metrics

Speech-DRAME-EvalBench distinguishes between two strategy domains:

• Archetype Evaluation: Top-down assessment of adherence to broad cultural and scenario-specific archetypes, operationalized through rigorous Content Pass filtering and focused rubrics.
• Realism Evaluation: Bottom-up assessment grounded in real human delivery, applying the ten-dimensional rubric to both synthetic and real human speech, with gating to ensure that subjective emotion labels are only scored when base accuracy is sufficient.

Evaluation protocols include:

• Stratified data splits to avoid scenario overlap across training and test sets.
• Prompt-based (zeroshot/fewshot) or supervised SEM training: JSON input encapsulates role, scene, or character, alongside waveform.
• Zero-shot/few-shot ALLMs (e.g., Gemini2.5Pro, GPT4o-audio) are prompted for distributional scores; public models return metrics via token-level expectation.

Metric suite:

• Content Pass: binary accuracy
• Scalar dimensions: Pearson correlation to human averages, $$\rho_{X,Y} = \frac{\sum_i (X_i - \bar X)(Y_i - \bar Y)}{\sqrt{\sum_i (X_i - \bar X)^2} \sqrt{\sum_i (Y_i - \bar Y)^2}}$$
• Agreement (Realism): transformed sample-wise standard deviation, clamped to $[0,1]$. Fleiss' kappa for inter-annotator agreement.

4. Benchmark Analysis and Results

Key empirical findings reveal:

• Archetype Evaluation: Model-level means (Mandarin Doubao, Appropriateness $3.82\pm0.67$; Human Likeness $3.66\pm0.64$; Audio Quality $4.37\pm0.70$; Content Pass 0.985. English ChatGPT-4o: Appropriateness $3.46\pm0.97$, Human Likeness $3.49\pm0.79$, Audio Quality $1.96\pm0.69$, Content Pass 0.966). Notably, "FantasyOccupation" and "NamedCharacters" scenarios yield low appropriateness (means $\approx1.8$), while "DailyOccupation" and "SocialIdentity" are more favorable ($2.5$–$2.7$).
• Realism Evaluation: Inter-annotator agreement (IAA) per dimension generally resides in $0.86$–$0.88$; lowest in Trait Embodiment, highest in Local Scene Fit/Semantic Match. Score distributions are unimodal in prosodic dimensions; emotion-dimensions display mass at 1 (flowering from gating). Professional actors’ samples skew higher with less variance compared to amateurs.
• Case Studies: Context-sensitivity is robust; for instance, opera-style battlefield narration in a “modern café” scenario triggers context/semantic score minima and observable drops in prosody and trait dimensions, demonstrating the evaluator’s sensitivity to scene/persona mismatches.

5. Model Training, Evaluation, and Benchmark Utility

The benchmark supports both public and proprietary model assessment under a unified training/inference paradigm:

• SEM Training: LoRA (rank=16, $\alpha=32$, dropout=0.1), full-parameter, or gradient-accumulation strategies are applied, with cosine LR schedules and bf16 precision. At inference, models return either full soft distributions ($[1,5]$) or expected value per dimension.
• Evaluation Regimes: Benchmarks support zero-shot, few-shot, and supervised settings, with standard practices of sampling few-shot exemplars from the train partition, matched to mean human scores.

Implications for audio and speech researchers: The benchmark enables rigorous, context-aware alignment of model outputs to nuanced human ratings, distinguishing artifact-driven errors from failures in prosody or scenario-appropriate delivery. It enables fine-grained diagnosis of model weaknesses along rubric axes and contextual conditions.

6. Exemplary Application and Impact

By integrating bilingual, scenario-diverse, and contextually-rich data with high-resolution rubrics and transparent protocols, Speech-DRAME-EvalBench:

• Establishes reproducible metrics for system-level comparison of speech foundation models in role-play settings.
• Facilitates model tuning by providing granular feedback that isolates specific dimensions (e.g., emotional expressiveness, appropriateness) for targeted improvement.
• Supports both academic and applied contexts in generative role-play, voice acting evaluation, dialogue systems, and human–AI interaction research.

Adoption of Speech-DRAME-EvalBench provides a pathway for future development in robust, human-aligned automatic speech evaluation, enabling more sensitive assessment of emerging models’ prosodic, expressive, and contextual competence (Shi et al., 3 Nov 2025).