Multi-turn Speech Interaction Benchmark

• Multi-turn Speech Interaction Benchmark (MSIB) is a comprehensive framework that evaluates spoken dialogue systems across extended multi-turn interactions.
• It rigorously tests capabilities such as context tracking, paralinguistic modeling, and resilience to spontaneous speech phenomena.
• The benchmark leverages hybrid dataset construction and diverse evaluation protocols to simulate realistic, audio-native human-machine conversations.

A Multi-turn Speech Interaction Benchmark (MSIB) is a structured evaluation suite designed to rigorously assess the capabilities of spoken dialogue systems in sustained, multi-turn, audio-native human–machine conversations. MSIB benchmarks probe context tracking, instruction compliance, paralinguistics, dynamic speech phenomena, and role maintenance, providing critical insight into modeling strategies, generalization boundaries, and failure points that are obscured in single-turn or synthetic settings (Tong et al., 15 Oct 2025, Du et al., 22 Aug 2025, Zhang et al., 13 Nov 2025, Gosai et al., 16 Dec 2025).

1. Conceptual Foundations and Benchmark Objectives

The central objective of MSIB is to enable reproducible, fine-grained diagnosis of dialogue system competencies over extended spoken interaction, simulating the demands of naturalistic, continuous audio exchange. Key evaluation axes include:

• Multi-turn memory: retention and utilization of context spanning several turns, including long-range recall challenges and dependency chains.
• Paralinguistic modeling: ability to produce and interpret emotion, prosody, speaker characteristics, and vocal non-lexical cues within ongoing exchanges.
• Robustness to spontaneous phenomena: tracking self-corrections, interruptions, ambient noise, and mid-utterance repairs typical in spoken dialogue.
• Instruction following and dynamic behavioral adaptation: obviating prompt drift and context confusion through conversational shifts, multi-step directives, and role-play fidelity.
• Task-oriented and open-domain coverage: real-world scenarios spanning goal-oriented, creative, and emotionally nuanced dialogue.

MSIB frameworks surpass prior single-turn metrics by centering on conversational interaction as a temporally and pragmatically structured process, requiring both semantic and non-semantic modeling for credible performance evaluation (Tong et al., 15 Oct 2025, Ma et al., 30 Jul 2025).

2. Dataset Construction and Scenario Design

MSIBs are typically constructed through a hybrid pipeline, combining LLM-driven dialogue generation, human or agentic audio recording, paralinguistic/ambient annotation, and post-hoc rubric authoring:

• Dialogue Sourcing: Generated or curated to span domains such as domestic assistance, healthcare, institutional inquiry, entertainment, psychological counseling, and role-playing. Turns per dialogue range from 2–16, with segment-level and session-level sampling (Gosai et al., 16 Dec 2025, Du et al., 22 Aug 2025).
• Speaker and Role Diversity: Inclusion of multiple speakers, with control over age, gender, accent, and emotional state, using zero-shot voice conversion and reference audio for role-playing (Jiang et al., 4 Aug 2025, Gosai et al., 16 Dec 2025).
• Phenomena Coverage: Incorporation of core phenomena—semantic ambiguity, omission, coreference, overlapping speech, disfluency, background noise—across multilingual and multi-modal contexts (Ma et al., 30 Jul 2025, Gosai et al., 16 Dec 2025).
• Realism and Naturalness: In-the-wild collection (e.g., MMedFD), expert improvisation (Audio MultiChallenge), and synthetic failure induction protocols expose system weaknesses under realistic, unscripted conditions (Gosai et al., 16 Dec 2025, Chen et al., 24 Sep 2025).
• Annotation and Quality Control: Machine- and human-in-the-loop multilayered annotation and validation (e.g., iterative LLM rubric authoring, human spot checks, adaptive sampling).

Table: Dataset Construction Key Parameters

Feature	Typical MSIB Value	Benchmark Examples
Dialogues	200–5,800+	AudioMC: 452, MMedFD: 5,805, MTalk-Bench: ~270
Turns per Dialogue	2–16	AudioMC: 3–8, C³: avg. 6–10, InteractiveOmni: 2–10
Language Coverage	English, Chinese, Bilingual	C³, MULTI-Bench, SpeechRole
Audio Duration/Quality	14.99 h @ 48 kHz, 16 kHz	AudioMC, MMedFD
Role/Scenario Diversity	6–98 roles/scenarios	SpeechRole: 98, MTalk-Bench: 9, InteractiveOmni: 6

3. Evaluation Protocols and Metrics

MSIBs utilize advanced evaluation pipelines, relying on model- and human-as-judge protocols, often combining absolute rubrics and relative pairwise comparisons:

• Arena-Style (Pairwise) Evaluation: Blind head-to-head matchups with human or LLM judges select the superior output, with model Elo scores reflecting conversational dominance (Du et al., 22 Aug 2025).
• Rubric-Based (Absolute) Evaluation: Responses scored against multi-level, axis-specific rubrics (content, paralinguistics, ambient, coherence). Per-instance binary criteria (criterion met) with aggregate Average Pass Rate (APR) and Average Rubric Score (ARS) (Gosai et al., 16 Dec 2025, Du et al., 22 Aug 2025).
• Mean Opinion Score (MOS): Perceptual 1–5 scale ratings for Speech Quality and Content Quality obtained via human rater or LLM judge (Tong et al., 15 Oct 2025).
• Contextual Probes: Recall tasks (e.g., re-ask initial question), self-consistency verification, and dynamic instruction adherence with accuracy, precision, recall, F1 (Ma et al., 30 Jul 2025, Shen et al., 2023).
• Specialized Measures: For healthcare or knowledge-specific domains, concept-level WER (e.g., HC-WER), and semantic F1 over extracted entities (Chen et al., 24 Sep 2025).
• Agreement Metrics: Cohens’ kappa, Krippendorff’s alpha for inter-annotator/judge reliability.

Representative Metric Formulas:

• APR: $$APR = \frac{1}{N} \sum_{i=1}^{N} \prod_{j=1}^{N_i} r_{i,j}$$ where $r_{i,j}$ is binary rubric outcome on instance $i$, $j$-th criterion.
• MOS: $MOS_{m} = \frac{1}{|D|} \sum_{i \in D} s_{i,m}$ for dimension $m$, $|D|$ instances.
• Entity F1 (ASR): precision/recall evaluated on concept extraction from multi-turn output (Chen et al., 24 Sep 2025).

4. Comparative Analysis and Model Behavior

Empirical findings across MSIBs reveal distinctive trends in model capabilities and limitations:

• Consistent Multi-Turn Degradation: Dialogue quality, feature tracking, and instruction following degrade over consecutive turns, especially under noise or context drift (Zhang et al., 13 Nov 2025, Gosai et al., 16 Dec 2025).
• Distinct Modality Gaps: Audio-native output models trail text-output configurations in pass rates; context-length increases exacerbate memory lapses and incoherence (Gosai et al., 16 Dec 2025, Du et al., 22 Aug 2025).
• Axis-Specific Challenges:
  • Paralinguistic and Ambient Reasoning: Significant performance drop in emotion, prosody, and ambient sound reasoning compared to semantic content (–20–30 ponits) (Du et al., 22 Aug 2025, Gosai et al., 16 Dec 2025).
  • Voice Editing / Self-Repair: Models fail to handle in-turn corrections or mid-dialogue overwrites, often ignoring self-repair and issuing inaccurate summarizations (Gosai et al., 16 Dec 2025).
  • Long-Range Consistency: Self coherence and instruction retention drop sharply over long context windows (>3–5 minutes cumulative audio) (Gosai et al., 16 Dec 2025).
• Robustness in Safety: Some full-duplex architectures demonstrate stable refusal rates across turns, outperforming in hazardous scenario detection relative to instruction following (Zhang et al., 13 Nov 2025).
• Systematic Model Comparisons: Arena and Rubric evaluations align strongly only when score differences are large; LLM-as-judge protocols approach human agreement in content judgments, less so for nuanced nonverbal assessment (Du et al., 22 Aug 2025, Gosai et al., 16 Dec 2025).

5. Task Families, Scenario Taxonomies, and Protocol Extensibility

MSIBs feature modular extension mechanisms for task and scenario addition:

• Task Family Taxonomy: Daily assistance, correction handling, entity tracking, and safety pressure-tests under staged multi-step goal structures; role-play and emotional expression via structured profiles (Lin et al., 9 Oct 2025, Jiang et al., 4 Aug 2025).
• Scenario Envelope: Integration of family- and scenario-specific pacing regimes (fast, slow), speaker overlap, barge-in, correction, cross-turn reference, and entity co-reference (Lin et al., 9 Oct 2025).
• Automated Examiner and Judge Modules: Streaming-native APIs with LLM-based examiners dynamically enforce conversational flows, interruptions, and semantic goal compliance, while scoring is performed at turn-by-turn and session levels (Lin et al., 9 Oct 2025).
• Multi-lingual and Multi-modal Expansion: Benchmarks such as C³ and MULTI-Bench demonstrate the incorporation of bilingual capabilities and extension to audio-visual exchanges (Ma et al., 30 Jul 2025, Deng et al., 2 Nov 2025).
• Custom Rubrics and Semantics: Supports per-task rubric customization for domain-specific evaluation, e.g., medical concepts, emotion categories, or prosody control.

6. Implementation Challenges and Recommendations

Analysis of MSIB experimental outcomes highlights implementation bottlenecks and best practices:

• Overlapping Speech and Repair: High confusion rates without explicit floor-control signals or repair-tracking modules; mitigation via stronger prosodic cues or “hold on” utterances (Lin et al., 9 Oct 2025).
• Memory Module and Context Handling: Dynamic re-summarization and explicit memory modules improve performance on entity tracking and recall tasks (Lin et al., 9 Oct 2025, Ma et al., 30 Jul 2025).
• Protocol Calibration: Statistical significance in Arena/Rubric rankings requires large performance gaps; inclusion of both absolute and pairwise protocols is recommended for robust assessment (Du et al., 22 Aug 2025, Gosai et al., 16 Dec 2025).
• Hybrid Modality Pipelines: Combining audio input for fresh cues with text memory for history yields improved context management and robustness (Du et al., 22 Aug 2025).
• Joint Objectives and Multi-task Learning: Training strategies that blend classification and generative goals (e.g., EI+response) enhance paralinguistic and emotional intelligence metrics (Deng et al., 2 Nov 2025).
• Scaling and Efficient Annotation: Combination of LLM-powered rubric generation and expert annotation streamlines dataset expansion and quality assurance (Gosai et al., 16 Dec 2025).

7. Representative Benchmarks and Resources

Several public MSIBs form the current foundation for multi-turn spoken dialogue benchmarking:

Benchmark	Focus Domains	Notable Metrics / Protocols	Reference
InteractiveOmni MSIB	Multi-turn speech, role, emotion	MOS (Human/LLM), attribute compliance	(Tong et al., 15 Oct 2025)
MTalk-Bench	S2S LLMs, ambient, paralinguistics	Arena-pairwise Elo, Rubric APR/ARS, scenario axes	(Du et al., 22 Aug 2025)
Audio MultiChallenge	Audio-native, repair, memory	APR/ARS, LLM-as-judge, adversarial blueprints	(Gosai et al., 16 Dec 2025)
Full-Duplex-Bench-v2	Full-duplex, correction, safety	TT, IF, TSC, streaming protocol, task family	(Lin et al., 9 Oct 2025)
C³	Bilingual, memory, recall	Accuracy (LLM-as-judge), phonology (human)	(Ma et al., 30 Jul 2025)
SpeechRole	Role-playing, prosody, personality	8-dim ratio MOS, reference normalization	(Jiang et al., 4 Aug 2025)
MULTI-Bench	EI, paralinguistics, interactive EI	Gemini/DeepSeek EI scoring, utterance/dialogue	(Deng et al., 2 Nov 2025)
MMedFD	Full-duplex ASR, healthcare	WER/CER/HC-WER, LLM G-Eval/PairEval	(Chen et al., 24 Sep 2025)
MultiTurnCleanup	Transcript-level coherence	Token-wise F1, category labeling, BERT models	(Shen et al., 2023)

Most benchmarks combine modular APIs, reproducible pipelines, and open datasets, facilitating extensibility and deep comparability.

8. Future Directions

Key recommendations and open research problems for advancing MSIB design include:

• Deeper turn depth and extended session evaluation to uncover cumulative context effects and scaling limitations (Gosai et al., 16 Dec 2025, Du et al., 22 Aug 2025).
• Multi-modal and cross-lingual coverage to probe generalization and robustness in truly heterogeneous settings (Ma et al., 30 Jul 2025, Deng et al., 2 Nov 2025).
• Objective, audio-native metrics and alignment of LLM-as-judge protocols with human nonverbal assessment through joint training and explicit tag support (Gosai et al., 16 Dec 2025, Du et al., 22 Aug 2025).
• Integration of scenario- and axis-specific rubrics for specialized domains (e.g., medical triage, legal interview, high-stakes scenarios) (Chen et al., 24 Sep 2025, Gosai et al., 16 Dec 2025).
• Continuous data augmentation leveraging in-the-wild, agentic, and adversarial protocols to surface rare or emerging failure modes (Gosai et al., 16 Dec 2025).

MSIB frameworks embody a paradigm shift toward holistic, scenario-driven, and fine-grained evaluation of speech dialogue systems, driving research toward more human-like, context-aware, and robust multi-turn spoken interaction.