Multi-Turn Speech Interaction Benchmark

• Multi-turn speech interaction benchmark is a systematic evaluation resource that measures spoken dialogue systems across extended, context-rich conversations.
• It integrates realistic multi-turn configurations including evolving instructions, emotional adaptation, and ambient cues using standardized datasets and metrics.
• The benchmark drives diagnostic error analysis and model improvements by assessing long-context reasoning, memory retention, self-coherence, and voice editing capabilities.

A multi-turn speech interaction benchmark is a systematic evaluation resource designed to measure and dissect the capabilities of spoken dialogue systems across extended, context-rich conversational settings involving multiple turns of speech. Such benchmarks occupy a foundational role in the development of robust, context-sensitive, and expressive speech models, providing standardized datasets, evaluation protocols, and metrics for end-to-end assessment. Core to this paradigm is the inclusion of realistic multi-turn configurations, diverse interaction phenomena (e.g., evolving instructions, emotional adaptation, voice edits), and multi-granular scoring schemes to probe spontaneous conversation, attribute retention, and nontrivial aspects of human-like speech interaction.

1. Benchmark Typology and Motivation

Multi-turn speech interaction benchmarks emerge from the need to transcend the limitations of single-turn or synthetic testbeds that typically fail to capture the cross-turn complexity, disfluencies, entity tracking, and evolving user goals inherent to natural spoken dialogue. Benchmarks such as Audio MultiChallenge (Gosai et al., 16 Dec 2025), Full-Duplex-Bench-v2 (Lin et al., 9 Oct 2025), InteractiveOmni’s Multi-Turn Speech Interaction Benchmark (Tong et al., 15 Oct 2025), MTalk-Bench (Du et al., 22 Aug 2025), Multi-Bench (Deng et al., 2 Nov 2025), and SpeechRole (Jiang et al., 4 Aug 2025), among others, have thus been developed to target:

• Long-context reasoning and instruction/constraint adherence;
• Robust handling of paralinguistic variation, correction, and role-playing;
• Audio-native understanding (including ambient cues and human disfluency);
• Empirical, instance-level evaluation grounded in high-fidelity datasets.

A benchmark of this class typically provides curated multi-turn conversational data, annotation schemas for dialogue phenomena, and rigorous metrics using both human and model-based raters, enabling quantitative comparison and diagnostic error analysis across S2S (speech-to-speech), cascaded, and end-to-end models.

2. Task Definitions and Challenge Axes

Multi-turn speech benchmarks formalize tasks along multiple axes reflecting realistic user–agent interactions. Typical axes include:

• Instruction Retention: Models must persistently apply or update global user instructions over subsequent turns, including evolving constraints, replacements, or additions (as formalized in SEQUOR (Canaverde et al., 7 May 2026)).
• Inference/Memory: Requires the model to recall scattered facts, context, or audio signals (e.g., paralinguistic cues, background events) from previous turns to inform current responses (Gosai et al., 16 Dec 2025, Sirdeshmukh et al., 29 Jan 2025).
• Self-Coherence: Ensures model consistency with prior assertions and corrects sycophancy or contradiction over time (Sirdeshmukh et al., 29 Jan 2025, Gosai et al., 16 Dec 2025).
• Correction Handling/Voice Editing: Tests the model’s ability to process mid-utterance repairs, retroactive edits, and disfluency-rich speech (e.g., “no, actually, make that…”), isolating final user intent (Gosai et al., 16 Dec 2025, Lin et al., 9 Oct 2025).
• Role-Playing and Persona: Evaluates the agent’s ability to assume and sustain a predefined character, including vocal style, personality, and knowledge grounding (Jiang et al., 4 Aug 2025, Tong et al., 15 Oct 2025).
• Emotional Intelligence: Measures emotional recognition, reasoning, and support in longitudinal spoken interaction (Deng et al., 2 Nov 2025).
• Ambient/Auditory Cues: Assesses recognition and reasoning over environmental audio, speaker overlap, and ambient noise robustness (Chen et al., 24 Sep 2025, Du et al., 22 Aug 2025).

Table 1 illustrates mapping between major benchmarks and challenge axes:

Benchmark	Instruction	Memory	Correction	Role/Persona	Emotion	Ambient
Audio MultiChallenge (Gosai et al., 16 Dec 2025)	✓	✓	✓	✓	—	✓
SEQUOR (Canaverde et al., 7 May 2026)	✓	✓	✓	—	—	—
Full-Duplex-Bench-v2 (Lin et al., 9 Oct 2025)	✓	✓	✓	—	—	✓
SpeechRole (Jiang et al., 4 Aug 2025)	✓	—	—	✓	✓	—
Multi-Bench (Deng et al., 2 Nov 2025)	—	✓	—	—	✓	—
MTalk-Bench (Du et al., 22 Aug 2025)	✓	✓	—	—	✓	✓

3. Dataset Construction and Scenario Realism

Multi-turn benchmarks are characterized by methodical data collection paradigms. Recent work emphasizes:

• Authentic Constraint Extraction: SEQUOR (Canaverde et al., 7 May 2026) mines 1.4k verifiable constraints from over a million LM–user exchanges, filtered for non-triviality and cross-task applicability (≥70% criteria), and categorizes them as linguistic, stylistic, format, or numeric.
• Persona-driven Simulation: SEQUOR generates 200 persona-driven agendas, yielding 50-turn dialogues per persona; SpeechRole (Jiang et al., 4 Aug 2025) covers 98 roles, each mapped to a real audio profile.
• Audio-native Generation: InteractiveOmni (Tong et al., 15 Oct 2025), Audio MultiChallenge (Gosai et al., 16 Dec 2025), and MTalk-Bench (Du et al., 22 Aug 2025) synthesize or record human-like, multi-turn speech with preserved disfluencies, emotion, and background noise.
• Multi-Speaker and Overlap Handling: MMedFD (Chen et al., 24 Sep 2025) and Full-Duplex-Bench-v2 (Lin et al., 9 Oct 2025) curate fully synchronous, mixed-channel dialogues exposing real-world speaker overlap, turn-taking, and barge-in events.
• Emotion and Support: Multi-Bench (Deng et al., 2 Nov 2025) integrates emotional counseling corpora and paralinguistic prototypes.

Such benchmarks ensure diversity in role, task, environmental conditions, and prompt engineering, providing robust stress tests for speech interaction models.

4. Evaluation Protocols and Metrics

Scoring frameworks are anchored in reproducible, discriminative, and, when possible, human-aligned metrics:

• Rubric-Based and LLM-as-Judge Protocols: Instance-specific binary or ordinal criteria (pass/fail/range) allow an LLM or human rater to score each model response (Sirdeshmukh et al., 29 Jan 2025, Gosai et al., 16 Dec 2025, Du et al., 22 Aug 2025, Lin et al., 9 Oct 2025). For Audio MultiChallenge, APR (Average Pass Rate) and ARS (Average Rubric Score) directly measure strict and soft task success.
• Aggregate MOS and Content Quality: InteractiveOmni’s MSIB (Tong et al., 15 Oct 2025) uses both model and human MOS for speech and content, on a 1–5 scale.
• Objective Speech Metrics: Classic ASR-derived metrics (WER, CER, HC-WER), prosodic similarity (KL divergence), and semantic similarity (embedding cosine) are used for end-to-end models (Chen et al., 24 Sep 2025, Jiang et al., 4 Aug 2025).
• Arena-Style and Elo Ranking: MTalk-Bench (Du et al., 22 Aug 2025) complements absolute scoring with pairwise head-to-head comparison and dynamic Elo updates for robust model ranking, controlling for position/length bias.
• Task-Specific Accuracy: SEQUOR (Canaverde et al., 7 May 2026) defines per-turn constraint adherence accuracy, averaged over task regimes. Multi-Bench (Deng et al., 2 Nov 2025) computes utterance-level precision, recall, and F₁ for emotion and paralinguistic recognition.

Table 2 summarizes representative metrics:

Protocol	Scalar Metric(s)	Judge Modality
APR, ARS (Gosai et al., 16 Dec 2025)	Pass Rate, Rubric Score	LLM, Human
MOS (Tong et al., 15 Oct 2025)	1–5 (Speech/Content)	Expert, Model-as-Judge
WER, HC-WER (Chen et al., 24 Sep 2025)	Error Rates	Alignment-Driven
Elo (Du et al., 22 Aug 2025)	Elo (pairwise win)	Human, LLM
Accuracy (Canaverde et al., 7 May 2026)	Per-Turn Constraint	LLM Judge

5. Empirical Findings and Model Limitations

Results across recent multi-turn speech benchmarks consistently reveal substantial performance gaps in current model architectures:

• Constraint Retention Degradation: In SEQUOR (Canaverde et al., 7 May 2026), even the strongest system (Gemini 3.1 Flash Lite) shows a 12% drop (86%→74%) from turn 1 to 50 on a single constraint, and up to 38–63 point drops when multiple constraints are introduced or updated.
• Axis-Specific Weaknesses: Audio MultiChallenge (Gosai et al., 16 Dec 2025) reports a maximum APR of 54.65% (Gemini 3 Pro Preview), with Voice Editing (APR=17.99%) and Self-Coherence over long context (APR=20%) particularly challenging.
• Memory and Reasoning Failures: In Multi-Bench (Deng et al., 2 Nov 2025), style and emotion inference (advanced track) drops to 26–64% accuracy, even as basic emotion or paralinguistic tasks reach 50–70%.
• Role-Playing and Persona Drift: End-to-end SRPAs struggle to maintain prosodic and personality consistency over turns (Jiang et al., 4 Aug 2025).
• Noise and Ambient Robustness: Turn overlap, background cues, and real-world latency constraints cause notable WER and task failures (Chen et al., 24 Sep 2025, Lin et al., 9 Oct 2025).

These findings collectively indicate that even frontier ASR and S2S models—despite near-perfect short-turn or synthetic-task accuracy—face intrinsic context-allocation, memory, and compositional generalization bottlenecks in multi-turn speech (Canaverde et al., 7 May 2026, Gosai et al., 16 Dec 2025, Sirdeshmukh et al., 29 Jan 2025).

6. Design Principles and Open Challenges

Key methodology advances include:

• Hybrid Human–Model Data Pipelines: Many benchmarks use LLM-based synthetic scaffolding with expert human review to generate diverse, challenging scenarios (MultiChallenge (Sirdeshmukh et al., 29 Jan 2025), Audio MultiChallenge (Gosai et al., 16 Dec 2025)).
• Explicit Memory and Context Packaging: To improve long-range retention, methods such as explicit context concatenation, dialogue memory states, and diarization-informed context windows are increasingly adopted (Chen et al., 24 Sep 2025, Canaverde et al., 7 May 2026).
• Specialized Rubric Engineering: Fine-grained, binary rubrics, often crowd- or expert-sourced and LLM-evaluated, improve scoring reliability and interpretability (Gosai et al., 16 Dec 2025, Sirdeshmukh et al., 29 Jan 2025).
• Scenario-Driven Expansion: Extensible architectures such as Full-Duplex-Bench-v2 (Lin et al., 9 Oct 2025) support rapid addition of new task families (e.g., safety, correction-handling, entity tracking).

Persistent open challenges include faithful modeling of mid-turn disfluency and correction, layering of overlapping instructions, dynamic persona/role maintenance, multimodal memory fusion, and real-time streaming evaluation—particularly under noisy, unscripted, and out-of-domain conversational settings.

7. Future Directions and Benchmark Evolution

Ongoing advancements in multi-turn speech interaction benchmarks point toward:

• Multimodal Integration: Expanding beyond audio to video, gesture, and shared context for richer interaction evaluation (Tong et al., 15 Oct 2025).
• Audio-Native Training and Evaluation: Increased emphasis on speech-to-speech objectives, fine-grained prosodic and paralinguistic modeling, and speech-native LLM-judge co-development (Gosai et al., 16 Dec 2025, Du et al., 22 Aug 2025).
• Scalability and Diversity: Scaling to thousands of natural dialogues, diverse languages, extended scenario duration, and wider coverage of user/population demographics (Canaverde et al., 7 May 2026, Deng et al., 2 Nov 2025).
• Automated Attribute Scoring: Development of automatic metrics for prosody, emotion, and attribute fidelity to complement or replace subjective MOS.
• Real-Time and Full-Duplex Probing: Increasing adoption of streaming, barge-in, and overlap-handling tasks (Lin et al., 9 Oct 2025, Chen et al., 24 Sep 2025).

Comprehensively, multi-turn speech interaction benchmarks are positioned as a central pillar in advancing end-to-end dialogue system robustness, enabling deep empirical analysis of memory, context integration, and naturalness—essential for deploying next-generation, humanlike conversational agents.