- The paper presents a novel hierarchical evaluation suite distinguishing speaker identification from advanced multi-speaker reasoning.
- It introduces 2,300 QA items across 16 diagnostic tasks, leveraging diverse referencing schemes and rigorous human curation.
- Comparative tests on multiple models reveal significant errors in temporal grounding and speaker attribution, guiding future LALM improvements.
Speaker-Centric Benchmarks for Multi-Speaker Conversational Understanding: An Analysis of MSU-Bench
Motivation and Background
MSU-Bench addresses a critical evaluation gap in Spoken Language Understanding (SLU) and Large Audio LLMs (LALMs): the absence of robust, speaker-centric benchmarks tailored for realistic multi-speaker conversational scenarios. Whereas prior benchmarks focus on single-speaker activities or limited subtasks (ASR, emotion recognition, diarization), they inadequately capture the complexity of speaker grounding, interaction dynamics, and context reasoning fundamental to natural, multi-party dialogue. With the progression toward unified audio-to-text LALMs, this deficiency hampers diagnosis and progress in building agents capable of nuanced conversational comprehension and multi-speaker coordination.
Benchmark Design and Task Taxonomy
MSU-Bench introduces a two-tier hierarchical evaluation suite that systematically disentangles speaker grounding from multi-speaker discourse reasoning. The benchmark comprises 2,300 meticulously curated QA items spanning 16 diagnostic tasks distributed across two major tiers:
- Tier 1 targets speaker identification and grounding: This includes tasks such as speaker retrieval, verification, profiling, attribute recognition (e.g., accent, age, gender, emotion), and viewpoint summarization. These demand precise alignment of linguistic and paralinguistic features with speaker indices, timestamps, or utterance snippets, reflecting real-world phenomena such as rapid turn-taking, overlap, or disfluency.
- Tier 2 encompasses higher-order multi-speaker reasoning: These tasks probe a model's capacity for background inference, role/identity identification, dialogue act recognition, emotion interaction tracing, viewpoint aggregation, and Q&A structure discrimination. The emphasis is on the inter-speaker relationships, context tracking, and reasoning over conversational structure.
The benchmark further introduces five explicit speaker-referencing schemes (audio-anchor, time-span reference, transcript quote, speaker index, and compound-cue combinations) to flexibly manipulate the complexity and grounding requirements per item. These are instantiated over eight corpora, covering English and Chinese, telephone, meeting, podcast, and movie domains, thus exposing models to a variety of acoustic and interactional conditions.
Data Construction Pipeline
Central to MSU-Bench is a scalable pipeline that integrates LLM-based (Gemini) dialogue quality filtering and prompt-based candidate generation with rigorous human-in-the-loop curation. Segments are pre-selected for informativeness and coherence, with audio and diarization annotations provided via automated and manual tools. Multiple-choice QA items are generated and iteratively revised to guarantee answer determinacy, realistic distractor balance, and robust format compliance. The final annotation protocol is evidenced to achieve over 95% validity and 96–98% ground-truth consensus.
Evaluation Protocol and Comparative Results
All evaluated systems are tested zero-shot, with no task- or data-specific fine-tuning. The main accuracy metric is exact-match to a single correct option per question, computed per task, tier, and referencing scheme. Nine representative models, including both open-source (e.g., Qwen3-Omni, MiMoAudio, AudioFlamingo-3, StepAudio2, Kimi-Audio) and closed-source (Gemini) families, are systematically assessed.
- Closed-source models (notably Gemini-3-Flash) achieve the highest overall accuracy (0.77 overall; 0.73 Tier 1; 0.84 Tier 2), but performance on heavy temporal and speaker-referential reasoning is still suboptimal.
- The best open-source model, MiMoAudio, attains 0.56 overall, showing relative strength especially on Tier 2 tasks (0.64), but a consistent gap to Gemini remains.
- Qwen3-Omni and AudioFlamingo-3, while multimodal, lag substantially in speaker-centric and higher reasoning capability, underscoring that general-purpose modality scaling does not guarantee robust multi-speaker dialogue understanding.
A clear finding is that performance drops are sharpest under temporal (Time Index) grounding, with speaker-attribute errors (wrong-speaker) dominating in highly capable models and unknown/indeterminate errors prevalent in weaker systems. Additional anchoring cues substantially improve attribution and reasoning performance, reinforcing the importance of multi-faceted speaker localization.
Diagnostic Error and Scheme Analysis
Error analysis reveals error stratification by model proficiency:
- Weaker systems (e.g., Qwen-Omni) default to "unknown" when required to reason across speaker-context, indicating limitations in grounding or context memory.
- Stronger systems (e.g., MiMoAudio, Gemini-3-Flash) shift failure modes toward fine-grained speaker-identity confusion (wrong-speaker attribution), particularly in complex, high-overlap exchanges or when references are indirect.
- Hallucination and format failures are infrequent but present, emphasizing the value of objective rather than free-form QA protocols for benchmarking.
Speaker-referencing schemes strongly modulate effective system performance, with "No Index" (acoustic snippet) enabling the highest accuracy, "Time Index" inducing maximal failures, and "Complex Index" allowing partial recovery via reference redundancy and cross-modal synergy.
Practical and Theoretical Implications
The systematic evaluation afforded by MSU-Bench highlights open research challenges central to deploying conversational AI in realistic, multi-party environments:
- Temporal speaker grounding and attribution remain non-trivial for both open and closed systems, impeding accurate downstream reasoning and agent interaction.
- Progress in LALMs must be further coupled with structured conversational understanding, granular context tracking, and robust handling of conversational phenomena such as interruptions or overlaps.
- The benefit of diverse grounding cues supports future model architectures that explicitly encode and align multi-view referencing (audio, text, time, speaker-index).
- Benchmark-driven diagnosis and stratification of error modes offer guidance for modular improvement (e.g., speaker-diarization heads, temporal localization, utterance-referencing modules).
Future Directions
MSU-Bench establishes a new diagnostic standard. Future work should pursue:
- Expansion toward more diverse language, domain, and acoustic settings.
- Inclusion of active speaker modeling and interactive dialogue dynamics.
- Development of open-source LALMs with explicit, trainable speaker-anchored inductive biases.
- Exploration of continual learning settings, robustification to noisy diarization, and multi-stream fusion.
Conclusion
MSU-Bench delivers a comprehensive, scalable, and speaker-centric benchmark for evaluating multi-speaker conversational understanding in LALMs. By revealing persistent gaps in temporal and speaker attribution and quantifying model weaknesses in higher-order reasoning, MSU-Bench provides actionable direction for both short-term model improvements and long-term research on genuinely interactive conversational intelligence (2606.22868).
