MSU-Bench: Benchmarking Audio and Music Understanding
- MSU-Bench is an ambiguous benchmark name covering speaker-centric multi-talker understanding in speech and complete score analysis in music.
- The speech benchmarks (2025 and 2026) employ distinct designs—open-ended vs. multiple-choice—to evaluate speaker grounding and multi-speaker interaction.
- The music benchmark assesses AI's ability to interpret full musical scores using both symbolic (ABC) and visual (PDF) modalities across multiple levels.
Searching arXiv for the cited MSU-Bench papers and related usage in Audio-Mind. MSU-Bench is an overloaded benchmark name in recent arXiv literature. In speech and audio research, it denotes a benchmark family for speaker-centric understanding of conversational multi-speaker scenarios; in music AI, it denotes the Musical Score Understanding Benchmark for complete-score reasoning. Within the speech literature, the name appears in two materially different public formulations: a 2025 four-tier, open-ended QA benchmark with 25 tasks and 1,232 QA pairs, and a 2026 two-tier, multiple-choice diagnostic benchmark with 16 tasks and 2,300 verified QA instances. A separate 2025 music benchmark uses the same acronym for 1,800 generative question-answer pairs over complete scores in ABC and PDF modalities (Wang et al., 11 Aug 2025, Sun et al., 22 Jun 2026, Dai et al., 24 Nov 2025).
1. Name, scope, and disambiguation
The acronym currently names at least three distinct benchmark formulations in arXiv-indexed work. Two belong to multi-speaker spoken language understanding, while one belongs to symbolic and visual music understanding.
| Referent | Domain | Core formulation |
|---|---|---|
| MSU-Bench (2025) | Conversational multi-talker audio | Four-tier speaker-centric benchmark; open-ended QA; 25 tasks; 1,232 QA pairs |
| MSU-Bench (2026) | Speaker-centric multi-speaker conversations | Two-tier diagnostic benchmark; multiple-choice QA; 16 tasks; 2,300 verified instances |
| Musical Score Understanding Benchmark | Music notation understanding | 1,800 generative QA pairs from 150 complete scores in ABC and PDF |
The two speech benchmarks share a common concern with speaker-centric understanding in realistic multi-speaker audio, but they differ in hierarchy, task inventory, prompting, and scoring. The 2025 version avoids multiple-choice and uses open-ended QA with LLM-based grading, whereas the 2026 version uses four-option multiple-choice QA with exact-match accuracy and explicit diagnostic distractors. The music benchmark is unrelated in modality and problem formulation, despite the shared acronym. This suggests that the term “MSU-Bench” must be interpreted through the associated paper title, task definition, and evaluation protocol rather than through the acronym alone (Wang et al., 11 Aug 2025, Sun et al., 22 Jun 2026, Dai et al., 24 Nov 2025).
2. Four-tier speaker-centric benchmark for conversational multi-talker understanding
The 2025 speech benchmark defines MSU-Bench as a hierarchical, speaker-centric benchmark for conversational multi-talker understanding. Its stated motivation is that real-world spoken interactions are multi-speaker, with interruptions, overlaps, role shifts, and social dynamics that exceed the scope of single-speaker or isolated-task benchmarks. It positions itself against prior resources such as AudioBench, AIR-Bench, MMSU, MMAU, VoiceBench, and SD-Eval, arguing that these are not designed to probe multi-speaker interaction logic or to evaluate speaker-centric abilities systematically in authentic dialogues (Wang et al., 11 Aug 2025).
Its design is explicitly hierarchical. Tier 1 covers single-speaker static attribute understanding, including Speaker Recognition, Speaker Attribute Comprehension, and Speaker Paralinguistic Analysis. Tier 2 covers single-speaker dynamic attribute understanding, including Speaker Dynamic Analysis and Speaker Cultural Identity Integration. Tier 3 addresses multi-speaker background understanding through Multi-Speaker Scene Inference and Multi-Speaker Relationship Inference. Tier 4 targets multi-speaker interaction understanding through Multi-Speaker Transcription, Multi-Speaker Interaction Analysis, and Multi-Speaker Contextual Reasoning. The progression is described as single-speaker to multi-speaker, static to dynamic, and perception to reasoning.
The benchmark is also formalized as a “5M” design: multi-tier, multi-speaker, multi-lingual, multi-scenario, and multi-task. It uses authentic multi-speaker recordings rather than TTS, spanning English and Chinese, and draws from six real-person corpora: Chinese and English near-field telephone data, Chinese far-field meetings, English far-field home conversations, and Chinese and English film or TV dialogues. Evaluation clips are 60–120 seconds long, contain at least two speakers, and include near-field, far-field, and film-like acoustic conditions. The benchmark contains 25 tasks totaling 1,232 QA pairs, and is presented as an evaluation-only testbed rather than a train/validation/test corpus.
A defining property of this version is its open-ended task formulation. Evaluated systems receive the audio and a task-specific instruction or prompt. Outputs may be labels, yes/no answers, per-speaker transcripts in a structured dictionary-like format, or concise textual rationales for reasoning tasks. Multiple-choice is explicitly avoided to reduce cueing. Scoring is delegated to an LLM grader, Deepseek v3, which evaluates relevance, accuracy, and causal soundness. The paper also lists standard formulas for accuracy, precision, recall, F1, WER, and DER for alternative evaluations, but reports rubric-based LLM scores as the official protocol.
The data creation pipeline is partly automated and partly manual. Speaker-attributed transcription is produced with SDASR, speaker attributes are tagged via VoxProfile, dialogue segments are selected with Deepseek v3, QA pairs are generated with capability-specific templates, and all selected items are manually checked. Inter-annotator agreement is not reported. The benchmark’s significance lies in its attempt to unify perception, temporal tracking, speaker attribution, social role inference, and cross-speaker reasoning within a single speaker-centric hierarchy.
3. Two-tier diagnostic benchmark for speaker grounding and dialogue reasoning
The 2026 speech benchmark, authored by Zhaokai Sun, Shuai Wang, Zhennan Lin, Chengyou Wang, Dehui Gao, Yuang Cao, Chunjiang He, Pan Zhou, and Lei Xie, redefines MSU-Bench as a diagnostic benchmark for speaker-centric understanding in conversational multi-speaker scenarios. Its core contribution is a two-tier framework progressing from speaker grounding to multi-speaker dialogue reasoning, with 16 speaker-centric tasks and 2,300 verified multiple-choice QA instances (Sun et al., 22 Jun 2026).
Tier 1, “Speaker grounding and identification,” contains 10 tasks: Accent Identification, Age Recognition, Gender Identification, Emotion Recognition, Speaker Profiling, Reverse Speaker Retrieval, Speaker Retrieval, Speaker-specific Viewpoint Summarization, Speaker Counting, and Speaker Verification. Tier 2, “Multi-speaker dialogue reasoning,” contains 6 tasks: Emotion Interaction Reasoning, Multi-speaker Viewpoint Summarization, Background Inference, Role/Identity Identification, Dialogue Act Recognition, and Q&A Structure Identification. The benchmark is multilingual in Chinese and English, multi-scenario across telephone calls, meetings, podcasts, and movies, and requires at least two speakers per instance.
A distinctive feature of this formulation is its speaker-referencing scheme. Items may anchor the target speaker through No Index, Time Index, Transcript Index, Speaker Index, or Complex Index. These schemes are designed to vary grounding difficulty. Time Index is reported as consistently hardest, while Complex Index often improves performance by providing complementary localization cues. This makes temporal localization and cross-cue alignment first-class components of the evaluation.
The annotation pipeline is Gemini-assisted and human-in-the-loop. Gemini first selects informative, coherent dialogue segments; diarization and transcripts are produced by the Volcano API; Gemini then tags speaker identity cues, paralinguistic events, and sound events; QA candidates are generated under the referencing schemes; and trained annotators verify metadata correctness, answer determinacy, and exact one-correct-option format. The paper reports initial QA validity of 95% for Tier 1 and 86% for Tier 2, and human–ground truth answer agreement of 98% for Tier 1 and 96% for Tier 2 on retained items.
Its evaluation protocol is deterministic. All tasks are four-option multiple-choice. Models must output exactly one letter among A, B, C, or D. The primary metric is exact-match accuracy, reported per task, capability group, tier, and referencing scheme:
The benchmark further encodes diagnostic error types through distractor design: wrong-speaker (WS), hallucination (HAL), unknown (UNK), and instruction-following failures (INS). This turns incorrect answers into structured evidence about whether a model failed at grounding, content inference, uncertainty calibration, or output compliance.
4. Empirical findings and diagnostic patterns in the speech benchmarks
The 2025 and 2026 speech formulations differ in protocol, so their scores are not directly comparable. Their empirical conclusions are nonetheless aligned in one respect: both report substantial degradation when the task requires robust speaker grounding, temporal tracking, or interaction-level reasoning rather than coarse semantic inference (Wang et al., 11 Aug 2025, Sun et al., 22 Jun 2026).
In the 2025 open-ended benchmark, Gemini-2.5-Pro achieves an overall average of 0.59, Gemini-2.5-Flash 0.55, GPT-4o-Audio 0.52, Qwen2.5-Omni 0.37, and Kimi-Audio 0.35. The paper emphasizes performance decline with task complexity, especially on Tier 4 multi-speaker interaction understanding. For Gemini-2.5-Pro, Tier 3 averages 0.76 while Tier 4 falls to 0.51. The benchmark also reports a persistent gap between commercial and open-source systems, with an overall difference of approximately 22–24 points absolute between Gemini-2.5-Pro and the leading open-source baselines. Error analysis shows that Gemini-2.5-Pro’s sampled mistakes are dominated by perceptual errors at 56.06%, followed by reasoning errors at 37.37%, while Kimi-Audio exhibits 48.24% answer extraction errors. The paper further notes similar difficulty trends across English and Chinese, with no clear over-optimization to one language.
In the 2026 multiple-choice diagnostic benchmark, overall exact-match accuracy ranges from 0.19 for Qwen2.5-Omni to 0.77 for Gemini-3-Flash. Gemini-3-Flash reports Tier 1 = 0.73, Tier 2 = 0.84, and Overall = 0.77; Gemini-2.5-Pro and Gemini-2.5-Flash follow at 0.70 and 0.69 overall. Among open-source systems, MiMoAudio is the strongest with Tier 1 = 0.52, Tier 2 = 0.64, and Overall = 0.56. The speaker-referencing ablation identifies Time Index as consistently hardest and Complex Index as frequently beneficial. Diagnostic error distributions show that weaker systems often default to UNK on harder reasoning problems, whereas stronger systems increasingly fail through WS, indicating that fine-grained speaker attribution remains the core bottleneck even after general capability improves.
Taken together, the two speech benchmarks indicate that current large audio-LLMs are relatively stronger on tasks driven by semantic content or broad contextual inference than on tasks requiring exact attribution of content, speaker state, turn structure, or causal interaction across speakers. A plausible implication is that the limiting factor is not only general reasoning capacity but also the stability of speaker-grounded intermediate representations under overlap, interruption, and rapid turn switching.
5. Use of MSU-Bench in agentic audio understanding
In the Audio-Mind framework, MSU-Bench is treated as a focused testbed for “information-dense multi-speaker understanding.” The paper adopts the benchmark’s official multiple-choice answer format and scores predictions by exact match against the labeled correct option. For MSU-Bench, Audio-Mind is instantiated with Gemini 2.5 Pro as the frontend large audio-LLM and Gemini 3.0 Flash as the planner, both with “thinking” enabled, a decoding temperature of 0.05, and at most 15 planner decision steps per example (Wang et al., 27 May 2026).
Audio-Mind’s architecture is organized around conditional evidence acquisition. The system receives audio inputs and a natural-language question , and aims to produce an answer grounded in the audio and satisfying the expected output format. The planner first constructs a question-oriented perception prompt that restates the task, identifies listening targets, and asks the frontend to report uncertainty. The frontend returns structured observations and, when possible, a tentative answer. The planner then writes a lightweight plan indicating which claims appear answerable by direct perception and which require targeted verification.
The evidence acquisition loop can call bounded tools, request targeted re-listening on selected audio after operations such as trimming or separation, proceed to answer generation, or stop with fail. Available tool categories include speech and speaker processing such as ASR, diarization, and speaker verification; temporal segmentation; audio derivation; acoustic and music feature analysis; metadata validation; and signal-level visualization or inspection. The planner is instructed to treat tool outputs as bounded evidence rather than as automatic ground truth. After evidence is deemed sufficient, the frontend receives a neutral evidence summary plus the original audio and produces the final answer; a separate format checker validates structure only.
On MSU-Bench, Audio-Mind reports 0.828 average accuracy. The direct Gemini 2.5 Pro baseline reaches 0.819, and the matched-backbone AudioGenie-Reasoner baseline reaches 0.789. Other reported baselines are Qwen3.5-Omni at 0.763, Gemini 2.5 Flash at 0.682, MiMoAudio at 0.600, StepAudio2 at 0.480, and Kimi-Audio at 0.460. The paper does not provide per-category breakdowns, confidence intervals, or significance tests for MSU-Bench. Its central interpretation is that, under strong audio frontends, always-on agentic decomposition can become an orchestration bottleneck when the workflow compresses or overrides holistic audio-grounded judgment. Audio-Mind attempts to avoid this by preserving the frontend as the final decision-maker over the original audio, invoking tools only to close specific evidence gaps, and separating evidence acquisition from answer synthesis.
The same auditable trace machinery used elsewhere in Audio-Mind is also produced on MSU-Bench. Recorded artifacts include the perception prompt, frontend observations and uncertainties, planner rationales, tool invocations and outputs, derived audio artifacts, targeted re-listening prompts, the neutral evidence summary, the final answer, and format-check feedback. In multi-talker scenes, this trace is intended to expose whether the system localized the right speaker or segment and whether ASR or diarization evidence was interpreted within category boundaries.
6. Musical Score Understanding Benchmark
A separate benchmark, also named MSU-Bench, stands for the Musical Score Understanding Benchmark. It is defined as a human-curated benchmark for evaluating whether contemporary AI models can understand complete musical scores rather than isolated notes or short excerpts. The dataset contains 1,800 generative QA pairs drawn from 150 complete scores and spans textual ABC notation and visual PDF modalities (Dai et al., 24 Nov 2025).
Its design is explicitly level-based. Level 1, Onset Information, covers composer, title, key and time signatures, clefs, instrumentation, tempo, expressive indications, and anacrusis. Level 2, Notation & Note, covers localized notational and pitch-level features such as accidentals, rests, ornaments, articulations, dynamics, and bar-level changes. Level 3, Chord & Harmony, covers harmonic analysis including chord qualities, functions, inversions, cadences, suspensions, anticipations, and modulation. Level 4, Texture & Form, targets motifs, theme organization, texture, orchestration, register, and large-scale structure. Each of the 150 scores receives 12 questions, three per level, yielding a balanced 25% per level across the benchmark.
A central methodological claim is that ABC notation provides a structured symbolic pathway that reduces visual mislocalization and hallucination. In this representation, metadata such as title, composer, tempo, meter, and key are explicit; voices and clefs are separated; bar positions can be marked; accidentals and chords are text tokens rather than pixels. The visual pathway, by contrast, exposes the full localization problem of native score images. The benchmark therefore evaluates both modalities to probe whether models can sustain correctness across symbolic depth rather than merely answer isolated questions.
Evaluation includes zero-shot and LoRA fine-tuned settings. Outputs are judged by a voting process involving ChatGPT-5, Claude Sonnet 4, and Gemini 2.5 Pro. Accuracy is reported per level and overall, and the paper introduces Level-wise Success Rate, a stricter measure of sustained multilevel correctness:
The zero-shot textual setting is led by Gemini 2.5 Pro at 49.44% overall, while the zero-shot visual setting is markedly lower, with Claude Opus 4 reaching 24.22% overall. The paper emphasizes that models can answer some isolated questions while still collapsing under multilevel consistency constraints. Fine-tuning yields large gains: for example, Qwen3-4B improves from 23.05% to 46.94% overall in textual QA, and Qwen2.5-VL-3B-Instruct improves from 24.72% to 50.00% in the PDF-only setting. Minimal forgetting is reported on MMLU after LoRA adaptation.
Because this music benchmark is entirely unrelated to conversational multi-speaker SLU, the shared acronym creates a genuine bibliographic ambiguity. A plausible implication is that any serious citation, leaderboard comparison, or benchmark integration should name the full title and modality rather than rely on “MSU-Bench” in isolation.
