MMAU Benchmark: Audio Intelligence Evaluation

Updated 5 March 2026

MMAU Benchmark is a large-scale evaluation suite that tests both audio perception and reasoning across speech, music, and environmental sounds.
It curates 10,000 human-annotated audio clips paired with 27 task archetypes to assess temporal sequences, causal links, and abstract attributes.
The benchmark provides rigorous diagnostics for audio-language models, revealing performance gaps and guiding future improvements in auditory AGI.

The Massive Multi-Task Audio Understanding and Reasoning (MMAU) benchmark is a large-scale evaluation suite created to probe both perception and reasoning capabilities in AI systems across the full span of real-world audio—speech, music, and environmental sounds. By curating diverse, expert-level tasks and enforcing classification and reasoning in a multiple-choice framework, MMAU catalyzes progress in general audio intelligence beyond conventional audio tagging or captioning tasks. MMAU’s rigor, skill diversity, and balanced domain coverage make it the central testbed for audio-language modeling research since 2024, with extensions (e.g., MMAU-Pro) that push the frontier toward holistic auditory AGI.

1. Motivation and Benchmark Genesis

Prior to MMAU, most audio benchmarks were domain-specific (ASR, ESC) or uni-modal, focusing on either perception (sound event tagging, speaker recognition) or simple captioning. These failed to probe structured reasoning, cross-domain inference, or multi-step logic over composite audio. MMAU was explicitly designed to address these deficiencies by introducing 10,000 human-annotated audio clips—balanced across speech, sound, music—paired with 27 task archetypes that require models not just to “hear” but to reason about complex scenes, temporal sequences, causal links, and abstract attributes (Sakshi et al., 2024).

A driving force in the benchmark’s creation was to serve as a high-quality, diagnostic testbed for audio-LLMs (LALMs), exposing their failure points in both low-level perception and high-level inference (Sakshi et al., 2024, Taheri et al., 9 Nov 2025). The deliberate inclusion of reasoning, temporal, and causal tasks fills gaps left by prior recognition and captioning corpora (Taheri et al., 9 Nov 2025).

2. Dataset Composition and Task Taxonomy

MMAU comprises 10,000 four-way multiple-choice QA items, sampled and balanced so that each of the three domains—Speech, Environmental Sound, and Music—accounts for roughly one-third of the corpus. Each audio clip (5–15 s, real-world diversity) is paired with a natural-language question and four expert-vetted distractors, resulting in 27 distinct skills covering both information extraction (e.g., “What instrument is playing?”) and reasoning (e.g., “What happened after the alarm?”) (Sakshi et al., 2024).

Table: Domains and Task Types (abridged)

Domain	Example Skills	Sample Questions
Speech	Emotion detection, counting, intent retrieval	"Which emotion is conveyed?"
Sound	Temporal ordering, source ID, causal inference	"Which event happened first?"
Music	Genre/chord/harmony, temporal structure, emotion	"Which chord progression?"

Complex question archetypes span temporal reasoning, causal chain inference, acoustic scene understanding, and domain-specific knowledge such as chord identification or sarcasm detection (Sakshi et al., 2024, Taheri et al., 9 Nov 2025, Mao et al., 28 Feb 2026).

An official “test-mini” subset (1,000 items) and a held-out 9,000-“test” set enable both fast prototyping and robust final evaluation. Test sets are unreleased for training to preserve benchmark integrity (Taheri et al., 9 Nov 2025).

3. Evaluation Protocols and Metrics

The benchmark is formatted for strict, reproducible evaluation:

Primary Metric: Micro-averaged accuracy over all items,

$\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl(\hat y_i = y_i\bigr)$

Domain- and skill-wise breakdowns expose strengths/weaknesses in sound, music, speech.
Protocol refinements: Variability is assessed via choice-ordering, paraphrase, and distractor perturbations, showing that single-run accuracy can fluctuate significantly; robust evaluation protocols include reporting both correctness rate (CoR) and consistency rate (CR) to measure stability and reliability (López et al., 6 Oct 2025).
Human ceiling and random/mode baselines: Human accuracy hovers around 82.2%, while random guess is approximately 26% (Sakshi et al., 2024).
Zero-shot, few-shot, and cascade protocols: Models are often tested zero-shot; recent work uses cascade (caption $\to$ LLM) to isolate perception from reasoning (Ma et al., 14 Oct 2025).

4. Model Performance and Baseline Comparisons

MMAU serves as the reference leaderboard for state-of-the-art LALMs as well as hybrid and cascaded systems. Early results revealed that even closed-source leaders such as Gemini Pro v1.5 yield only 52.97%, with open-source Qwen2-Audio-Instruct at 52.50% on full MMAU (Sakshi et al., 2024). Introduction of reasoning- and RL-augmented models rapidly pushed performance higher:

SAR-LM: Symbolic features with Gemini 2.5 Pro achieve 73.5% on mini-test, displacing dense-embedding baselines and enabling traceable, interpretable errors (Taheri et al., 9 Nov 2025).
Omni-CLST, MiMo-Audio: Selective chain-of-thought, curriculum, and large-scale pretraining approaches yield 73.8-74.9% (mini-test) (Zhao et al., 14 Sep 2025, Team et al., 29 Dec 2025).
MPAR²: Structured perception-reasoning pipeline, reinforced via GRPO, achieves 74.59% and mitigates perception decay (Mao et al., 28 Feb 2026).
Other leading models: Tools such as Audio-Maestro (Lee et al., 13 Oct 2025), R1-AQA (Rouditchenko et al., 14 May 2025), and Audio-Reasoner (Xie et al., 4 Mar 2025) leverage either tool-augmented, reinforcement, or chain-of-thought methods to improve and analyze performance.

Table: Task-Wise and Overall Accuracy on MMAU mini-test (abridged) (Taheri et al., 9 Nov 2025)

Model	Sound	Music	Speech	Overall
SAR-LM (symbolic)	73.27	64.97	82.28	73.50
Audio-Reasoner	60.06	64.30	60.70	61.71
Gemini-2.5-Pro(raw)	80.18	72.46	83.18	78.60

A plausible implication is that recent performance gains are driven by routing and curriculum mechanisms, hybrid perception–reasoning architectures, and robust RL finetuning rather than sheer parameter scaling.

5. Design Decisions, Extensions, and Analysis

MMAU’s explicit skill and domain structuring enables ablation and detailed error analysis:

Symbolicity vs. Dense Embeddings: SAR-LM and similar pipelines replace black-box embeddings with explicit features—speech transcripts, chord sequences, event timestamps—yielding interpretability and actionable error tracing (Taheri et al., 9 Nov 2025).
Tool-Augmentation: Audio-Maestro shows that integrating specialized external tool outputs, such as ASR or diarization outputs, into LLM pipelines reliably improves accuracy, especially on speech and music tasks (Lee et al., 13 Oct 2025).
Curriculum and Chain-of-Thought: Omni-CLST pioneers error-aware curricula and selective chain-of-thought dropout, boosting both efficiency and performance (Zhao et al., 14 Sep 2025).
Perception Decay and Dynamic Budgeting: MPAR² directly targets audio perception decay in long reasoning chains by enforcing multi-step perception–review loops, dynamically adjusting reasoning length and attention (Mao et al., 28 Feb 2026).
Robustness to Evaluation Perturbations: Systematic studies demonstrate that typical models are sensitive to choice orderings, paraphrase, and distractor modifications, necessitating more robust metrics and reporting (López et al., 6 Oct 2025).

MMAU-Pro, a direct extension, increases challenge and realism:

Scale and Scope: 5,305 QA pairs (with multi-audio, spatial, long-form, mixed-modality items), 49 skills (Kumar et al., 19 Aug 2025).
Question diversity: Beyond 4-option MCQs, MMAU-Pro incorporates open-ended responses, spatial reasoning, STEM oral QA, multicultural music, and multi-clip fusion.
Evaluation Methodology: Embedding-based answer validation, LLM-driven open-ended scoring, and deterministic instruction-following checks (Kumar et al., 19 Aug 2025).
Performance: Even with these advances, top closed-source models achieve just 59.2% (Gemini 2.5 Flash); open-source and hybrid models lag further behind.

This suggests that MMAU and MMAU-Pro together systematically delimit the current frontiers of audio-language AGI by exposing failure modes in duration, context fusion, instruction following, and cross-cultural transfer.

7. Community Impact and Future Directions

MMAU and its successors have become the de facto standard for benchmarking LALMs and cascaded audio-language systems. Key future directions highlighted in the literature include:

Mixed- and Multi-Skill Tasks: Extending to QA tasks requiring simultaneous extraction and reasoning (Sakshi et al., 2024).
Open-ended Generation and Error Taxonomy: Moving beyond MCQs to free-form answers and detailed error typologies.
Bias and Robustness: Ongoing vetting to reduce annotation, data, and LLM-induced biases (Sakshi et al., 2024, López et al., 6 Oct 2025).
Holistic Audio Intelligence: MMAU-Pro’s focus on multi-hop, long-form, multicultural, and spatial tasks marks a paradigm shift toward comprehensive auditory AGI (Kumar et al., 19 Aug 2025).
Deployment Relevance: Lightweight systems combining intent-classified expert modules and high-accuracy LLMs (e.g., Phi-3.5+ACD) demonstrate viability for on-device inference at competitive accuracy (Naveen et al., 2024).

In sum, MMAU has fundamentally reoriented the trajectory of audio-language research by providing a rigorous, multi-domain, multi-skill benchmark. Its protocols, task design, and analytic depth invite the development of agents and models that are both perceptually grounded and reasoning-capable, setting a new baseline for progress in audio general intelligence (Sakshi et al., 2024, Taheri et al., 9 Nov 2025, Team et al., 29 Dec 2025, Kumar et al., 19 Aug 2025, Mao et al., 28 Feb 2026).