MAEB: Massive Audio Embedding Benchmark

Abstract: We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasoning in 100+ languages. We evaluate 50+ models and find that no single model dominates across all tasks: contrastive audio-text models excel at environmental sound classification (e.g., ESC50) but score near random on multilingual speech tasks (e.g., SIB-FLEURS), while speech-pretrained models show the opposite pattern. Clustering remains challenging for all models, with even the best-performing model achieving only modest results. We observe that models excelling on acoustic understanding often perform poorly on linguistic tasks, and vice versa. We also show that the performance of audio encoders on MAEB correlates highly with their performance when used in audio LLMs. MAEB is derived from MAEB+, a collection of 98 tasks. MAEB is designed to maintain task diversity while reducing evaluation cost, and it integrates into the MTEB ecosystem for unified evaluation across text, image, and audio modalities. We release MAEB and all 98 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.

• The paper introduces MAEB, a unified benchmark evaluating diverse audio embedding models across varied tasks, languages, and domains.
• It standardizes evaluation using few-shot probes, text prompts, clustering, and retrieval metrics to diagnose model specialization and performance trade-offs.
• Results reveal no universal audio model, highlighting challenges in multilinguality, semantic clustering, and balancing acoustic and linguistic features.

Massive Audio Embedding Benchmark: A Comprehensive Evaluation Framework for Universal Audio Representation

Motivation and Scope

The "MAEB: Massive Audio Embedding Benchmark" (2602.16008) addresses a persistent fragmentation in the evaluation of audio embedding models. Prior benchmarks—such as those for audio classification or speech recognition—have traditionally focused on isolated domains and narrow task scopes, impeding progress on universal audio representations. MAEB introduces a unified, extensible, and large-scale evaluation protocol incorporating diverse domains (speech, music, environmental sounds, bioacoustics), task families (classification, clustering, retrieval, reranking, cross-modal matching), multilinguality (100+ languages), and cross-modal reasoning to robustly assess the generalization and transferability of learned audio representations.

Figure 1: Overview of task types and subtypes in MAEB+, with numbers denoting task counts for both MAEB+ and the core MAEB benchmark.

Benchmark Design and Coverage

MAEB is derived from the broader MAEB+ suite, subsampling 30 highly-informative tasks from an initial pool of 98. Selection criteria include maximizing domain/task/linguistic diversity, minimizing redundancy (using rank correlation analysis), and ensuring computational efficiency. Tasks are grouped into seven categories—including supervised classification, zero-shot classification, unsupervised clustering, audio/text retrieval, pairwise similarity, and reranking—each of which exposes distinct embedding properties in supervised and transfer settings. The benchmark is fully integrated into the MTEB ecosystem, facilitating unified evaluation across modalities and consistent protocol enforcement.

MAEB’s dataset pool covers a broad spectrum, including high- and low-resource languages, multiple music genres, varied acoustic scenes, and paralinguistic cues such as emotion and speaker identity. In addition, cross-modal tasks (e.g., audio-text retrieval, zero-shot classification via language prompts) probe multimodal alignment, a key desideratum for future audio LLMs (Audio LLMs).

Model Coverage and Evaluation Pipeline

Fifty-three models representing four architectural classes are benchmarked:

• Audio encoders (e.g., Wav2Vec2, AST, CNN14): Pretrained for speech or sound event representation.
• Sequence-to-sequence models (Whisper, MMS, SeamlessM4T): Trained for speech recognition/translation and used for representation extraction.
• Contrastive alignment models (CLAP, MS-CLAP, MuQ-MuLan): Jointly learn audio-text spaces via contrastive losses.
• Audio LLMs (Qwen2-Audio, LCO-Embedding): Large, instruction-tuned generative models, evaluated via hidden state pooling.

Embedding extraction is rigorously standardized: audio is truncated/zero-padded to 30s, resampled as needed, converted to mono, and pooled following model-specific strategies. All supervised evaluations use few-shot linear probes; zero-shot evaluations employ text prompts; clustering is assessed using MiniBatchKMeans/V-measure; retrieval and reranking use standard recall and average precision metrics.

Results and Analysis: No Universal Audio Model

MAEB evaluation results reveal several critical findings:

• No model achieves universal audio representation. Specialization remains pronounced: speech-pretrained models (Whisper, Wav2Vec2) excel in speech/minor paralinguistic tasks but underperform in music/environmental sound domains; contrastive models (CLAP variants) lead on environmental/multimodal benchmarks but perform near random on many multilingual speech tasks.
• Clustering exposes semantic limitations. All models exhibit degraded clustering performance (best: 22.7%), indicating that current embedding spaces lack inherent semantic structure for unsupervised organization.
• Multilingual audio understanding is unresolved. Despite coverage of 100+ languages, models show acute bias toward high-resource languages. Cross-modal retrieval for low-resource languages is near chance for all but a few models, underscoring severe limitations in existing alignment and transfer strategies.
• Acoustic/linguistic trade-off persists. Models strongly optimized for acoustic surface features (e.g., gender, timbre, speaker) underperform on linguistic content tasks, and vice versa. Attempts at balancing both (e.g., large, multimodal LLMs) have yet to close this gap.

  Figure 2: Domain-level performance for each of five acoustic domains; no model approaches the universal target performance (dashed line), and strong inter-domain specialization is evident.

This domain specialization is further visualized in the radial plot (Figure 2), where each model achieves peak performance in only a subset of domains, and the aggregated scores fall short of the target for "universal" audio understanding.

Downstream Correlation and Practical Implications

The study empirically demonstrates that MAEB scores are strongly correlated with downstream Audio LLM performance on the MMAU benchmark (Figure 3), with preliminary $R^2=0.86$ (albeit with a small $n=4$). This connection validates the use of MAEB as a diagnostic for downstream cross-modal reasoning and instruction following abilities.

Figure 3: Positive correlation between MAEB+ classification performance and Audio LLM downstream performance (MMAU), supporting the utility of MAEB for encoder diagnostic prediction.

The practical implications are direct for both research and deployment:

• Benchmark-driven model selection enables practitioners to select domain- or capability-specific models rather than assuming transferability, avoiding failure cases in cross-lingual or cross-domain applications.
• Task/unit alignment with MTEB/MMTEB/MIEB enables universal encoder evaluation pipelines across vision, audio, and language modalities.

Dataset and Linguistic Distribution

MAEB’s design exposes resource concentration and language imbalance, as observed in the language distribution histogram (Figure 4): English-centricity is severe, but over 30 non-English languages are present, with efforts to tag data lacking linguistic content.

Figure 4: Task distribution by language in the MAEB+ collection, highlighting English dominance and the “zxx” (no linguistic content) tag for pure acoustic datasets.

Domain representation across speech, music, environment, bioacoustics, and emotion is also visualized relative to MAEB+ and the audio-only benchmark in Figure 5.

Figure 5: Comparison of acoustic domain coverage for MAEB+, MAEB, and the audio-only MAEB subset.

Limitations and Open Problems

Despite the breadth of its coverage, MAEB has known limitations:

• Resource skew and underrepresentation for minority languages and non-Western music traditions persist.
• Lack of real-world audio artifacts (e.g., noisy and reverberant data): Most tasks remain “clean,” limiting ecological validity.
• Absence of generation-based or real-time evaluation tasks: Current focus is on representational quality; generative and low-latency applications remain outside the current benchmark.
• High compute requirements for the largest models (e.g., Whisper-large, XLS-R-2B) may restrict accessibility for some groups, though MAEB is deliberately constructed for tractability (~2 GPU hours for small models).

Broader Implications and Research Directions

MAEB sets a rigorous new standard for audio embedding evaluation and exposes contradictory trends that should inform the next wave of research:

• Specialization gap: There is an urgent need for architecture and objective innovations that can jointly capture acoustic and linguistic properties without mutual degradation.
• Multilingual objectives: Cross-lingual contrastive learning and universal (acoustic, linguistic, paralinguistic) transfer objectives should be prioritized, with special attention to low-resource language robustness.
• Semantic structure: Clustering and unsupervised organization require alternative training objectives (e.g., neighborhood-aware contrastive learning), potentially leveraging self-supervision informed by large-scale clustering/partitioning signals.
• Cross-modal scaling: Models like LCO-Embedding-Omni-7B demonstrate that cross-modal, large-scale encoder training can elevate zero-shot and retrieval capabilities, but performance bifurcation with “pure” audio or speech tasks remains unresolved.
• Unified evaluation: Integration into the MTEB suite ensures future extensibility and long-term maintenance, allowing MAEB to evolve alongside rapid advances in audio representation and multimodal pretraining.

Conclusion

MAEB provides an authoritative, large-scale, heterogeneous benchmark for audio representations, exposing the lack of any universal model and identifying four critical weaknesses—domain specialization, poor clustering, underdeveloped multilinguality, and irreconcilable acoustic-linguistic trade-offs. Its extensible design, unified task taxonomy, and rigorous multi-domain/multilingual coverage offer a robust platform for driving fundamental progress in universal audio representation, and it strongly correlates with practical downstream Audio LLM performance. Addressing the limitations surfaced by MAEB will require advances in cross-modal learning, low-resource adaptation, and architectural unification across the acoustic-linguistic spectrum.