Massive Audio Embedding Benchmark (MAEB)

• MAEB is a multilingual benchmark suite that evaluates audio embedding models across 30 diverse tasks in over 100 languages.
• It employs unified, fine-grained metrics like Recall@k and mAP to compare performance across speech, music, environmental, and cross-modal tasks.
• MAEB advances audio research by benchmarking cascaded and end-to-end architectures while promoting reproducible evaluations and transparency.

The Massive Audio Embedding Benchmark (MAEB) is a large-scale, publicly maintained benchmark suite designed to evaluate the representational strength and generalization of audio embedding models across a comprehensive range of speech, music, environmental, and cross-modal audio–text retrieval tasks in over 100 languages. MAEB advances the field by providing unified, fine-grained metrics and a late-2020s task suite that stresses core capabilities of both specialized audio encoders and "audio-native" multimodal LLMs, facilitating direct comparison between cascaded and end-to-end architectures (Assadi et al., 17 Feb 2026, Allauzen et al., 6 May 2026, Li et al., 8 Jun 2026).

1. Scope, Design, and Task Composition

MAEB is a curated subset of MAEB+, which itself aggregates 98 downstream audio tasks to maximize diversity and maintain strong rank correlation while reducing computational burden by 2.2–3.3×. The final MAEB suite consists of 30 tasks, stratified across four major categories:

• Speech: Multilingual speech classification, speaker identification, accent/language ID
• Music: Genre retrieval, onset detection, artist/timbre analysis
• Environmental Sounds: Sound event classification, scene recognition, urban/environmental detection
• Audio–Text Matching: Cross-modal retrieval—audio→text, text→audio, and zero-shot prompt-based tasks

Task selection follows a systematic, multi-step pipeline focused on semantic validity, unique domain coverage, linguistic breadth, redundancy pruning (using Spearman's ρ for rank correlation), and runtime efficiency. The resulting suite is highly multilingual, covering tasks such as SIB-FLEURS (94 languages), FLEURS (102 languages), CommonVoice (114), and specialized domains such as bioacoustics and paralinguistic emotion in MAEB+ (Assadi et al., 17 Feb 2026).

Representative public datasets include AudioCaps, AudioSetStrong, UrbanSound8K, FSDKaggle, LibriSpeech, GTZAN, MagnaTagATune, FSD50K, BirdSet, and Speech-MASSIVE. The official MAEB splits rigorously hold out queries and candidate indices to prevent contamination from pretraining or fine-tuning (Li et al., 8 Jun 2026, Allauzen et al., 6 May 2026).

2. Evaluation Methodology and Metrics

MAEB leverages and extends the MTEB ecosystem, adopting a standardized evaluation protocol with explicit task families and metrics:

Retrieval Tasks (audio→audio/text, text→audio):

• Recall@k: Fraction of queries where a relevant item is within the top k retrieved results.

$$\mathrm{Recall@}k = \frac{1}{Q} \sum_{q=1}^Q \bigl[ \max_{i\leq k} \mathrm{rel}(q, r_i) = 1 \bigr]$$

• Precision@k, Mean Average Precision (mAP), NDCG: Used for both 1–1 and multi-relevant retrieval scenarios, following standard IR definitions (Li et al., 8 Jun 2026).

Classification:

• Accuracy via logistic probe (8-shot):

$$\text{Accuracy} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}(\hat y_i = y_i)$$

• Zero-Shot Classification: Prompt-based cosine matching of audio and candidate text label embeddings (Assadi et al., 17 Feb 2026).

Clustering:

• V-measure (harmonic mean of homogeneity and completeness):

$$V = \frac{2 h c}{h + c}, \quad h = 1 - \frac{H(C \mid K)}{H(C)}, \quad c = 1 - \frac{H(K \mid C)}{H(K)}$$

where H(·) denotes (conditional) entropy (Assadi et al., 17 Feb 2026).

Pair Classification:

• Average Precision (AP) for binary matching (e.g., same speaker):

$$\mathrm{AP} = \sum_{k=1}^N P(k) \mathrm{rel}(k) / \#\mathrm{relevant}$$

Reranking:

• MAP@1000 for candidate ranking.

Correlation Analyses:

• Pearson r and Spearman ρ for benchmarking encoder and downstream large audio LLM (audio LLM) task correlation.

Other tasks (in the broader MAEB/MSEB ecosystem): segmentation (NDCG for event localization), reasoning (QA with F1/geometric mean), reconstruction (LSD, FAD/KAD).

The primary MAEB score is the unweighted mean of the main retrieval/classification metric per task; "task-type averages" aggregate scores within each domain then over the four domains (Li et al., 8 Jun 2026).

3. Baselines, Architectures, and Leaderboard Results

MAEB encompasses a broad model taxonomy:

• Self-supervised Speech Models: Wav2Vec2, HuBERT, WavLM, XLS-R, UniSpeech, Data2Vec, MCTCT
• Spectrogram Transformers/CNNs: AST, CNN14, VGGish, YAMNet
• Contrastive Audio–Text Models: CLAP, MS-CLAP, Wav2CLIP, MuQ-MuLan
• Large Audio–LLMs: Qwen2-Audio, LCO-Embedding, BidirLM-Omni
• Omni-modal Models: Conan-embedding-v3, Jina Embeddings, LCO-Embedding-Omni (Li et al., 8 Jun 2026, Assadi et al., 17 Feb 2026).

Leaderboard analysis reveals no universal champion. CLAP variants lead environmental sound tasks (ESC50 > 90%), while speech-specialized models dominate multilingual speech tasks (SIB-FLEURS > 60%). Large audio-LLMs excel in cross-modal and zero-shot settings: LCO-Embedding-Omni-7B achieves the best overall cross-modal retrieval (50.3%). Clustering lags markedly, with top V-measure scores below 23%.

The table below summarizes representative model performance on the 30-task MAEB suite (Li et al., 8 Jun 2026):

Model	Type	Params	MAEB	Task-type Avg.
clap-htsat-fused	audio encoder	0.15B	33.47	41.30
Qwen2-Audio-7B	audio MLLM	7.00B	34.54	37.01
jina-embeddings-v5-omni-nano	omni embedding	0.99B	50.14	55.31
LCO-Embedding-Omni-7B	omni embedding	8.93B	53.54	57.06
Conan-embedding-v3	omni embedding	8.8B	55.61	59.32

Ablation studies (e.g., in Conan-embedding-v3) demonstrate that naïve backbone fusion degrades MAEB scores due to projector–backbone mismatch (“projector drift”). Recovery via projector fine-tuning and balanced multimodal rehearsal is essential for restoring state-of-the-art results (Li et al., 8 Jun 2026).

4. Cross-Domain Insights and Diagnostic Findings

MAEB exposes structural strengths and deficits in leading approaches:

• Trade-offs: Environmental and speech domains show uncorrelated model ranks—embeddings optimized for one domain show near-random performance on the other (Assadi et al., 17 Feb 2026).
• Clustering: Remains the weakest link; all encoder classes lack effective unsupervised group structure (best V-measure 22.7%); supervised tuning does not compensate.
• Multilinguality: MAEB sustains extensive coverage, yet most models incur significant degradation on low-resource languages and non-Western datasets (Allauzen et al., 6 May 2026).
• Correlation with Audio-LLMs: Encoder scores on MAEB are strong predictors of downstream LLM QA and reasoning outcomes; $R^2\approx0.86$ and Spearman ρ comparable for four major architectures.

5. System Integration, Extensibility, and Benchmark Infrastructure

MAEB is integrated into the “embeddings-benchmark/mteb” platform, enabling:

• Unified APIs: Standardized dataset, task, runner, and evaluator interfaces; submission ready for public leaderboard integration.
• Reproducibility and Transparency: Versioned datasets, splits, scripts, model configs, documented scoring practices.
• Expansion: Modular registration of new encoders (via MultiModalEncoder interface), tasks (Task subclassing), and datasets.
• Computational Accessibility: Runtime costs curtailed with the reduced 30-task MAEB suite (∼2 GPU-hours for small models); public codebase and result leaderboards at github.com/embeddings-benchmark/mteb and github.com/maeb-benchmark/MAEB (Assadi et al., 17 Feb 2026, Allauzen et al., 6 May 2026).

6. Limitations, Open Problems, and Future Directions

MAEB advances evaluation methodology but several challenges and limitations persist:

• Compute and Content: Very large models and long-form audio (>30 s) remain cost-intensive. Domain coverage is skewed to major language groups and Western music.
• Missing Task Classes: Real-time streaming, generative audio modeling, and low-latency applications are excluded from standard MAEB.
• Benchmark Contamination: Measures to prevent test leakage from pretraining corpora remain an ongoing concern (Allauzen et al., 6 May 2026).
• Robustness: Noise robustness metrics (e.g., SNR curves), far-field robustness, and evaluation protocols for adversarial and degraded audio require further standardization.
• Universal Representation: No model yet approaches universal high performance; the pursuit of cross-domain audio representations and compressed, retrieval-robust embeddings continues to be a principal research direction.
• Expanded Relevance: MAEB’s task and evaluation design have direct implications for LLM-powered multimodal systems by providing predictive value for audio-reasoning abilities (Assadi et al., 17 Feb 2026, Allauzen et al., 6 May 2026).

7. Impact and Research Significance

MAEB fills a longstanding gap as the first large-scale, multi-domain, multilingual audio representation benchmark with systematic cross-modal, cross-domain, and zero-shot task support. It enables objective measurement of audio encoder progress, reveals fundamental representation trade-offs, and has demonstrably high rank correlation with downstream capability in audio LLMs. MAEB provides an open infrastructure for community-driven task extension, reproducible experimentation, and ongoing calibration of embedding model progress across an evolving spectrum of audio and multimodal AI research (Assadi et al., 17 Feb 2026, Li et al., 8 Jun 2026, Allauzen et al., 6 May 2026).