---
title: Massive Audio Embedding Benchmark (MAEB)
url: https://www.emergentmind.com/topics/massive-audio-embedding-benchmark-maeb
type: topic
---

# Massive Audio Embedding Benchmark (MAEB)

The Massive Audio Embedding Benchmark (MAEB) is a large-scale, publicly maintained benchmark suite designed to evaluate the representational strength and generalization of audio embedding models across a comprehensive range of speech, music, environmental, and cross-modal audio–text retrieval tasks in over 100 languages. MAEB advances the field by providing unified, fine-grained metrics and a late-2020s task suite that stresses core capabilities of both specialized audio encoders and "audio-native" multimodal large language models, facilitating direct comparison between cascaded and end-to-end architectures [2602.16008][2605.04556][2606.09331].

## 1. Scope, Design, and Task Composition

MAEB is a curated subset of MAEB+, which itself aggregates 98 downstream audio tasks to maximize diversity and maintain strong rank correlation while reducing computational burden by 2.2–3.3×. The final MAEB suite consists of 30 tasks, stratified across four major categories:

- **Speech:** Multilingual speech classification, speaker identification, accent/language ID
- **Music:** Genre retrieval, onset detection, artist/timbre analysis
- **Environmental Sounds:** Sound event classification, scene recognition, urban/environmental detection
- **Audio–Text Matching:** Cross-modal retrieval—audio→text, text→audio, and zero-shot prompt-based tasks

Task selection follows a systematic, multi-step pipeline focused on semantic validity, unique domain coverage, linguistic breadth, redundancy pruning (using Spearman's ρ for rank correlation), and runtime efficiency. The resulting suite is highly multilingual, covering tasks such as SIB-FLEURS (94 languages), FLEURS (102 languages), CommonVoice (114), and specialized domains such as bioacoustics and paralinguistic emotion in MAEB+ [2602.16008].

Representative public datasets include AudioCaps, AudioSetStrong, UrbanSound8K, FSDKaggle, LibriSpeech, GTZAN, MagnaTagATune, FSD50K, BirdSet, and Speech-MASSIVE. The official MAEB splits rigorously hold out queries and candidate indices to prevent contamination from pretraining or fine-tuning [2606.09331][2605.04556].

## 2. Evaluation Methodology and Metrics

MAEB leverages and extends the MTEB ecosystem, adopting a standardized evaluation protocol with explicit task families and metrics:

**Retrieval Tasks (audio→audio/text, text→audio):**
- *Recall@k*: Fraction of queries where a relevant item is within the top k retrieved results.
  $$
  \mathrm{Recall@}k = \frac{1}{Q} \sum_{q=1}^Q \bigl[ \max_{i\leq k} \mathrm{rel}(q, r_i) = 1 \bigr]
  $$
- *Precision@k*, *Mean Average Precision (mAP)*, *NDCG*: Used for both 1–1 and multi-relevant retrieval scenarios, following standard IR definitions [2606.09331].

**Classification:**  
- *Accuracy* via logistic probe (8-shot):  
  $$
  \text{Accuracy} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}(\hat y_i = y_i)
  $$
- *Zero-Shot Classification*: Prompt-based cosine matching of audio and candidate text label embeddings [2602.16008].

**Clustering:**  
- *V-measure* (harmonic mean of homogeneity and completeness):
  $$
  V = \frac{2 h c}{h + c}, \quad h = 1 - \frac{H(C \mid K)}{H(C)}, \quad c = 1 - \frac{H(K \mid C)}{H(K)}
  $$
  where H(·) denotes (conditional) entropy [2602.16008].

**Pair Classification:**  
- *Average Precision (AP)* for binary matching (e.g., same speaker):  
  $$
  \mathrm{AP} = \sum_{k=1}^N P(k) \mathrm{rel}(k) / \#\mathrm{relevant}
  $$

**Reranking:**  
- *MAP@1000* for candidate ranking.

**Correlation Analyses:**  
- *Pearson r* and *Spearman ρ* for benchmarking encoder and downstream large audio language model (audio LLM) task correlation.

**Other tasks** (in the broader MAEB/MSEB ecosystem): segmentation (NDCG for event localization), reasoning (QA with F1/geometric mean), reconstruction (LSD, FAD/KAD).

The primary MAEB score is the unweighted mean of the main retrieval/classification metric per task; "task-type averages" aggregate scores within each domain then over the four domains [2606.09331].

## 3. Baselines, Architectures, and Leaderboard Results

MAEB encompasses a broad model taxonomy:

- **Self-supervised Speech Models:** Wav2Vec2, HuBERT, WavLM, XLS-R, UniSpeech, Data2Vec, MCTCT
- **Spectrogram Transformers/CNNs:** AST, CNN14, VGGish, YAMNet
- **Contrastive Audio–Text Models:** CLAP, MS-CLAP, Wav2CLIP, MuQ-MuLan
- **Large Audio–Language Models:** Qwen2-Audio, LCO-Embedding, BidirLM-Omni
- **Omni-modal Models:** Conan-embedding-v3, Jina Embeddings, LCO-Embedding-Omni [2606.09331][2602.16008].

Leaderboard analysis reveals no universal champion. CLAP variants lead environmental sound tasks (ESC50 > 90%), while speech-specialized models dominate multilingual speech tasks (SIB-FLEURS > 60%). Large audio-LLMs excel in cross-modal and zero-shot settings: LCO-Embedding-Omni-7B achieves the best overall cross-modal retrieval (50.3%). Clustering lags markedly, with top V-measure scores below 23%.

The table below summarizes representative model performance on the 30-task MAEB suite [2606.09331]:

| Model                        | Type              | Params | MAEB  | Task-type Avg. |
|------------------------------|-------------------|--------|-------|---------------|
| clap-htsat-fused             | audio encoder     | 0.15B  | 33.47 | 41.30         |
| Qwen2-Audio-7B               | audio MLLM        | 7.00B  | 34.54 | 37.01         |
| jina-embeddings-v5-omni-nano | omni embedding    | 0.99B  | 50.14 | 55.31         |
| LCO-Embedding-Omni-7B        | omni embedding    | 8.93B  | 53.54 | 57.06         |
| Conan-embedding-v3           | omni embedding    | 8.8B   | 55.61 | 59.32         |

Ablation studies (e.g., in Conan-embedding-v3) demonstrate that naïve backbone fusion degrades MAEB scores due to projector–backbone mismatch (“projector drift”). Recovery via projector fine-tuning and balanced multimodal rehearsal is essential for restoring state-of-the-art results [2606.09331].

## 4. Cross-Domain Insights and Diagnostic Findings

MAEB exposes structural strengths and deficits in leading approaches:

- **Trade-offs:** Environmental and speech domains show uncorrelated model ranks—embeddings optimized for one domain show near-random performance on the other [2602.16008].  
- **Clustering:** Remains the weakest link; all encoder classes lack effective unsupervised group structure (best V-measure 22.7%); supervised tuning does not compensate.
- **Multilinguality:** MAEB sustains extensive coverage, yet most models incur significant degradation on low-resource languages and non-Western datasets [2605.04556].
- **Correlation with Audio-LLMs:** Encoder scores on MAEB are strong predictors of downstream LLM QA and reasoning outcomes; $R^2\approx0.86$ and Spearman ρ comparable for four major architectures.

## 5. System Integration, Extensibility, and Benchmark Infrastructure

MAEB is integrated into the “embeddings-benchmark/mteb” platform, enabling:

- **Unified APIs:** Standardized dataset, task, runner, and evaluator interfaces; submission ready for public leaderboard integration.
- **Reproducibility and Transparency:** Versioned datasets, splits, scripts, model configs, documented scoring practices.
- **Expansion**: Modular registration of new encoders (via MultiModalEncoder interface), tasks (Task subclassing), and datasets.
- **Computational Accessibility:** Runtime costs curtailed with the reduced 30-task MAEB suite (∼2 GPU-hours for small models); public codebase and result leaderboards at github.com/embeddings-benchmark/mteb and github.com/maeb-benchmark/MAEB [2602.16008][2605.04556].

## 6. Limitations, Open Problems, and Future Directions

MAEB advances evaluation methodology but several challenges and limitations persist:

- **Compute and Content:** Very large models and long-form audio (>30 s) remain cost-intensive. Domain coverage is skewed to major language groups and Western music.
- **Missing Task Classes:** Real-time streaming, generative audio modeling, and low-latency applications are excluded from standard MAEB.
- **Benchmark Contamination:** Measures to prevent test leakage from pretraining corpora remain an ongoing concern [2605.04556].
- **Robustness:** Noise robustness metrics (e.g., SNR curves), far-field robustness, and evaluation protocols for adversarial and degraded audio require further standardization.
- **Universal Representation:** No model yet approaches universal high performance; the pursuit of cross-domain audio representations and compressed, retrieval-robust embeddings continues to be a principal research direction.
- **Expanded Relevance:** MAEB’s task and evaluation design have direct implications for LLM-powered multimodal systems by providing predictive value for audio-reasoning abilities [2602.16008][2605.04556].

## 7. Impact and Research Significance

MAEB fills a longstanding gap as the first large-scale, multi-domain, multilingual audio representation benchmark with systematic cross-modal, cross-domain, and zero-shot task support. It enables objective measurement of audio encoder progress, reveals fundamental representation trade-offs, and has demonstrably high rank correlation with downstream capability in audio LLMs. MAEB provides an open infrastructure for community-driven task extension, reproducible experimentation, and ongoing calibration of embedding model progress across an evolving spectrum of audio and multimodal AI research [2602.16008][2606.09331][2605.04556].

Source: https://www.emergentmind.com/topics/massive-audio-embedding-benchmark-maeb