SUPERB Benchmark for Speech SSL

• SUPERB Benchmark is a comprehensive evaluation suite for SSL in speech that uses fixed protocols with frozen upstream encoders to ensure methodical comparisons.
• It covers a wide range of tasks including ASR, semantic analysis, speaker recognition, generative tasks, and deepfake detection, with extensions for multilingual evaluation.
• The benchmark’s community-driven design and reproducible methodology foster fair model comparisons and advance research in robust and efficient speech technologies.

The Speech processing Universal PERformance Benchmark (SUPERB) is a comprehensive, community-driven benchmarking suite designed to evaluate the generality and utility of self-supervised learning (SSL) representations for speech. Initiated in 2021, SUPERB unified the scattered landscape of speech SSL evaluation into a task-rich, reproducible framework, modeling the impact of GLUE for NLP and VTAB for computer vision. Since its inception, SUPERB has rapidly expanded—first in task coverage (semantic, generative, and robustness-oriented tasks), then into multilingual and multimodal domains—becoming the canonical evaluation substrate for both wave-based and multimodal foundation models in speech. Recent extensions such as ML-SUPERB, SUPERB-SG, Dynamic-SUPERB, and benchmark spin-offs for security and deepfake detection have further entrenched SUPERB as a cornerstone resource for academic and practical research in speech technology.

1. Origins, Motivation, and Core Principles

SUPERB was introduced to resolve the lack of systematic, comparable evaluation for SSL speech models. Prior to its launch, speech SSL research was fragmented by heterogeneous datasets, task definitions, and scoring conventions, preventing fair cross-architecture comparison. Inspired by NLP and CV benchmarks, SUPERB established a fixed evaluation protocol based on three core tenets: (i) frozen SSL encoders (no parameter update, isolating representational quality), (ii) lightweight, task-specific heads (to test discriminative content in SSL feature spaces), and (iii) broad compositional task coverage spanning content (ASR, PR, KS, QbE), speaker (SID, ASV, SD), semantics (IC, SF), and paralinguistics (ER) (Yang et al., 2021). The evaluation metric suite includes Phoneme Error Rate (PER), Word Error Rate (WER), Character Error Rate (CER), accuracy, F1, Equal Error Rate (EER), Diarization Error Rate (DER), and maximum term-weighted value (MTWV).

Key design criteria stress public, modest-sized datasets and rigorous, reproducible task recipes employing fixed data splits, model architectures, and tunable hyperparameters. All upstream SSL models are subject to the same protocol, enabling direct, interpretable comparisons focused on transferability and generalizability.

2. Task Spectrum and Benchmark Extensions

SUPERB originally targeted ten English-language, discriminative tasks, with each task probing a theoretical dimension of speech information. The suite rapidly extended along several dimensions:

• Semantic-Generative Expansion: SUPERB-SG introduced deep-semantic (e.g., speech translation, out-of-domain ASR) and generative tasks (voice conversion, speech separation, speech enhancement), maintaining frozen-backbone evaluation (Tsai et al., 2022). These tasks required more expressive downstream heads (CTC-based encoder–decoder, BLSTM mask predictors), pushing the limits of SSL model representational depth and robustness to domain shift.
• Efficiency and Generalization: The SUPERB @ SLT 2022 challenge formalized computational trade-offs—model PARAm count, theoretical multiply-accumulate operations (MACs)—and introduced out-of-distribution (OOD) “hidden” test sets, compelling the community to optimize both representational quality and resource cost (Feng et al., 2022).
• Zero-Shot and Instruction Tuning: Dynamic-SUPERB evaluated instruction-tuned and multimodal models in a unified zero-shot, generative framework, incorporating text instructions, generative label outputs, and classification for audio, speech, and paralinguistic tasks (Huang et al., 2023). Dynamic-SUPERB Phase-2 further expanded the taxonomy to 180 tasks, covering regression and sequence-generation tasks, music and environmental audio, and open-vocabulary outputs (Huang et al., 2024).
• Security/Deepfake: Spoof-SUPERB applies the constrained, frozen-feature protocol to audio deepfake detection, providing a systematic, multi-dataset testbed measuring cross-corpus generalization of SSL models for ASVspoof-style attacks (Ali et al., 2 Mar 2026).

3. Methodologies: Protocol, Model Design, and Scoring

SUPERB and its derivatives standardize model evaluation by mandating a two-part system. The first is the frozen upstream—typically a wave-based SSL Transformer (wav2vec 2.0, HuBERT, XLS-R, WavLM, etc.) pre-trained on speech corpora (mostly LibriSpeech or LibriLight, with extensions to larger, multilingual setups). The second is the lightweight prediction head, generally a combination of layer-weighted sum, sub-sampling, and compact sequence models (linear, BLSTM, Transformer, or MLP), sometimes with task-adaptive losses (CTC, cross-entropy, MSE, etc.).

Model ranking is aggregated in the “SUPERB_s” score, a normalized metric averaging per-task improvements over FBANK baselines up to the current SOTA: $$\mathrm{SUPERB}_s(u) = \frac{1000}{|T|}\sum_{t\in T}\frac{1}{|I_t|}\sum_{i\in I_t}\frac{s_{t,i}(u)-s_{t,i}(\mathrm{FBANK})}{s_{t,i}(\mathrm{SOTA})-s_{t,i}(\mathrm{FBANK})}$$ where $s_{t,i}(u)$ is the user model’s score, $T$ is the task set, and $I_t$ task metrics (Shi et al., 2023, Feng et al., 2022).

The frozen-feature constraint is sometimes relaxed in ML-SUPERB 2.0 and Dynamic-SUPERB Phase-2: larger downstreams, partial/full fine-tuning, parameter-efficient adaptation (LoRA, adapters), and hybrid decoders (CTC+ATT) are compared with traditional protocols to analyze how much head design, SSL layer choice, and adaptation affect per-task and per-language robustness (Shi et al., 2024).

4. Multilingual and Cross-Modal Evolution

SUPERB was originally English-centric. The introduction of ML-SUPERB addressed the need for benchmarking in multilingual and low-resource regimes (Shi et al., 2023). ML-SUPERB’s public release evaluated ASR and LID over 143 languages (from high-resource to endangered), using controlled data slices (10 minutes and 1 hour; few-shot regimes) drawn from CommonVoice, VoxPopuli, Fleurs, and more.

The challenge protocol introduced:

• Monolingual ASR (CER or PER per language)
• Multilingual ASR (macro-averaged CER, normal and few-shot)
• Language identification (143-way accuracy)
• Joint ASR+LID (shared CTC and softmax losses)

More recent iterations (ML-SUPERB Challenge 2023, 2.0) increased language coverage to 154+, adopted open language submission tracks, and injected real-world conditions (conversational, singing voice, diverse phone types) to challenge model robustness (Shi et al., 2023). Benchmark results show that broad-coverage SSL (XLS-R, MMS) dominates average CER and LID accuracy, but model scaling alone does not guarantee generalization, especially for challenging domains or in long-tail low-resource cases. Downstream architecture, mid-layer tuning, and parameter-efficient adaptation all affect target-language and per-dataset performance differentially (Shi et al., 2024).

Dynamic-SUPERB’s collaborative, instruction-tuned evaluation—leveraging zero-shot generalization and task instructions—bridges the gap between task-specific benchmarking and foundation-model-style universal evaluation, supporting expansion to music and environmental audio (Huang et al., 2024).

5. Empirical Findings and Performance Trends

Across public and private leaderboards, recent SUPERB-style benchmarks have yielded several key empirical observations:

1. SSL dominates FBANK and supervised baselines: SSL representations (especially masked prediction—wav2vec 2.0, HuBERT, XLS-R, WavLM) consistently and vastly outperform frame-level log-Mel filterbanks (FBANK) for content, semantic, and speaker/prosody tasks, both under constrained and relaxed adaptation (Yang et al., 2021, Tsai et al., 2022, Shi et al., 2023, Shi et al., 2024).
2. Multilingual coverage improves macro performance, not always per-language: Models pre-trained on broad language sets (XLS-R, MMS-1B) yield lower average CER and higher LID accuracy, but “selective” or regional models may underperform strong monolingual baselines on their language(s) (Shi et al., 2023, Shi et al., 2023).
3. Downstream architecture and adaptation are critical: ML-SUPERB 2.0 demonstrates that CTC-ATT hybrids, E-Branchformer encoders, and mid-layer partial fine-tuning can improve macro-CER by 5–10 pts over frozen transformer heads. Adaptation (LoRA, Houlsby adapters) narrows but does not close the gap to full fine-tuning—especially in few-shot, long-tail languages (Shi et al., 2024).
4. Modal and domain robustness remains a weakness: Speech generation and enhancement tasks, cross-domain OOD evaluation, and real-world speech variants (conversational, singing, telephony) cause all models to degrade, with significant variances by per-language resource, dataset, and upstream pre-training objective (Feng et al., 2022, Shi et al., 2023, Huang et al., 2024).
5. No universal winner: Unified benchmarks (Dynamic-SUPERB Phase-2) show that no foundation model achieves state-of-the-art across all tasks; for example, SALMONN-13B excels in English ASR but is weak on paralinguistics and music, while Qwen2-Audio-7B-Instruct leads in emotion recognition (Huang et al., 2024).

6. Benchmark Ecosystem, Community Processes, and Expansion

The SUPERB ecosystem includes the s3prl toolkit (PyTorch), public leaderboards, and extensible benchmarking recipes for both classic and new tasks (Yang et al., 2021, Shi et al., 2023). Dynamic-SUPERB introduces a dynamic workflow for community-driven task addition, embracing continuous versioning, automated validation, and REST APIs for scalable model evaluation (Huang et al., 2023, Huang et al., 2024). All code, data splits, and evaluation servers are open-sourced with permissive licensing to enable reproducibility and encourage field-wide adoption.

ML-SUPERB and Dynamic-SUPERB have each established contribution protocols for new languages, tasks, and corpora: researchers submit standardized data splits and metadata; editors review and test; approved tasks are merged and public test splits released (with training and dev sets often withheld for fair benchmarking) (Shi et al., 2023, Huang et al., 2024).

A non-exhaustive table summarizing key extensions and coverage is as follows:

Benchmark/Phase	Languages	Tasks	Modalities
SUPERB (Yang et al., 2021)	1 (EN)	10	Speech (content, speaker, semantics)
SUPERB-SG (Tsai et al., 2022)	1 (EN)	15	+Generation (translation, conversion)
ML-SUPERB (Shi et al., 2023)	143–154	2–4	Multilingual ASR, LID, Joint
ML-SUPERB 2.0 (Shi et al., 2024)	142	2–4	+Hybrid/fine-tuned/LoRA setups
Dynamic-SUPERB (Huang et al., 2023)	Any	33–180	Speech, music, audio, zero-shot
Spoof-SUPERB (Ali et al., 2 Mar 2026)	1 (EN; multi-accent)	1	Deepfake detection (security)

The table omits older non-SUPERB benchmarks, and task count reflects distinct scoring dimensions.

7. Limitations, Open Problems, and Future Directions

Despite the breadth and rigor of SUPERB-style evaluation, several challenges remain:

• Incomplete modality and domain coverage: Conversational, noisy, code-switched, and singing speech tasks expose consistent performance deficits across model classes (Shi et al., 2023, Huang et al., 2024).
• Language–dataset mismatch: Empirically, macro-averaged metrics can mask large per-language and per-dataset disparities (e.g., Urdu's CER varies by >30 points between datasets) (Shi et al., 2024). A plausible implication is that further pre-training on underrepresented conditions, aggressive domain-targeted augmentation, and language/dataset-specific adapters may be essential.
• Instruction-tuning generalization: Dynamic-SUPERB shows significant gaps between seen and unseen task/instruction performance, with models often exploiting superficial instruction patterns rather than deep semantic understanding (Huang et al., 2023).
• No one-size-fits-all adaptation: Recent studies suggest partial fine-tuning of intermediate SSL layers is often optimal in few-shot and low-resource contexts, balancing performance, parameter efficiency, and overfitting (Shi et al., 2024).

Future benchmark priorities include the continued expansion of real-world, diverse speech situations (environmental, medical, low-resource), expanding generative and sequence-prediction tasks, improved evaluation for sequence, regression, and open-vocabulary outputs, and broader inclusion of music and audio for unified audio-LLMs (Huang et al., 2024). Benchmarking will need to coordinate with advances in parameter-efficient adaptation, speech-aware LLM adapters, and better multimodal foundations.

• SUPERB: Speech Processing Universal PERformance Benchmark (Yang et al., 2021)
• ML-SUPERB: Multilingual Speech Universal PERformance Benchmark (Shi et al., 2023)
• Findings of the 2023 ML-SUPERB Challenge (Shi et al., 2023)
• ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints (Shi et al., 2024)
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark (Tsai et al., 2022)
• SUPERB @ SLT 2022: Challenge on Generalization and Efficiency (Feng et al., 2022)
• Dynamic-SUPERB: Towards a Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech (Huang et al., 2023)
• Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark (Huang et al., 2024)
• Spoof-SUPERB: A SUPERB-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detection (Ali et al., 2 Mar 2026)