Dynamic SUPERB Phase-2: A Universal Audio Benchmark

• Dynamic SUPERB Phase-2 is an evolving benchmark that evaluates instruction-following audio models across 180 tasks spanning speech, music, and environmental domains.
• It introduces regression and sequence generation alongside classification to comprehensively assess modern audio-language systems without task-specific fine-tuning.
• The benchmark employs a dynamic, community-driven expansion process and LLM-based metrics that closely align with human judgments.

Dynamic-SUPERB Phase-2 is an open, evolving benchmark for evaluating instruction-based spoken LLMs that accept audio and natural-language instructions and respond in text without task-specific fine-tuning. It extends the first generation of Dynamic-SUPERB from an instruction-following speech benchmark centered on classification to a substantially broader suite spanning speech, music, and environmental audio, with classification, regression, and sequence generation tasks. In its current form, it contains 180 tasks and is described as the largest benchmark for speech and audio evaluation, with a design explicitly oriented toward community expansion, zero-shot evaluation, and capability profiling across heterogeneous audio domains (Huang et al., 2024, Huang et al., 2023).

1. Origins, motivation, and phase transition

Dynamic-SUPERB Phase-2 emerged from a problem already identified in the first Dynamic-SUPERB release: traditional speech benchmarks such as SUPERB, SLUE, ESB, FLEURS, ML-SUPERB, and LeBenchmark are largely static, assume task-specific models or heads, and do not provide a unified natural-language instruction interface for evaluating general-purpose audio models. The first Dynamic-SUPERB therefore introduced a dynamic, collaborative benchmark for speech in which models receive speech as input, text instructions, and are evaluated for zero-shot task generalization; its initial release covered 55 evaluation instances by combining 33 tasks and 22 datasets, with emphasis on instruction-following classification (Huang et al., 2023).

Phase-2 preserves that core premise but broadens it in scope and ambition. It is designed for modern “spoken LLMs” and related audio-language systems that are expected to operate more like multimodal foundation models: they ingest raw audio, interpret natural-language instructions, and answer in free text. Relative to the first generation, Phase-2 expands from a speech-only, classification-focused setup to a multi-modality benchmark covering speech, music, and environmental audio, and it introduces regression and sequence generation in addition to classification. The expansion is not only numerical but conceptual: it reframes the benchmark as a general evaluation platform for universal speech and audio models rather than a narrowly speech-processing benchmark (Huang et al., 2024).

The benchmark is termed “dynamic” in both a conceptual and procedural sense. Conceptually, it is intended to grow as new tasks become relevant. Procedurally, it incorporates a formal community contribution process. This dynamic structure is central to the project’s positioning: benchmark extension is treated primarily as an evaluation problem for instruction-following models, rather than as a repeated cycle of defining a new task and training a new task-specific system (Huang et al., 2024).

2. Task space, taxonomy, and benchmark composition

Dynamic-SUPERB Phase-2 contains 180 tasks organized under two top-level fields, each subdivided into domains that are meant to map onto capability clusters rather than isolated datasets. This taxonomy is one of the benchmark’s defining design decisions because it supports domain-level diagnosis while retaining task-level granularity (Huang et al., 2024).

Field	Domains
Speech	Speech Recognition; Speaker & Language; Spoken Language Understanding; Phonetics, Phonology, Prosody; Paralinguistics; Speech Enhancement; Speech / Voice / Hearing Disorders; Safety and Security
Music & audio	Music Classification; Harmony & Pitch; Rhythm Analysis; Singing Analysis; Quality Assessment; Sound Event; Safety; Spatial Audio Analysis; Signal Characteristics Analysis

Within these domains, the benchmark includes a wide range of task families. Speech recognition tasks include multilingual ASR on MLS, code-switched ASR on ASCEND and NTUML2021, AAVE recognition, Brazilian Portuguese recognition, and reformulated SUPERB and out-of-distribution ASR tasks. Paralinguistic tasks include emotion recognition on IEMOCAP and CREMA-D, RAVDESS-based emotion change detection, MELD conversational emotion recognition, vocal event detection, and COVID-19 cough detection. Speaker-related tasks include speaker verification, diarization, speaker counting, and speaker characteristic prediction such as age group and gender. Spoken language understanding tasks include intent classification, slot filling, dialogue acts, sentiment, semantic textual similarity, and sarcasm detection. Phonetics and prosody tasks include phoneme recognition, phoneme segment counting, phonological feature classification, stress detection, accent classification, pronunciation quality scoring, fluency scoring, and prosody naturalness (Huang et al., 2024).

The audio and music side is similarly broad. It includes genre, instrument, emotion, and tagging tasks; harmony and pitch tasks such as key detection and MAESTRO transcription; rhythm analysis through beat tracking; singing analysis via lyric recognition, lyric translation, vocal technique classification, and transcript verification; quality assessment through singing MOS prediction; sound event recognition across environmental, animal, and human non-speech sounds; spatial audio tasks such as position prediction and distance estimation; and signal characteristics analysis such as duration prediction and audio effect detection (Huang et al., 2024).

Phase-2 also introduces “core tasks,” which are lighter-weight subsets reformulated from SUPERB, MARBLE, and HEAR. These subsets allow fast evaluation of representative speech, music, and audio capabilities while preserving the benchmark’s instruction-based format. This suggests a two-level usage pattern: quick diagnostic evaluation through core tasks and broader capability auditing through the full 180-task suite (Huang et al., 2024).

3. Instruction-based formulation and evaluation protocol

Every task in Dynamic-SUPERB Phase-2 is framed as instruction-following. A model is given a text instruction, one or more audio inputs, and sometimes additional text such as candidate labels or target languages; it must generate a textual response. This design aligns the benchmark with the operational interface of contemporary audio-capable LLMs, where the model is not asked for logits over a fixed label set but for natural-language outputs that must nevertheless be evaluated consistently (Huang et al., 2024).

The benchmark is explicitly test-only. No training split is provided for benchmark tasks, and models are evaluated strictly in zero-shot mode. Each task may be instantiated with multiple possible instructions to probe sensitivity to prompt wording. For models that cannot ingest multiple audio files natively, the evaluation pipeline concatenates audio segments with $0.5$ seconds of silence between them. Models using Whisper encoders are limited to audio of length $\le 30$ seconds, and the authors report that only about $7\%$ of benchmark audio files exceed $30$ seconds, so model defaults are retained for fairness and efficiency (Huang et al., 2024).

The evaluation pipeline distinguishes three task types. For classification, model outputs are judged by GPT-4o at temperature $T=0$, which receives the instruction, ground truth, and model output and returns a deterministic YES/NO alignment decision. Accuracy is then computed as

$$\mathrm{Accuracy} = \frac{\text{count of YES decisions}}{\text{total instances}}.$$

The authors report that this LLM-based judging procedure aligns strongly with human annotation, with GPT-4o achieving about $98.67\%$ agreement against human labels, while LLaMA-3.1-70B-Instruct also performs well as an alternative evaluator (Huang et al., 2024).

For regression, models are again free to answer in natural language, but GPT-4o is used as a post-processor to extract a number or return “N/A” when extraction fails. The benchmark defines the N/A rate as

$$\mathrm{NAR} = \frac{\#\text{invalid outputs}}{\#\text{instances}}.$$

To penalize failures to produce parseable outputs, scaled scores are used. For higher-is-better metrics such as accuracy, F1, or correlation,

$s' = s \cdot (1 - \mathrm{NAR}),$

and for lower-is-better metrics such as WER, CER, or MSE,

$s' = \frac{s}{1 - \mathrm{NAR}}.$

For sequence generation tasks, such as ASR, captioning, and translation, the benchmark applies standard metrics directly to raw model outputs without LLM post-processing. For ASR-like tasks this includes metrics such as

$\le 30$0

with $\le 30$1, $\le 30$2, and $\le 30$3 denoting substitutions, deletions, and insertions, respectively (Huang et al., 2024).

At the aggregation stage, the benchmark avoids collapsing all tasks into a single global number because the tasks use heterogeneous metrics. Instead, it computes relative improvements against a cascaded Whisper-LLaMA baseline on a per-task basis and averages those relative scores within domains. The authors explicitly warn that outliers, especially tasks with very large errors, can distort domain means; task-level reporting therefore remains important for detailed analysis (Huang et al., 2024).

4. Evaluated models and empirical findings

The benchmark evaluates several publicly available models representing different design strategies for audio-language processing. These include SALMONN-7B and SALMONN-13B, LTU-AS, Qwen-Audio, Qwen2-Audio, WavLLM, MU-LLaMA, GAMA, and a cascaded Whisper-LLaMA baseline that uses Whisper-v3-large for ASR and LLaMA3.1-8B for downstream language reasoning over transcriptions (Huang et al., 2024).

The central empirical conclusion is that no model performs well universally across the 180 tasks. Instead, the results show strong specialization. Domain-level comparisons indicate that no model consistently surpasses the Whisper-LLaMA baseline on speech recognition and spoken language understanding, and the reported interpretation is that ASR plus text-LM reasoning remains strong for semantic tasks because text representation is explicit. By contrast, all universal models surpass the cascaded baseline in speaker and paralinguistics domains, where ASR pipelines are liable to discard information about speaker identity, pitch, and emotion. On audio and music domains, spoken LLMs such as Qwen, SALMONN, and WavLLM often outperform music-specific models such as MU-LLaMA and GAMA on harmony and pitch, music classification, and rhythm analysis. Safety and specialized tasks remain comparatively weak across the board, with some models producing unparseable outputs and therefore high N/A rates (Huang et al., 2024).

The core-task results make this specialization concrete. On reformulated SUPERB-style ASR, SALMONN-13B achieves $\le 30$4 WER and WavLLM $\le 30$5, whereas Whisper-LLaMA records $\le 30$6. On speaker verification, SALMONN-13B reaches $\le 30$7 accuracy and is clearly strongest among the reported models. On emotion recognition, WavLLM records $\le 30$8 accuracy and Qwen2-Audio $\le 30$9, confirming that paralinguistic sensitivity is distributed differently from ASR competence. On intent classification, however, Whisper-LLaMA leads with $7\%$0 accuracy, which underscores the continuing competitiveness of transcription-based pipelines for semantically dominated tasks. Query-by-example and speaker diarization remain difficult for all models, with diarization often suffering from $7\%$1 N/A because systems fail to produce parseable outputs (Huang et al., 2024).

A plausible implication is that present audio-language systems are already heterogeneous enough that a single aggregate notion of “best model” is misleading. Dynamic-SUPERB Phase-2 instead exposes a capability profile: English ASR and speaker verification are strengths for SALMONN-13B, emotion recognition is stronger for WavLLM and Qwen2-Audio, and transcription-mediated reasoning remains surprisingly strong for some SLU tasks (Huang et al., 2024).

5. Relation to Phase-1 and other benchmarks

Dynamic-SUPERB Phase-2 is best understood as both a continuation and a redefinition of the first Dynamic-SUPERB release. The first release established the instruction-following paradigm for speech, emphasized seen versus unseen task generalization, and restricted evaluation to classification for simplicity. Phase-2 broadens that agenda to regression and sequence generation, extends coverage beyond speech to music and environmental audio, and replaces the earlier six-dimension organization with a finer-grained capability taxonomy (Huang et al., 2023, Huang et al., 2024).

Relative to prior benchmarks, its distinctiveness lies in the combination of scale, task formulation, and output diversity. SUPERB focuses on speech encoders and roughly 13 tasks, requires task-specific heads and fine-tuning, and is mainly English speech. SLUE targets a smaller set of spoken language understanding tasks. HEAR evaluates audio representations over 19 tasks but is not instruction-based. MARBLE focuses on music representation tasks. AIR-bench evaluates universal audio-LLMs in a generative setting but with lower task count and narrower coverage. Phase-2 integrates task families from SUPERB, SLUE, HEAR, and MARBLE into a single instruction-based evaluation framework and extends beyond them with regression, sequence generation, and community-contributed tasks across speech, music, and general audio (Huang et al., 2024).

The project also changes the benchmarking unit from fixed supervised tasks to instruction-grounded capabilities. In the first Dynamic-SUPERB release, this shift was already visible in the move from static evaluation instances to a dynamic benchmark meant to accommodate unseen tasks. In Phase-2, the same logic is generalized: the benchmark is intended not merely to compare encoders or downstream heads, but to evaluate whether a model can interpret task descriptions in natural language and transfer that competence across modalities and domains (Huang et al., 2023, Huang et al., 2024).

6. Collaborative expansion, limitations, and research significance

The collaborative mechanism is integral to Dynamic-SUPERB Phase-2 rather than ancillary. In March 2024 the maintainers opened a GitHub-based call for tasks. Contributors submit a proposal with task motivation, datasets and licenses, a README describing task definition, challenges, metrics, and state-of-the-art results, and a JSON specification for the evaluation pipeline. Editors review submissions for duplication, licensing, and consistency with design principles, suggest refinements, and standardize accepted tasks through a Hugging Face-based data workflow. From approximately 145 proposals, 91 tasks were accepted for Phase-2, while others remained under review for future phases (Huang et al., 2024).

The benchmark is open-sourced with task data and evaluation pipeline, and its model interface is deliberately simple: conceptually, a system needs to implement an audio-plus-instruction to text-output mapping. This makes the benchmark suitable both as a shared evaluation standard and as a diagnostic instrument for model development. Researchers can use the core tasks for rapid testing, the domain taxonomy for capability-focused diagnosis, and N/A rates for identifying instruction-following failures rather than raw recognition failures (Huang et al., 2024).

The authors also identify several limitations. Phase-2 focuses mainly on audio understanding rather than generation, because few universal speech or audio generation models currently exist. The taxonomy is explicitly evolving and may require revision as the field diversifies. The LLM-based evaluation pipeline works well for current tasks but may not transfer unchanged to more complex dialogue or structured outputs. The benchmark also inherits dataset biases in language, accent, and demographics from its source corpora (Huang et al., 2024).

Its broader significance lies in making the gap between current systems and a genuinely universal spoken LLM empirically visible. The benchmark shows that strong performance in ASR, speaker analysis, emotion recognition, music understanding, or safety does not yet cohere into a single broadly competent model. This suggests that progress in audio-language modeling will likely depend not only on larger multimodal pretraining, but also on more robust instruction following, better structured output control, and stronger cross-domain transfer. In that sense, Dynamic-SUPERB Phase-2 functions simultaneously as a benchmark, a taxonomy of capabilities, and an agenda-setting instrument for research on universal speech and audio models (Huang et al., 2024).