USAD 2.0: Universal Audio Encoder

• USAD 2.0 is a universal audio encoder that unifies speech, music, and environmental sound representation using multi-expert SSL distillation and effective depth scaling.
• It employs domain-aware weighting and a dedicated music expert to balance performance across heterogeneous audio signals while reducing self-attention costs.
• The system integrates a second-stage supervised distillation for audio LLM alignment, achieving state-of-the-art results on HEAR, MARBLE, and XARES-LLM benchmarks.

USAD 2.0 is a universal audio encoder introduced to provide a single frontend for speech, environmental sounds, and music in modern audio-language systems that increasingly rely on one encoder for diverse inputs (Chang et al., 4 Jun 2026). It integrates knowledge from multiple self-supervised learning (SSL) experts and supervised foundation models, extends prior multi-domain coverage to the music domain, and scales to 1 billion parameters through depth scaling. The system is positioned against a landscape in which domain-specific SSL encoders such as WavLM for speech and MuQ for music are strong within their own domains but fail on out-of-domain audio, while prior multi-domain approaches such as USAD and SPEAR remain limited in coverage and evaluation.

1. Scope, motivation, and problem setting

The central objective of USAD 2.0 is to build a universal audio encoder by distilling from multiple SSL experts and supervised foundation models, extending coverage to music, and scaling model capacity under limited compute. The motivating deployment setting is a modern audio-LLM that requires a single frontend rather than a domain-switching stack.

Three motivations are explicit. First, the model seeks to leverage the complementary strengths of speech, general-audio, and music SSL models. Second, it incorporates supervised encoders, specifically Whisper and Audio Flamingo 3, because recent studies suggest supervised encoders align better with audio LLM tasks. Third, it pursues efficient capacity scaling through depth scaling and reduced feature rate.

Within this framing, USAD 2.0 addresses a recurrent tension in audio representation learning: domain-specialized encoders often achieve high in-domain performance, but a universal frontend must preserve competence across heterogeneous signals with different statistical structure, annotation regimes, and downstream objectives. A plausible implication is that the model is not merely a larger encoder, but an attempt to reconcile heterogeneous teacher priors within one representational space.

2. Encoder backbone and parameter scaling

The encoder takes a 16 kHz waveform as input and converts it to a 128-bin Mel spectrogram. Its backbone contains a CNN feature extractor with 2 convolutional layers, using kernel size 3 and strides adjusted for framerate, followed by a convolutional positional encoder with 5 layers of 1D convolutions as in wav2vec 2.0. The main sequence model is a transformer encoder composed of multi-head self-attention and feed-forward layers.

USAD 2.0 is instantiated in Small, Base, Large, XLarge, and XXLarge configurations, which vary hidden size, feed-forward size, number of layers, and number of heads. The paper does not enumerate all such values in the provided summary, but it does specify the largest scaled variant. The XXLarge+ configuration has 1 036 M parameters, hidden size 1 280, FFN size 5 120, 48 layers, 20 heads, and a 25 Hz framerate.

Efficient scaling is achieved through two mechanisms. The first is temporal resolution reduction: the CNN stride is increased so that the feature rate drops from 50 Hz to 25 Hz, halving sequence length and self-attention cost. The second is depth up-scaling in the style of “Solar 10.7 b” by Kim et al. (2024): lower and upper blocks of a pretrained 32-layer model are copied and stacked to form a 48-layer model, after which the resulting network is fine-tuned for a few thousand steps. This design links parameter growth to manageable inference cost rather than to a proportional increase in sequence-processing complexity.

3. Multi-expert SSL distillation and domain-aware weighting

The first training stage distills from $M=3$ SSL experts: WavLM for speech, ATST-Frame for general audio, and MuQ for music (Chang et al., 4 Jun 2026). In the standard USAD formulation, the student minimizes an averaged layerwise squared-error objective over teachers and layers:

$$\mathcal{L}_{\text{USAD}} = \frac{1}{M K}\sum_{m=1}^{M}\sum_{k=1}^{K} \left\lVert h_k^s(x) - h_k^{t_m}(x)\right\rVert_2^2,$$

where $h_k^s$ and $h_k^{t_m}$ are the student and $m$-th teacher hidden representations at layer $k$.

USAD 2.0 modifies this with domain-aware weighting to address teacher mismatch. For a training clip with domain label $m_{\text{data}}$, the loss becomes

$$\mathcal{L}_{\text{USAD 2.0}} = \sum_{m=1}^{M} w_m(m_{\text{data}})\,\mathcal{L}_m, \quad w_m(m_{\text{data}})= \begin{cases} \frac{\alpha}{\alpha+M-1}, & m=m_{\text{data}} \ \frac{1}{\alpha+M-1}, & m\neq m_{\text{data}} \, , \end{cases}$$

with $\alpha>1$, and the reported choice is $\alpha=10$.

The weighting scheme soft-emphasizes the matching teacher while preserving contributions from mismatched teachers. In the paper’s wording, mismatched teachers still contribute small gradients, preserving cross-domain cues. This is a substantial design distinction from a hard routing scheme: the model remains universally trained even when domain labels are available, rather than collapsing into independent per-domain objectives.

4. Music-domain extension and second-stage supervised alignment

A major extension over prior USAD is the inclusion of a dedicated music expert and explicit music-domain training data. The combined first-stage training set comprises 116 K h speech, 21 K h general-audio, and 13 K h music, with domain labels assigned per corpus. The music corpora listed are FMA, MTG-Jamendo, Music4All, NSynth, MagnaTagATune, and MAESTRO.

The empirical effect reported for this extension is specific and large: including the music teacher and music data yields a +30% relative gain on pitch classification. Conversely, the ablation without music teacher or data incurs a $$\mathcal{L}_{\text{USAD}} = \frac{1}{M K}\sum_{m=1}^{M}\sum_{k=1}^{K} \left\lVert h_k^s(x) - h_k^{t_m}(x)\right\rVert_2^2,$$0 relative pitch accuracy result. These findings situate music not as auxiliary coverage but as a structurally necessary domain for universal audio encoding.

USAD 2.0 also introduces a second-stage supervised distillation procedure intended to align the encoder with audio LLM objectives. The teachers in this stage are the Whisper Large-v3 encoder for multilingual ASR and the Audio Flamingo 3 AF-Whisper encoder for captioning and ASR. The student is initialized from first-stage USAD 2.0, and only the final layer is distilled:

$$\mathcal{L}_{\text{USAD}} = \frac{1}{M K}\sum_{m=1}^{M}\sum_{k=1}^{K} \left\lVert h_k^s(x) - h_k^{t_m}(x)\right\rVert_2^2,$$1

This stage formalizes the paper’s claim that supervised encoders align better with audio LLMs. It also differentiates the model family terminologically: the notation “USAD 2.0+” refers to systems after this supervised distillation stage.

5. Evaluation protocols and quantitative performance

Evaluation is organized around both frozen-encoder probing and audio-LLM-based assessment (Chang et al., 4 Jun 2026). The probing benchmarks are HEAR, consisting of 21 tasks across speech, environmental sound, and music; MARBLE, which is music-centric and includes genre, key, emotion, singer, instrument tagging, and pitch; and SUPERB, which is speech-only and includes phoneme recognition, ASR, speaker diarization, keyword spotting, intent, emotion, speaker identification, and spoof detection.

The audio-LLM benchmark is XARES-LLM. Track A contains classification tasks such as keyword spotting, speaker and language identification, spoof detection, intent, emotion, sound, genre, instrument classification, and event detection. Track B contains understanding tasks: English and Mandarin ASR, audio captioning, and music captioning. The evaluation protocol trains a multitask audio LLM on frozen encoder outputs and reports per-task and per-track averages.

The main reported encoder-only results are as follows:

Model	Params	Reported scores
USAD 2.0 Large	336 M	HEAR 82.9; MARBLE 75.8; A 0.667; B 0.473
USAD 2.0 XLarge	695 M	HEAR 82.5; MARBLE 75.7; A 0.708; B 0.485
USAD 2.0+ Large+	336 M	HEAR 84.0; MARBLE 75.1; A 0.769; B 0.580
USAD 2.0+ XLarge+	695 M	HEAR 84.4; MARBLE 75.0; A 0.772; B 0.611
USAD 2.0+ XXLarge+	1036 M	HEAR 84.4; MARBLE 75.6; A 0.783; B 0.624

The paper states that USAD 2.0 outperforms prior universal and domain-specific SSL encoders at comparable sizes. It further states that second-stage supervised distillation boosts HEAR by approximately 1.5 points and XARES-LLM Track B by more than 0.10, and that the 1 B XXLarge+ model achieves state-of-the-art performance across probing and LLM tasks.

The ablation studies isolate several effects. The best trade-off for the domain-aware scale is at $$\mathcal{L}_{\text{USAD}} = \frac{1}{M K}\sum_{m=1}^{M}\sum_{k=1}^{K} \left\lVert h_k^s(x) - h_k^{t_m}(x)\right\rVert_2^2,$$2. Removing domain weighting causes a +50% PER on phoneme recognition and degrades sound and pitch tasks. Supervised initialization improves Track B from 0.574 to 0.611 relative to training from scratch. Among depth-scaling strategies, depth up-scaling outperforms uniform duplication, which in turn outperforms adding new top layers.

6. Inference efficiency, interpretation, and future directions

The reported inference measurements use an A5000 GPU and a 30 s input. Large at 50 Hz attains a real-time factor of 0.0029 with 1.2 GB memory. XLarge at 25 Hz attains 0.0018, approximately $$\mathcal{L}_{\text{USAD}} = \frac{1}{M K}\sum_{m=1}^{M}\sum_{k=1}^{K} \left\lVert h_k^s(x) - h_k^{t_m}(x)\right\rVert_2^2,$$3 faster than the 50 Hz setting, with 1.7 GB memory. XXLarge at 25 Hz attains 0.0026 with 2.4 GB memory and is reported as faster than Large at 50 Hz despite having $$\mathcal{L}_{\text{USAD}} = \frac{1}{M K}\sum_{m=1}^{M}\sum_{k=1}^{K} \left\lVert h_k^s(x) - h_k^{t_m}(x)\right\rVert_2^2,$$4 more parameters.

These figures are consequential because they separate parameter count from end-to-end deployment cost. The framerate reduction cuts sequence length, and therefore self-attention cost, sufficiently that a deeper and much larger model can remain operationally competitive. This suggests that the architecture’s scaling strategy is as much about sequence economics as about representational capacity.

The paper’s concluding characterization is that USAD 2.0 integrates multi-expert SSL distillation, domain-aware weighting, a music expert, supervised alignment, and efficient depth scaling to produce a universal audio encoder with strong cross-domain balance. Its stated limitations and future directions are fourfold: incorporating video-audio joint distillation for audiovisually grounded representations; exploring dynamic routing or adapters for domain adaptation at inference; extending second-stage distillation to additional supervised tasks such as music transcription; and investigating continual or lifelong distillation to integrate new domain experts without retraining from scratch.

A common oversimplification would be to view the reported gains as a consequence of scale alone. The ablations do not support that reading. They attribute distinct effects to domain-aware weighting, music-domain coverage, supervised second-stage distillation, and the specific depth up-scaling strategy, indicating that USAD 2.0 is a compound method rather than a single scaling intervention.