Universal Speech and Audio Distillation

• USAD is a unified self-supervised learning method that distills knowledge from domain-specific models into one robust, general-purpose audio encoder.
• It employs an efficient sparse layer-to-layer distillation technique that aligns selective teacher layers to optimize performance and reduce computational overhead.
• USAD simplifies deployment across speech, sound, and music applications while achieving near state-of-the-art results on benchmarks like SUPERB and HEAR.

Universal Speech and Audio Distillation (USAD) is a unified methodology in self-supervised learning that transfers knowledge from specialized domain-expert models (teachers) into a single student model to create robust, general-purpose audio representations encompassing diverse audio types—speech, sound, and music. By leveraging efficient sparse layer-to-layer distillation, USAD enables a single encoder to achieve near state-of-the-art results across multiple benchmarks, significantly streamlining deployment for broad audio processing applications.

1. Distillation Framework and Methodology

USAD employs a sparse layer-to-layer (L2L) distillation approach, wherein a student model is trained to match the hidden representations of pretrained domain-specific teacher models at selected encoding layers. Unlike dense L2L distillation, which aligns all layers, USAD selects $K$ informative layers (e.g., layers 3, 6, 9, 12 in a 12-layer model), substantially improving training and inference efficiency.

Training simultaneously involves two teacher models: one trained for speech tasks and one for general audio (music, environmental sound). For each batch drawn from a comprehensive, balanced audio dataset, the model uses loss functions to align student and teacher representations:

$$l^{(\mathrm{S})}_{k} = \left\lfloor \frac{kL^{(\mathrm{S})}}{K} \right\rfloor,\quad l^{(\mathrm{T1})}_{k} = \left\lfloor \frac{kL^{(\mathrm{T1})}}{K} \right\rfloor,\quad l^{(\mathrm{T2})}_{k} = \left\lfloor \frac{kL^{(\mathrm{T2})}}{K} \right\rfloor$$

for $k=1,\ldots, K$, where $L^{(\mathrm{S})}$, $L^{(\mathrm{T1})}$, $L^{(\mathrm{T2})}$ are the depths of the student and teacher networks.

The distillation loss is:

$$\mathcal{L}_{k,t}^{(\mathrm{T})} = \frac{1}{D} \left\| \tilde{\boldsymbol{z}}^{(\mathrm{T})}_{k, t} - \boldsymbol{z}^{(\mathrm{T})}_{k, t} \right\|_1 - \log \sigma \left[ \cos \left( \tilde{\boldsymbol{z}}^{(\mathrm{T})}_{k, t}, \boldsymbol{z}^{(\mathrm{T})}_{k, t} \right) \right]$$

where $\sigma$ is the sigmoid function and $D$ the feature dimension. Separate multi-layer perceptron (MLP) heads project the student to each teacher’s feature dimension, and loss is summed across layers, time steps, and both teacher models.

2. Model Architecture and Data Strategy

The USAD architecture integrates several components for universal representation:

• Input preprocessing: 128-D Mel-spectrogram extraction from raw audio.
• Core encoder: Transformer with relative positional encoding, scaling from 24M up to 330M parameters.
• Positional encoding: Five-layer convolution after frame/patch embedding.
• Decoder heads: Distinct MLP projections per distilled layer and per teacher.
• Dataset: Training on Mix126k-B—a large, balanced mixture of speech, sound, and music—ensures robust cross-domain generalization.

Frame-based features (rather than patch-based) are preferred during distillation, as these preserve the high temporal resolution demanded by speech tasks and also confer benefits for other audio modalities.

3. Performance Across Benchmarks

USAD’s single encoder achieves near state-of-the-art results on both speech and non-speech tasks, as validated by major evaluation suites:

• SUPERB: Covers speech-centric tasks such as automatic speech recognition (ASR), speaker diarization (SD), keyword spotting (KS), intent classification (IC), speaker identification (SID), and emotion recognition (ER).
• HEAR: Comprises 19 tasks spanning speech, sound event detection, music classification, and more.

Notable results include:

• USAD Large (330M): SUPERB instance-level score 948.3, matching or exceeding WavLM Base+ and outperforming most audio SSL baselines in average scores.
• AS-20K (AudioSet event tagging): mAP 37.4—comparable to top-performing patch-based models.
• ESC-50 (environmental sound classification): 92.7% accuracy—on par with domain experts.
• HEAR: An average score of 79.7, surpassing naive concatenation of teacher model features.

Smaller USAD variants (24M and 94M) outperform other compact models such as DPWavLM and DistilHuBERT, demonstrating strong efficiency gains.

4. Technical Innovations

USAD introduces several advancements in universal audio modeling:

• Sparse L2L distillation: Efficiently aligns student and teacher only at select layers, reducing computational and memory demands while maintaining high-fidelity knowledge transfer.
• Multi-teacher integration: Concurrent distillation from both speech and general audio teachers yields highly transferable representations, eliminating domain-specific limitations.
• Frame-wise feature alignment: Provides critical improvement in speech (and overall audio) generalization, confirmed by ablation studies.
• Balanced multi-domain batching: Ensures that training fully exploits diverse data, allowing the student to develop robust, domain-generalized representations.

5. Applications and Practical Impact

USAD serves as a universal backbone across a wide range of applications:

• Speech processing: ASR, SD, ER, intent and keyword recognition.
• Sound event detection: General-purpose tagging on datasets such as AudioSet and ESC-50.
• Music and sound analysis: Instrument or genre classification and music information retrieval.
• Multimodal and audio LLMs: Acts as a shared encoder for audio-driven conversational or generative systems.

By removing the need for separate models for each audio type, USAD simplifies deployment and maintenance, particularly in resource-constrained settings.

6. Directions for Future Research

Several avenues are identified for further refinement of USAD:

• Robustness: Additional noise, reverberation, and distributional shift handling.
• Multilinguality: Extension to cross-lingual and code-switching speech.
• Audio-LLM Integration: As a universal audio representation backbone for audio-first or multimodal LLMs.
• Dynamic teacher/layer selection: Data- or task-driven optimization of which teacher and layer combinations yield maximal performance and efficiency.
• Scaling and adaptation: Validation on larger, more diverse, and dynamic datasets; adaptation to streaming and online inference.

7. Contributions to the Field

USAD is the first approach to achieve universal audio representation by distilling from both domain-specific (speech) and general (sound/music) SSL models with a sparse, computationally efficient methodology. The empirical outcomes indicate USAD matches or exceeds specialized models on their respective benchmarks, setting a new standard for efficient, scalable, and robust all-in-one audio encoders. This positions USAD as a foundational component for the next generation of multimodal and audio-driven AI systems.