S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models

Abstract: General audio foundation models have recently achieved remarkable progress, enabling strong performance across diverse tasks. However, state-of-the-art models remain extremely large, often with hundreds of millions of parameters, leading to high inference costs and limited deployability on edge devices. Knowledge distillation is a proven strategy for model compression, but prior work in audio has mostly focused on supervised settings, relying on class logits, intermediate features, or architecture-specific techniques. Such assumptions exclude models that output only embeddings, such as self-supervised or metric-learning models. We introduce S-SONDO (Self-Supervised KnOwledge DistillatioN for General AuDio FOundation Models), the first framework to distill general audio models using only their output embeddings. By avoiding the need for logits or layer-level alignment, S-SONDO is architecture-agnostic and broadly applicable to embedding-based teachers. We demonstrate its effectiveness by distilling two audio foundation models into three efficient students that are up to 61 times smaller while retaining up to 96% of teacher performance. We also provide practical insights on loss choice and clustering-based balanced data sampling. Code is available here: https://github.com/MedAliAdlouni/ssondo.

• The paper introduces S-SONDO, a self-supervised framework that distills large audio models by aligning final embeddings without relying on supervised signals or class logits.
• It uses a projection head and clustering-based balanced data sampling to transfer knowledge effectively to lightweight student networks.
• Experimental results demonstrate that distilled models retain up to 96.4% of teacher performance while significantly reducing parameters for real-world deployment.

S-SONDO: A Self-Supervised Embedding-Based Knowledge Distillation Framework for General Audio Foundation Models

Introduction and Problem Motivation

Contemporary general audio foundation models have achieved high performance on a range of downstream tasks by leveraging large-scale self-supervised learning (SSL) on massive unlabeled datasets. However, these models are often characterized by high parameter counts—on the order of hundreds of millions—which translates to substantial inference costs and limited deployability for edge or resource-constrained devices. Existing knowledge distillation (KD) methods for audio typically presuppose access to supervised data, class logits, or require teacher-student architectural compatibility, which restricts their application to embedding-only models. The work “S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models” (2604.24933) addresses these limitations by proposing the first SSL KD framework for general audio that relies exclusively on the alignment of final embeddings, without recourse to class logits, intermediate layers, or supervision.

Methodological Framework

The S-SONDO framework generalizes the classic response-based KD approach by enforcing the alignment between the output embeddings of a large self-supervised teacher and a smaller student network. The method is fully architecture-agnostic due to the use of a learned mapping head, which projects student embeddings into the dimensionality of the teacher’s latent space.

Given a batch of input audio $X$, the teacher $g_\gamma$ and student $f_\theta$ generate embeddings $Z_t$ and $Z_s$ respectively. These may differ in dimension, so a multi-layer perceptron (MLP) mapping head $h_\beta$ transforms $Z_s$ to $Z'_s$ in the teacher’s embedding space. The core training signal is a self-supervised distillation loss $\mathcal{L}_{KD}(Z'_s, Z_t)$ evaluated between these embeddings.

Multiple candidates for $\mathcal{L}_{KD}$ are explored, including cosine similarity, CLAP loss (contrastive), L1, MSE, and KL divergence. The empirical analysis reveals that cosine similarity achieves the most robust alignment, outperforming contrastive and elementwise losses except at large batch sizes.

Figure 1: S-SONDO aligns student and teacher embeddings in the teacher’s latent space via self-supervised distillation, with architecture-agnosticity enabled by a projection head.

To address sample imbalance, the work introduces a clustering-based Balanced Data Sampling (BDS) strategy. Pseudo-labels are generated by k-means clustering of teacher embeddings, and batch composition is adjusted to upsample under-represented clusters, allowing the distilled student to better capture the semantic diversity of the teacher’s embedding space.

Experimental Setup

Evaluation is performed by distilling two Transformer-based SSL teacher models (M2D and MATPAC++, each ≈86M parameters) into three distinct student architectures—MobileNetV3 (2.9M), DyMN (8.7M), and ERes2Net (1.4M). The distillation and baseline supervised training are both conducted on AudioSet using the same augmentation and sampling protocols. Downstream generalization is evaluated across seven standard audio classification datasets spanning music and environmental sound tagging: OpenMIC, NSynth, GTZAN, MTT, FSD50K, ESC-50, and US8K. The comparative analysis includes both parameter and performance (mean average precision, classification accuracy) metrics.

Results and Analysis

Quantitative Performance

S-SONDO demonstrates strong retention of teacher accuracy: distilled students consistently reach at least 92.4% of their respective teacher’s average performance. Notably, S-SONDO students in 4/6 cases outperform their equivalent-dimensional, fully supervised baselines. For example, MobileNetV3 distilled from MATPAC++ achieves 73.0 average downstream score, corresponding to 96.4% of teacher performance and surpassing the supervised student (72.0), while being approximately 30× smaller than the teacher network.

Distillation Loss Comparison

Cosine loss outperforms other candidates in the default regime due to its alignment with the geometric structure of SSL embeddings, while CLAP loss (contrastive) improves at very large batch sizes but offers limited benefit for small batches. Losses such as L1, MSE, and KL divergence are less effective, as their underlying assumptions (elementwise correspondence, probabilistic outputs) are not well-matched to the semantics of SSL representations.

Impact of Balanced Data Sampling

Clustering-based BDS contributes to higher student generalization—especially for models with lower capacity (e.g., ERes2Net), which otherwise tend to collapse or overfit. The effect of $g_\gamma$0 (number of clusters) is dataset-dependent, and the gains are most prominent on single-label tagging tasks where the embedding space clusters are more semantically coherent.

Figure 2: S-SONDO ablation shows the effect of varying the number of clusters for BDS on downstream test accuracy, with a fixed random baseline shown for reference.

Practical and Theoretical Implications

S-SONDO establishes that self-supervised knowledge can be distilled solely from final embeddings, allowing for the construction of lightweight general audio models that retain most of the semantic richness of large foundation models. This finding challenges the presumption that direct alignment of final SSL representations is inadequate as a training signal without layerwise adaptation or architectural similarity.

The practical implications are significant for real-world deployment: S-SONDO enables the derivation of sub-10M parameter models that approach the representational and task performance of 86M-parameter teachers, facilitating mobile and low-power application scenarios.

On a theoretical level, the work provides evidence that the structure of SSL-derived embedding spaces is sufficiently robust to guide efficient, label-agnostic distillation, and that clustering-based strategies meaningfully supplement limited-capacity students.

Future Directions

Opportunities for further advancement include the use of more sophisticated unsupervised clustering schemes to capture multi-label semantic complexity, adaptive or learned batch construction strategies, and the integration of advanced contrastive objectives able to leverage positive/negative pairs beyond naive batch sampling. Extension to multimodal SSL teachers or streaming/online KD regimes is an open prospect.

Conclusion

S-SONDO constitutes a significant step in generalizing embedding-based knowledge distillation to the audio domain, establishing that SSL teacher alignment—augmented with balanced sampling—is performant, architecture-agnostic, and deployable for general audio modeling. The approach enables compression ratios up to 61× with up to 96.4% of the teacher’s performance retained. This fills a critical gap in the model distillation literature and portends the widespread availability of efficient foundation models for real-world applications.