S-SONDO: Self-Supervised Audio Distillation

• S-SONDO is a framework that distills audio foundation models by aligning output embeddings, bypassing the need for layer-wise or logit matching.
• It employs a simple two-layer MLP mapping head and balanced data sampling from AudioSet to achieve compression ratios up to 61× with high retention.
• Empirical results demonstrate that using cosine similarity loss with smaller batch sizes yields robust performance, highlighting the importance of teacher quality.

Searching arXiv for S-SONDO and closely related cited models to ground the article in current papers. S-SONDO, short for Self-Supervised KnOwledge DistillatioN for General AuDio FOundation Models, is a framework for compressing large audio foundation models into substantially smaller students by aligning output embeddings rather than logits or intermediate feature maps. It is presented as the first framework to distill general audio models using only their output embeddings, thereby targeting teachers that expose only a fixed embedding, including self-supervised and metric-learning models. The framework is explicitly described as simple and architecture-agnostic, because it requires only teacher–student output embeddings and does not depend on internal architectural correspondence or pre-training objective. Empirically, it distills two audio foundation models into three efficient students that are up to 61 times smaller while retaining up to 96% of teacher performance (Adlouni et al., 27 Apr 2026).

1. Problem Setting and Scope

Modern self-supervised or metric-learning audio foundation models, including M2D (Niizumi et al. ’23) and MATPAC++ (Quelennec et al. ’25), achieve state-of-the-art performance across speech, music, and environmental sound tasks, but typically contain parameters on the order of $10^8$ (Adlouni et al., 27 Apr 2026). The central deployment problem is therefore not only representational quality, but also inference cost and edge-device feasibility.

S-SONDO is formulated against a specific limitation of prior knowledge distillation practice in audio. Standard knowledge distillation methods, exemplified in the data by Hinton et al. ’15 and Romero et al. ’15, match class logits or internal feature maps. That assumption excludes teachers that output only embeddings. S-SONDO addresses this gap by training a smaller student $f_\theta$ to produce embeddings $Z_s$ that, after a learned projection $h_\beta$, match the teacher’s embeddings $Z_t$, and by sampling data in a balanced way through clustering of the teacher’s embeddings (Adlouni et al., 27 Apr 2026).

A common misconception is that knowledge distillation necessarily requires class probabilities or explicit layer-wise correspondence. In the formulation used here, neither is required. The framework operates solely on representation-level alignment at the output embedding level. This suggests that S-SONDO is best understood not as a variant of classifier compression, but as a representation distillation method for embedding-based audio models.

2. Mathematical Formulation

Let $X$ be an input audio clip. The student network $f_\theta$ outputs

$Z_s = f_\theta(X) \in \mathbb{R}^{N \times d_s},$

and the teacher network $g_\gamma$ outputs

$Z_t = g_\gamma(X) \in \mathbb{R}^{N \times d_t},$

where $f_\theta$0 is the batch size and $f_\theta$1 are the student and teacher embedding dimensions, respectively (Adlouni et al., 27 Apr 2026).

A small two-layer MLP projection, termed the mapping head $f_\theta$2, produces

$f_\theta$3

Training minimizes a single self-supervised distillation loss

$f_\theta$4

with several candidate choices. Writing $f_\theta$5 and $f_\theta$6 for the $f_\theta$7-th rows of $f_\theta$8 and $f_\theta$9, the losses considered are:

$Z_s$0

$Z_s$1

$Z_s$2

$Z_s$3

$Z_s$4

where $Z_s$5 as in Elizalde et al. ’23, and $Z_s$6 are softmax-normalized embeddings over $Z_s$7 dimensions (Adlouni et al., 27 Apr 2026).

Because neither teacher nor student provides logits or labels, the total objective is simply

$Z_s$8

This formulation is technically notable for its minimalism: there are no auxiliary classification losses, no contrastive negatives beyond those implicit in the CLAP variant, and no architecture-specific feature matching. A plausible implication is that the framework’s generality derives from reducing distillation to embedding-space alignment plus a low-capacity dimensionality adapter.

3. Architectural Components and Architecture-Agnosticism

The framework is instantiated with two teachers and three students (Adlouni et al., 27 Apr 2026).

Model	Role	Parameters / embedding dimension
M2D	Teacher	$Z_s$9 M parameters, $h_\beta$0
MATPAC++	Teacher	$h_\beta$1 M parameters, $h_\beta$2
MobileNetV3-Small	Student	$h_\beta$3 M parameters, $h_\beta$4
DynamicMobileNet (DyMN)	Student	$h_\beta$5 M parameters
ERes2Net	Student	$h_\beta$6 M parameters

The teacher side comprises M2D, described as a Transformer, and MATPAC++, described as an enhanced masked-latent Transformer. The student side comprises MobileNetV3-Small, DynamicMobileNet (DyMN), and ERes2Net. The mapping head $h_\beta$7 is a 2-layer MLP with hidden size 1 280 and ReLU (Adlouni et al., 27 Apr 2026).

The term architecture-agnostic has a precise meaning in this context. It does not mean that all model components are identical or interchangeable; rather, it means that no layer-wise alignment or logits are required, and the method is applicable to any teacher–student pair that produces embeddings. This is a narrower but operationally useful notion of agnosticism: the compatibility condition is the existence of output embeddings, not architectural homology.

4. Training Procedure and Balanced Data Sampling

Training uses AudioSet, described as 1.8 M ten-second clips, pre-processed into 128-bin log-Mel spectrograms (50–16 kHz) at 32 kHz (Adlouni et al., 27 Apr 2026). The sampling strategy is a central component of the method.

The framework introduces Balanced Data Sampling (BDS). First, teacher embeddings $h_\beta$8 are clustered over the entire AudioSet into $h_\beta$9 clusters via k-means. Each training example $Z_t$0 is then assigned a weight

$Z_t$1

Each epoch samples 100 000 distinct clips without replacement proportional to $Z_t$2 (Adlouni et al., 27 Apr 2026).

Optimization is specified as follows: 200 epochs, batch size $Z_t$3, Adam with $Z_t$4, $Z_t$5, and base LR = $Z_t$6 with cosine annealing. Only $Z_t$7 is minimized, with no additional regularization terms beyond standard weight decay (Adlouni et al., 27 Apr 2026).

The procedural role of BDS is not merely data balancing in the conventional class-frequency sense. Because the clustering is performed in the teacher embedding space, it functions as a representation-space balancing mechanism. This suggests that S-SONDO attempts to equalize exposure to teacher-defined acoustic structure rather than to externally annotated labels.

5. Empirical Performance

Evaluation is conducted on 7 audio-tagging benchmarks: OpenMIC, NSynth, GTZAN, Magnatagatune, FSD50K, ESC-50, and UrbanSound8K. Performance is measured as mAP for multi-label tasks and accuracy (%) for single-label tasks, then averaged (Adlouni et al., 27 Apr 2026).

The principal quantitative findings are:

Distillation setting	Average score	Retention / compression
MATPAC++ $Z_t$8 MobileNetV3 (2.9 M)	73.0	96.4% retention, $Z_t$9 compression ratio
MATPAC++ $X$0 DyMN (8.7 M)	72.6	95.9% retention
MATPAC++ $X$1 ERes2Net (1.4 M)	70.8	93.5% retention, $X$2 compression
M2D teacher	68.7–69.2	92.4–93.1% retention

For comparison, MATPAC++ $X$3 MobileNetV3 attains avg. 73.0 vs. 72.0 for supervised training, indicating that the distilled student can exceed its supervised-training baseline (Adlouni et al., 27 Apr 2026).

Compression ratio and retention are defined by

$X$4

and

$X$5

These results establish the empirical profile of the method: strong retention under aggressive compression, especially when the teacher is MATPAC++. A plausible implication is that the quality of the teacher representation materially affects the absolute ceiling attainable by embedding-only distillation, since the M2D teacher yields similarly strong retention but lower absolute scores.

6. Ablations, Practical Recommendations, and Interpretive Boundaries

The loss-function ablation reports the following averages: cosine (72.7) $X$6 CLAP (72.3) $X$7 (65.6) $X$8 MSE (62.7) $X$9 KL (66.2) (Adlouni et al., 27 Apr 2026). In practice, $f_\theta$0 yielded the best trade-off for $f_\theta$1, whereas CLAP surpassed it only when $f_\theta$2. The practical recommendation is therefore to use cosine similarity $f_\theta$3 as a default distillation loss when batch sizes $f_\theta$4, and to consider contrastive CLAP loss when one can afford $f_\theta$5 batch size (Adlouni et al., 27 Apr 2026).

The data-sampling ablation indicates that BDS is crucial to avoid collapse when student capacity is low. The clearest example is ERes2Net on MATPAC++ without BDS, which fails to converge and yields avg. 44.8 vs 70.8 with BDS. For larger students, gains are characterized as modest but consistent (Adlouni et al., 27 Apr 2026).

Additional implementation guidance is explicit. Pseudo-label clustering with $f_\theta$6 strikes a good balance. The data further states that one may increase $f_\theta$7 for finer semantics in single-label tasks, though careful tuning is needed for multi-label domains. The paper also reports that a simple 2-layer MLP mapping head suffices, and that more complex heads did not yield significant gains (Adlouni et al., 27 Apr 2026).

These findings delimit both the strengths and the boundaries of the method. S-SONDO does not claim that any embedding loss is equally effective, nor that balanced sampling is optional. Rather, its reported performance depends materially on the choice of cosine similarity and on clustering-based balanced data sampling. In this sense, the framework’s simplicity should not be confused with insensitivity to training design. Its contribution lies in showing that output-embedding alignment alone can be sufficient for high-retention compression of general audio foundation models, provided that the geometry of the distillation loss and the sampling distribution are chosen appropriately (Adlouni et al., 27 Apr 2026).