---
title: 'S-SONDO: Self-Supervised Audio Distillation'
url: https://www.emergentmind.com/topics/s-sondo
type: topic
---

# S-SONDO: Self-Supervised Audio Distillation

Searching arXiv for S-SONDO and closely related cited models to ground the article in current papers.
S-SONDO, short for **Self-Supervised KnOwledge DistillatioN for General AuDio FOundation Models**, is a framework for compressing large audio foundation models into substantially smaller students by aligning **output embeddings** rather than logits or intermediate feature maps. It is presented as the **first framework to distill general audio models using only their output embeddings**, thereby targeting teachers that expose only a fixed embedding, including self-supervised and metric-learning models. The framework is explicitly described as **simple** and **architecture-agnostic**, because it requires only teacher–student output embeddings and does not depend on internal architectural correspondence or pre-training objective. Empirically, it distills two audio foundation models into three efficient students that are **up to 61 times smaller** while retaining **up to 96% of teacher performance** [2604.24933].

## 1. Problem Setting and Scope

Modern self-supervised or metric-learning audio foundation models, including **M2D** (Niizumi et al. ’23) and **MATPAC++** (Quelennec et al. ’25), achieve state-of-the-art performance across speech, music, and environmental sound tasks, but typically contain parameters on the order of $10^8$ [2604.24933]. The central deployment problem is therefore not only representational quality, but also inference cost and edge-device feasibility.

S-SONDO is formulated against a specific limitation of prior knowledge distillation practice in audio. Standard knowledge distillation methods, exemplified in the data by Hinton et al. ’15 and Romero et al. ’15, match **class logits** or **internal feature maps**. That assumption excludes teachers that output only embeddings. S-SONDO addresses this gap by training a smaller student $f_\theta$ to produce embeddings $Z_s$ that, after a learned projection $h_\beta$, match the teacher’s embeddings $Z_t$, and by sampling data in a **balanced** way through clustering of the teacher’s embeddings [2604.24933].

A common misconception is that knowledge distillation necessarily requires class probabilities or explicit layer-wise correspondence. In the formulation used here, neither is required. The framework operates solely on representation-level alignment at the output embedding level. This suggests that S-SONDO is best understood not as a variant of classifier compression, but as a representation distillation method for embedding-based audio models.

## 2. Mathematical Formulation

Let $X$ be an input audio clip. The student network $f_\theta$ outputs

$$
Z_s = f_\theta(X) \in \mathbb{R}^{N \times d_s},
$$

and the teacher network $g_\gamma$ outputs

$$
Z_t = g_\gamma(X) \in \mathbb{R}^{N \times d_t},
$$

where $N$ is the batch size and $d_s, d_t$ are the student and teacher embedding dimensions, respectively [2604.24933].

A small two-layer MLP projection, termed the **mapping head** $h_\beta : \mathbb{R}^{d_s} \to \mathbb{R}^{d_t}$, produces

$$
Z'_s = h_\beta(Z_s) \in \mathbb{R}^{N \times d_t}.
$$

Training minimizes a single self-supervised distillation loss

$$
\mathcal{L}_{KD}(Z'_s, Z_t),
$$

with several candidate choices. Writing $Z_s'^{(i)}$ and $Z_t^{(i)}$ for the $i$-th rows of $Z'_s$ and $Z_t$, the losses considered are:

$$
\mathcal{L}_{MSE} = \frac{1}{N} \sum_{i=1}^{N} \| Z'_s{}^{(i)} - Z_t^{(i)} \|_2^2
$$

$$
\mathcal{L}_{\ell_1} = \frac{1}{N} \sum_{i=1}^{N} \| Z'_s{}^{(i)} - Z_t^{(i)} \|_1
$$

$$
\mathcal{L}_{cos} = \frac{1}{N} \sum_{i=1}^{N} \left[ 1 - \frac{\langle Z'_s{}^{(i)}, Z_t^{(i)} \rangle}{\| Z'_s{}^{(i)} \|_2 \| Z_t^{(i)} \|_2} \right]
$$

$$
\mathcal{L}_{CLAP} = -\frac{0.5}{N} \sum_{i=1}^{N} \log \left[ l_{Z'_s \to Z_t}^{(i)} + l_{Z_t \to Z'_s}^{(i)} \right]
$$

$$
\mathcal{L}_{KL} = \frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{d_t} P_t^{(i,j)} \log \left( \frac{P_t^{(i,j)}}{P'_s{}^{(i,j)}} \right),
$$

where $l_{a \to b} = \operatorname{diag}(\operatorname{softmax}(\langle a, b \rangle))$ as in Elizalde et al. ’23, and $P_t, P'_s$ are softmax-normalized embeddings over $d_t$ dimensions [2604.24933].

Because neither teacher nor student provides logits or labels, the total objective is simply

$$
\mathcal{L}_{total}(\theta,\beta) = \mathcal{L}_{KD}(Z'_s, Z_t).
$$

This formulation is technically notable for its minimalism: there are no auxiliary classification losses, no contrastive negatives beyond those implicit in the CLAP variant, and no architecture-specific feature matching. A plausible implication is that the framework’s generality derives from reducing distillation to embedding-space alignment plus a low-capacity dimensionality adapter.

## 3. Architectural Components and Architecture-Agnosticism

The framework is instantiated with two teachers and three students [2604.24933].

| Model | Role | Parameters / embedding dimension |
|---|---|---|
| M2D | Teacher | $\sim 86$ M parameters, $d_t \approx 1024$ |
| MATPAC++ | Teacher | $\sim 86$ M parameters, $d_t \approx 1024$ |
| MobileNetV3-Small | Student | $2.9$ M parameters, $d_s \approx 576$ |
| DynamicMobileNet (DyMN) | Student | $8.7$ M parameters |
| ERes2Net | Student | $1.4$ M parameters |

The teacher side comprises **M2D**, described as a Transformer, and **MATPAC++**, described as an enhanced masked-latent Transformer. The student side comprises **MobileNetV3-Small**, **DynamicMobileNet (DyMN)**, and **ERes2Net**. The mapping head $h_\beta$ is a **2-layer MLP with hidden size 1 280 and ReLU** [2604.24933].

The term **architecture-agnostic** has a precise meaning in this context. It does not mean that all model components are identical or interchangeable; rather, it means that **no layer-wise alignment or logits are required**, and the method is applicable to **any teacher–student pair that produces embeddings**. This is a narrower but operationally useful notion of agnosticism: the compatibility condition is the existence of output embeddings, not architectural homology.

## 4. Training Procedure and Balanced Data Sampling

Training uses **AudioSet**, described as **1.8 M ten-second clips**, pre-processed into **128-bin log-Mel spectrograms (50–16 kHz) at 32 kHz** [2604.24933]. The sampling strategy is a central component of the method.

The framework introduces **Balanced Data Sampling (BDS)**. First, teacher embeddings $g_\gamma(X)$ are clustered over the entire AudioSet into **$k = 50$ clusters via k-means**. Each training example $i$ is then assigned a weight

$$
w_i = \frac{1}{\mathrm{freq}(\mathrm{cluster}(i)) + 100}.
$$

Each epoch samples **100 000 distinct clips without replacement proportional to $w_i$** [2604.24933].

Optimization is specified as follows: **200 epochs**, **batch size $N = 64$**, **Adam** with $\beta_1 = 0.9$, $\beta_2 = 0.999$, and **base LR = $8 \times 10^{-4}$ with cosine annealing**. Only $\mathcal{L}_{KD}$ is minimized, with **no additional regularization terms beyond standard weight decay** [2604.24933].

The procedural role of BDS is not merely data balancing in the conventional class-frequency sense. Because the clustering is performed in the teacher embedding space, it functions as a representation-space balancing mechanism. This suggests that S-SONDO attempts to equalize exposure to teacher-defined acoustic structure rather than to externally annotated labels.

## 5. Empirical Performance

Evaluation is conducted on **7 audio-tagging benchmarks**: **OpenMIC, NSynth, GTZAN, Magnatagatune, FSD50K, ESC-50, and UrbanSound8K**. Performance is measured as **mAP for multi-label tasks** and **accuracy (%) for single-label tasks**, then averaged [2604.24933].

The principal quantitative findings are:

| Distillation setting | Average score | Retention / compression |
|---|---|---|
| MATPAC++ $\to$ MobileNetV3 (2.9 M) | 73.0 | 96.4% retention, $\approx 30\times$ compression ratio |
| MATPAC++ $\to$ DyMN (8.7 M) | 72.6 | 95.9% retention |
| MATPAC++ $\to$ ERes2Net (1.4 M) | 70.8 | 93.5% retention, $\approx 61\times$ compression |
| M2D teacher | 68.7–69.2 | 92.4–93.1% retention |

For comparison, **MATPAC++ $\to$ MobileNetV3** attains **avg. 73.0 vs. 72.0 for supervised training**, indicating that the distilled student can exceed its supervised-training baseline [2604.24933].

Compression ratio and retention are defined by

$$
CR = \frac{|\Theta_{teach}|}{|\Theta_{stud}|},
$$

and

$$
R(\%) = \left( \frac{\mathrm{Score}_{student}}{\mathrm{Score}_{teacher}} \right) \times 100.
$$

These results establish the empirical profile of the method: strong retention under aggressive compression, especially when the teacher is MATPAC++. A plausible implication is that the quality of the teacher representation materially affects the absolute ceiling attainable by embedding-only distillation, since the M2D teacher yields **similarly strong retention** but **lower absolute scores**.

## 6. Ablations, Practical Recommendations, and Interpretive Boundaries

The loss-function ablation reports the following averages: **cosine (72.7) $\approx$ CLAP (72.3) $\gg \ell_1$ (65.6) $\gg$ MSE (62.7) $\gg$ KL (66.2)** [2604.24933]. In practice, **$\mathcal{L}_{cos}$ yielded the best trade-off for $N = 64$**, whereas **CLAP surpassed it only when $N \ge 1024$**. The practical recommendation is therefore to use **cosine similarity $\mathcal{L}_{cos}$ as a default distillation loss when batch sizes $\lesssim 64$**, and to consider **contrastive CLAP loss** when one can afford **$> 1 000$ batch size** [2604.24933].

The data-sampling ablation indicates that **BDS is crucial to avoid collapse when student capacity is low**. The clearest example is **ERes2Net on MATPAC++ without BDS**, which **fails to converge** and yields **avg. 44.8 vs 70.8 with BDS**. For larger students, gains are characterized as **modest but consistent** [2604.24933].

Additional implementation guidance is explicit. **Pseudo-label clustering with $k \approx 50$ strikes a good balance**. The data further states that one may **increase $k$ for finer semantics in single-label tasks**, though **careful tuning is needed for multi-label domains**. The paper also reports that a **simple 2-layer MLP mapping head suffices**, and that **more complex heads did not yield significant gains** [2604.24933].

These findings delimit both the strengths and the boundaries of the method. S-SONDO does not claim that any embedding loss is equally effective, nor that balanced sampling is optional. Rather, its reported performance depends materially on the choice of **cosine similarity** and on **clustering-based balanced data sampling**. In this sense, the framework’s simplicity should not be confused with insensitivity to training design. Its contribution lies in showing that output-embedding alignment alone can be sufficient for high-retention compression of general audio foundation models, provided that the geometry of the distillation loss and the sampling distribution are chosen appropriately [2604.24933].

Source: https://www.emergentmind.com/topics/s-sondo