---
title: AudioFM Framework for Audio Representation Learning
url: https://www.emergentmind.com/topics/audiofm
type: topic
---

# AudioFM Framework for Audio Representation Learning

AudioFM is a self-supervised framework for general-purpose audio representation learning introduced in “AudioFM: Towards General Purpose Audio Representation Learning” [2610.34183]. It is designed to preserve both holistic recording-level information—such as acoustic scenes and global soundscapes—and fine-grained local information, including brief events, temporal transitions, and frequency-specific structures. AudioFM combines recording-level contrastive learning with patch-wise masked latent discrimination and weaker supervision from eligible temporal neighbors. The resulting encoder is evaluated through linear probing, fine-tuning, frame-level event detection, and zero-shot transfer across environmental, bioacoustic, medical, speech-emotion, and general audio domains.

## 1. Conceptual foundations and objectives

AudioFM addresses a limitation of recording-level self-supervised learning. Conventional audio SSL systems commonly generate multiple views of the same recording and optimize agreement between pooled representations. Such objectives encourage invariance to augmentations, cropping, and masking, but the supervisory signal is applied after temporal and frequency information has been aggregated. A model may therefore learn a useful scene-level embedding without learning sufficiently discriminative representations for brief or localized acoustic events.

AudioFM combines two complementary objectives:

1. **Global recording contrast**: two independently masked views of the same recording are required to produce similar pooled embeddings, while views from other recordings serve as negatives.
2. **Masked latent discrimination (MLD)**: a student predicts the representation of a masked time-frequency patch using visible acoustic context and identifies the corresponding full-view teacher target among candidates from other recordings.
3. **Temporal-neighbor supervision**: teacher targets from eligible neighboring temporal patches are included as weaker positives, encouraging local temporal consistency.

The local targets are representations produced by a teacher encoder from the corresponding unmasked spectrogram, rather than raw spectrogram patches. This provides semantically richer supervision than spectrogram-level reconstruction. The framework is therefore intended to bridge scene-level understanding and event-level discrimination without requiring downstream labels during pretraining.

The central methodological claim is that global contrast alone is insufficient for versatile audio representations. AudioFM’s local objective forces the encoder to model acoustic structure that is absent from the student’s masked input, while the global objective preserves information about the recording as a whole.

## 2. Audio representation and model architecture

Unless otherwise specified, audio is resampled to 16 kHz, converted to mono, represented as a 128-band log-Mel spectrogram with 1,024 time frames, and divided into non-overlapping $16\times16$ time-frequency patches. The patch grid is indexed by temporal and frequency coordinates, $(t,f)$. The default temporal patch step corresponds to approximately 160 ms.

The student encoder is a Vision Transformer applied to the spectrogram patch sequence. The default AudioFM model is a ViT-Base with 12 Transformer layers, hidden dimension $D=768$, 12 attention heads, and learnable two-dimensional positional embeddings. The teacher has the same general representation structure and is updated from the student by exponential moving average rather than direct backpropagation.

The teacher parameters obey

$$
\bar\theta \leftarrow m\bar\theta+(1-m)\theta,
$$

where the EMA momentum increases during pretraining from $m=0.990$ to $m=0.9995$. The teacher projection-head parameters are updated analogously. Teacher outputs use stop-gradient, so they act as stable targets for the student.

AudioFM maintains separate global and local branches. The global branch pools the encoder output and applies a shared global projection head producing 128-dimensional, $\ell_2$-normalized recording embeddings. For view $v$ of recording $i$, the global embedding is denoted $\mathbf z_i^v$.

The local branch removes the class token, restores visible patch features to their original grid positions, inserts a shared learnable mask token at masked positions, and applies a two-dimensional Transformer predictor. The predictor projects the 768-dimensional encoder features to a 384-dimensional bottleneck, adds two-dimensional sinusoidal positional encodings and a view-specific masking-branch embedding, applies full time-frequency self-attention, and projects back to dimension $D$.

A shared two-layer local projection head maps

$$
768\rightarrow512\rightarrow128,
$$

followed by $\ell_2$ normalization. The resulting local student predictions are denoted $\mathbf p_{i,t,f}^v$, while the corresponding teacher targets are $\mathbf q_{i,t,f}^v$. After pretraining, only the student encoder is retained; the predictor, pretraining projection heads, and EMA teacher are discarded.

## 3. Masked views and local latent discrimination

For each recording, two independently augmented waveform versions are converted into log-Mel spectrograms and divided into the same patch grid. Complementary structured masks are then applied:

- **Temporal mask**: one view masks temporal slices across all frequency bands.
- **Frequency mask**: the other view masks frequency-patch bands across all time positions.

The default masking ratios are 60% of temporal slices in one view and 40% of frequency-patch bands in the other. The student processes the masked spectrogram, whereas the teacher processes the corresponding full, unmasked spectrogram.

For a masked position $(t,f)$, the student prediction $\mathbf p_{i,t,f}^v$ is contrasted with teacher targets at the same coordinate from every recording in the minibatch. The positive is the target from the same recording, and targets from other recordings are negatives:

$$
\ell_i^v(\mathbf p_{i,t,f}^v;(t,f))
=
-\log
\frac{
\exp\left((\mathbf p_{i,t,f}^v)^\top\mathbf q_{i,t,f}^v/\tau_p\right)
}{
\displaystyle
\sum_{j=1}^{B}
\exp\left((\mathbf p_{i,t,f}^v)^\top\mathbf q_{j,t,f}^v/\tau_p\right)
},
$$

with local temperature $\tau_p=0.2$.

A defining feature is that negatives are selected from other recordings at the same grid coordinate. Teacher targets from other positions within the same recording are not treated as negatives. This avoids explicitly forcing related moments within one recording to become dissimilar solely because they have different temporal or frequency coordinates.

The student must therefore infer the missing local representation from visible acoustic context well enough to distinguish the correct full-view target from targets belonging to other recordings. The objective is discriminative rather than reconstructive: the system does not attempt to reproduce the missing spectrogram values.

### Temporal-neighbor supervision

AudioFM extends same-position discrimination with weaker positives from nearby temporal patches. For temporal radius $K$, the neighborhood is

$$
\mathcal T=\{-K,\ldots,0,\ldots,K\}.
$$

The default is $K=1$, yielding the center position and its immediate temporal neighbors. With a 160-ms patch step, the side neighbors are approximately $\pm160$ ms from the anchor.

A neighbor is eligible only if it lies within the patch grid and is also masked in the student view. The latter condition prevents the predictor from copying a visible neighboring feature. For each eligible offset, the same student prediction is contrasted with the teacher target at the shifted coordinate. The default prior weights are

$$
(w_{-1},w_0,w_{+1})=(0.1,0.8,0.1).
$$

These weights are renormalized over valid offsets. The temporal-neighbor objective is thus a weighted set of cross-recording InfoNCE losses, not an explicit squared-distance smoothness penalty between adjacent embeddings. The center target remains dominant, while nearby targets provide weaker local temporal supervision.

## 4. Global contrastive learning and complete objective

The global branch uses symmetric NT-Xent. For a global embedding $\mathbf z_i^v$, the positive is the other masked view of the same recording, $\mathbf z_i^{3-v}$. All views from other recordings are negatives. The global temperature is $\tau_g=0.1$.

For normalized embeddings, similarity is cosine similarity:

$$
\operatorname{sim}(\mathbf a,\mathbf b)=\mathbf a^\top\mathbf b.
$$

The symmetric global loss averages the contrastive loss over both views and all recordings. The complete AudioFM objective is

$$
\mathcal L
=
\mathcal L_{\mathrm{global}}
+
\lambda_{\mathrm{local}}\mathcal L_{\mathrm{MLD}},
$$

with default coefficient

$$
\lambda_{\mathrm{local}}=1.0.
$$

The student encoder, local predictor, global projection head, and local projection head are optimized using this objective. The teacher is updated only through EMA.

This division of labor gives the framework two representational targets. Global contrast encourages invariance to complementary masking patterns and captures recording-level acoustic identity. MLD retains coordinate-specific information and requires inference of missing local content. Temporal-neighbor supervision extends local discrimination without imposing uniformity over acoustically unrelated distant patches.

The framework therefore produces an encoder that can be pooled for recording-level classification or retained as a time-indexed sequence for event detection. This dual use is a central architectural consequence of combining global and local supervision.

## 5. Pretraining data, masking, and optimization

AudioFM is pretrained on an available version of AudioSet containing approximately 1.68 million valid clips, corresponding to about 4,665 hours. The corpus is smaller than the approximately 1.91–2.0 million clips commonly used in earlier AudioSet SSL work because of YouTube attrition.

Two independently augmented versions of each recording are generated. The pretraining augmentation uses random temporal starts with cyclic rolling. Downstream augmentation configurations are distinct: fine-tuning may use SpecAugment, mixup, and additive noise for speech-command tasks, whereas linear probing uses no data augmentation.

The default masking configuration is 60% temporal masking and 40% frequency masking. Sensitivity experiments report that reducing temporal masking harms performance, while increasing temporal masking above 60% produces nearly unchanged performance over the tested range. Both alternatives to 40% frequency masking reduce performance, with the higher frequency-masking ratio producing the larger decline.

Pretraining uses AdamW with $\beta_1=0.9$, $\beta_2=0.999$, weight decay $0.05$, peak learning rate $6\times10^{-4}$, minimum learning rate $10^{-6}$, linear warm-up for 2,500 steps, half-cycle cosine decay, and gradient clipping at 1.0. Training runs for 125,000 steps with batch size 6,144 on a Cloud TPU v4-128 slice using 64 chips. The paper evaluates Small, Base, and Large model variants and reports consistent gains with increasing capacity.

## 6. Evaluation, results, and transfer behavior

AudioFM is evaluated through fine-tuning, frozen linear probing, frame-level event detection, and zero-shot transfer. Fine-tuning is performed on AudioSet-2M, AudioSet-20K, FSD50K, ESC-50, and Speech Commands V1 and V2. Multilabel tasks use mAP, while ESC-50 and Speech Commands use accuracy.

The principal fine-tuning results are:

| Task | Metric | AudioFM |
|---|---:|---:|
| AudioSet-20K | mAP | 42.3 |
| AudioSet-2M | mAP | 50.2 |
| FSD50K | mAP | 71.4 |
| ESC-50 | Accuracy | 98.2 |
| Speech Commands V2 | Accuracy | 98.5 |
| Speech Commands V1 | Accuracy | 99.1 |

AudioFM obtains 42.3 mAP on AudioSet-20K versus 41.8 for AudioMosaic, and 71.4 mAP on FSD50K versus 70.5 for AudioMosaic. On a shared AudioSet-2M evaluation subset, it exceeds the re-evaluated AudioMosaic checkpoint by approximately 1.0 mAP and matches the highest published AudioSet-2M result reported in the comparison table. These results use approximately 13% fewer pretraining and fine-tuning clips than the larger conventional AudioSet setups.

### Frozen linear probing

The frozen encoder is evaluated with a classifier trained on top of its representations. Three random seeds are used, and two variants are reported: a bare linear classifier and LayerNorm followed by a linear classifier.

| Dataset | Metric | Bare | LayerNorm |
|---|---|---:|---:|
| AudioSet-20K | mAP | 31.2 | 35.0 |
| FSD50K | mAP | 55.3 | 58.6 |
| ESC-50 | Accuracy | 94.5 | 94.7 |
| BEANS-Watkins | Accuracy | 89.1 | 85.9 |
| BEANS-Dogs | Accuracy | 85.9 | 83.5 |
| BirdSet-HSN | mAP | 11.7 | 11.9 |
| PSG-Audio binary | Balanced accuracy | 79.1 | 78.1 |
| PSG-Audio subtype | Macro-F1 | 37.9 | 33.1 |
| CirCor Murmur | Macro-F1 | 72.7 | 68.9 |
| CREMA-D | Macro-F1 | 68.7 | 64.9 |
| RAVDESS | Macro-F1 | 58.9 | 56.8 |
| TESS | Macro-F1 | 99.8 | 99.5 |

Under strict bare probing, AudioFM achieves the best or tied-best result on 10 of 12 tasks. It leads all three general-audio tasks under both probing conditions and all three medical tasks under bare probing. Reported improvements over the strongest baseline include +3.2 macro-F1 on PSG-Audio subtype classification and +4.7 macro-F1 on CirCor Murmur.

These medical results indicate transfer of information useful for the evaluated acoustic classification tasks, but they do not establish clinical validity.

### Frame-level event detection

AudioSet Strong evaluates whether frozen representations retain temporal information. A linear classifier operates on frozen time-indexed representations, with frame mAP, PSDS1, and PSDS2 as metrics. AudioFM outperforms the evaluated baselines on all three metrics. Exact numerical values are presented in the referenced figure rather than in the supplied textual results.

This evaluation is particularly relevant to AudioFM’s local objective: the encoder must preserve temporal detail rather than only recording-level information.

### Zero-shot transfer

For RealDESED, an AudioSet-trained event-tagging predictor is transferred directly to a new domestic soundscape dataset. AudioFM achieves 79.9% mAP and outperforms the evaluated baselines. The encoder also ranks first on three of four reported emotion-transfer directions and second on the remaining direction.

The paper presents t-SNE visualizations suggesting that AudioFM embeddings separate normal breathing from apnea and organize apnea subtypes more clearly than AudioMosaic. This is qualitative evidence of representational organization, not a clinical performance claim.

### Ablation findings

Adding MLD improves AudioSet-20K frozen probing over global-only training for all tested positive values of $\lambda_{\mathrm{local}}$, with $\lambda_{\mathrm{local}}=1.0$ performing best among the tested configurations. The default $K=1$ temporal neighborhood outperforms center-only supervision on AudioSet-20K classification, AudioSet Strong frame mAP, PSDS1, and PSDS2, whereas increasing the radius to $K=2$ reduces performance.

Removing the transformation layers reduces AudioCaps from 0.58 to 0.55, GTZAN from 69.1 to 64.3, MuchoMusic from 56.5 to 51.8, and MMAU from 69.0 to 59.0 in the reported downstream comparison. Removing RoPE reduces AudioCaps from 0.58 to 0.56, GTZAN from 69.1 to 67.2, MMAU from 69.0 to 59.1, and LongAudioBench from 6.4 to 4.3.

The ablations support the role of local prediction, short-range temporal supervision, structured masking, positional information, and the transformation modules. They do not isolate every architectural or augmentation choice through a complete factorial study.

## 7. Limitations and research significance

AudioFM’s pretraining and scaling experiments focus on AudioSet. Although AudioSet contains diverse audio, this does not establish uniform performance across all acoustic domains. The framework also converts recordings to mono, discarding spatial and multichannel cues.

Robustness to controlled microphone, codec, reverberation, channel, and device shifts is not systematically evaluated. Specialized-domain coverage remains limited despite the inclusion of bioacoustic, medical, and speech-emotion tasks. The reported medical transfer results should not be interpreted as evidence of clinical deployment readiness.

Temporal-neighbor supervision assumes that nearby patches may contain related acoustic information. This assumption is less reliable at abrupt event boundaries, where adjacent patches may represent different events. The mask-aware eligibility rule and lower side-neighbor weights reduce this risk but do not remove it.

Same-coordinate teacher targets from different recordings are not semantically aligned. The method relies on contrastive discrimination rather than an assumption that coordinates have fixed acoustic meaning. This candidate construction may therefore introduce noisy negatives when recordings are highly heterogeneous.

The evaluation does not provide a systematic study of multichannel audio, extreme channel distortion, broader clinical and physiological tasks, or arbitrary domain shifts. The available results support the narrower conclusion that joint global-local self-supervision improves the versatility of a frozen audio encoder relative to recording-level contrast alone.

AudioFM’s principal significance is the integration of three forms of supervision: recording-level view invariance, local teacher-target discrimination, and short-range temporal consistency. Its reported performance across fine-tuning, frozen probing, frame-level detection, and zero-shot transfer indicates that a single encoder can retain both holistic acoustic context and localized temporal-frequency information. The framework thus represents a general design pattern for audio SSL in which global invariance and local predictive discrimination are optimized jointly rather than treated as competing objectives.

Source: https://www.emergentmind.com/topics/audiofm