Papers
Topics
Authors
Recent
Search
2000 character limit reached

AudioFM Framework for Audio Representation Learning

Updated 28 October 2026
  • AudioFM is a self-supervised learning framework designed to preserve both holistic recording-level audio information and fine-grained local details using contrastive learning, masked latent discrimination, and temporal-neighbor supervision.
  • The AudioFM model employs recording-level contrastive learning which emphasizes global scene embeddings while masked latent discrimination forces the encoder to model acoustic structures based on visible context.
  • Temporal-neighbor supervision provides additional local consistency ensuring semantically richer embeddings which contribute to robust performance in diverse downstream tasks.

AudioFM is a self-supervised framework for general-purpose audio representation learning introduced in “AudioFM: Towards General Purpose Audio Representation Learning” (Li et al., 25 Oct 2026). It is designed to preserve both holistic recording-level information—such as acoustic scenes and global soundscapes—and fine-grained local information, including brief events, temporal transitions, and frequency-specific structures. AudioFM combines recording-level contrastive learning with patch-wise masked latent discrimination and weaker supervision from eligible temporal neighbors. The resulting encoder is evaluated through linear probing, fine-tuning, frame-level event detection, and zero-shot transfer across environmental, bioacoustic, medical, speech-emotion, and general audio domains.

1. Conceptual foundations and objectives

AudioFM addresses a limitation of recording-level self-supervised learning. Conventional audio SSL systems commonly generate multiple views of the same recording and optimize agreement between pooled representations. Such objectives encourage invariance to augmentations, cropping, and masking, but the supervisory signal is applied after temporal and frequency information has been aggregated. A model may therefore learn a useful scene-level embedding without learning sufficiently discriminative representations for brief or localized acoustic events.

AudioFM combines two complementary objectives:

  1. Global recording contrast: two independently masked views of the same recording are required to produce similar pooled embeddings, while views from other recordings serve as negatives.
  2. Masked latent discrimination (MLD): a student predicts the representation of a masked time-frequency patch using visible acoustic context and identifies the corresponding full-view teacher target among candidates from other recordings.
  3. Temporal-neighbor supervision: teacher targets from eligible neighboring temporal patches are included as weaker positives, encouraging local temporal consistency.

The local targets are representations produced by a teacher encoder from the corresponding unmasked spectrogram, rather than raw spectrogram patches. This provides semantically richer supervision than spectrogram-level reconstruction. The framework is therefore intended to bridge scene-level understanding and event-level discrimination without requiring downstream labels during pretraining.

The central methodological claim is that global contrast alone is insufficient for versatile audio representations. AudioFM’s local objective forces the encoder to model acoustic structure that is absent from the student’s masked input, while the global objective preserves information about the recording as a whole.

2. Audio representation and model architecture

Unless otherwise specified, audio is resampled to 16 kHz, converted to mono, represented as a 128-band log-Mel spectrogram with 1,024 time frames, and divided into non-overlapping 16×1616\times16 time-frequency patches. The patch grid is indexed by temporal and frequency coordinates, (t,f)(t,f). The default temporal patch step corresponds to approximately 160 ms.

The student encoder is a Vision Transformer applied to the spectrogram patch sequence. The default AudioFM model is a ViT-Base with 12 Transformer layers, hidden dimension D=768D=768, 12 attention heads, and learnable two-dimensional positional embeddings. The teacher has the same general representation structure and is updated from the student by exponential moving average rather than direct backpropagation.

The teacher parameters obey

θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,

where the EMA momentum increases during pretraining from m=0.990m=0.990 to m=0.9995m=0.9995. The teacher projection-head parameters are updated analogously. Teacher outputs use stop-gradient, so they act as stable targets for the student.

AudioFM maintains separate global and local branches. The global branch pools the encoder output and applies a shared global projection head producing 128-dimensional, ℓ2\ell_2-normalized recording embeddings. For view vv of recording ii, the global embedding is denoted ziv\mathbf z_i^v.

The local branch removes the class token, restores visible patch features to their original grid positions, inserts a shared learnable mask token at masked positions, and applies a two-dimensional Transformer predictor. The predictor projects the 768-dimensional encoder features to a 384-dimensional bottleneck, adds two-dimensional sinusoidal positional encodings and a view-specific masking-branch embedding, applies full time-frequency self-attention, and projects back to dimension (t,f)(t,f)0.

A shared two-layer local projection head maps

(t,f)(t,f)1

followed by (t,f)(t,f)2 normalization. The resulting local student predictions are denoted (t,f)(t,f)3, while the corresponding teacher targets are (t,f)(t,f)4. After pretraining, only the student encoder is retained; the predictor, pretraining projection heads, and EMA teacher are discarded.

3. Masked views and local latent discrimination

For each recording, two independently augmented waveform versions are converted into log-Mel spectrograms and divided into the same patch grid. Complementary structured masks are then applied:

  • Temporal mask: one view masks temporal slices across all frequency bands.
  • Frequency mask: the other view masks frequency-patch bands across all time positions.

The default masking ratios are 60% of temporal slices in one view and 40% of frequency-patch bands in the other. The student processes the masked spectrogram, whereas the teacher processes the corresponding full, unmasked spectrogram.

For a masked position (t,f)(t,f)5, the student prediction (t,f)(t,f)6 is contrasted with teacher targets at the same coordinate from every recording in the minibatch. The positive is the target from the same recording, and targets from other recordings are negatives:

(t,f)(t,f)7

with local temperature (t,f)(t,f)8.

A defining feature is that negatives are selected from other recordings at the same grid coordinate. Teacher targets from other positions within the same recording are not treated as negatives. This avoids explicitly forcing related moments within one recording to become dissimilar solely because they have different temporal or frequency coordinates.

The student must therefore infer the missing local representation from visible acoustic context well enough to distinguish the correct full-view target from targets belonging to other recordings. The objective is discriminative rather than reconstructive: the system does not attempt to reproduce the missing spectrogram values.

Temporal-neighbor supervision

AudioFM extends same-position discrimination with weaker positives from nearby temporal patches. For temporal radius (t,f)(t,f)9, the neighborhood is

D=768D=7680

The default is D=768D=7681, yielding the center position and its immediate temporal neighbors. With a 160-ms patch step, the side neighbors are approximately D=768D=7682 ms from the anchor.

A neighbor is eligible only if it lies within the patch grid and is also masked in the student view. The latter condition prevents the predictor from copying a visible neighboring feature. For each eligible offset, the same student prediction is contrasted with the teacher target at the shifted coordinate. The default prior weights are

D=768D=7683

These weights are renormalized over valid offsets. The temporal-neighbor objective is thus a weighted set of cross-recording InfoNCE losses, not an explicit squared-distance smoothness penalty between adjacent embeddings. The center target remains dominant, while nearby targets provide weaker local temporal supervision.

4. Global contrastive learning and complete objective

The global branch uses symmetric NT-Xent. For a global embedding D=768D=7684, the positive is the other masked view of the same recording, D=768D=7685. All views from other recordings are negatives. The global temperature is D=768D=7686.

For normalized embeddings, similarity is cosine similarity:

D=768D=7687

The symmetric global loss averages the contrastive loss over both views and all recordings. The complete AudioFM objective is

D=768D=7688

with default coefficient

D=768D=7689

The student encoder, local predictor, global projection head, and local projection head are optimized using this objective. The teacher is updated only through EMA.

This division of labor gives the framework two representational targets. Global contrast encourages invariance to complementary masking patterns and captures recording-level acoustic identity. MLD retains coordinate-specific information and requires inference of missing local content. Temporal-neighbor supervision extends local discrimination without imposing uniformity over acoustically unrelated distant patches.

The framework therefore produces an encoder that can be pooled for recording-level classification or retained as a time-indexed sequence for event detection. This dual use is a central architectural consequence of combining global and local supervision.

5. Pretraining data, masking, and optimization

AudioFM is pretrained on an available version of AudioSet containing approximately 1.68 million valid clips, corresponding to about 4,665 hours. The corpus is smaller than the approximately 1.91–2.0 million clips commonly used in earlier AudioSet SSL work because of YouTube attrition.

Two independently augmented versions of each recording are generated. The pretraining augmentation uses random temporal starts with cyclic rolling. Downstream augmentation configurations are distinct: fine-tuning may use SpecAugment, mixup, and additive noise for speech-command tasks, whereas linear probing uses no data augmentation.

The default masking configuration is 60% temporal masking and 40% frequency masking. Sensitivity experiments report that reducing temporal masking harms performance, while increasing temporal masking above 60% produces nearly unchanged performance over the tested range. Both alternatives to 40% frequency masking reduce performance, with the higher frequency-masking ratio producing the larger decline.

Pretraining uses AdamW with θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,0, θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,1, weight decay θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,2, peak learning rate θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,3, minimum learning rate θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,4, linear warm-up for 2,500 steps, half-cycle cosine decay, and gradient clipping at 1.0. Training runs for 125,000 steps with batch size 6,144 on a Cloud TPU v4-128 slice using 64 chips. The paper evaluates Small, Base, and Large model variants and reports consistent gains with increasing capacity.

6. Evaluation, results, and transfer behavior

AudioFM is evaluated through fine-tuning, frozen linear probing, frame-level event detection, and zero-shot transfer. Fine-tuning is performed on AudioSet-2M, AudioSet-20K, FSD50K, ESC-50, and Speech Commands V1 and V2. Multilabel tasks use mAP, while ESC-50 and Speech Commands use accuracy.

The principal fine-tuning results are:

Task Metric AudioFM
AudioSet-20K mAP 42.3
AudioSet-2M mAP 50.2
FSD50K mAP 71.4
ESC-50 Accuracy 98.2
Speech Commands V2 Accuracy 98.5
Speech Commands V1 Accuracy 99.1

AudioFM obtains 42.3 mAP on AudioSet-20K versus 41.8 for AudioMosaic, and 71.4 mAP on FSD50K versus 70.5 for AudioMosaic. On a shared AudioSet-2M evaluation subset, it exceeds the re-evaluated AudioMosaic checkpoint by approximately 1.0 mAP and matches the highest published AudioSet-2M result reported in the comparison table. These results use approximately 13% fewer pretraining and fine-tuning clips than the larger conventional AudioSet setups.

Frozen linear probing

The frozen encoder is evaluated with a classifier trained on top of its representations. Three random seeds are used, and two variants are reported: a bare linear classifier and LayerNorm followed by a linear classifier.

Dataset Metric Bare LayerNorm
AudioSet-20K mAP 31.2 35.0
FSD50K mAP 55.3 58.6
ESC-50 Accuracy 94.5 94.7
BEANS-Watkins Accuracy 89.1 85.9
BEANS-Dogs Accuracy 85.9 83.5
BirdSet-HSN mAP 11.7 11.9
PSG-Audio binary Balanced accuracy 79.1 78.1
PSG-Audio subtype Macro-F1 37.9 33.1
CirCor Murmur Macro-F1 72.7 68.9
CREMA-D Macro-F1 68.7 64.9
RAVDESS Macro-F1 58.9 56.8
TESS Macro-F1 99.8 99.5

Under strict bare probing, AudioFM achieves the best or tied-best result on 10 of 12 tasks. It leads all three general-audio tasks under both probing conditions and all three medical tasks under bare probing. Reported improvements over the strongest baseline include +3.2 macro-F1 on PSG-Audio subtype classification and +4.7 macro-F1 on CirCor Murmur.

These medical results indicate transfer of information useful for the evaluated acoustic classification tasks, but they do not establish clinical validity.

Frame-level event detection

AudioSet Strong evaluates whether frozen representations retain temporal information. A linear classifier operates on frozen time-indexed representations, with frame mAP, PSDS1, and PSDS2 as metrics. AudioFM outperforms the evaluated baselines on all three metrics. Exact numerical values are presented in the referenced figure rather than in the supplied textual results.

This evaluation is particularly relevant to AudioFM’s local objective: the encoder must preserve temporal detail rather than only recording-level information.

Zero-shot transfer

For RealDESED, an AudioSet-trained event-tagging predictor is transferred directly to a new domestic soundscape dataset. AudioFM achieves 79.9% mAP and outperforms the evaluated baselines. The encoder also ranks first on three of four reported emotion-transfer directions and second on the remaining direction.

The paper presents t-SNE visualizations suggesting that AudioFM embeddings separate normal breathing from apnea and organize apnea subtypes more clearly than AudioMosaic. This is qualitative evidence of representational organization, not a clinical performance claim.

Ablation findings

Adding MLD improves AudioSet-20K frozen probing over global-only training for all tested positive values of θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,5, with θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,6 performing best among the tested configurations. The default θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,7 temporal neighborhood outperforms center-only supervision on AudioSet-20K classification, AudioSet Strong frame mAP, PSDS1, and PSDS2, whereas increasing the radius to θˉ←mθˉ+(1−m)θ,\bar\theta \leftarrow m\bar\theta+(1-m)\theta,8 reduces performance.

Removing the transformation layers reduces AudioCaps from 0.58 to 0.55, GTZAN from 69.1 to 64.3, MuchoMusic from 56.5 to 51.8, and MMAU from 69.0 to 59.0 in the reported downstream comparison. Removing RoPE reduces AudioCaps from 0.58 to 0.56, GTZAN from 69.1 to 67.2, MMAU from 69.0 to 59.1, and LongAudioBench from 6.4 to 4.3.

The ablations support the role of local prediction, short-range temporal supervision, structured masking, positional information, and the transformation modules. They do not isolate every architectural or augmentation choice through a complete factorial study.

7. Limitations and research significance

AudioFM’s pretraining and scaling experiments focus on AudioSet. Although AudioSet contains diverse audio, this does not establish uniform performance across all acoustic domains. The framework also converts recordings to mono, discarding spatial and multichannel cues.

Robustness to controlled microphone, codec, reverberation, channel, and device shifts is not systematically evaluated. Specialized-domain coverage remains limited despite the inclusion of bioacoustic, medical, and speech-emotion tasks. The reported medical transfer results should not be interpreted as evidence of clinical deployment readiness.

Temporal-neighbor supervision assumes that nearby patches may contain related acoustic information. This assumption is less reliable at abrupt event boundaries, where adjacent patches may represent different events. The mask-aware eligibility rule and lower side-neighbor weights reduce this risk but do not remove it.

Same-coordinate teacher targets from different recordings are not semantically aligned. The method relies on contrastive discrimination rather than an assumption that coordinates have fixed acoustic meaning. This candidate construction may therefore introduce noisy negatives when recordings are highly heterogeneous.

The evaluation does not provide a systematic study of multichannel audio, extreme channel distortion, broader clinical and physiological tasks, or arbitrary domain shifts. The available results support the narrower conclusion that joint global-local self-supervision improves the versatility of a frozen audio encoder relative to recording-level contrast alone.

AudioFM’s principal significance is the integration of three forms of supervision: recording-level view invariance, local teacher-target discrimination, and short-range temporal consistency. Its reported performance across fine-tuning, frozen probing, frame-level detection, and zero-shot transfer indicates that a single encoder can retain both holistic acoustic context and localized temporal-frequency information. The framework thus represents a general design pattern for audio SSL in which global invariance and local predictive discrimination are optimized jointly rather than treated as competing objectives.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to AudioFM.