AudioMosaic: Contrastive Masked Audio Representation Learning

Abstract: Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data. While recent advances have been driven mainly by generative reconstruction objectives, contrastive approaches remain less explored, partly due to the difficulty of designing effective audio augmentations and the large batch sizes required for contrastive pre-training. We introduce \textbf{AudioMosaic}, a contrastive learning-based audio encoder for general audio understanding. During pre-training, AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches, which reduces memory usage and enables efficient large-batch training. Compared with generative approaches, the AudioMosaic encoder learns more discriminative utterance-level representations that demonstrate strong transferability across datasets, domains, and acoustic conditions. Extensive experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks under both linear probing and fine-tuning. We further show that integrating the pretrained AudioMosaic encoder into audio-LLMs improves performance on audio-language tasks. The code is publicly available in our \href{https://github.com/HanxunH/AudioMosaic}{GitHub repository}.

• The paper introduces AudioMosaic, a novel contrastive self-supervised pre-training framework leveraging structured time–frequency masking for enhanced audio representations.
• It achieves state-of-the-art performance on benchmarks like AudioSet-20K and ESC-50, with robust transfer learning capabilities validated by deep linear probing.
• Its efficient design reduces memory usage and integrates smoothly with multimodal language models to improve acoustic perception in audio–language tasks.

AudioMosaic: Contrastive Masked Audio Representation Learning

Introduction

The paper "AudioMosaic: Contrastive Masked Audio Representation Learning" (2605.14231) introduces AudioMosaic, a contrastive learning framework for self-supervised pre-training of general-purpose audio encoders. Unlike established generative approaches relying on masked reconstruction of spectrograms, AudioMosaic frames masking as a mechanism for constructing informative and complementary positive pairs in a contrastive setting. This shift leverages structured, independent time–frequency masking of spectrogram patches, improving representation discriminability, transferability, and computational efficiency.

Figure 1: Overview of AudioMosaic and its contrastive time–frequency masking approach versus traditional unstructured masking.

Methodology

Structured Time–Frequency Masking

AudioMosaic constructs positive pairs by applying independent structured masking along the time and frequency dimensions to spectrogram patch tokens derived from augmented audio waveforms. This process creates two complementary masked views that expose the encoder to broader utterance-level invariants, rather than trivial or overly local co-occurrences. Only visible (unmasked) patches are encoded, and the order of tokens is randomized to augment invariance. The resultant sequences are processed by a shared Transformer encoder with a contrastive loss, using a lightweight MLP projection head.

Key distinctions of this approach include:

• Contrastive vs. Generative Pre-training: Masking is designed to intentionally reduce view redundancy in the contrastive framework, avoiding dimension collapse and improving utilization of the representation space.
• Efficiency: By omitting masked patches at the encoder input, inference and memory costs are reduced; high masking ratios further compress the sequence, enabling the use of large batch sizes necessary for robust contrastive learning.
• Domain-tailored Augmentation: Unlike image or waveform-based methods, time–frequency masking leverages unique spectrotemporal structures inherent to audio spectrograms.

Analysis of Representation Quality

The effective rank of the encoder output distribution is utilized as a metric to diagnose the intrinsic dimensionality and richness of learned embeddings. Higher effective rank correlates with more expressive, less degenerate representations, which are empirically associated with superior downstream performance.

Figure 2: Effective rank analysis reveals that AudioMosaic's time–frequency masking yields higher-rank representations than unstructured masking, indicating richer latent space usage.

AudioMosaic's structured time–frequency masking consistently produces higher effective rank than both unstructured contrastive and generative masking (e.g., Audio-MAE). This demonstrates mitigation of dimensional collapse and improved global information encoding in the latent space.

Experimental Evaluation

Fine-tuning and Transfer Performance

AudioMosaic achieves state-of-the-art results on a suite of standard benchmarks, including AudioSet-20K (42.5 mAP), ESC-50 (97.5% accuracy), and robust performance on Speech Commands and environmental sound deepfake detection. Notably, these gains generalize across domains and tasks, indicating successful capture of high-level, non-trivial audio invariants.

Linear probing experiments further substantiate these findings: AudioMosaic significantly surpasses all tested masked modeling baselines under frozen encoder evaluation, an especially notable result since prior methods sometimes show improvements under fine-tuning but degrade in linear probe regimes. This underscores the practical advantage of the learned representations for transfer learning.

Figure 3: Layer-wise linear probe performance demonstrates that AudioMosaic's deeper layers maintain robust semantics, while masked modeling baselines peak in mid-layers.

Layer-wise probing reveals that, unlike competitive methods whose final layers show degradation, AudioMosaic's encoder continues to improve representational strength up to the top layers, with aggregation strategies (weighted sum or attention) providing further gains.

Ablations and Efficiency

Ablation studies conclusively show that:

• Positive pair formation using both time and frequency masking produces superior results to masking either axis alone or using unstructured masking. The best result is achieved with a masking ratio of 0.6 (time) and 0.4 (frequency).
• Larger batch sizes consistently improve downstream performance, enabled by the framework's efficient masking.
• Memory consumption scales linearly and stays low, outperforming methods such as EAT, which imposes prohibitive GPU requirements.

Figure 4: Comparison of masking strategies shows that time–frequency masking for positive pair construction is critical for contrastive audio representation learning.

Effectiveness for Deepfake Detection and Audio-LLMs

AudioMosaic excels in environmental sound deepfake detection, outperforming prior art by a substantial margin in equal error rate, both with linear and more complex heads. This demonstrates the representations' robustness to out-of-distribution generative artifacts.

Integrating the pretrained AudioMosaic encoder as the audio front-end of LLMs (e.g., LLaMA-7B) systematically boosts the performance of audio–language foundation models across both classification and captioning tasks, including zero-shot regimes. Qualitative analysis reveals the encoder enables finer-grained acoustic perception, which is critical for long-context, open-domain audio–LLM applications.

Implications and Future Directions

This work demonstrates that reimagining masking as a view generation mechanism for contrastive learning, utilizing structured time–frequency ablations, yields audio representations with improved rank, discriminability, and universality. The results challenge the primacy of generative masked modeling for general-purpose audio pre-training and suggest a practical alternative for efficient, scalable, and transferable auditory representation learning.

Practically, these findings lower the pre-training memory barrier, enable larger contrastive batch regimes, and align well with modern trends towards multimodal and audio–language foundation models. Theoretically, the approach highlights the importance of positive pair correlation structure, intrinsic dimensionality, and spectral domain invariances for robust representation learning in audio.

Ongoing directions include extending view-generation paradigms (including more sophisticated structured masks), fine-grained alignment between audio and other modalities, and the development of more precise diagnostics for degeneracy and overfitting in high-dimensional SSL models.

Conclusion

AudioMosaic establishes a contrastive masked spectrogram learning approach wherein the construction of positive pairs via structured time–frequency masking leads to superior audio representations, validated by effective rank analysis and strong downstream results. The architecture is both memory and data efficient, generalizes robustly, and integrates synergistically with multimodal LLMs, providing a compelling framework for future general-purpose audio understanding and audio-centric AI systems.