AudioMosaic: Contrastive Audio SSL

• The paper introduces AudioMosaic as a contrastive masked spectrogram framework that reframes masking into a view-construction mechanism for discriminative audio representation.
• It leverages structured time-frequency masking to drop tokens before encoding, reducing computation and improving large-batch efficiency with a shared Transformer backbone.
• Empirical evaluations demonstrate high transferability and linear separability, with significant gains in mAP, robustness in deepfake detection, and improvements in audio-language tasks.

Searching arXiv for the specified AudioMosaic paper and closely related entries. AudioMosaic is a contrastive masked spectrogram learning framework for audio self-supervised learning that targets general-purpose utterance-level audio representations for transfer across datasets, domains, and acoustic conditions (Huang et al., 14 May 2026). Its central design move is to use masking not for reconstruction but for view construction: two complementary spectrogram views of the same utterance are created by structured masking along time and frequency, encoded with a shared Transformer, and optimized with a symmetric contrastive objective. In this formulation, masked patches are omitted before encoding, so the method couples a discriminative SSL objective with reduced token count and large-batch efficiency.

1. Conceptual position and problem setting

AudioMosaic is situated against the dominant masked-reconstruction line in audio SSL. The paper identifies recent reconstruction-oriented methods such as Audio-MAE, BEATs, EAT, and SSLAM as effective but biased toward local time-frequency correlations, because neighboring patches often suffice to infer missing content (Huang et al., 14 May 2026). That inductive bias is useful for reconstruction, but the stated target of AudioMosaic is different: discriminative utterance-level representations that remain transferable under dataset shift and varying acoustic conditions.

The paper argues that contrastive learning is a better fit for this target, but that audio contrastive pre-training has been constrained by three issues: audio augmentation design is difficult and domain-specific; naive positive-pair construction on spectrograms preserves too much local redundancy; and contrastive learning benefits from large batch sizes, which are computationally expensive. AudioMosaic addresses all three by constructing positive pairs through structured time-frequency masking and by dropping masked patches before the encoder, thereby reducing memory use and making large-batch contrastive training practical (Huang et al., 14 May 2026).

A key conceptual distinction follows from this design. In reconstruction-based masked modeling, masking corrupts an input so that the model predicts or reconstructs the hidden content. In AudioMosaic, masking defines two nontrivial but semantically matched views of the same utterance. This shifts the objective from local recovery to utterance-level invariance and discrimination.

2. Representation pipeline and masked-view construction

For pre-training, AudioMosaic uses unlabeled AudioSet audio. Inputs are resampled to mono at 16 kHz and converted into log-Mel spectrograms with 128 Mel bins, a 25 ms Hann window, and a 10 ms hop size (Huang et al., 14 May 2026). A 10-second clip yields a spectrogram of size

$1 \times 1024 \times 128.$

Before spectrogram formation, the paper applies waveform-level augmentations to produce two raw views of the same utterance. The appendix lists the following augmentation pipeline: polarity inversion with probability 0.5; time stretch with probability 0.7 and rate in $[0.7,1.25]$; Gaussian noise with probability 0.5 and SNR in $[5,40]$ dB; gain with probability 0.3 in $[-12,12]$ dB; high-pass filtering with probability 0.3 and cutoff in $[20,2400]$ Hz; band-stop filtering with probability 0.5 and center frequency in $[200,4000]$ Hz; and pitch shift with probability 0.6 in $[-4,4]$ semitones. The appendix ablation explicitly shows that masking alone is not sufficient (Huang et al., 14 May 2026).

Each spectrogram is partitioned into non-overlapping patches of size $p_t \times p_f$. In the instantiated model, AudioMosaic uses AST / ViT-B with $16 \times 16$ patches. Since a 10-second spectrogram is $1024 \times 128$, patching yields $[0.7,1.25]$0 patches along time and $[0.7,1.25]$1 along frequency, for a total of

$[0.7,1.25]$2

patch tokens.

Two augmented waveforms $[0.7,1.25]$3 and $[0.7,1.25]$4 are transformed into spectrogram patch sequences, after which masking is applied independently: one branch receives time masking and the other frequency masking. The masked sequences are

$[0.7,1.25]$5

The main setting is $[0.7,1.25]$6 and $[0.7,1.25]$7. The paper emphasizes that masked patches are omitted, not replaced by zeros or mask tokens for encoding (Huang et al., 14 May 2026).

Before encoding, 2D positional embeddings are added, and patch tokens are randomly shuffled. The encoder is a shared AST backbone instantiated as a 12-layer ViT-B/16 Transformer:

$[0.7,1.25]$8

A lightweight two-layer MLP projection head $[0.7,1.25]$9 maps the encoder outputs into the contrastive space:

$[5,40]$0

The head uses a linear layer to 512 hidden units, BatchNorm, ReLU, a linear layer to projection dimension, final bias-free BatchNorm, and projection dimension 128. The embeddings are $[5,40]$1-normalized. For downstream use, the projection head is discarded, and the paper uses the average-pooled output of the last encoder layer over the token sequence as the utterance representation in linear probing (Huang et al., 14 May 2026).

3. Contrastive objective and computational efficiency

AudioMosaic uses a symmetric batch contrastive loss with no reconstruction term, no BYOL-style target network, and no VICReg-style variance or covariance regularizer (Huang et al., 14 May 2026). For batch size $[5,40]$2, the objective is

$[5,40]$3

This is the paper’s bidirectional InfoNCE-style formulation: the first term predicts the matching frequency-masked view given the time-masked anchor, and the second predicts the matching time-masked view given the frequency-masked anchor.

The paper’s efficiency claim is inseparable from its masking design. Because masked patches are dropped before the Transformer, attention cost is reduced from

$[5,40]$4

to

$[5,40]$5

The paper notes that masking 50% of patches reduces quadratic attention computation by 75%, and further states that with the structured time-frequency masking used in AudioMosaic, each view is reduced to about 24% of the full token sequence, while still using two views per sample (Huang et al., 14 May 2026).

This leads to a low-overhead pre-training configuration. The main comparison table lists 86M pre-training parameters and 86M fine-tuning parameters for AudioMosaic, compared with 137M/86M for Audio-MAE, 182M/91M for BEATs, 93M/88M for EAT, and 93M/88M for SSLAM. Peak pre-training memory on a single NVIDIA L40S (48 GB), with ViT-Base and 10-second spectrograms, is reported as 3.3 GB, 6.3 GB, 12.3 GB, and 24.3 GB for batch sizes 64, 128, 256, and 512, respectively; the corresponding AudioMAE numbers are 3.7, 6.8, 13.0, and 25.4 GB (Huang et al., 14 May 2026).

The default pre-training configuration uses AdamW, learning rate 6e-4, weight decay 0.01, 400 epochs, batch size 6144, and 12 GPUs. The appendix further reports warmup of 1 epoch, half-cycle cosine decay, layer decay 1.0, drop path 0.1, dropout 0, no mixup during pre-training, and random start with cyclic rolling in time (Huang et al., 14 May 2026).

4. Empirical performance and representation analysis

AudioMosaic is pretrained on AudioSet without labels, using the unbalanced and balanced splits. The paper evaluates transfer on AS-20K, AS-2M, ESC-50, Speech Commands, EnvSDD, and a set of audio-language tasks following LTU (Huang et al., 14 May 2026).

The strongest headline results are the fine-tuning and linear-probing numbers. Under fine-tuning, AudioMosaic reports 42.5 mAP on AS-20K, 50.2 mAP on AS-2M, 97.5 accuracy on ESC-50, 98.4 on SPC-2, and 99.0 on SPC-1. The paper notes that AS-2M may be near saturation because AudioSet is both the pre-training and fine-tuning domain. Under linear probing, AudioMosaic reports 29.4 on AS-20K, 28.7 on AS-2M, and 93.0 on ESC-50, substantially above the listed reconstruction-oriented baselines (Huang et al., 14 May 2026).

Benchmark	Fine-tuning	Linear probing
AS-20K	42.5 mAP	29.4
AS-2M	50.2 mAP	28.7
ESC-50	97.5 Acc	93.0

The paper treats the linear-probe result as especially diagnostic. Its interpretation is that AudioMosaic embeddings are more linearly separable and more discriminative at the utterance level, whereas some strong fine-tuning methods depend more heavily on downstream adaptation. Layer-wise probing supports that reading: AudioMosaic improves monotonically with depth and peaks at layer 10: 30.2 mAP on AS-20K, with minimal final-layer degradation. Attentive aggregation across all layers yields 33.5 mAP, which is +3.3 over the best single layer (Huang et al., 14 May 2026).

To diagnose collapse, the paper analyzes effective rank. For a matrix $[5,40]$6 with singular values $[5,40]$7, it defines

$[5,40]$8

and

$[5,40]$9

Applied to a batch representation matrix $[-12,12]$0, higher effective rank indicates that representations span a richer subspace. The paper reports that contrastive training yields higher effective rank than Audio-MAE reconstruction, that time-frequency masking yields higher effective rank than unstructured masking, and that AudioMosaic achieves the highest effective rank overall (Huang et al., 14 May 2026).

The method also shows strong robustness on deepfake detection. On EnvSDD, AudioMosaic with a linear classifier reports average EER of 1.30 in the TTA setting and 0.02 in ATA; AudioMosaic combined with AASIST reports 1.41 and 0.003, respectively. The paper presents this as evidence of strong out-of-distribution generalization (Huang et al., 14 May 2026).

A separate transfer result concerns audio-language modeling. In the LTU setup, the paper replaces LTU’s original CAV-MAE audio encoder with the pretrained AudioMosaic encoder and aligns it with LLaMA-7B on OpenAQA, using 5.4M of the original 5.6M samples. The resulting system improves most reported tasks, including ESC-50 from 82.0 to 86.5, VocalSound from 55.7 to 68.2, VGGSound from 38.4 to 54.6, FSD50K from 45.8 to 46.9, AudioSet from 18.2 to 21.0, AudioCaps SPICE from 16.0 to 17.1, and Clotho SPICE from 12.0 to 12.5, with a small drop on DCASE from 50.5 to 48.9 (Huang et al., 14 May 2026).

The ablations reinforce the core design. Time-frequency masking is best; frequency-only and unstructured masking are worse; time-only is competitive at low mask ratios. The best mask-ratio setting is $[-12,12]$1, $[-12,12]$2. Larger batch sizes continue improving performance up to the default batch size of 6144. The augmentation ablation shows AS-20K fine-tuning mAP of 24.8 with time-frequency masking and no waveform augmentation, 34.2 after adding high-pass and band-stop filters, 42.5 with the full augmentation pipeline, and 30.1 with full augmentation but no masking (Huang et al., 14 May 2026).

5. Terminology, adjacent usages, and name collisions

The label “AudioMosaic” can be confused with several unrelated or only partially related arXiv entities. In the strict sense of the term used here, AudioMosaic denotes the contrastive masked audio representation-learning method introduced in 2026 (Huang et al., 14 May 2026).

A separate but conceptually adjacent system is AudioMorphix, a training-free audio editor built on frozen pretrained latent diffusion backbones such as AudioLDM and Tango (Liang et al., 21 May 2025). AudioMorphix performs localized spectrogram editing with binary masks, reference audio, inference-time latent manipulation, energy-based guidance, and self-attention cache reuse. It supports addition, removal, replacement, time shifting, time stretching, and pitch shifting, but it is not a representation-learning encoder and does not implement a multi-fragment audio mosaicing engine.

Another nearby term is MOSA, the dataset introduced in “MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processing” (Huang et al., 2024). MOSA is a cross-modal music dataset comprising 742 performances by 23 professional musicians, totaling 30.7 hours and more than 570K notes, with aligned audio, full-body 3-D motion capture, and note-by-note semantic annotations. It is a benchmark infrastructure for cross-modal MIR and generation rather than an audio SSL encoder.

A third distinct usage is MedMosaic, a medical audio question-answering benchmark with 46,701 question-answer pairs and 16,815 audio files, spanning sound-only physiological audio, speech-only clinical audio, speech+sound audio, long-form conversations, multi-turn QA, open-ended QA, and voice-based QA (Rajgarhia et al., 1 May 2026). MedMosaic uses “mosaic” in the sense of heterogeneous benchmark composition, not in the sense of contrastive masked spectrogram learning.

A practical implication is that “AudioMosaic” in current arXiv usage can refer either to a specific SSL method or, more loosely, to a broader family of mosaic-style dataset or editing concepts. The 2026 method is distinguished by its explicit combination of structured time-frequency masking, contrastive learning, and efficient large-batch pre-training (Huang et al., 14 May 2026).

6. Limitations, assumptions, and reproducibility considerations

The paper presents AudioMosaic as a spectrogram-based method rather than a raw-waveform architecture (Huang et al., 14 May 2026). It relies on careful waveform augmentation in addition to structured masking, and the appendix ablations show that masking alone is not enough. It also depends on large-batch training: the method is efficient enough to permit such batches, but its best reported performance continues improving up to batch size 6144.

The reported evaluation scope is centered on AST / ViT-B/16 and AudioSet pre-training. The paper explicitly notes that broader architecture-scaling studies are limited. It also states that gains on AS-2M are modest because in-domain fine-tuning may be saturated, and that while AudioMosaic improves utterance-level discriminativeness, it does not deeply analyze tasks requiring precise local temporal reconstruction, where reconstructive methods may still have advantages (Huang et al., 14 May 2026).

Reproduction requires attention to an internal inconsistency: the appendix training table lists pre-training loss as “MSE,” but the method section and Eq. (1) clearly define AudioMosaic as contrastive pre-training using the symmetric InfoNCE-style loss. The paper advises following the main method section and released code rather than the appendix typo (Huang et al., 14 May 2026).

Within the space of audio SSL, the method’s broader significance lies in reframing masking from a reconstruction device into a contrastive view-construction mechanism. This design yields no reconstruction decoder, only a lightweight projection head, reduced token count before encoding, higher effective rank than the reconstruction baselines examined in the paper, and strong frozen-feature behavior in linear probing and out-of-distribution audio forensics (Huang et al., 14 May 2026). This suggests that AudioMosaic is particularly well suited to transfer settings in which utterance-level discrimination, frozen encoders, or lightweight downstream adaptation are primary objectives.