BYOL-A: Self-Supervised Audio Learning

• The paper introduces a self-supervised audio method that learns robust representations from two augmented views of a single audio segment without relying on temporal proximity.
• It employs a dual-network architecture with an online network and an EMA-updated target network, combined with audio-specific augmentations like mixup and random resize crop.
• Empirical results show that BYOL-A transfers effectively across various tasks such as sound event classification, speaker identification, and music recognition, outperforming prior methods on benchmarks like AudioSet and FSD50K.

BYOL-A, pronounced “viola,” is a self-supervised learning method for general-purpose audio representation learning that adapts Bootstrap Your Own Latent (BYOL) to audio by learning from two augmented views of a single audio segment, rather than from assumed relationships between nearby or distant segments. It is designed for unlabeled audio, operates on log-mel spectrograms, and uses an online network, a target network updated by exponential moving average, and audio-specific augmentations to learn representations that are both robust to perturbations and useful across heterogeneous downstream tasks such as sound event classification, speaker identification, language identification, speech commands, music genre recognition, and instrument recognition (Niizumi et al., 2021, Niizumi et al., 2022).

1. Origins, motivation, and problem formulation

BYOL-A was introduced against a background in which many self-supervised audio methods relied on temporal structure: nearby segments were treated as similar, distant segments as dissimilar, and losses were contrastive or triplet-based. The motivating objection was that temporal distance is an unreliable proxy for semantic similarity in audio. Repetitive content at distant times is common in music, while adjacent segments can differ sharply when short events such as knocks, gunshots, or footsteps occur back-to-back. BYOL-A therefore proposed learning solely from a single audio segment, without expecting relationships between different time segments of audio samples, and without contrastive or triplet loss (Niizumi et al., 2021).

This design places BYOL-A within the non-contrastive joint-embedding family. The supervision signal is created by applying two augmentations to the same segment and training the model to predict one augmented view from the other. In the 2021 formulation, the emphasis was on general-purpose audio representation from unlabeled data, with the intended downstream scope spanning sound event classification, acoustic scenes, musical instruments, speakers, languages, and speech commands. A later study sharpened the motivation by arguing that general audio representations should expose multiple aspects of the signal while remaining robust to perturbations such as varying pitch, timbre, time shift, time stretch, and temporal amplitude changes (Niizumi et al., 2022).

2. Objective, networks, and input representation

BYOL-A retains the standard BYOL decomposition into an online network and a target network. The online network, with parameters $\theta$, comprises an encoder $f_\theta$, a projector $g_\theta$, and a predictor $q_\theta$. The target network, with parameters $\xi$, comprises an encoder $f_\xi$ and projector $g_\xi$ of the same architecture, but no predictor. The target parameters are updated by exponential moving average: $\xi \leftarrow \tau \xi + (1 - \tau)\theta,$ with $\tau = 0.99$ in the 2021 experiments (Niizumi et al., 2021).

The training signal is the BYOL loss between two augmented views $v=t(x)$ and $f_\theta$0 of the same audio segment. With

$f_\theta$1

and

$f_\theta$2

the one-direction loss is

$f_\theta$3

and the final objective is the symmetrized sum over the two view directions (Niizumi et al., 2021).

The initial BYOL-A formulation used log-mel spectrograms derived from 16 kHz audio, with STFT window size 64 ms, hop size 10 ms, 64 mel bins, and frequency range 60–7800 Hz. A pretraining example was a random crop of shape $f_\theta$4, corresponding to approximately $f_\theta$5 s. The encoder in that formulation was a small CNN based on a DCASE 2020 audio captioning model, with embedding dimension $f_\theta$6, and projector and predictor MLPs producing 256-dimensional outputs (Niizumi et al., 2021).

3. Augmentation pipeline and the multi-aspect encoder

A defining feature of BYOL-A is its audio-specific augmentation pipeline. In the original formulation, the augmentation module consisted of four blocks: pre-normalization, mixup with a memory bank, random resize crop on the spectrogram, and post-normalization. Pre-normalization standardized each spectrogram with global training statistics. Mixup was implemented as log-mixup-exp: $f_\theta$7 where $f_\theta$8 was drawn from a FIFO memory bank of 2048 recent normalized inputs and $f_\theta$9. In that study, $g_\theta$0. The intended effect was to preserve the original segment as foreground while perturbing its background. Random Resize Crop then approximated pitch scaling and time scaling in the time–frequency plane, using crop scale ranges $g_\theta$1 to $g_\theta$2 in both dimensions and bicubic interpolation back to the original size. Post-normalization re-standardized the augmented spectrogram using batch statistics (Niizumi et al., 2021).

A later BYOL-A study retained Mixup and Random Resize Crop and added Random Linear Fader, which applies a linear gain envelope over time to simulate fade-in and fade-out. That study also placed stronger emphasis on encoder design. Its encoder had 6,333,376 parameters and consisted of two Conv2d blocks, a reshape from $g_\theta$3 to $g_\theta$4, an MLP producing 2048-dimensional per-frame global features, concatenation of local and global per-frame features, and temporal mean+max pooling. For an input $g_\theta$5, the Conv blocks produced $g_\theta$6, reshaping yielded $g_\theta$7, the MLP yielded $g_\theta$8, concatenation yielded $g_\theta$9, and pooling produced a final $q_\theta$0 representation (Niizumi et al., 2022).

The rationale of this later encoder was explicitly multi-aspect. The reshaped $q_\theta$1-dimensional feature was treated as a local per-frame descriptor that preserves channel and frequency structure, while the MLP output provided a more abstract global per-frame descriptor. Temporal mean pooling captures average tendencies across time, whereas temporal max pooling captures rare but salient events. The final representation is the elementwise sum of the temporal mean and temporal max statistics, which the paper argued better serves the diverse demands of general audio tasks than either statistic alone (Niizumi et al., 2022).

4. Empirical performance across downstream tasks

The 2021 BYOL-A paper pretrained on unlabeled AudioSet, using balanced_train_segments and unbalanced_train_segments, totaling 1,963,807 clips. Linear evaluation was performed by freezing the encoder and training a single linear layer on downstream tasks. In the 2048-dimensional setting, BYOL-A achieved 74.1% on NSynth, 79.1% on UrbanSound8K, 40.1% on VoxCeleb1, 90.2% on VoxForge, 91.0% on SPCV2/12, and 92.2% on SPCV2, for an average of 77.8%. Under that protocol, COLA' with the same embedding dimension achieved an average of 68.7%, so BYOL-A exceeded it by 9.1 points; the largest gains were on SPCV2 (+15.4), VoxForge (+10.7), and VoxCeleb1 (+9.7) (Niizumi et al., 2021).

The 2022 study evaluated BYOL-A in a broader benchmark spanning ESC-50, UrbanSound8K, FSD50K, Speech Commands V2, VoxCeleb1, VoxForge, CREMA-D, GTZAN, NSynth, and Surge. In that protocol, BYOL-A achieved the best average result of 72.4% across the nine single-label tasks and the best VoxCeleb1 result of 57.6%. On FSD50K, BYOL-A reached mAP $q_\theta$2 and AUC $q_\theta$3, which was the best result among the non-AudioSet-supervised models reported there (Niizumi et al., 2022).

These results should be read as protocol-specific rather than directly interchangeable, because the later study used a different task suite and a more elaborate encoder analysis. What is consistent across both studies is that BYOL-A transferred well across environmental sound, speech-related, and music-related tasks, and that speaker and language identification were particularly strong downstream settings (Niizumi et al., 2021, Niizumi et al., 2022).

A smaller-data experiment in the 2021 paper also illustrated scale sensitivity. Using FSD50K development data with 40,966 samples for pretraining yielded average accuracy 70.1%, whereas using the AudioSet $q_\theta$4 subset with 210,315 samples yielded 72.3% under the same 512-dimensional evaluation protocol. The method still outperformed prior methods with AudioSet pretraining despite the 5× smaller dataset, but the result supported the general observation that BYOL-A benefits from larger unlabeled corpora (Niizumi et al., 2021).

5. Ablations, critical components, and implementation lessons

The ablation studies consistently identified the combination of audio-specific augmentations, the predictor-based BYOL objective, and the encoder architecture as the main determinants of performance. In the 2021 augmentation study on AudioSet $q_\theta$5 with a 512-dimensional encoder, Mixup+RRC achieved 72.3% average accuracy, Mixup only 63.9%, RRC only 68.4%, Gaussian-only 23.2%, Gaussian+RRC 69.3%, and Mixup+Gaussian+RRC 70.5%. Gaussian noise as the mixup source was therefore markedly inferior to mixing with real samples, and RRC was the single strongest augmentation component (Niizumi et al., 2021).

Normalization mattered asymmetrically. In the same 2021 study, full BYOL-A achieved 72.3%; removing Post-Norm yielded 72.1%, whereas removing Pre-Norm reduced performance to 70.5% at $q_\theta$6, 70.3% at $q_\theta$7, and 68.9% at $q_\theta$8. Pre-Normalization was therefore critical, especially because log-mixup-exp makes the effective strength of Mixup sensitive to the scale of log-mel values (Niizumi et al., 2021).

The 2022 paper attributed most of BYOL-A’s performance to encoder design and only the final critical portion to the BYOL framework and augmentations. A random BYOL-A encoder already achieved 61.7% average accuracy. Replacing the full encoder with global-only features reduced average performance to 68.3%; local-only features reduced it to 64.8%. Mean-only temporal pooling yielded 67.7%, max-only 69.9%, and mean+max 70.3%. Removing the predictor reduced performance to 52.8%, confirming that the predictor remains essential even though the method was otherwise relatively robust to the target decay rate $q_\theta$9 and to batch size (Niizumi et al., 2022).

The same study also found that BYOL-A was far less sensitive to $\xi$0 than the original visual BYOL literature might suggest. With $\xi$1 the method achieved 70.3%; $\xi$2 yielded 69.7%; $\xi$3 yielded 70.0%; $\xi$4 yielded 69.7%; $\xi$5, corresponding to a fixed random target, still yielded 66.9%; and $\xi$6, corresponding to target=online, yielded 69.5%. This robustness should not be confused with predictor dispensability: the no-predictor variant remained substantially weaker (Niizumi et al., 2022).

Practically, BYOL-A is therefore characterized by a stable recipe rather than a single immutable implementation: log-mel spectrogram input, an online and an EMA target network, a predictor head, aggressive time–frequency augmentations that preserve semantics, and an encoder that preserves local spectral detail while also exposing higher-level global statistics (Niizumi et al., 2021, Niizumi et al., 2022).

6. Taxonomic position, ambiguities, and later interpretations

In a later taxonomy of self-supervised learning, BYOL is placed in the non-contrastive alignment sub-category of alignment-based SSL. Its learning signal is “View alignment,” its prediction target is “View embeddings,” and its objective symmetry is “Symmetric.” In that taxonomy, it is explicitly distinguished from Predictive Representation Learning, where the target is an unobserved component $\xi$7 predicted from an observed context $\xi$8. A comparative implementation in that study reported BYOL augmentation similarity $\xi$9 and occlusion robustness $f_\xi$0, whereas I-JEPA obtained augmentation similarity $f_\xi$1 and occlusion robustness $f_\xi$2. This suggests that BYOL-A inherits the strengths of alignment-based learning—strong invariance under augmentations and strong transfer to classification-style tasks—while also illuminating a limitation: it does not explicitly model latent prediction of unseen audio components in the sense of Predictive Representation Learning (Dutta et al., 15 Apr 2026).

The nomenclature is not fully stable across fields. In audio representation learning, BYOL-A denotes “BYOL for Audio.” In reinforcement learning, however, “BYOL-A” can also refer to action-conditional self-predictive learning, more formally BYOL-AC, where a predictor is conditioned on actions and trained to predict future latent representations. That usage is conceptually distinct from BYOL for Audio, even though both inherit the online/target, predictor, and stop-gradient mechanics of the BYOL family (Khetarpal et al., 2024).

Within audio, the main open directions stated or implied by the BYOL-A literature are larger backbones, multimodal pretraining, fine-tuning beyond linear probing, and more systematic study of robustness under noise and domain shift. The taxonomic analysis from later self-supervised learning work adds a further implication: if the objective is not only invariance but also long-horizon predictive structure or robustness under partial observability, then BYOL-A is a natural starting point but not necessarily the final form of the model (Niizumi et al., 2021, Niizumi et al., 2022, Dutta et al., 15 Apr 2026).