EnCodecMAE: Universal Audio Representation

Updated 25 November 2025

EnCodecMAE is a self-supervised audio framework that reconstructs perceptually-relevant EnCodec tokens to generate universal, task-general embeddings.
It fuses a Masked Autoencoder paradigm with neural codec targets, achieving high temporal resolution and state-of-the-art performance in heterogeneous audio tasks.
The framework not only boosts downstream recognition tasks but also aligns its representations with brain activity, enhancing interpretability and transferability.

EnCodecMAE is a self-supervised learning framework for universal audio representation, designed to produce robust, task-general embeddings applicable across speech, music, and environmental sounds. Distinct from prior audio SSL approaches, EnCodecMAE reconstructs perceptually-motivated discrete codes produced by the EnCodec neural audio codec rather than handcrafted features or random quantizer tokens. Its architecture fuses the Masked Autoencoder (MAE) paradigm with neural codec targets, achieving high temporal resolution essential for speech and event analysis and setting new benchmarks in heterogeneous audio task performance (Pepino et al., 2023).

1. Motivation and Conceptual Foundations

Universal audio representation learning seeks to derive a single backbone model, pre-trained on large amounts of unlabeled audio, yielding embeddings that serve a broad family of downstream tasks via frozen-feature or fine-tuned pipelines. Early SSL techniques for audio, such as wav2vec 2.0, HuBERT, and BEATs, employed frame-wise contrastive or clustering objectives, typically predicting MFCC clusters or random codebook tokens from masked segments. Vision-inspired audio MAEs often reconstruct spectrogram pixels from patch-masked inputs, leading to decreased temporal resolution and suboptimal performance on tasks relying on precise time structure.

EnCodecMAE departs from these conventions by employing masking on frame-level embeddings and reconstructing EnCodec discrete units—high-fidelity, perceptually-relevant codes that encode information necessary for high-quality neural audio synthesis. This enables better temporal resolution (75 Hz—frame shift ≈13 ms or up to 100 Hz, depending on feature choice) and greater alignment of the learned representation with the needs of both temporally precise and class-general tasks (Pepino et al., 2023, Pepino et al., 20 Nov 2025).

2. Architecture and Training Objective

Input and Feature Extraction

Raw audio is converted to either EnCodec encoder outputs (128-dimensional vectors, 75 Hz) or mel-spectrograms (128 or 256 bins, Hanning window of 640 samples, hop 320).
Each time-frame embedding is linearly projected to model dimension $D$ (768 for Base, 1024 for Large) and summed with sinusoidal positional encodings.

Masking Strategy

A fraction $M_\mathrm{prop} = 0.5$ of time-frames is masked in contiguous spans of $M_\mathrm{gap}=15$ frames.
Masked frames are dropped prior to the encoder, and their positions are re-inserted as special mask tokens before decoding.

Encoder-Decoder MAE

The visible sequence is processed by a transformer encoder (5, 10, or 20 layers for Small, Base, and Large, respectively; 12–16 attention heads; MLP dimension 4 $D$ ).
The lightweight transformer decoder (2 layers) reconstructs discrete code indices for each masked position and each of $Q=8$ selected EnCodec codebooks (each with 1024 entries).

Reconstruction Loss

For each time $t$ and codebook $q$ , EnCodec provides one-hot targets $y_q(t)\in\{1,...,1024\}$ .
The main training objective is a weighted cross-entropy loss:

$\mathcal{L} = \sum_{q=1}^{Q} \gamma_q \left[\alpha \sum_{t\in M} C(y_q(t), \hat{y}_q(t)) + \beta \sum_{t\notin M} C(y_q(t), \hat{y}_q(t)) \right]$

where $\gamma_q$ is codebook-dependent (proportional to quantization residual error), $\alpha = \delta/|M|$ , $\beta = (1-\delta)/(T_f - |M|)$ , and $\delta=0.9$ .

Pretraining Regimen

Data: AudioSet (≈4.5k h), Free Music Archive (800 h), Libri-Light (6k h speech); 4s random crops.
Optimizer: AdamW, learning rate $1 \times 10^{-4}$ , weight decay $0.05$, batch size 128.
Pretraining: 500k steps; followed by a self-training stage (additional 150k steps) predicting k-means clusters over backbone embeddings (Pepino et al., 2023).

3. Benchmark Results and Comparisons

Global Performance

EnCodecMAE achieves superior global accuracy as evaluated on the HEAREval suite (music, speech, environment):

Model	Global Score
BEATs (Iter 3)	96.0
AudioMAE (PT, patch-based)	–32
BYOL-A	82.6
MSM-MAE-512	91.7
EnCodecMAE Base Mixture	95.9
EnCodecMAE Base + Self-Train	97.6
EnCodecMAE Large + ST	100.0

On individual tasks, EnCodecMAE Large+ST achieves pitch accuracy ≈83.4%, speech command accuracy ≈97.0%, FSD50K mAP ≈51.0%, ESC-50 accuracy ≈84.1%. These figures match or exceed patch-based MAEs for environmental sounds and significantly surpass them on speech-oriented tasks, reflecting the architectural prioritization of high temporal resolution (Pepino et al., 2023).

Automatic Speech Recognition (ASR)

EnCodecMAE Base+ST: WER = 14.41% (no LM), 9.90% (with 4-gram LM)
EnCodecMAE Large+ST: WER = 12.44% (no LM), 8.59% (with 4-gram LM)
By contrast, HuBERT Base/Large: WER = 6.42/3.62 (no LM), 4.79/2.94 (with LM) (Pepino et al., 2023)

4. Brain Alignment and Representational Analysis

Recent studies demonstrate a positive correlation between EnCodecMAE's downstream performance and the alignment of its internal representations with human auditory cortex as measured by fMRI. Using both voxel-wise regression and Representational Similarity Analysis (RSA), the model's best layers achieve $R^2_m \approx 0.32$ –$0.37$ (median across subjects, posterior regions) and RSA $\rho_m$ up to $\sim 0.37$ after 300k steps of pretraining (Pepino et al., 20 Nov 2025).

Layerwise RSA increases steadily throughout pretraining, with brain-like representations (as quantified by RSA relative to auditory cortex RDMs) emerging within the first 5k steps.
Higher transformer layers show stronger alignment, and “pre-post norm” enables retention of both local and global information.
Direct performance–brain similarity: Pearson correlations up to $r=0.91$ (downstream global score and voxel-wise fit) and $r=0.82$ (global score and RSA) (Pepino et al., 20 Nov 2025).

This suggests that learning to reconstruct masked audio from naturalistic data, as in EnCodecMAE, induces representations reflecting core features of human auditory perception—an emergent alignment, not explicitly optimized for.

5. Downstream Use Cases and Extensions

Transfer and Interpretability

The architecture supports various practical transfer settings:

Frozen-backbone MLP classifiers for instance-level recognition tasks.
Speech, music, and event recognition, including high-temporal-resolution applications.
Recent work leverages EnCodecMAE as a backbone for interpretable prototype learning, extracting 768-d embeddings that inform diffusion-based sonification of learned prototypes without nearest-neighbor dependencies (Alonso-Jiménez et al., 14 Feb 2024).

Prototype Learning (PECMAE Framework)

Uses non-quantized EnCodec frame embeddings (128-d, 75 Hz) as MAE input.
A prototype network is trained on frozen EnCodecMAE features with a combination of cross-entropy and prototype-proximity losses.
Diffusion models map prototype vectors back to EnCodec feature sequences and ultimately reconstruct audio waveforms.

On genre and instrument recognition datasets, performance with 5–20 prototypes per class approaches or exceeds prior prototype methods, while enabling direct interpretability of learned class representations (Alonso-Jiménez et al., 14 Feb 2024).

6. Limitations and Ongoing Research Directions

Known Limitations

ASR performance, while competitive with older SSL approaches, is inferior to HuBERT and other dedicated, speech-specialized backbones. This indicates that the EnCodecMAE target space may not be optimally correlated with phonetic or linguistic content.
Slight specialization trade-offs are observed depending on the domain composition during pretraining; for certain mixtures, environment and music performance can increase at minor expense to speech recognition (Pepino et al., 2023).

Future Directions

Incorporation of alternative or multi-granular targets (e.g., joint EnCodec and phoneme-aligned clusters).
Extension to cross-modal audio applications (e.g., speech-to-MIDI) and integration into curriculum learning regimens.
Direct inclusion of brain similarity metrics as regularization or early stopping criteria during pretraining.
Investigations into the ecological validity and biological plausibility of the learned representations, including adaptation to animal vocalization modeling (Pepino et al., 20 Nov 2025).

7. Significance and Impact

EnCodecMAE establishes a new state-of-the-art in universal audio representation, unifying high perceptual fidelity from neural codecs with the efficient, high-resolution context modeling of transformer MAEs. Its strong global task performance, emergent brain-like representations, and modular transfer into interpretable and generative systems underscore its relevance both as a scientific model of auditory computation and as a practical foundation for diverse ML audio pipelines (Pepino et al., 2023, Pepino et al., 20 Nov 2025, Alonso-Jiménez et al., 14 Feb 2024).