Masked Spectro-Temporal Prediction (MSTP)

• Masked Spectro-Temporal Prediction is a self-supervised approach that reconstructs masked log-Mel spectrogram patches using an encoder-decoder transformer architecture.
• It partitions spectrograms into square patches and applies a static 75% random masking strategy to train the model for accurate spectro-temporal prediction.
• Empirical results demonstrate MSTP’s effectiveness in enhancing transformer performance on tasks like audio tagging, sound classification, and speech command recognition.

Masked Spectro-Temporal Prediction (MSTP) is a self-supervised pre-training approach designed for audio transformer models. MSTP operates on log-Mel spectrogram representations of audio, applying a masking and reconstruction scheme analogous to masked language modeling for speech and masked image modeling for vision, but adapted to the two-dimensional spectro-temporal domain. Its prime realization, Masked Spectrogram Prediction (MaskSpec), demonstrates that transformer-based models can learn audio-specific representations from large-scale unlabeled datasets and outperform models pretrained on non-audio modalities with respect to several downstream audio tasks (Chong et al., 2022).

1. Spectrogram Representation and Patchification

MSTP utilizes log-Mel spectrograms computed from raw, mono audio sampled at 32 kHz. For each 10 s segment, the following steps are performed:

• Short-Time Fourier Transform (STFT) with a Hamming window of 1024 samples (32 ms) and hop size of 320 samples (10 ms).
• The magnitude spectrum is projected onto 128 Mel filterbanks, then the logarithm of (magnitude + $\epsilon$) is taken to produce a log-Mel spectrogram $T \in \mathbb{R}^{N_t \times N_f}$ with $N_f = 128$ Mel bins.
• The time dimension is fixed at $N_t = 992$ (≈9.92 s), yielding spectrograms of shape $992 \times 128$.

The spectrogram is partitioned into non-overlapping square patches of size $p \times p$ in the time–frequency domain. For $p = 16$, the number of patches is:

$$n = \left\lfloor \frac{N_t}{p} \right\rfloor \times \left\lfloor \frac{N_f}{p} \right\rfloor = 62 \times 8 = 496$$

Each patch $e_i \in \mathbb{R}^{p \times p}$ serves as a basic token for prediction.

2. Masking Strategy

MSTP employs a random uniform masking procedure:

• A fixed masking ratio $\alpha \in [0.05, 0.95]$ is chosen ($\alpha = 0.75$ is default).
• $N = \lfloor \alpha \cdot n \rfloor$ patches are sampled uniformly at random without replacement to form the masked set $\bar{E}$.
• The remaining $n - N$ patches comprise the visible set.
• Masking is static and uniform at each training step—there is no use of block-shaped masks, curriculum schedules, or dynamic annealing of the mask ratio.

This strategy is notable for its simplicity: large random subsets (e.g., 75%) of spectro-temporal patches are withheld from the model for prediction, without structured or progressive masking schemes.

3. Model Architecture

MSTP as instantiated by MaskSpec adopts an asymmetric encoder–decoder transformer architecture:

Encoder:

• Receives only the unmasked patches, each flattened to length $p^2$ and linearly projected to an embedding of dimension $D_\text{emb}$ (768 in the base model).
• Employs standard 1-D sinusoidal positional encodings shared across frequency and time.
• Stacks $N_d = 12$ transformer blocks, each with $N_h = 12$ self-attention heads and intermediate feed-forward dimension $D_\text{ffn} = 2048$.

Decoder:

• Used exclusively for pre-training reconstruction.
• Re-inserts $N$ learned "mask tokens" $S$ (each vector of size 512) at masked patch locations.
• Concatenates mask tokens and encoded visible patches, sorts by original patch positions, adds positional embeddings, and processes them via 8 transformer blocks (16 heads, embedding size 512, feed-forward 2048).
• Outputs are mapped back through a final linear layer to recover tensors in $\mathbb{R}^{p \times p}$ for each masked patch.

This design ensures efficiency by restricting the encoder to visible patches and relegating the masked patch reconstruction to the decoder sub-network.

4. Objective Function and Pre-Training Protocol

The MSTP learning objective is a mean-squared error reconstruction loss imposed only on the masked patches:

$$L(\theta) = \sum_{i \in I_\text{mask}} \| e_i - y_i \|_2^2$$

where $e_i$ are ground-truth patch vectors and $y_i$ are the predicted reconstructions. The formulation abstains from contrastive, adversarial, or auxiliary regularization objectives.

Training employs AdamW with a cosine learning rate decay and linear warm-up over the first 40 epochs (initial learning rate $1 \times 10^{-3}$, weight decay 0.05). The pre-training corpus is the ~1.9M-clip, 10 s/clip unlabeled AudioSet. Training takes 80 epochs (4 days, 8 × V100 32 GB GPUs). Batch size is not strictly specified but is compatible with 64 or 128 patches per GPU.

5. Downstream Tasks and Evaluation

After pre-training, a task-specific linear output head is attached to the encoder, and fine-tuning is performed for 80–100 epochs. Optimizer settings match pre-training but with a 5-epoch warm-up and additional data augmentations (mixup in time/frequency, time shifting, layer-wise learning rate decay).

Performance is assessed on several standard datasets and benchmarks:

Dataset	Task	Metric	Result (MaskSpec-Base)
AudioSet (full)	Audio tagging	mAP	0.471
ESC-50	Environmental sound classification	Accuracy	0.982
DCASE2019 Task1A	Acoustic scene classification	Accuracy	0.823
OpenMIC2018	Polyphonic instrument recognition	mAP	0.853
SCV2	Speech command recognition	Accuracy	0.976

MaskSpec matches or surpasses prior work, including audio models initialized from ImageNet-pretrained vision transformers (such as AST and PaSST), and consistently exceeds other self-supervised methods like SSAST. Smaller MaskSpec variants (Small, Tiny) exceed corresponding from-scratch and self-supervised baselines.

6. Ablation Studies and Empirical Insights

Empirical analysis of MSTP via MaskSpec yields the following findings:

• Masking ratio: Robust pre-training is observed for $\alpha$ in $[15\%, 85\%]$, optimal at $\alpha \approx 75\%$.
• Model scale: Smaller architectures (fewer layers or reduced embedding sizes) maintain the transfer benefits of MaskSpec, sometimes achieving comparable results to the base variant on moderate-size downstream sets.
• Masking strategy: No improvement is observed with structured or curriculum masks over static random masking.

These results indicate that complex masking schemes are superfluous and that the MSTP protocol is broadly robust to hyperparameter choices related to mask proportion and model scaling.

7. Significance and Context

MSTP, as realized by MaskSpec, represents an efficient and domain-adapted form of masked prediction pre-training for audio. By patchifying log-Mel spectrograms and applying aggressive, random masking, transformer encoders acquire spectro-temporal acoustic representations rivaling those obtained by transferring features from large-scale, non-audio pre-training. Notably, MSTP's reliance on native audio corpora and absence of contrastive or adversarial losses simplify the pre-training pipeline, with empirical validation across diverse downstream benchmarks (Chong et al., 2022).

A plausible implication is that further advances in self-supervised audio representation learning may stem from continued refinement of spectro-temporal masking and transformer architectures, rather than from increased algorithmic complexity or multimodal pre-training dependencies.