Attention-based Spectrogram Transformers

• AST is a pure Transformer model that processes 2D time–frequency spectrograms using multi-head self-attention and learnable positional embeddings.
• FlexiAST extends AST by employing randomized patch-size sampling and mathematically controlled weight resizing to achieve consistent performance across various input granularities.
• The architecture achieves state-of-the-art results on audio classification tasks, demonstrating flexibility for deployment in diverse scenarios including on-device and streaming applications.

An Attention-based Spectrogram Transformer (AST) is a pure Transformer architecture for audio signal understanding that processes 2D time–frequency representations (spectrograms) using multi-head self-attention and learnable positional embeddings. AST models, inspired by Vision Transformers (ViT), have established a new state of the art in audio classification, demonstrating capacity to learn global context directly from raw spectrograms, without convolutional inductive bias or need for additional CNN front-ends. A key technical advance, FlexiAST, extends the AST framework by allowing robust inference across a wide range of time–frequency patch sizes through patch-size randomization and mathematically principled parameter reparameterization, enabling flexible deployment without architectural modification (Feng et al., 2023).

1. Standard AST Architecture and Patch Dependence

In its canonical instantiation, AST takes an audio waveform, computes a log-Mel spectrogram $x\in\mathbb{R}^{F\times T}$, and divides it into non-overlapping (or mildly overlapping) $p\times p$ 2D patches. Each patch $x_i\in\mathbb{R}^{p\times p}$ is vectorized and linearly projected to a $D$-dimensional token $e_i = E\;\mathrm{vec}(x_i)$, using a learnable $\omega=E\in\mathbb{R}^{D\times p^2}$. Patch location is encoded by a learnable positional embedding $\pi\in\mathbb{R}^{N\times D}$, where $N=(F/p)\times (T/p)$. The input sequence, typically prepended with a [CLS] token, is processed by a stack of $L$ standard Transformer encoder layers (multi-head self-attention and MLP with residuals and LayerNorm).

Both patch embedding weights $\omega$ and positional encodings $p\times p$0 are strictly dependent on the patch size $p\times p$1, making the AST’s operational patch granularity essentially “hard-coded” at training time. Empirical results demonstrate that naively resizing these tensors to a new patch size $p\times p$2 at test time results in catastrophic failure: on AudioSet, for example, $p\times p$3 yields $p\times p$4 at the training patch size, but drops below $p\times p$5 at $p\times p$6 (Feng et al., 2023).

2. Patch-Size Flexibility via Randomized Training (FlexiAST)

FlexiAST enables robust evaluation across arbitrary patch sizes within a prescribed set, without any architectural modification. The core procedure includes:

• Random patch-size sampling: At each training iteration, a patch size $p\times p$7 is uniformly sampled from a candidate set $p\times p$8.
• Parameter resizing:
  • Positional embeddings: The original $p\times p$9 ($x_i\in\mathbb{R}^{p\times p}$0, $x_i\in\mathbb{R}^{p\times p}$1) are reshaped to a 2D grid. Bilinear interpolation indices $x_i\in\mathbb{R}^{p\times p}$2 to $x_i\in\mathbb{R}^{p\times p}$3, $x_i\in\mathbb{R}^{p\times p}$4, then flattened to $x_i\in\mathbb{R}^{p\times p}$5.
  • Patch-embedding weights (PI-Resize): A pseudo-inverse resize matrix $x_i\in\mathbb{R}^{p\times p}$6 is constructed such that for any patch $x_i\in\mathbb{R}^{p\times p}$7, the new embedding $x_i\in\mathbb{R}^{p\times p}$8 best matches the original $x_i\in\mathbb{R}^{p\times p}$9 under MSE. Explicitly, $D$0 with $D$1, where $D$2 bilinearly (up/down)-samples patches (Feng et al., 2023).
• Training: The minibatch is tokenized using the current patch size, embedded with the resized weights, and fed to the (unchanged) AST backbone. Loss is computed and backpropagated. No architectural change is required; only weight resizing and patch regridding.

This procedure exposes the model during training to all patch granularities in $D$3, leading to high transferability: the resulting AST backbone is able to maintain a nearly flat performance curve across the entire patch-size range at inference.

3. Empirical Performance Across Patch-Sizes and Tasks

Systematic benchmarking on standard audio classification datasets underscores the effectiveness of FlexiAST:

Patch Size	AST-B/16 (Trained @16)	FlexiAST-Sup
8	0.006 mAP	0.399 mAP
12	0.100 mAP	0.400 mAP
16	0.396 mAP	0.396 mAP
24	0.297 mAP	0.390 mAP
32	0.371 mAP	0.355 mAP
48	0.074 mAP	0.305 mAP

On AudioSet, the standard AST exhibits severe collapse outside its training patch. FlexiAST, however, sustains accuracy within a $D$4 mAP margin from the peak over all six tested patch sizes. Analogous behavior was confirmed on VGGSound, ESC-50, and Speech Commands. For speaker identification (e.g., VoxCeleb), frequency-axis resizing was avoided to preserve fine frequency-encoded speaker cues; resizing only the time dimension allows flexibility without degrading performance (Feng et al., 2023).

4. Analysis of Methodological Impact

FlexiAST’s strengths arise from two principles:

• Randomized exposure: By training across all desired patch scales, the model does not become overfit to one frequency–time decomposition, retaining the required geometric and statistical alignment at all supported scales.
• Mathematically controlled reparameterization: PI-Resize of patch and positional embeddings guarantees that the tokenization at a new patch size produces features that match the corresponding patch’s “meaning” under the MSE-optimal linear map, minimizing embedding distortion.

Between-batch randomization ensures all patch-size statistics are sufficiently covered by optimizer updates, while mathematically justified parameter resizing allows inference at arbitrary $D$5.

No architectural or computational cost is incurred. The Transformer encoder and classification head remain unchanged. The only runtime difference is patch repartitioning and online weight resizing, which can be implemented efficiently.

5. Applications, Limitations, and Use Cases

Practical implications of FlexiAST are substantial for deployment scenarios:

• Single-model flexibility: A trained model seamlessly adapts to hardware and context constraints by adjusting patch size. Larger patches can be used when memory or compute is restricted and finer patches when detailed event detection is required.
• On-device and streaming scenarios: Enables dynamic patching in real-time or memory-constrained inference, supporting streaming audio and event-driven switching of granularity.
• Multi-task front-ends: The same AST backbone can be shared across tasks with differing time–frequency resolution needs (e.g., environmental sound classification and acoustic event detection).

Limitations include:

• Frequency-distortive tasks: For tasks highly reliant on frequency resolution (e.g., speaker ID), frequency-axis resizing can be detrimental—suggesting axis-aware resizing or patching is essential.
• Large patch sizes: Coarse patches reduce temporal resolution, potentially missing short-duration or high-frequency events.

This suggests that for specialized tasks (e.g., those requiring invariance to fine time or frequency structure), specialized patch axis selection or model augmentation is required.

6. Broader Significance and Future Developments

FlexiAST establishes that with judicious data augmentation (randomized patch scaling) and mathematical alignment of embeddings and positional information (PI-Resize), pure attention-based models can be equipped with static-architecture, dynamic-resolution capabilities. This matches the flexibility previously attained mainly by multi-resolution or convolutional schemes, but retains all the modeling advantages of attention (global context, lack of translation bias).

A plausible implication is that FlexiAST-like strategies could be generalized to other data modalities—such as image, time series, or multimodal audio-visual tasks—where variable input resolution or aspect ratio is desirable but architecture changes are expensive. The clean separation between tokenization/embedding and subsequent model stages points to broader architectures where input adaptability is handled entirely in preprocessing and embedding, not by stacking multiple subnetworks or resizing the backbone (Feng et al., 2023).