Replay Attention Masks

• Replay Attention Masks are a selective mechanism that iteratively refines audio attention by replaying crucial segments using slot attention and temporal mask extraction.
• They leverage a multi-playback pipeline to progressively zoom in on informative audio segments, thereby improving classification accuracy.
• This approach balances computational cost and resolution by dynamically adjusting spectrogram granularity and focusing processing on key audio cues.

Replay attention masks are a mechanism designed for temporal audio understanding within attention-based end-to-end architectures, specifically exemplified by the PlayItBack model. The technique iteratively identifies and focuses computation on the most discriminative segments of an audio sequence, leveraging successive “playbacks” and increasingly fine temporal resolution. The strategy draws inspiration from human auditory cognition, where listeners often mentally replay crucial moments to increase categorical confidence. The replay attention mask framework is grounded in slot attention, Transformers, and temporal mask extraction, supporting state-of-the-art results in large-scale audio recognition benchmarks (Stergiou et al., 2022).

1. Architecture Overview

The replay attention mask mechanism operates within the PlayItBack pipeline as follows. Given a raw waveform $w$ of length $L$, a log-mel spectrogram $\mathbf{X}_1$ is computed at an initial hop length $h_1 = 10\,\mathrm{ms}$: $$\mathbf{X}_1 = \operatorname{MelSpectrogram}\bigl(w;~\mathrm{hop}=h_1\bigr) \in \mathbb{R}^{F \times T_1}$$

$\mathbf{X}_1$ is segmented into $k$ non-overlapping patches, embedded, and augmented with 2D positional encodings. The sequence of vectors $\{\mathbf{x}_{1,i}\}_{i=1}^k$ is then processed by a Vision Transformer encoder $\mathcal{B}$: $$\mathbf{z}_1 = \mathcal{B}\bigl(\{\mathbf{x}_{1,i}+P_i\}\bigr) \in \mathbb{R}^{d\times C}$$

This representation forms the input to slot attention, which is applied for mask generation.

2. Slot-Attention-Based Mask Extraction

The mask generation relies on slot attention with two slots—one signifying “informative,” the other “uninformative.” The process unfolds over $L$0 iterations indexed by $L$1. For slot $L$2 at iteration $L$3: $L$4 With random initialization $L$5, slot states after $L$6 iterations are denoted $L$7 (informative) and $L$8 (uninformative).

A contrastive single-headed attention matrix is formed: $L$9 Obtaining the diagonal $\mathbf{X}_1$0 yields a $\mathbf{X}_1$1-dimensional relevance score vector, which is interpolated to length $\mathbf{X}_1$2, normalized to $\mathbf{X}_1$3, and thresholded at $\mathbf{X}_1$4: $\mathbf{X}_1$5 This binary vector $\mathbf{X}_1$6 constitutes the initial replay attention mask.

3. Iterative Mask Refinement and Temporal Zooming

For subsequent iterations (playbacks $\mathbf{X}_1$7), the model progressively “zooms in” on the previously selected segments:

1. The indices $\mathbf{X}_1$8 denote active regions.
2. Corresponding waveform segments are concatenated to yield a new, shortened audio $\mathbf{X}_1$9.
3. A higher resolution log-mel spectrogram is computed with $h_1 = 10\,\mathrm{ms}$0 where $h_1 = 10\,\mathrm{ms}$1.
4. This spectrogram is patchified, encoded, and processed as before, producing a new mask $h_1 = 10\,\mathrm{ms}$2.

Each additional playback narrows attention to finer-grained audio details, up to a pre-specified maximum number of playbacks ($h_1 = 10\,\mathrm{ms}$3 in empirical optimum).

4. Training and Inference Procedures

Training and inference proceed through tightly synchronized steps, as expressed in the following high-level pseudocode:

$$\mathbf{X}_1 = \operatorname{MelSpectrogram}\bigl(w;~\mathrm{hop}=h_1\bigr) \in \mathbb{R}^{F \times T_1}$$6

Key loss components include classification loss $h_1 = 10\,\mathrm{ms}$4 at each playback and a ranking loss $h_1 = 10\,\mathrm{ms}$5 encouraging monotonic confidence improvements across playbacks.

5. Empirical Design Considerations

Experimental findings indicate optimal trade-offs among several hyperparameters:

• Number of playbacks $h_1 = 10\,\mathrm{ms}$6 balances computational cost and classification accuracy.
• Slot attention iterations $h_1 = 10\,\mathrm{ms}$7 confer a marginal accuracy gain ($h_1 = 10\,\mathrm{ms}$8) over $h_1 = 10\,\mathrm{ms}$9 with minimal added compute (+2.6 GFLOPs).
• Hop length starts at $$\mathbf{X}_1 = \operatorname{MelSpectrogram}\bigl(w;~\mathrm{hop}=h_1\bigr) \in \mathbb{R}^{F \times T_1}$$0ms, subtracting $$\mathbf{X}_1 = \operatorname{MelSpectrogram}\bigl(w;~\mathrm{hop}=h_1\bigr) \in \mathbb{R}^{F \times T_1}$$1ms per playback, slowing down selected segments for finer detail.
• Ranking loss margin $$\mathbf{X}_1 = \operatorname{MelSpectrogram}\bigl(w;~\mathrm{hop}=h_1\bigr) \in \mathbb{R}^{F \times T_1}$$2 and mixture weighting $$\mathbf{X}_1 = \operatorname{MelSpectrogram}\bigl(w;~\mathrm{hop}=h_1\bigr) \in \mathbb{R}^{F \times T_1}$$3 are empirically chosen.

Ablation studies establish that increasing the sampling rate to $$\mathbf{X}_1 = \operatorname{MelSpectrogram}\bigl(w;~\mathrm{hop}=h_1\bigr) \in \mathbb{R}^{F \times T_1}$$4kHz or adding further playbacks ($$\mathbf{X}_1 = \operatorname{MelSpectrogram}\bigl(w;~\mathrm{hop}=h_1\bigr) \in \mathbb{R}^{F \times T_1}$$5) is ineffective; excess playbacks over-focus on short fragments and degrade accuracy. The PlayItBackX3 configuration achieves consistent state-of-the-art results on AudioSet, VGG-Sound, and EPIC-KITCHENS-100 (Stergiou et al., 2022).

6. Practical Implementation and Performance Considerations

In practice, the extraction and replay steps benefit from spectrogram-domain interpolative gathering, negating a need for explicit waveform slicing and concatenation of all selected segments. The model operates end-to-end, with all mask generation, signal upsampling, and iterative refinement steps integrated in the attention-based audio recognition pipeline.

The replay attention mask procedure can be summarized in terms of its distinctive workflow components:

Stage	Operation	Purpose
Initial play (t=1)	Coarse mask extraction via slot attention	Highlights informative time-bins
Subsequent playbacks (t>1)	Higher-res spectrograms using segment replay	Focuses on temporally local discriminativity
Training/inference schedule	Classification and ranking losses across playbacks	Guarantees iterative confidence refinement

The architecture’s design enables selective computation over informative regions, allowing for iterative resolution enhancement and efficient discrimination of fine-grained audio categories.

7. Context, Significance, and Extensions

The replay attention mask mechanism extends the paradigm of attention in audio recognition by incorporating selective, temporally focused replay, and resolution adjustment, directly inspired by human listening strategies. Its state-of-the-art performance demonstrates the utility of iterative attention in large-scale audio classification settings. A plausible implication is that similar iterative mask-based replay could be adapted to other sequential domains where fine-grained discrimination is essential, provided appropriately defined slot-attention modules and replay schedules.

For detailed empirical results, architectural diagrams, and full experimental procedures, see "Play It Back: Iterative Attention for Audio Recognition" (Stergiou et al., 2022).