Iterative Audio Pre-Training Framework

• The paper introduces an iterative framework that cyclically refines acoustic tokenizers and self-supervised models, resulting in enhanced semantic audio representations and improved classification metrics.
• It leverages discrete token prediction, knowledge distillation, and aggressive masking techniques to optimize performance across diverse audio, speech, and multimodal tasks.
• The framework’s modular design addresses reconstruction shortcomings and supports robust domain adaptation and low-resource learning through continuous co-optimization.

Iterative audio pre-training frameworks constitute a class of representation learning methodologies in which audio models and discretization modules are co-optimized over multiple alternate cycles. These frameworks were originally motivated by the need to obtain semantically rich, discrete audio labels analogous to those used in text and vision, thereby addressing the shortcomings of conventional reconstruction-based self-supervised learning (SSL) objectives in the audio domain. The iterative formulation fosters mutual refinement between the audio encoder and the acoustic tokenizer, driving state-of-the-art performance across a range of audio, speech, and multimodal tasks.

1. Core Principles of Iterative Audio Pre-Training

Iterative audio pre-training frameworks formalize audio representation learning as a cyclic process in which the SSL model and the label-generating tokenizer are alternately updated. The archetype of this approach—exemplified by BEATs—proceeds as follows: given a collection of unlabeled audio clips $\mathbf X$, an acoustic tokenizer $T_k$ generates discrete token sequences $\hat{Z}_k = \{\hat{z}^{(k)}_t\}$. An SSL model $M_k$ is then trained via a discrete prediction objective, commonly Masked Acoustic Modeling (MAM), to predict these tokens from masked acoustic inputs. Upon convergence, the model's intermediate activations $\hat{\mathbf O}_k$ are used to supervise the next tokenizer $T_{k+1}$ by knowledge distillation, creating new and potentially semantically richer token targets for the subsequent SSL iteration. This interplay—prediction of discrete tokens and tokenizer self-distillation—continues for a fixed number of iterations or until empirical gains saturate (Chen et al., 2022).

A representative pseudocode summary:

Initialize tokenizer T₁ with random projection; for k = 1 to K do Generate token targets Zₖ = Tₖ(X) for all data X; Train SSL model Mₖ by masking 75% patches: minimize Masked Acoustic Modeling loss; Use Mₖ to compute teacher outputs Ōₖ = Mₖ(X); Train next tokenizer Tₖ₊₁ to minimize distillation loss to Ōₖ; end for Return final tokenizer T_K and model M_K.

2. Mathematical Objectives and Tokenizer Architectures

The design of iterative audio pre-training frameworks hinges on explicit, stage-dependent losses. The initial tokenizer is typically a non-parametric random projection: given input patch $\mathbf{x}_t$, token assignment is

$$\hat{z}_t^{(1)} = \arg\min_i \|\mathbf{v}_i - \mathbf{W} \mathbf{x}_t\|^2_2,$$

where $\mathbf W$ is a fixed random linear map, and $\mathbf V$ is a codebook of $K$ code vectors. Subsequent tokenizers typically employ transformer encoders, mapping $\mathbf{x}_t$ to an embedding $\mathbf{e}_t$, which is quantized over the learned codebook.

The SSL model is trained to minimize

$$\mathcal{L}_{\mathrm{MAM}} = -\sum_{t\in \mathcal{M}} \log P(z_t = \hat{z}_t^{(k)}\,|\,\mathbf{X}^U;\,\theta_k),$$

where $\mathcal{M}$ is the set of masked positions and $\theta_k$ are the model parameters.

For the self-distilled tokenizer, the loss combines cosine alignment to the teacher's output, vector quantization commitment, and straight-through gradients for quantization:

$$\mathcal{L}_{\mathrm{KD}} = -\sum_{t=1}^T \cos(\mathbf{o}_t, \hat{\mathbf{o}}_t^{(k)}) + \lambda \sum_{t=1}^T \Big\|\ell_2(\mathbf{e}_t) - \ell_2(\mathbf{v}_{\hat{z}_t})\Big\|^2_2 + \lambda \sum_{t=1}^T \Big\|\mathrm{sg}[\ell_2(\mathbf{e}_t)] - \ell_2(\mathbf{v}_{\hat{z}_t})\Big\|^2_2.$$

Here $\mathrm{sg}[\cdot]$ denotes the stop-gradient operator, and EMA-based updates are used for robust codebook learning.

Related frameworks (e.g., Baichuan-Audio) extend tokenization to multi-codebook (residual vector quantization, RVQ), for finer control of semantic vs. acoustic detail, and optimize commitment and reconstruction losses over multi-stage Mel-spectrogram outputs (Li et al., 24 Feb 2025).

3. Variants and Extensions Across Modalities

Several variants operationalize the iterative audio pre-training concept for different settings and tasks:

• SSL for General Audio (BEATs, EAT): Standard iterative alternation between discrete target generation and masked prediction. BEATs achieves state-of-the-art metrics (ensemble mAP 50.6% on AudioSet-2M, 98.1% top-1 on ESC-50) using a 12-layer transformer encoder, with further efficiency improvements in EAT using a student-teacher bootstrap loop and aggressive masking (Chen et al., 2022, Chen et al., 2024).
• Continual Domain Adaptation (SONAR): Iterative pre-training is augmented with dual-level self-distillation, domain-aware stratified sampling, and a dynamic codebook that reinitializes underused codes on novel data, yielding significant robustness against catastrophic forgetting as new domains are introduced (Zhang et al., 19 Sep 2025).
• Audio-Text Multimodal/Low-Resource (IDP): Iterative Denoising Process (IDP) utilizes iterative translation and denoising in feature space to refine pseudo-parallel samples in the absence of ample parallel audio-text data. This approach enables effective cross-modal pre-training even in extremely data-scarce regimes by progressively denoising pseudo-translations via a dual cross-modal Transformer architecture (Kang et al., 2022).
• Multimodal LLMs (Baichuan-Audio): A two-stage iterative strategy first warms up the audio-head with discrete token prediction while freezing the pretrained language backbone, then jointly adapts the model on interleaved audio-text tasks. To maintain language competence, the architecture employs staged optimization and RVQ tokenizers, achieving strong results on speech-based QA and TTS generation (Li et al., 24 Feb 2025).

4. Empirical Performance, Convergence, and Optimization

The iterative approach results in consistently monotonic or near-monotonic improvements across multiple audio classification and understanding tasks, as measured on AudioSet-2M, ESC-50, and CLAS benchmarks:

Iteration	BEATs mAP (AudioSet-2M)	BEATs Accuracy (ESC-50)
Iter 1	47.9	94.0%
Iter 2	48.1	95.1%
Iter 3	48.0	95.6%
Iter 3⁺	48.6	97.1%
Ensemble	50.6	—

Progressive tokenization and model refinement yield increasing semantic coherence in discrete audio token assignments, as demonstrated by t-SNE visualization and robustness to noise, distinguishing the approach from conventional waveform or spectrogram regression-based SSL (Chen et al., 2022). In continual learning scenarios, frameworks such as SONAR retain near-zero forgetting rates on base domains—a critical advantage over naive direct continual pre-training baselines (Zhang et al., 19 Sep 2025).

Efficiency-oriented designs, such as EAT, introduce aggressive masking rates (up to 80%), multi-clone masking, and shallow decoders, achieving up to 15× speedup in pre-training wall-clock time without degrading downstream accuracy (Chen et al., 2024).

5. Data Selection, Regularization, and Stability Strategies

Iterative frameworks employ advanced data selection (e.g., task-relevant stratified sampling) and regularization techniques to optimize plasticity-stability tradeoffs:

• Sampling: Task-Relevant Stratified Sampling (TRSS) clusters new domain data and retrieves representative samples from a growing memory buffer to ensure diversity and relevance in each mini-batch. This is particularly effective for domain adaptation and resistance to catastrophic forgetting (Zhang et al., 19 Sep 2025).
• Regularization: Dual-level feature regularization (tokenizer and encoder) via strong $\ell_2$ penalties to anchor new model outputs to previous domain features, governed by hyperparameters such as $\lambda_{\rm tok}$ and $\mu_{\rm enc}$.
• Codebook Adaptation: Online Clustered Codebook (OCC) strategies adaptively re-center rarely used token vectors onto current feature centroids, capturing novel acoustic phenomena without unbounded codebook expansion (Zhang et al., 19 Sep 2025).

6. Expansion to Multimodal and Low-Resource Regimes

Iterative pre-training is adaptable across resource-constrained and multimodal settings:

• Audio-Text Alignment (IDP): IDP exploits iterative cross-modal denoising to bootstrap robust audio-text alignment in the absence of parallel data, by refining translation targets (pseudo-parallel samples) after each round of intra- and cross-modal denoising training. Ablations confirm the effectiveness of iterative refinement in lowering translation noise and boosting downstream task accuracy (Kang et al., 2022).
• Audio-LLMs (Baichuan-Audio): The two-stage iterative paradigm allows audio heads to align with discrete token streams before joint fine-tuning with textual modalities, supporting real-time multimodal conversational agents and text-guided audio generation (Li et al., 24 Feb 2025).

7. Challenges, Limitations, and Future Directions

Key open challenges include the design of acoustic tokenizers capable of capturing both high-level semantics and low-level acoustic variability, optimal codebook management strategies in the face of expanding datasets, and robust transfer to novel domains. Particular attention is required when expanding to cross-modal or continual learning, as improper regularization or sampling can result in semantic drift or feature collapse.

A plausible implication is that future frameworks may integrate more sophisticated, dynamically reconfigurable tokenizers, or meta-learning strategies to automate codebook adaptation and sampling. Further, extending iterative paradigms to non-parallel multimodal or downstream transfer settings remains a fertile area of research.

---

References:

• "BEATs: Audio Pre-Training with Acoustic Tokenizers" (Chen et al., 2022)
• "SONAR: Self-Distilled Continual Pre-training for Domain Adaptive Audio Representation" (Zhang et al., 19 Sep 2025)
• "Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction" (Li et al., 24 Feb 2025)
• "EAT: Self-Supervised Pre-Training with Efficient Audio Transformer" (Chen et al., 2024)
• "Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data" (Kang et al., 2022)