AudioPG: Procedural Audio Pre-training

• AudioPG is a procedural synthesis framework that generates waveforms from basic acoustic primitives for self-supervised audio pre-training.
• It trains a Transformer-based masked autoencoder on synthetic spectrograms, achieving competitive and fast transfer performance on real-world benchmarks.
• The framework enables explicit control over physical generative factors, reducing data curation burdens and promoting interpretable latent representations.

AudioPG is a procedural synthesis framework for self-supervised audio representation learning that eliminates real audio recordings during pre-training by generating waveforms on the fly from basic acoustic primitives and composition rules. In its canonical formulation, AudioPG trains a Transformer-based masked autoencoder on synthetic audio derived from a source–filter process, then transfers the encoder to real-world benchmarks. The framework is positioned as a first-principles alternative to data-centric pre-training regimes that depend on massive real-audio corpora, with the stated aims of reducing computational cost, curation burden, licensing and privacy barriers, and dataset-induced bias while exposing explicit physical generative factors for analysis and interpretation (Liu et al., 11 Jun 2026).

1. Concept and motivation

AudioPG is defined by two coupled commitments: pre-training without real recordings, and doing so with a controllable generator grounded in elementary acoustic structure. The framework replaces large curated corpora with a procedural synthesizer that emits waveforms online, so each batch can be generated afresh without storage overhead. The encoder is then optimized through masked reconstruction rather than supervised labels, aligning the method with spectrogram-based masked autoencoding while changing the source of pre-training data from collected recordings to synthetic signals (Liu et al., 11 Jun 2026).

The central motivation is not merely efficiency. AudioPG is designed so that the generative factors behind each waveform are known, sampled, and manipulable. This makes the pre-training distribution physically interpretable in a way that typical web-scale audio corpora are not. The paper explicitly frames this as a route to representations whose latent geometry can be studied against variables such as fundamental frequency and relative intensity. A plausible implication is that AudioPG treats controllability and representation analysis as coequal goals rather than viewing interpretability as a secondary diagnostic.

The framework also addresses a practical constraint common in audio research: many settings lack the scale, legal clearance, or privacy permissions needed for large self-supervised corpora. AudioPG therefore proposes procedural synthesis as a viable pre-training signal when large-scale corpora are unavailable. In the reported setup, pre-training completes in under 20 minutes on a single NVIDIA RTX 4090 GPU for a ViT-Base model trained for 100 epochs with batch size 64, making the approach explicitly oriented toward low-friction experimentation (Liu et al., 11 Jun 2026).

2. Procedural synthesis and physical parameterization

AudioPG follows a source–filter model with a parameterized generator $G(\theta)$ that maps parameter sets $\theta$ to waveforms $y(t)$. A source signal is built by superposing event-based excitations, shaping them with envelopes and optional transient bursts, convolving the result with a damping filter, adding a background noise floor, and finally applying peak normalization. The basic normalization step is

$$y(t)=\frac{\tilde y(t)}{\max_t |\tilde y(t)|+\epsilon},$$

where $\tilde y(t)=S_{\mathrm{src}}(t)+A_n n(t)$ and $A_n$ controls background-noise intensity. Because peak normalization removes absolute gain, changes in event amplitudes $A_i$ primarily adjust effective signal-to-noise ratio rather than absolute loudness (Liu et al., 11 Jun 2026).

The source term is a superposition of $N_e$ acoustic events, with event count sampled from $N_e\in[1,5]$ and random temporal positions:

$$S_{\mathrm{src}}(t)=\left[\sum_{i=1}^{N_e} A_i e_i(t)\left(y_{\mathrm{osc}}^{(i)}(t)+y_{\mathrm{trans}}^{(i)}(t)\right)\right]*h_{\mathrm{damp}}(t).$$

Here $\theta$0 is an ADSR envelope, $\theta$1 is an optional transient burst, and $\theta$2 is a stochastic low-pass biquad filter used to model damping. Damping and transients are applied stochastically with probabilities $\theta$3 and $\theta$4, respectively; the paper states that these values are sampled and are not fixed (Liu et al., 11 Jun 2026).

The excitation primitives are deliberately simple but spectrally diverse. AudioPG uses harmonic additive synthesis,

$\theta$5

with $\theta$6; two-operator frequency modulation,

$\theta$7

where $\theta$8 is the modulation index and $\theta$9; and broadband pulse synthesis using geometric waveforms such as sawtooth and square waves. The default fundamental-frequency range is 50–2000 Hz, with sensitivity tests on 50–500 Hz and 50–4000 Hz (Liu et al., 11 Jun 2026).

All audio is resampled to 16 kHz and converted to log-Mel spectrograms $y(t)$0 using a 25 ms Hamming window, 10 ms hop, $y(t)$1 Mel bins, and $y(t)$2 frames, corresponding to approximately 10.24 s. Global standardization using dataset-level mean and variance is then applied. Because generation occurs on the fly for each batch, the paper characterizes the curriculum as effectively infinite and specifically notes that this prevents memorization (Liu et al., 11 Jun 2026).

3. Masked autoencoding formulation

AudioPG adopts a spectrogram-based masked autoencoder that reconstructs masked log-Mel patches. Inputs of shape $y(t)$3 are split into non-overlapping $y(t)$4 patches, producing a sequence of tokens. Random masking is applied at ratio $y(t)$5; only visible patches are passed to the encoder, while masked positions are represented by mask tokens in the decoder. The backbone is a Transformer MAE with a ViT-Base architecture and patch size $y(t)$6, implemented in PyTorch (Liu et al., 11 Jun 2026).

The training objective is mean squared reconstruction error over masked patches:

$y(t)$7

where $y(t)$8 indexes masked patches, and $y(t)$9 and $$y(t)=\frac{\tilde y(t)}{\max_t |\tilde y(t)|+\epsilon},$$0 are the ground-truth and reconstructed normalized spectrogram patches. Optimization uses AdamW. The pre-training configuration reported for the main setting is 100 epochs, batch size 64, and a single RTX 4090, with total time under 20 minutes (Liu et al., 11 Jun 2026).

Downstream fine-tuning uses unmasked inputs, batch size 32, SpecAugment, and either central cropping or loop-padding to the 10.24 s pre-training length. The main benchmark results are obtained after 300 epochs of fine-tuning, while a fast 50-epoch fine-tuning protocol is used for ablations to isolate component contributions. This distinction is important: the ablation numbers are intentionally lower in absolute terms than the main transfer results because the fine-tuning budget is reduced (Liu et al., 11 Jun 2026).

The reconstruction task is heavy by construction. The combination of 75% masking, multi-event composition, transient structure, damping, and noise creates a nontrivial inverse problem on synthetic data. The paper argues that this supports transfer because the model must recover structured time–frequency regularities rather than memorize fixed templates. A plausible implication is that AudioPG leverages difficulty induced by composition and partial observability rather than diversity supplied by real-world semantics.

4. Transfer performance, baselines, and ablations

With full fine-tuning, the pre-trained encoder transfers to multiple real-world benchmarks despite using zero real audio during pre-training. The reported main results are shown below (Liu et al., 11 Jun 2026).

Benchmark	Task/protocol	AudioPG result
ESC-50	2,000 clips, 50 classes, 5-fold protocol	90.60% accuracy
FSD50K	51,197 clips, multi-label, standard dev/eval split	0.546 mAP
UrbanSound8K	8,732 clips, 10 classes, official split	88.17% accuracy
Speech Commands V2	105,829 one-second utterances, 35 keywords	97.03% accuracy

These numbers substantially exceed the reported supervised-from-scratch baselines of 54.00% on ESC-50, 75.34% on UrbanSound8K, 0.398 mAP on FSD50K, and 96.30% on Speech Commands V2. They also place AudioPG in a competitive range relative to several real-audio pre-training references reported from prior work, including SSAST, MAE-AST, MaskSpec, MSM-MAE, PaSST, and BEATs, although the paper also records that the strongest real-data systems retain an advantage on some benchmarks and in high-resource regimes (Liu et al., 11 Jun 2026).

A central empirical claim of the paper is cold-start efficiency. Under a time-matched pre-training budget, AudioPG reaches 82.00% on ESC-50 and 85.75% on UrbanSound8K, whereas a real-data time-matched baseline reaches 72.00% and 83.57%, respectively. Validation curves are reported to show faster convergence for AudioPG under constrained compute. When pre-training is extended to 500 epochs, AudioPG reaches 85.40% on ESC-50 and 88.17% on UrbanSound8K with pre-train time of approximately 11,000 s; a real-data full-training baseline reaches 87.00% and 88.38% with pre-train time 13,044 s. The paper interprets this as evidence that a remaining sim-to-real domain gap appears in the high-resource regime (Liu et al., 11 Jun 2026).

The ablations make the compositional argument explicit:

Pre-training variant	ESC-50	US8K
Sine-only frequency baseline	42.25%	70.97%
+ Harmonic timbre	69.25%	75.99%
+ ADSR dynamics	73.25%	78.73%
Full AudioPG (+ spectral damping + background noise)	82.00%	85.75%

These results attribute transfer gains to timbre, temporal envelope, and spectral shaping rather than to frequency variation alone. Sensitivity analyses further show modest ESC-50 differences across $$y(t)=\frac{\tilde y(t)}{\max_t |\tilde y(t)|+\epsilon},$$1 ranges—76.50% for 50–500 Hz, 76.75% for 50–2000 Hz, and 77.00% for 50–4000 Hz—while increasing event overlap from one event to 5–10 events improves ESC-50 from 70.75% to 78.75%. The paper states that increasing overlap creates a more challenging reconstruction signal that improves transfer (Liu et al., 11 Jun 2026).

5. Latent organization and interpretability

AudioPG is unusual among audio pre-training schemes in that its latent analysis is defined against known physical generative factors. Using a frozen encoder, the paper estimates mutual information between individual neurons and sampled parameters with a Kraskov k-NN estimator. The highest reported mutual information values are 1.7299 nats for frequency at neuron #343, 1.4660 nats for relative intensity at neuron #666, and 0.4896 nats for temporal position at neuron #437. At the same time, low mutual-information gaps across frequency (0.0062), amplitude (0.1273), and temporal position (0.1277) are presented as evidence of distributed encoding with redundancy (Liu et al., 11 Jun 2026).

Linear decodability is correspondingly strong. Ridge regression on frozen embeddings yields $$y(t)=\frac{\tilde y(t)}{\max_t |\tilde y(t)|+\epsilon},$$2, $$y(t)=\frac{\tilde y(t)}{\max_t |\tilde y(t)|+\epsilon},$$3, and $$y(t)=\frac{\tilde y(t)}{\max_t |\tilde y(t)|+\epsilon},$$4. Principal-component analysis further shows that the top 20 principal components explain 96.49% of variance; amplitude or intensity dominates PC1 with mutual information 1.0343 nats, while frequency peaks at PC5 with mutual information 1.4124 nats. The paper interprets this as support for orthogonality between physical axes, writing conceptually that $$y(t)=\frac{\tilde y(t)}{\max_t |\tilde y(t)|+\epsilon},$$5 and that dot products across principal components encoding different physical factors are near zero (Liu et al., 11 Jun 2026).

The representational interpretation is not restricted to latent vectors. Patch-embedding filters are reported to evolve from noise-like patterns to frequency-selective bands and transient-sensitive edges over the course of training. This is consistent with learned time–frequency templates that align with the structure of the synthetic generator. Taken together, these analyses suggest that the masked reconstruction objective on explicit physical data can induce representations in which physically meaningful variables are both linearly decodable and partially factorized.

A recurring misconception in discussions of synthetic pre-training is that strong transfer would necessarily imply latent entanglement or opaque heuristics because the input distribution is artificial. AudioPG directly contests that view at the level of measurements: the framework reports both competitive transfer and high decodability of known physical parameters. This does not mean the representation is semantically complete, but it does show that synthetic pre-training need not be synonymous with uninterpretable pretext learning.

6. Limits, extensions, and terminological ambiguity

The paper is explicit about what AudioPG does not model. Its procedural generator captures physical acoustics but not high-level semantics such as human vocal-tract formants and linguistic articulation. Error analyses highlight confusions driven by acoustic similarity, including footsteps versus fireworks, helicopter versus engine, vacuum cleaner versus washing machine, and phonetic overlap such as forward versus four and tree versus three. FSD50K, with polyphonic multi-label content, is identified as particularly challenging because the curriculum is predominantly monophonic (Liu et al., 11 Jun 2026).

Several extensions are proposed. These include richer excitation sources, more complex mixing and stochastic rules, and semantics-aware generators such as AudioGen or latent diffusion systems to output structural configurations, including temporal distributions and modulation parameters, for semantic-aware procedural curricula. The paper also notes preliminary domain-adaptive masked autoencoding results in which FSD50K adaptation initialized from AudioPG yields 91.80% on UrbanSound8K, 90.80% on ESC-50, and 0.617 mAP, suggesting that procedural initialization may remain useful even when later adapted toward real data (Liu et al., 11 Jun 2026).

The name “AudioPG” is also potentially ambiguous in the wider literature. Related abbreviations occur in other areas of audio research, including audio-driven performance video generation (Zhu et al., 2020), audio adversarial perturbation generation for steganography (Yan et al., 28 May 2025), and audio prompt adherence measurement (Grachten, 2024). This suggests that references should be resolved by arXiv identifier and title rather than acronym alone. In the context of current audio representation learning, however, “AudioPG” most specifically denotes the procedural synthetic pre-training framework introduced in “From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation” (Liu et al., 11 Jun 2026).

For reproducibility, the framework provides code and reports a concrete reference configuration: 16 kHz audio; 128-bin log-Mel spectrograms with 25 ms window and 10 ms hop; 1024 frames per clip; a ViT-Base MAE with $$y(t)=\frac{\tilde y(t)}{\max_t |\tilde y(t)|+\epsilon},$$6 patches and 75% random masking; AdamW optimization; and the fine-tuning protocol described above. Within those boundaries, AudioPG is best understood as a physics-grounded program for audio pre-training: it replaces large curated corpora with synthetic composition, treats masked reconstruction as the transfer vehicle, and uses explicit control over generative factors to make the resulting representations analyzable as well as useful (Liu et al., 11 Jun 2026).