From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation

Abstract: Self-supervised learning advances audio representation for multimedia analysis. However, prevailing data-centric approaches rely on massive real-world corpora, increasing training costs, curation burdens, and privacy barriers. To address this, we present AudioPG, a procedural synthesis framework eliminating real audio recordings during pre-training. AudioPG trains a Transformer-based masked autoencoder on waveforms generated on-the-fly from basic acoustic primitives and composition rules. The encoder transfers effectively to real audio benchmarks, achieving 90.60% accuracy on ESC-50, 0.546 mAP on FSD50K, 88.17% on UrbanSound8K, and 97.03% on Speech Commands V2. Notably, pre-training completes in under 20 minutes on a single GPU. Latent space analysis reveals physical factors, including fundamental frequency and relative intensity, emerge in orthogonal subspaces, making representations linearly decodable. These results establish procedural synthesis as an efficient, interpretable pre-training signal when large-scale corpora are unavailable. Our code is available at: https://github.com/Freyliu0516/audioPG.

• The paper introduces AudioPG, which leverages physics-based procedural audio synthesis and masked autoencoder training to learn audio representations without relying on real data.
• It demonstrates high transferability by achieving up to 97% accuracy on benchmarks, while converging rapidly and efficiently on a single GPU.
• The study reveals that physical parameters create interpretable latent spaces, though a semantic gap remains in accurately capturing human-level audio nuances.

Physics-Driven Procedural Pre-Training for Audio Representation Learning

Overview

The paper "From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation" (2606.14791) presents AudioPG, a framework that eschews the need for real audio corpora during pre-training by leveraging physics-based procedural audio synthesis. AudioPG synthesizes waveforms using parameterized acoustic primitives and composition rules, pre-trains a masked autoencoder (MAE) to reconstruct heavily masked log-Mel spectrogram patches, and demonstrates transferability of the resulting representations to standard downstream benchmarks. The approach decouples inherent biases and data requirements typically associated with data-driven pipelines, enabling controlled curriculum generation, interpretable latent space analysis, and highly efficient pre-training.

Methodology

Procedural Audio Synthesis

AudioPG utilizes a parametric synthesizer informed by physical principles, allowing specification and systematic variation of generative factors. The generator implements multiple compositional mechanisms:

• Harmonic additive synthesis: Control over fundamental frequency, amplitude, and power-law roll-off.
• Frequency modulation and broadband pulse trains: Parameter modulation enables rich spectral and temporal diversity.
• Transient bursts and dynamic envelopes: ADSR envelopes and impulsive noise sculpt time-domain event structure.
• Spectral damping and background noise: Stochastic filtering and signal-to-noise ratio manipulation model realistic attenuation and intensity variation.

Continual parameter sampling produces an infinite and diverse curriculum, ensuring robust representation learning and preventing memorization.

Masked Autoencoder Training

Pre-training operates on log-Mel spectrograms partitioned into $16 \times 16$ patches, applying a 75% random masking ratio. A Transformer encoder processes the visible patches, and the decoder reconstructs masked regions. The reconstruction objective leverages compositional regularities induced by the synthesizer, pushing the network to exploit underlying physical attributes. Mean-squared error is used as the loss function for spectrogram reconstruction.

Empirical Evaluation

Downstream Benchmark Transfer

The encoder, trained solely on synthetic, procedurally generated waveforms, is fine-tuned and tested on four real-audio benchmarks: ESC-50, UrbanSound8K, FSD50K, and Speech Commands V2. AudioPG achieves:

• ESC-50: 90.60% accuracy
• UrbanSound8K: 88.17% accuracy
• FSD50K: 0.546 mean average precision (mAP)
• Speech Commands V2: 97.03% accuracy

Pre-training is completed in under 20 minutes on a single GPU, demonstrating significant computational efficiency.

Ablation and Efficiency Analysis

Component ablation indicates additive synthesis primitives, temporal dynamics, and noise are critical for transferability. Cold-start efficiency analysis shows AudioPG converges faster and achieves higher performance under matched compute compared to real-data-based pre-training. Increasing event density and synthesizer complexity notably improves downstream accuracy and robustness. In high-resource regimes, procedural pre-training narrows but does not fully eliminate the domain gap relative to real data.

Latent Space Disentanglement

AudioPG's latent representations encode physical parameters in orthogonal and linearly decodable subspaces. Mutual information and principal component analysis reveal distinct directions in the latent space corresponding to fundamental frequency, amplitude, and temporal position. Ridge regression probes confirm high $R^2$ for decoding these physical factors, supporting the claim of emergent disentanglement. Visualizations of patch embedding evolution further corroborate alignment between low-level filters and compositional acoustic structures.

Error Analysis and Semantic Limitations

Cross-dataset error attribution pinpoints the semantic gap inherent to physical-based synthesis. Misclassifications arise from acoustic similarity (e.g., impulsive events, sustained resonance, broadband texture) rather than human-defined semantic cues, illustrating limitations in fine-grained linguistic or ontological discrimination. Speech-based errors are dominated by phonetic overlap due to the absence of explicit vocal tract modeling.

Preliminary Domain Adaptation

Continued MAE pre-training on unlabeled real-audio domains (FSD50K) further improves performance, demonstrating robust initialization and resistance to domain bias relative to models initialized from out-of-domain visual corpora (e.g., ImageNet).

Implications

AudioPG introduces a paradigm shift in audio representation learning by replacing data-centric pipelines with physics-grounded procedural generation. Practically, the method relieves burdens related to dataset curation, privacy, licensing, and compute, and provides extensive controllability over generative factors. Theoretically, this explicit control enables in-depth analysis of feature space structure, disentanglement of latent factors, and connection between reconstruction objectives and physical audio attributes.

However, the semantic gap remains significant. Procedural synthesis aligns latent space structure with physical principles, but fails to encode human-level semantics without more sophisticated generative models. Future extensions could integrate generative foundation models (e.g., AudioGen, AudioLDM) to compose curricula reflecting conceptual and ontological richness, possibly mapping abstract semantic descriptors to acoustic parameterizations, thus bridging the divide between physical and semantic representations in audio.

Conclusion

AudioPG establishes procedural synthesis as a viable, efficient, and interpretable pre-training signal for audio representation learning. The framework delivers strong transferability across multiple real audio tasks, converges rapidly, and enables physically interpretable latent structure without reliance on real recordings. Further research should explore hybrid approaches integrating high-level generative priors and richer compositional source models to enhance semantic acuity and generalization, especially in multimedia and cross-modal settings.