Hierarchical Audio Augmentation Strategy

• Hierarchical Audio Augmentation Strategy is a multi-stage pipeline that sequentially exploits spatial (ACS, MCS), temporal (TDM), and feature (TFM) transformations to enrich audio datasets.
• The approach increases data diversity—up to an 8× expansion—and systematically reduces SELD errors by refining DOA labels and event overlaps.
• Integrated with a ResNet-Conformer architecture, the method significantly improves detection and localization accuracy, achieving state-of-the-art SELD scores in DCASE challenges.

A hierarchical audio augmentation strategy is a structured data augmentation pipeline that systematically enhances audio datasets for tasks such as Sound Event Localization and Detection (SELD), explicitly increasing both the diversity and complexity of training data at each successive stage. The approach detailed in "A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection" (Wang et al., 2021) implements a stepwise augmentation sequence—Audio Channel Swapping (ACS), Multi-Channel Simulation (MCS), Time-Domain Mixing (TDM), and Time-Frequency Masking (TFM)—to improve generalization and robustness of acoustic models, particularly when combined with a ResNet-Conformer architecture. This multi-stage methodology leverages physical and statistical properties of spatial audio to enrich the distribution of direction-of-arrival (DOA) labels and event overlaps, addressing data sparsity and variation inherent in spatial audio scene analysis.

1. Spatial Augmentation: Audio Channel Swapping (ACS) and Multi-Channel Simulation (MCS)

The hierarchical strategy initiates with spatial augmentations that exploit the geometric symmetries and spatial encoding of multi-channel audio formats.

Audio Channel Swapping (ACS):

ACS leverages the rotational symmetry of First-Order Ambisonics (FOA) and tetrahedral microphone (MIC) arrays. A FOA 4-channel signal, comprising channels $[W, Y, Z, X]$, encodes source locations parametrized by azimuth $\phi$ and elevation $\theta$. ACS applies one of eight valid spatial transformations, each mapping $(\phi, \theta)$ to $(\phi', \theta')$ with corresponding permutations and sign adjustments to the audio channels. For instance, a transformation $$(\phi \rightarrow \phi + \pi, \theta \rightarrow \theta)$$ for MIC reorders $[C_1, C_2, C_3, C_4]$ as $[C_4, C_3, C_2, C_1]$; for FOA, $[W, Y, Z, X]$ becomes $[W, -Y, Z, -X]$. Table I in (Wang et al., 2021) enumerates all eight transformation patterns. Applied in a single pass to every 60 s audio clip—regardless of event overlap or movement—ACS provides an $8\times$ dataset expansion, e.g., 8 h of data becomes $\approx$55 h, with the only label adjustment performed on $(\phi, \theta)$ as prescribed by the transformation.

Multi-Channel Simulation (MCS):

MCS generates new spatial instances by decomposing and recombining spectral and spatial characteristics from isolated static events. It operates in two stages: first, a complex-Gaussian mixture model (CGMM) estimates time-frequency masks $\lambda_{f,t}^{(s)}$, facilitating extraction of source covariance matrices $\mathcal{R}_f^{(s)}$ and a distortionless spectral vector $\hat{S}_{f,t}$ via Generalized Eigenvalue (GEV) beamforming. Second, for simulation, spectral content $\hat{S}$ from one segment is randomly paired with spatial covariance $S$ from another, followed by eigen-decomposition and reconstruction of a new multi-channel Short-Time Fourier Transform (STFT):

$$\hat{x}_{f,t} = \sum_{m=1}^M \sqrt{\lambda_{f,m}} \cdot \hat{S}_{f,t} \cdot e^{2\pi j \xi_{t,m}} u_{f,m}$$

with random phase $\xi_{t,m} \sim \text{Uniform}(0,1)$ for $m\geq2$. This process is array-agnostic and computationally intensive, applied only to static isolated-event segments post-ACS, expanding 55 h to 155 h of material.

2. Temporal Composition: Time-Domain Mixing (TDM)

Following spatial enhancements, TDM increases event overlap diversity by constructing novel mixtures at the waveform level. For two non-overlapping single-event waveforms $x^{(1)}(l)$ and $x^{(2)}(l)$, TDM offsets one by $\Delta$, forming

$$x_{\text{mix}}(l) = x^{(1)}(l) + x^{(2)}(l - \Delta)$$

with corresponding SED label as a union of event classes and DOA labels as the union of their respective $(\phi, \theta)$ values. This step is computationally efficient and leverages the pool of augmented single-event segments produced by ACS and MCS, increasing the training corpus from 155 h to approximately 255 h.

3. Feature Regularization: Time-Frequency Masking (TFM)

TFM introduces stochastic feature-level perturbations analogous to SpecAugment, further regularizing model training. In each minibatch, for the first 11 feature maps (comprising Mel-spectrogram, intensity vectors, or generalized cross-correlation features), random frequency bands ($F' \leq 30$ Mel bins) and temporal frames ($T' \leq 35$ per 100 frames) are zeroed out. DOA-related channels remain unmasked to preserve localization cues. TFM is applied in the final model training phase on the 255 h dataset.

4. Rationale for Hierarchical Staging and Integration with Acoustic Models

The augmentation stages are strictly ordered:

• ACS precedes all others since it operates directly on unprocessed waveforms.
• MCS employs static segments from the ACS-expanded dataset for cross-segment recombination.
• TDM demands a maximally diverse pool of events, thus follows ACS and MCS.
• TFM, as a feature-level manipulation, is reserved for the final training set.

This "coarse-to-fine" sequencing—progressing from spatial to temporal to feature-level transformations—effectively maximizes label diversity and mitigates redundancy (Wang et al., 2021).

The augmentation pipeline directly interfaces with a ResNet-Conformer architecture. Feature extraction yields 64-dimensional log-Mel spectrograms per channel plus event-localization features (intensity vectors for FOA, GCC-PHAT for MIC), combining MIC and FOA inputs for 17 feature maps. The ResNet backbone learns local, shift-invariant structure, feeding into a stack of 8 Conformer blocks that alternate between Feedforward, Multi-Headed Self-Attention (8 heads, $d=512$, $d_k = d_v = 64$), and depthwise convolution (kernel size 51), yielding both local and global contextual representations. Dual output branches provide sigmoid SED output with binary cross-entropy loss and Sound Source Localization (SSL) output with masked mean-squared error—using a joint loss function $$L = \alpha_1 \cdot \text{BCE} + \alpha_2 \cdot \text{MSE}$$, where $\alpha_1 = 1$, $\alpha_2 = 10$. Training employs 60 s waveform chunks and Adam optimizer, with learning rate adaptation and early stopping on SELD_score.

5. Quantitative Outcomes and Ablation Analysis

The hierarchical augmentation regime delivers significant improvements across the DCASE 2020 and 2022 SELD benchmarks. Results from (Wang et al., 2021) are summarized as follows:

Stage	Training Hours	SELD_score	Relative Reduction
None	8	0.40	—
+ ACS	55	0.27	-32.5%
+ MCS	155	0.24	-40%
+ TDM	255	0.22	-45%
+ TFM	255	0.18	-55%
Final (ResNet-Conformer)	255	0.17	—

Each augmentation stage yields a 5–10% relative gain in SELD_score. Isolated ablations confirm the unique contributions of each augmentation: ACS alone (0.27), MCS alone (0.28), TDM alone (0.31), and TFM alone (0.36) when evaluated on a ResNet-GRU baseline. Qualitative analyses reveal that, for example, short events such as "barking dog" and overlapping sources in late segments are correctly detected only when TDM and TFM are employed.

In the DCASE 2022 evaluations (FOA only), the pipeline achieves SELD_score 0.32 (a 31.9% relative reduction vs. baseline 0.55), with final ensemble SELD_score 0.28, ranking first place.

6. Mechanistic Insights and Impact

ACS and MCS strategically expand the space of observed DOA labels, exposing the SSL branch of the model to a broader spectrum of localization cues. TDM compels the network to accurately resolve overlapping events, a primary challenge in real-world sound scenes. TFM enhances generalization by regularizing feature learning through stochastic masking. The combined ResNet-Conformer architecture is capable of capturing both local (convolutional) and global (self-attentive) contexts, fully leveraging the diversity introduced at each augmentation layer.

The hierarchical audio augmentation strategy orchestrates spatial, temporal, and spectral variations in a principled order, markedly improving both detection (increased F-score) and localization (decreased angular error) on standard benchmarks, and has set new state-of-the-art results for SELD in consecutive DCASE challenges (Wang et al., 2021).