SynSonic: Controlled Audio Synthesis for SED

• SynSonic is a data augmentation method for SED that generates temporally controlled, strongly labeled synthetic events using text-to-audio diffusion and energy-envelope ControlNet.
• It employs dual-classifier filtering with CLAP and Dasheng-AS to rank and retain the top 50% of generated samples, ensuring class accuracy and reduced noise.
• Integration into the FDY-CRNN pipeline alongside real data enhances both temporal localization and class discrimination, as evidenced by improved PSDS1 and PSDS2 scores.

Searching arXiv for the cited SynSonic-related papers to ground the article. SynSonic is a data augmentation method for Sound Event Detection (SED) that uses text-to-audio diffusion models, an energy-envelope ControlNet, and dual-classifier sample filtering to generate temporally coherent, class-accurate synthetic events for strongly labeled training mixtures (Hai et al., 23 Sep 2025). It was introduced to address a central limitation in SED: training strong-label detectors requires precise start/stop annotations for each event, which are expensive and time-consuming to obtain. Within the method, foreground events are generated by a text-to-audio diffusion transformer, filtered by complementary scoring models, and mixed with real backgrounds to form a new strongly labeled subset that is then integrated into a standard SED pipeline. In separate contexts, the term also appears in the EgoSonics literature as the synchronization essence of a video-to-audio diffusion system, and as a descriptive label for a synesthetic music-visualization concept (Rai et al., 2024, Adiletta et al., 2020).

1. Problem setting and motivation

SynSonic targets a core bottleneck in sound event detection: training strong-label detectors requires precise start/stop annotations for each event, which are expensive and time-consuming to obtain (Hai et al., 23 Sep 2025). Traditional augmentations such as SpecAugment, which masks time/frequency regions, and Mix-up, which linearly mixes existing clips, help diversification but remain bounded by the limited variability of the original dataset. This motivates the use of generative models, since text-to-audio systems can create new audio rather than only perturbing existing material.

The direct application of generative models to SED, however, is described as difficult for two specific reasons. First, text-to-audio generative models typically lack precise temporal control, so onsets and offsets in synthetic events are not well aligned. Second, naïve filtering of generated content, such as filtering with a single scoring model, can introduce noise or bias. SynSonic is designed specifically around these two failure modes: controllable generation addresses temporal alignment, and joint score filtering addresses sample quality.

The method is therefore not framed as a replacement for the existing SED stack. Instead, it is layered on top of mean-teacher, Mix-up, and SpecAugment baselines. This suggests that SynSonic is best understood as a temporally controlled generative augmentation layer for strongly labeled SED training rather than as a standalone event detector.

2. Pipeline and training integration

The SynSonic pipeline has four linked stages: prompt-to-audio generation with temporal control, dual-classifier filtering, labeled mixture synthesis, and integration into FDY-CRNN training batches (Hai et al., 23 Sep 2025). For each target class, a text prompt specifies the semantic class and an energy envelope encodes the desired onset, offset, duration, and intensity profile. Foreground events are then generated by EzAudio-L augmented with an energy-envelope ControlNet.

Each generated foreground sample is scored by two complementary models. CLAP measures text–audio semantic alignment through a similarity score, while Dasheng-AS provides an AudioSet-based classifier logit for the target class. For each class, SynSonic independently ranks samples by both metrics and combines the ranks through a weighted sum,

$$s_i = w \cdot R_{\mathrm{CLAP},i} + (1-w) \cdot R_{\mathrm{Cls},i},$$

where lower ranks are better and $w \in [0,1]$. Samples are sorted by $s_i$, and only the top $p\%$ per class are retained. Empirically, $w = 0.5$ and $p = 50\%$ yield the best results.

The retained foregrounds are mixed with real backgrounds using Scaper to produce strongly labeled mixtures. The onset and offset labels are inherited from the energy envelopes used during generation, forming the “SynSonic strong subset.” That subset is then integrated into the training batches of FDY-CRNN alongside real weak, real strong, real unlabeled, and the original synthetic strong subsets. The best-performing regime adds SynSonic on top of existing data without substantially reducing real data. Training follows the FDY-CRNN pipeline under Mean Teacher, Mix-up, and SpecAugment, and evaluation uses PSDS1 and PSDS2 with median-filtered outputs and window size 7.

3. Energy-envelope ControlNet and controllable generation

The generative backbone is EzAudio-L, used as a text-to-audio diffusion transformer, with a ControlNet built on top of EzAudio and trained or fine-tuned using AudioCaps (Hai et al., 23 Sep 2025). The ControlNet follows a standard design for diffusion transformers: the first half of the backbone’s blocks are duplicated into ControlNet blocks, and these ControlNet blocks connect via long skip connections to the latter half of the main transformer. The envelope pathway begins with a zero-initialized 1D convolution that processes the energy envelope and injects it into the ControlNet stream.

SynSonic uses waveform energy as a simple time-varying control signal to encode onsets, offsets, duration, and intensity profile. Although no explicit formula is given in the paper, the envelope is conceptually described by short-time energy,

$E(t) = \sum_{n \in \mathrm{window}(t)} x[n]^2.$

The envelope is aligned to the audio latent representation used by EzAudio-L. It passes through the zero-initialized 1D convolution and is then added to the ControlNet input, while the ControlNet’s long skip connections guide the latter half of the main transformer to enforce temporal coherence.

The reported outcome is a particular balance between timing precision and spectral diversity. Compared to unguided text-to-audio diffusion, which produces loose timing, and audio-to-audio diffusion, which retains original spectral characteristics and therefore has lower diversity, energy-envelope guidance provides both precise timing and high diversity. The paper states that the generated audio closely follows the envelope in intensity and time structure, with clear onsets and offsets, while preserving diversity in frequency content such as timbre and spectral texture.

Training and inference settings are explicitly specified. The ControlNet is trained on AudioCaps with batch size 16, learning rate $1 \times 10^{-5}$, and 10 epochs. Inference uses classifier-free guidance with scale 3.5 and 50 diffusion sampling steps. The paper does not provide explicit diffusion SDE or Markov chain formulas, instead stating that SynSonic adheres to the standard diffusion training and inference regimen used by EzAudio-L.

4. Filtering strategy and synthetic subset construction

A defining component of SynSonic is its rank-based dual-classifier filtering, which is introduced to mitigate single-model bias and reduce generative noise (Hai et al., 23 Sep 2025). CLAP can favor semantic alignment but miss acoustic misclassifications, while Dasheng-AS can catch class confidence but overlook prompt–audio mismatch. Because the two models operate on different scoring scales, SynSonic does not fuse raw scores; it converts each model’s output into per-class ranks before combining them.

The acceptance policy is controlled by the keep threshold $p$. The paper reports that lower thresholds, such as 75%, let too much noise through, whereas higher thresholds, such as 25%, cut diversity. The acceptance rate after filtering is 50%, meaning that half of the generated samples per class are retained in the best configuration. This retained set is then used to pre-generate a 10,000-sample “SynSonic strong subset,” which is mixed with real backgrounds through Scaper.

The design of this stage is important for the semantics of the labels. The synthetic foregrounds become strongly labeled not because the generator outputs annotations directly, but because the labels are inherited from the envelopes that specified the event timing during generation. This means that temporal annotation is externalized into the control signal rather than inferred post hoc from the generated waveform. A plausible implication is that SynSonic shifts the annotation problem from expensive manual event boundary labeling toward controlled synthesis with explicit timing specifications.

5. Experimental setting and quantitative results

The experiments are conducted on DESED with 10 domestic classes (Hai et al., 23 Sep 2025). The training splits are 1,578 real weakly labeled clips, 3,470 real strongly labeled clips, 10,000 synthetic strongly labeled clips from DCASE, and 14,412 real unlabeled clips, with a development set of 1,168 real clips. SynSonic adds 10,000 strongly labeled mixtures built from generated foregrounds and real backgrounds. The SED model is FDY-CRNN with Mean Teacher, Mix-up, and SpecAugment, and most hyperparameters follow the official FDY-SED repository. Evaluation uses PSDS1 and PSDS2, scores averaged across three random seeds, with a median filter window of 7 on predictions.

PSDS1 emphasizes temporal localization under operating points that prioritize accurate onset and offset timing, whereas PSDS2 emphasizes event class discrimination by balancing class-wise performance and false positives. SynSonic improves both.

Method	PSDS1	PSDS2
Baseline (no additional data)	0.4417	0.6639
Simple T2A diffusion	0.4507	0.6859
Audio-to-audio (A2A) diffusion	0.4556	0.6799
Energy-envelope ControlNet (SynSonic)	0.4641	0.6868

The paper interprets these numbers as showing that energy-envelope ControlNet delivers the best balance, improving timing and class discrimination. The reported temporal localization gain is from PSDS1 0.4417 to 0.4641, and the reported class-discrimination gain is from PSDS2 0.6639 to 0.6868.

Filtering ablations further isolate the contribution of joint ranking. CLAP-only filtering gives 0.4565 / 0.6750, the 0.7 / 0.3 weighting gives 0.4590 / 0.6748, the 0.5 / 0.5 weighting gives 0.4641 / 0.6868, the 0.3 / 0.7 weighting gives 0.4573 / 0.6874, AS-only gives 0.4574 / 0.6784, no filtering gives 0.4436 / 0.6690, Top 75% gives 0.4500 / 0.6776, and Top 25% gives 0.4589 / 0.6802. The paper’s interpretation is that joint ranking is critical and that equal weighting with a 50% keep threshold best balances quality and diversity.

Additional ablations concern how the generated material is used. For mixing strategy, “Generated foreground only” yields 0.4641 / 0.6868, “+ Real foreground” yields 0.4484 / 0.6826, and “+ Generated background” yields 0.4613 / 0.6866. The paper states that reusing known foregrounds hurts generalization and that synthetic backgrounds provide no clear benefit because real backgrounds are already diverse. For batch composition, the best PSDS1 is obtained by “+Gen. (add on top): 6/6/6/6/12 → 0.4641 / 0.6868,” where the reported ratios are SynSonic strong / synthetic strong / real strong / real weak / real unlabeled. The paper notes that adding SynSonic samples on top while retaining real data yields the strongest gains in temporal localization.

The reported results are averages over three seeds, but the paper does not include confidence intervals or formal significance tests. It also does not report per-class breakdowns, stating only that improvements are consistent across overall metrics.

6. Reproducibility, adoption, and limitations

The implementation details given for reproducibility are concise but specific (Hai et al., 23 Sep 2025). The architecture is EzAudio-L with ControlNet for generation and FDY-CRNN with Mean Teacher for SED. CLAP and Dasheng-AS official checkpoints are used for filtering. The ControlNet is trained on AudioCaps with batch size 16, learning rate $1 \times 10^{-5}$, and 10 epochs; inference uses classifier-free guidance 3.5 and 50 steps. Scores are averaged over three seeds, and code is released at https://github.com/JHU-LCAP/SynSonic.

Several practical recommendations are stated directly. Prompts should be concise and class-focused, such as “a single dog bark” or “vacuum cleaner noise in a room,” and long descriptive narratives should be avoided. Envelopes should be derived from desired onset, offset, and duration, aligned to the generator’s latent representation, and kept simple as single-event shapes with clear boundaries. Filtering should use equal weights between CLAP and classifier ranks with the top 50% per class retained. Mixing should prefer generated foreground plus real background, and SynSonic samples should be added on top of existing data rather than replacing real data.

The limitations are correspondingly concrete. Without ControlNet, unguided text-to-audio generation can produce loose temporal structures and unsmooth onsets and offsets, limiting PSDS1. Audio-to-audio diffusion preserves temporal pattern but leaks spectral characteristics, reducing diversity and limiting PSDS2 gains. Even with ControlNet, some generated samples can be off-class or unnatural, and dual-classifier filtering mitigates but does not eliminate all errors. The evaluation is restricted to DESED, so broader cross-dataset robustness is not reported. Compute requirements such as GPU or TPU type, training duration, and per-sample inference time are also not reported.

The future directions named in the paper follow directly from these limitations. Richer control signals beyond energy envelopes, such as time-varying multi-attribute controls, multi-event conditioning for controlled overlap and co-occurrence, quality estimation with multiple scoring models or uncertainty estimates, and direct generation of full strongly labeled mixtures from text plus structured temporal specifications are all identified as possible extensions.

7. Related uses of the term

The term “SynSonic” is not confined to the 2025 SED augmentation paper. In EgoSonics, “SynSonic” denotes the synchronization essence of a system for generating semantically meaningful and synchronized audio tracks conditioned on silent egocentric videos (Rai et al., 2024). There, a latent diffusion backbone is augmented with SyncroNet, a ControlNet-derived module that injects time-aware control signals extracted from video into the diffusion UNet to enforce alignment between visual events and audio onsets or transients. The task is formulated as conditional generation of audio encodings from video embeddings, and the method reports FID 41.14, IS 54.41, and VAAS 92.77 on 10-second Ego4D clips, outperforming prior video-to-audio baselines in both audio quality and synchronization. In that setting, “SynSonic” refers to synchronized multimodal generation rather than SED augmentation.

A further, looser usage appears in the paper “An Artistic Visualization of Music: Modeling a Synesthetic Experience,” where “SynSonic” is presented as an apt description of an audio-to-visual pipeline rooted in chromesthesia but is not the paper’s formal system name (Adiletta et al., 2020). That project uses audio input, FFT-based feature extraction, 12 analysis bands, volatility-based triggers, and GPU particle simulation in Max 8 to map sound to color, motion, and spatial structure. The paper explicitly states that the implementation does not carry the SynSonic name, but that the term aptly describes its essence as a synesthetic sonic visualization.

Taken together, these usages define a narrow and a broad sense of the term. In the narrow sense, SynSonic is the SED augmentation method built from EzAudio-L, an energy-envelope ControlNet, CLAP and Dasheng-AS filtering, Scaper mixture synthesis, and FDY-CRNN integration (Hai et al., 23 Sep 2025). In the broader sense, the term has been used for synchronization-centric multimodal generation and for synesthetic audiovisual mapping. The common thread across these contexts is controlled alignment between sound content and another structure: event timing in SED, visual timing in egocentric video, or perceptual color-motion structure in synesthetic visualization.