SynthCloner: Synthesizer Preset Conversion via Factorized Codec with ADSR Envelope Control

Abstract: Electronic synthesizer sounds are controlled by presets, parameters settings that yield complex timbral characteristics and ADSR envelopes, making preset conversion particularly challenging. Recent approaches to timbre transfer often rely on spectral objectives or implicit style matching, offering limited control over envelope shaping. Moreover, public synthesizer datasets rarely provide diverse coverage of timbres and ADSR envelopes. To address these gaps, we present SynthCloner, a factorized codec model that disentangles audio into three attributes: ADSR envelope, timbre, and content. This separation enables expressive synthesizer preset conversion with independent control over these three attributes. Additionally, we introduce SynthCAT, a new synthesizer dataset with a task-specific rendering pipeline covering 250 timbres, 120 ADSR envelopes, and 100 MIDI sequences. Experiments show that SynthCloner outperforms baselines on both objective and subjective metrics, while enabling independent attribute control. The code, model checkpoint, and audio examples are available at https://buffett0323.github.io/synthcloner/.

• The paper introduces a factorized codec model that disentangles audio into ADSR envelope, timbre, and content, enabling precise synthesizer preset conversion.
• The methodology employs perturbation-based disentanglement and attribute-specific encoders, validated by superior metrics over baseline models like SS-VAE and CTD.
• The study utilizes the SynthCAT dataset, comprising around 3 million audio samples, to demonstrate robust audio synthesis performance and independent attribute manipulation.

SynthCloner: Synthesizer Preset Conversion via Factorized Codec with ADSR Envelope Control

Introduction

The "SynthCloner" paper presents a novel approach for synthesizer preset conversion by leveraging a factorized codec model that disentangles synthesizer audio into distinct components of ADSR (Attack, Decay, Sustain, Release) envelope, timbre, and content. This approach addresses existing limitations in synthesizer preset manipulation, offering independent control over each attribute for precise audio synthesis. A key innovation in this study is the introduction of SynthCAT, a comprehensively designed dataset encompassing 250 timbres, 120 ADSR envelopes, and 100 MIDI sequences, facilitating extensive experimentation and evaluation.

Data Collection

SynthCAT was developed with the aim of providing broad coverage of timbral and envelope diversity, which are lacking in current public datasets. The dataset construction employed the Serum synthesizer, incorporating commercial presets to ensure varied audio renderings (Figure 1).

Figure 1: Data rendering pipeline of SynthCAT. A sustained-note segment is processed through pitch shifting, duration alignment, and ADSR envelope shaping.

The SynthCAT dataset includes combinations of pitch-shifted and duration-aligned samples, enriched by a diverse range of ADSR parameters. This constitutes a substantial dataset of approximately 3 million monophonic audio samples, which is invaluable for training and evaluating synthesizer preset conversion models.

SynthCloner Model Framework

SynthCloner, inspired by the FACodec architecture, introduces ADSR envelope modeling and a separate pathway for disentanglement of audio attributes, namely timbre, content, and the ADSR envelope (Figure 2).

Figure 2: SynthCloner model architecture. Synthesizer preset conversion is depicted by replacing $\mathbf{x}_\text{e}$ and $\mathbf{x}_\text{t}$ with reference audio inputs.

The model employs perturbation-based disentanglement, ensuring the separation of audio features through specific training regimes. For ADSR, content, and timbre embeddings, the system uses attribute-specific encoders with training focused on maintaining the integrity of each isolated aspect while perturbing non-target attributes.

Experiment and Results

SynthCloner was evaluated against baseline models, SS-VAE and CTD, demonstrating superior performance across objective metrics like MSTFT, LRMSD, and F0RMSE as well as subjective measures such as TMOS, ADSRMOS, and CMOS. The model's ability to faithfully replicate target timbres and ADSR envelopes while preserving the original content is noteworthy.

The incorporation of explicit ADSR envelope modeling, as evidenced by a decrease in performance when it is omitted, highlights its critical role in achieving high-fidelity preset conversion. SynthCloner's capacity for independent attribute manipulation was validated through controlled trials, reinforcing its applicability for precise audio synthesis tasks.

Conclusion

SynthCloner marks a significant advancement in the field of synthesizer preset conversion by integrating a factorized codec approach that efficiently separates and controls ADSR envelopes, timbre, and content. This model, underpinned by the extensive SynthCAT dataset, offers promising directions for future research, such as enhancing generalization capabilities and incorporating more complex synthesis behaviors including LFO modulations and intricate envelope configurations. The presented methodologies provide a robust framework for further developments in the nuanced control of synthesized audio outputs.