Automatic Synthesizer Programming (ASP)

• ASP is the automatic inference of synthesizer controls from sound specifications, enabling parameter matching from audio or text prompts.
• Deep learning models, including transformer-based systems and neural proxies, are used to tackle challenges such as non-differentiability and complex architecture integration.
• Recent advancements incorporate evolutionary search, iterative programming, and differentiable synthesis to improve parameter prediction and perceptual sound quality.

Searching arXiv for papers on automatic synthesizer programming and synthesizer sound matching. Automatic Synthesizer Programming (ASP) denotes the automatic inference of synthesizer controls from a specification of the desired sound. In synthesizer sound matching, systems automatically set the parameters of a synthesizer to emulate an input sound; in text-conditioned settings, ASP refers to inferring the set of synthesizer parameters that, when rendered, result in audio semantically aligned with a given text prompt (Bruford et al., 2024, Cherep et al., 2024). Across these formulations, the underlying motivation is consistent: the enormous variety of synthesizers in the marketplace, and the complexity of many of them, make general-purpose systems that function with minimal knowledge or prior assumptions about the underlying synthesis architecture particularly desirable (Bruford et al., 2024). At the same time, deep learning appears as an appealing solution for ASP, but integrating software synthesizers into training pipelines is challenging due to their potential non-differentiability (Combes et al., 9 Sep 2025).

1. Problem formulations and representational assumptions

ASP research in music technology has centered on several distinct but related formulations. The most direct is synthesizer sound matching, in which a model predicts synthesizer parameters from audio so that the synthesizer can reconstruct the target sound (Bruford et al., 2024). A second formulation uses target audio only indirectly during training: a model predicts parameters, a synthesizer or a learned proxy evaluates the result, and the optimization objective is defined in a perceptually informed embedding space rather than only in parameter space (Combes et al., 9 Sep 2025). A third formulation replaces the target audio with a text prompt and optimizes synthesizer parameters so that synthesized audio aligns with the prompt in a joint audio-text embedding space (Cherep et al., 2024). A fourth formulation is explicitly iterative and pedagogical: SerumRNN provides step-by-step instructions for applying audio effects to change a user’s input audio towards a desired sound, rather than producing a single one-shot preset estimate (Mitcheltree et al., 2021).

These formulations differ in what they assume about the synthesizer. The AST-based sound matching model is designed to remove the need for knowledge of or differentiability in the synthesizer’s internal architecture (Bruford et al., 2024). NAS-FM instead adopts neural architecture search to build a differentiable frequency modulation synthesizer, thereby extending ASP from parameter inference to automatic construction of the synthesizer’s structure itself (Ye et al., 2023). The neural proxy line of work occupies an intermediate position: arbitrary synthesizers are treated as black boxes during rendering, but a preset encoder is trained as a differentiable proxy that maps presets directly to a pretrained audio embedding space (Combes et al., 9 Sep 2025). This partition between black-box inference, differentiable synthesis, and gradient-free optimization is one of the defining structural features of the field.

A recurring misconception is that high-quality ASP requires direct differentiability of the target synthesizer. The recent literature does not support that view. The AST model explicitly removes the need for knowledge of or differentiability in the synthesizer’s internal architecture; CTAG uses a gradient-free optimization strategy due to unstable gradients in the synthesizer; and neural proxies are introduced precisely to enable audio embedding loss for black-box synthesizers (Bruford et al., 2024, Cherep et al., 2024, Combes et al., 9 Sep 2025).

2. Direct regression from audio to synthesizer parameters

A representative example of direct parameter estimation is the Audio Spectrogram Transformer model for synthesizer sound matching (Bruford et al., 2024). The input audio is converted to a 64-bin Mel spectrogram, which is split into $16 \times 16$ patches with stride of 6; each patch is flattened and mapped to a 768-dimensional embedding vector, learnable positional embeddings are added, and a 12-layer transformer encoder with 12 attention heads and hidden dimension 768 processes the sequence. Instead of a classification head, the model ends with a three-layer MLP, each layer width 768, mapping the transformer output to synthesizer parameter predictions. Mean Squared Error is used between predicted and target parameters, and no external pretraining is used (Bruford et al., 2024).

The training data were generated from Native Instruments’ Massive synthesizer. The model reconstructs parameters of samples generated from a set of 16 parameters. These parameter sets were rendered as one-shot audio of 4 seconds with 1 second note-on at C4 and maximum velocity; samples with low loudness $(<-60\mathrm{dB})$ were discarded; and the resulting dataset comprised 1 million parameter-audio pairs, 22GB total, generated automatically via Python scripting and Pedalboard (Bruford et al., 2024).

The reported baselines were a 5-layer MLP with 2048 units per layer, batch norm, and dropout, and a 5-layer 2D CNN with 128 channels, kernel size 7 and stride 2, followed by a 5-layer MLP. On a test set of 10,000 examples, the transformer improved both parameter and perceptual reconstruction metrics (Bruford et al., 2024).

Model	Parameter MSE	Spectral Convergence
MLP	0.077	4.608
CNN	0.094	5.372
AST	0.031	0.616

The same model was also evaluated out of domain on vocal imitation, sounds from other synthesizers, and various musical instrument one-shots. The reported observation is that it is capable of capturing timbre and envelope aspects of arbitrary input sounds, while noted failure modes are mainly in oscillator pitch estimation and detuning, where small parameter errors can cause perceptually large differences (Bruford et al., 2024). This suggests that direct regression can already support query-by-example and query-by-vocalization workflows, but pitch-sensitive controls remain a disproportionately difficult subproblem.

3. Differentiable synthesis and automatic architecture search

NAS-FM extends ASP beyond preset estimation by searching over the structure of a frequency modulation synthesizer itself (Ye et al., 2023). The basic FM module is defined by

$y(t) = a(t) \sin(2\pi f_c t + I\sin(2\pi f_m t)),$

where $a(t)$ is the amplitude envelope, $f_c$ and $f_m$ are carrier and modulator frequencies, and $I$ is the modulation index. More complex FM algorithms are represented as directed acyclic graphs with oscillator outputs

$$x^{(i)}(t) = a_i(t) \sin \left(2\pi f_i t + \sum_{j \in \mathbb{M}} x^{(j)}(t) \right),$$

and full synthesizer output

$y(t) = \sum_{i \in \mathbb{C}} x^{(i)}(t).$

This DAG parameterization is central because it makes architecture search over single, nested, double, and formant FM-like organizations technically tractable (Ye et al., 2023).

The search procedure has two components. First, a supernet is trained with a specifically designed search space, including predicting the envelopes of carriers and modulators with different frequency ratios. The search space is organized into a carrier layer and one or more modulator layers, and an envelope-sharing strategy ties envelopes among oscillators within a layer that share a frequency ratio. Pitch and loudness are extracted from real audio using CREPE and A-weighted loudness estimators, and a Temporal Convolutional Network predicts oscillator envelopes. Proxy oscillators are used to reduce the training burden during supernet training (Ye et al., 2023).

Second, an evolutionary search algorithm with adaptive oscillator size is developed to find the optimal relationship between oscillators and the frequency ratio of FM. Each candidate architecture is encoded as

$$\left\{f_{ol_1}, f_{ol_2}, ..., f_{ol_N};\ l_{ol_1}, l_{ol_2}, ..., l_{ol_N}\right\},$$

and fitness is measured by Fréchet Audio Distance:

$(<-60\mathrm{dB})$0

The reported setup used a six-oscillator synthesizer, an evolutionary search population size of 1000, and max iterations of 50 (Ye et al., 2023).

On URMP recordings of flute, violin, and trumpet, NAS-FM was compared with DDX7, a differentiable FM synthesizer with expert-designed FM algorithms and parameter search only (Ye et al., 2023).

Instrument	DDX7	NAS-FM
Flute	7.841	7.077
Violin	3.497	3.255
Trumpet	4.442	3.384

The paper’s interpretation is that NAS-FM can build a synthesizer fully automatically and achieves better results than handcrafted synthesizers (Ye et al., 2023). A plausible implication is that, in FM settings, architecture search is not merely a compression of manual design practice; it changes the ASP problem from parameter inversion into joint structure-and-parameter synthesis.

4. Neural proxies, preset encoders, and black-box synthesizers

Neural proxy methods address a specific limitation of many ASP pipelines: software synthesizers are often black boxes, and parameter-space losses do not correlate well with perceptual similarity (Combes et al., 9 Sep 2025). The proposed solution is to train a preset encoder $(<-60\mathrm{dB})$1 that maps a preset $(<-60\mathrm{dB})$2 to an audio embedding space defined by a pretrained model $(<-60\mathrm{dB})$3. The preset encoder is trained so that its output matches the pretrained model’s embedding of the audio rendered by the preset, using an $(<-60\mathrm{dB})$4 objective that was empirically superior for audio embeddings (Combes et al., 9 Sep 2025).

This formulation is explicitly evaluated with synthetic and hand-crafted presets from three popular software synthesizers: Dexed, Diva, and TAL-NoiseMaker. The pretrained audio models benchmarked include EfficientAT, PaSST, AudioMAE, OpenL3, CLAP, DAC, and M2L. The reported finding is that EfficientAT offered the best overall correlation with perceptually relevant sound attributes, and that hybrid models combining hand-crafted and learned features provided compact and effective representations (Combes et al., 9 Sep 2025).

Several preset encoder architectures were compared: a shallow feedforward MLP with one-hot parameter encoding; deep highway networks with one-hot or learnable token embeddings; a bidirectional GRU; and a transformer-based model. The transformer-based preset encoder outperformed all others, especially on more complex synthesizers such as Dexed and Diva, in both Mean Reciprocal Rank and embedding error. Recurrent and highway architectures were also competitive, whereas simpler MLPs and architectures with bottlenecks underperformed, especially on complex synthesizers (Combes et al., 9 Sep 2025).

The downstream use case is synthesizer sound matching. A sound-to-parameter estimator is trained with both parameter loss and audio embedding loss, where the embedding term is provided by the preset encoder rather than by direct differentiation through the synthesizer. The reported outcome is better perceptual similarity in matched sounds, especially for out-of-domain audio targets. At the same time, all preset encoders showed a drop in performance on real hand-crafted presets versus synthetic presets, and UMAP visualizations showed that hand-crafted and synthetic presets occupy different regions in embedding space, particularly for complex FM synthesizers (Combes et al., 9 Sep 2025). This suggests that synthetic-data scale alone does not eliminate the domain mismatch between procedurally sampled presets and musically curated preset libraries.

5. Iterative and text-conditioned programming

Two influential extensions of ASP move beyond one-shot sound matching: iterative instruction generation and text-to-audio synthesizer programming. SerumRNN addresses Xfer Records Serum, a complex VST synthesizer used in audio production, and frames the task as an iterative sequence of effect applications (Mitcheltree et al., 2021). The system is an ensemble of neural network models with an effect selection model and one effect parameter model per effect. Inputs include Mel spectrograms with 128 bands and MFCCs using the first 30 coefficients, computed from 1-second audio at 44.1kHz with STFT window 2048 and hop 512. The training corpus comprised 1,536,000 audio clips generated from 12 Serum presets in three complexity groups and five effects: compressor, distortion, equalizer, phaser, and hall reverb (Mitcheltree et al., 2021).

The reported behavior of SerumRNN is explicitly sequential. At each step, the effect selection model predicts the next effect, the corresponding parameter model predicts settings, Serum processes the audio, and the loop continues until no further improvement is detected. The paper reports stepwise accuracy for effect selection greater than 98%, error reduction between input and target audio by up to approximately 80%, and inference speed of approximately 300ms per effect step (Mitcheltree et al., 2021). The authors further report that SerumRNN learns to prioritize effects and can discover more efficient effect order sequences than a variety of baselines (Mitcheltree et al., 2021). In ASP terms, this converts preset programming into a policy over interpretable editing actions.

CTAG generalizes the target specification from audio to language (Cherep et al., 2024). Its synthesizer is implemented via SynthAX in JAX and uses a virtual modular architecture with only 78 continuous-valued parameters, including 2 Low-Frequency Oscillators, 6 ADSR envelopes, a sine VCO, a square-saw VCO with a shape parameter, a noise generator, VCAs, and modulation and audio mixers. LAION-CLAP provides a shared embedding space for text and audio, and the optimization objective is

$(<-60\mathrm{dB})$5

The optimization loop is gradient-free and proceeds by generating candidate parameter sets, synthesizing audio, computing CLAP similarity to the prompt, and updating the optimizer. Learned Evolution Strategies outperformed other tested optimizers in ablation (Cherep et al., 2024).

CTAG’s evaluation emphasizes recognizability and artistic character rather than literal waveform reconstruction. In a user study, reported prompt identification accuracy was 56% for CTAG, 59.5% for AudioGen, and 34% for AudioLDM, while artistic rating on a 1–5 scale was 3.54 for CTAG, 2.32 for AudioGen, and 2.90 for AudioLDM (Cherep et al., 2024). The paper characterizes CTAG outputs as more abstract, capturing essential conceptual features over fine-grained acoustic details, and positions synthesizer programming as a complementary tool to neural audio synthesis rather than a replacement (Cherep et al., 2024).

6. Evaluation criteria, open technical issues, and terminological ambiguity

The evaluation landscape of ASP is notably heterogeneous. Direct sound matching work reports parameter MSE and Spectral Convergence (Bruford et al., 2024). NAS-FM uses Fréchet Audio Distance as the search objective and evaluation metric (Ye et al., 2023). Neural proxy work adds Mean Reciprocal Rank, $(<-60\mathrm{dB})$6 embedding error, and audio similarity metrics such as STFT, Mel distance, and MFCC (Combes et al., 9 Sep 2025). CTAG relies on CLAP similarity, classifier-based recognizability, spectral audio features, and human evaluation of identifiability and artistic interpretativeness (Cherep et al., 2024). SerumRNN evaluates effect programming with MSE, MAE, Log-Spectral Distance, MFCCD, Pearson correlation, and Multi-Scale Spectral Mean Absolute Error (Mitcheltree et al., 2021). This suggests that ASP is not yet evaluated by a single canonical criterion; metrics depend strongly on whether the goal is preset recovery, perceptual similarity, semantic alignment, or interpretable editing.

Several technical limitations recur across the literature. AST-based matching identifies failure modes mainly in oscillator pitch estimation and detuning (Bruford et al., 2024). The neural proxy work reports a generalization gap between synthetic and hand-crafted presets, especially for Dexed (Combes et al., 9 Sep 2025). The transformer-based sound matching paper identifies extension to categorical and discrete parameters, modeling parameter interdependencies such as modulations and routing, and more robust pitch and musical content awareness as future directions (Bruford et al., 2024). CTAG adopts a gradient-free strategy because gradients in the synthesizer are unstable (Cherep et al., 2024). NAS-FM is restricted to the FM family, though it makes that family tunable and interpretable without prior expert knowledge (Ye et al., 2023). Across all of these systems, training data generation remains central: 1 million Massive examples for AST, millions of random presets for neural proxies, and 1,536,000 rendered Serum clips for SerumRNN (Bruford et al., 2024, Combes et al., 9 Sep 2025, Mitcheltree et al., 2021).

A separate source of confusion is terminological. In a different research tradition, “ASP” denotes Answer Set Programming rather than Automatic Synthesizer Programming. Papers on generating imperative programs from Answer Set Programming specifications (Varanasi, 2019), composing ASP code from natural language specifications (Borroto et al., 2024), debugging and repair of ASP submissions (Brancas et al., 2024), deriving synchronization code with ASP (Varanasi et al., 2023), and pointer data structure synthesis from ASP specifications (Varanasi et al., 2020) concern logic programming rather than musical synthesizers. This suggests that literature searches for “ASP” require explicit disambiguation.

Within music technology, however, the term has converged on a family of inverse and search problems over interpretable synthesizer controls. The field now includes direct transformer regression, differentiable FM architecture search, neural proxies for black-box synthesizers, iterative effect programming, and text-to-audio optimization in shared embedding spaces (Bruford et al., 2024, Ye et al., 2023, Combes et al., 9 Sep 2025, Mitcheltree et al., 2021, Cherep et al., 2024). What unifies these otherwise different systems is the same core objective: to make synthesizer programming automatic while preserving the modular, inspectable, and editable structure that has long distinguished synthesizers from large latent generative models.