MIDI-DDSP: Hierarchical Neural Audio Synthesis

• MIDI-DDSP is a hierarchical neural synthesis paradigm that fuses symbolic MIDI control with differentiable DSP to generate expressive and interpretable musical sound.
• It decomposes audio synthesis into note, performance, and synthesis parameter stages, enabling transparent intervention and fine-grained control.
• Leveraging deep neural networks and real-time DSP, MIDI-DDSP bridges traditional synthesis with modern machine learning for enhanced musical expressiveness.

MIDI-DDSP is a hierarchical neural synthesis paradigm for musical sound generation, combining symbolic MIDI control with differentiable digital signal processing (DDSP) to achieve expressive neural audio synthesis with interpretable and fine-grained performance parameters. It enables both high-level musical structure and low-level timbral nuance, leveraging the strengths of deep neural networks, digital audio signal modeling, and interpretable control. The MIDI-DDSP approach is developed to address the limitations of conventional synthesizers and black-box neural models by providing transparent, editable, and musically meaningful interfaces for control and intervention across multiple representational levels (Wu et al., 2021).

1. Architectural Decomposition and Information Flow

MIDI-DDSP factorizes the MIDI-to-audio task into three explicit stages: note (symbolic) representation, performance-level expressive control, and DDSP parameter synthesis:

1. Note Level: Input is a sequence of note events $n = \{n_1, ..., n_I\}$, each with pitch $p_i$, onset $t_{\text{on},i}$, and offset $t_{\text{off},i}$, forming the symbolic piano roll or MIDI input.
2. Performance Level: Each note is augmented with a learned or heuristically extracted vector of continuous expression controls $z_{p,i} \in \mathbb{R}^6$ (e.g., dynamics, vibrato, brightness, articulation), representing interpretive nuance.
3. Synthesis Parameter Level: For each frame, a set of DDSP parameters $\theta(t) = \{f_0(t), a(t), h(t), \eta(t)\}$ is predicted, directly driving a harmonic-plus-noise oscillator model. This synthesis stage is differentiable, enabling end-to-end learning (Wu et al., 2021).

This separation allows users or systems to intervene at any representational layer (e.g., edit MIDI, adjust performance, or manually modify synthesis parameters), providing a transparent workflow for audio generation and manipulation. The generative model is defined as: $$p(n,\,z_p,\,\theta,\,x)=p(n)\left[\prod_i p(z_{p,i}|n)\right] \left[\prod_t p(\theta(t)|z_p,n)\right] p(x|\theta)$$ and used for both inference (audio to controls) and synthesis (controls to audio).

2. DDSP-Based Neural Synthesis Methodology

DDSP-based synthesizers parameterize the waveform as the sum of harmonically structured signals and filtered noise: $$y(t)=\sum_{k=1}^K A_k(t)\,\sin\!\bigl(\phi_k(t)\bigr) + B(t)\,\text{noise}(t)$$ where $A_k(t)$ are harmonic amplitudes, $\phi_k(t)=2\pi k \int_0^t f_0(\tau)d\tau$ are instantaneous harmonic phases, and $B(t)$ is a band-limited noise envelope. The synthesis is further augmented by learned reverb, implemented as frequency-domain convolution with a trainable impulse response (Wu et al., 2021, Ganis et al., 2021, Jonason et al., 2023).

The DDSP parameter prediction network is typically a stack of recurrent (RNN) and convolutional layers; these are conditioned on continuous control signals, either inferred from audio (for analysis/synthesis tasks) or generated from symbolic input (for synthesis generation).

3. MIDI-to-Expressive Sound: Control Mechanisms and Mapping

MIDI-DDSP's control chain transforms symbolic or real-time MIDI input into interpretable control signals, enabling high-fidelity, expressive synthesis. The mapping comprises:

• Note-to-frequency mapping: MIDI note $N$ is mapped to $f_0=440\cdot 2^{(N-69)/12}$.
• Velocity-to-loudness mapping: MIDI velocity $V$ yields loudness as $ld=V/127$ or scaled to dB.
• Continuous controller mapping: MIDI CC values are linearly or nonlinearly mapped to global amplitude, pitch shift, harmonic stretch, noise color, and reverb mix parameters (Ganis et al., 2021).

MIDI-based input can drive full expressive synthesis by feeding note and velocity (or additional controls) through a neural control model that outputs frame-wise $f_0$, loudness, and other expressive features. These controls are further used by the DDSP parameter prediction module(s), and then rendered using the harmonic-plus-noise synthesizer.

Practical implementations support both full MIDI control (individual note events, CC mapping) and audio-based input modes, with inference conducted in real-time via efficient threading and optimized block-wise DSP (Ganis et al., 2021).

4. Learning Paradigms: Hierarchical, Classification, and End-to-End Strategies

Several training paradigms for MIDI-DDSP have emerged:

• Three-stage hierarchy (Wu et al., 2021): Separates note-to-performance (expression generator), performance-to-parameters (synthesis generator), and DSP synthesis. This affords isolated training, modular interpretability, and editable intervention at any layer.
• Control Model Variants (Jonason et al., 2023):
  • Regression: Predicts continuous $f_0$, loudness, periodicity, etc., using mean-squared error; however, prone to pitch averaging errors.
  • Classification: Quantizes controls into bins and trains using cross-entropy, improving pitch robustness and subjective quality (MIDI pitch accuracy $>0.9$ in best models).
  • Joint/Unified Training: End-to-end joint optimization of control front-end and synthesis back-end with losses combining cross-entropy (pitch), multi-scale spectral loss, and adversarial training (LSGAN). The unified architecture merges all stages, simplifying optimization and yielding the best overall fidelity (Jonason et al., 2023).

Key loss functions:

• Multi-scale spectral loss for waveform fidelity.
• Cross-entropy for discretized pitch/controls.
• LSGAN and feature-matching losses for adversarial enhancement.

Objective and subjective evaluations demonstrate that classification-based and unified training pipelines yield substantial improvements in both spectral loss and perceptual match to ground-truth recordings, outpacing regression controls and baseline samplers (Jonason et al., 2023).

5. Implementation Details and Real-Time Constraints

Practical MIDI-DDSP systems have been deployed as DAW plug-ins (via JUCE) and MATLAB/C++ real-time engines:

• Real-time inference: Decoder runs in a dedicated low-latency inference thread (TensorFlow C API), while synthesis operates at audio block rates (≤4096 samples), with phase accumulation and buffer management to prevent discontinuities (Ganis et al., 2021).
• Block-wise DSP: Additive, subtractive (filtered noise), and reverb modules run per audio block. MIDI and GUI controls are synchronized with DSP state.
• GUI and Automation: Modular single-page interfaces expose all synthesis and control parameters, allowing host automation of expressive macros (e.g., harmonic editor, noise color, MPE modulation). MIDI CC binding is implemented via JUCE parameter trees (Ganis et al., 2021).
• Latency: Measured round-trip latencies at 512-sample buffer sizes (44.1 kHz) are ~11.6 ms, supporting live applications without dropouts.

For polyphonic models, as in guitar synthesis, string-wise MIDI input is transformed using bidirectional LSTMs with self-attention and decoded per-string to harmonic/noise/reverb parameters, summed at the output (Jonason et al., 2023).

6. Evaluation, Limitations, and Future Directions

Objective and Subjective Evaluation

• Objective: Multi-scale spectral loss (MSSL), CREPE-based pitch accuracy (typically $>0.89$ in classification/unified models), and note-level RMSE for expression controls.
• Subjective: Mean opinion scores (MOS) on realism, naturalness, and pitch appropriateness place the best MIDI-DDSP systems above alternative neural models and comparable to commercial sample-based libraries, though still lagging in absolute fidelity (Jonason et al., 2023, Wu et al., 2021).
• User Interface Feedback: GUI design is rated highly for appeal, although some features (e.g., harmonic editor) demand clearer affordances (Ganis et al., 2021).

Limitations

• Current MIDI-DDSP models are most robust for monophonic and string-wise polyphonic sources; full polyphonic instrument modeling requires accurate multi-pitch tracking.
• Expression controls are currently hand-specified; future work aims at learning these controls end-to-end (e.g., variational inference on DDSP parameters) (Wu et al., 2021).
• Adversarial training improves details but can add noise; spectral losses alone may smooth critical textures.

Future Research Directions

• Extension to multi-instrument mixtures, singing voice, and more complex timbral modeling.
• Improved low-latency, multi-block context inference for better temporal coherence in real-time synthesis.
• End-to-end learning of expression, synthesis, and note representations within unified architectures.

7. Comparative Landscape and Significance

MIDI-DDSP bridges a key gap between black-box neural audio synthesis and traditional, highly controllable DSP-based synthesizers. The hierarchical separation and explicit mapping from MIDI through expressive controls to DDSP synthesis enables:

• Transparent, editable, and interpretable interfaces, allowing artistic or automatic intervention at every stage.
• State-of-the-art perceptual and objective audio quality with the capacity for nuanced expressive variation.
• Real-time, plug-in integration in digital audio production workflows, expanding the creative and research toolkit for musicians and engineers.

The approach has informed subsequent work in neural instrument modeling, expressive control, and DAW-integrable neural synthesis, fostering a new generation of musically meaningful machine learning audio tools (Wu et al., 2021, Ganis et al., 2021, Jonason et al., 2023).