---
title: 'MIDI-DDSP: Hierarchical Neural Audio Synthesis'
url: https://www.emergentmind.com/topics/midi-ddsp
type: topic
---

# MIDI-DDSP: Hierarchical Neural Audio Synthesis

MIDI-DDSP is a hierarchical neural synthesis paradigm for musical sound generation, combining symbolic MIDI control with differentiable digital signal processing (DDSP) to achieve expressive neural audio synthesis with interpretable and fine-grained performance parameters. It enables both high-level musical structure and low-level timbral nuance, leveraging the strengths of deep neural networks, digital audio signal modeling, and interpretable control. The MIDI-DDSP approach is developed to address the limitations of conventional synthesizers and black-box neural models by providing transparent, editable, and musically meaningful interfaces for control and intervention across multiple representational levels [2112.09312].

## 1. Architectural Decomposition and Information Flow

MIDI-DDSP factorizes the MIDI-to-audio task into three explicit stages: note (symbolic) representation, performance-level expressive control, and DDSP parameter synthesis:

1. **Note Level**: Input is a sequence of note events $n = \{n_1, ..., n_I\}$, each with pitch $p_i$, onset $t_{\text{on},i}$, and offset $t_{\text{off},i}$, forming the symbolic piano roll or MIDI input.
2. **Performance Level**: Each note is augmented with a learned or heuristically extracted vector of continuous expression controls $z_{p,i} \in \mathbb{R}^6$ (e.g., dynamics, vibrato, brightness, articulation), representing interpretive nuance.
3. **Synthesis Parameter Level**: For each frame, a set of DDSP parameters $\theta(t) = \{f_0(t), a(t), h(t), \eta(t)\}$ is predicted, directly driving a harmonic-plus-noise oscillator model. This synthesis stage is differentiable, enabling end-to-end learning [2112.09312].

This separation allows users or systems to intervene at any representational layer (e.g., edit MIDI, adjust performance, or manually modify synthesis parameters), providing a transparent workflow for audio generation and manipulation. The generative model is defined as:
\[
p(n,\,z_p,\,\theta,\,x)=p(n)\left[\prod_i p(z_{p,i}|n)\right] \left[\prod_t p(\theta(t)|z_p,n)\right] p(x|\theta)
\]
and used for both inference (audio to controls) and synthesis (controls to audio).

## 2. DDSP-Based Neural Synthesis Methodology

DDSP-based synthesizers parameterize the waveform as the sum of harmonically structured signals and filtered noise:
\[
y(t)=\sum_{k=1}^K A_k(t)\,\sin\!\bigl(\phi_k(t)\bigr) + B(t)\,\text{noise}(t)
\]
where $A_k(t)$ are harmonic amplitudes, $\phi_k(t)=2\pi k \int_0^t f_0(\tau)d\tau$ are instantaneous harmonic phases, and $B(t)$ is a band-limited noise envelope. The synthesis is further augmented by learned reverb, implemented as frequency-domain convolution with a trainable impulse response [2112.09312, 2103.07220, 2309.07658].

The DDSP parameter prediction network is typically a stack of recurrent (RNN) and convolutional layers; these are conditioned on continuous control signals, either inferred from audio (for analysis/synthesis tasks) or generated from symbolic input (for synthesis generation).

## 3. MIDI-to-Expressive Sound: Control Mechanisms and Mapping

MIDI-DDSP's control chain transforms symbolic or real-time MIDI input into interpretable control signals, enabling high-fidelity, expressive synthesis. The mapping comprises:

- **Note-to-frequency mapping**: MIDI note $N$ is mapped to $f_0=440\cdot 2^{(N-69)/12}$.
- **Velocity-to-loudness mapping**: MIDI velocity $V$ yields loudness as $ld=V/127$ or scaled to dB.
- **Continuous controller mapping**: MIDI CC values are linearly or nonlinearly mapped to global amplitude, pitch shift, harmonic stretch, noise color, and reverb mix parameters [2103.07220].

MIDI-based input can drive full expressive synthesis by feeding note and velocity (or additional controls) through a neural control model that outputs frame-wise $f_0$, loudness, and other expressive features. These controls are further used by the DDSP parameter prediction module(s), and then rendered using the harmonic-plus-noise synthesizer.

Practical implementations support both full MIDI control (individual note events, CC mapping) and audio-based input modes, with inference conducted in real-time via efficient threading and optimized block-wise DSP [2103.07220].

## 4. Learning Paradigms: Hierarchical, Classification, and End-to-End Strategies

Several training paradigms for MIDI-DDSP have emerged:

- **Three-stage hierarchy** ([2112.09312]): Separates note-to-performance (expression generator), performance-to-parameters (synthesis generator), and DSP synthesis. This affords isolated training, modular interpretability, and editable intervention at any layer.
- **Control Model Variants** ([2309.07658]):
  - *Regression*: Predicts continuous $f_0$, loudness, periodicity, etc., using mean-squared error; however, prone to pitch averaging errors.
  - *Classification*: Quantizes controls into bins and trains using cross-entropy, improving pitch robustness and subjective quality (MIDI pitch accuracy $>0.9$ in best models).
  - *Joint/Unified Training*: End-to-end joint optimization of control front-end and synthesis back-end with losses combining cross-entropy (pitch), multi-scale spectral loss, and adversarial training (LSGAN). The unified architecture merges all stages, simplifying optimization and yielding the best overall fidelity [2309.07658].
  
Key loss functions:
- Multi-scale spectral loss for waveform fidelity.
- Cross-entropy for discretized pitch/controls.
- LSGAN and feature-matching losses for adversarial enhancement.

Objective and subjective evaluations demonstrate that classification-based and unified training pipelines yield substantial improvements in both spectral loss and perceptual match to ground-truth recordings, outpacing regression controls and baseline samplers [2309.07658].

## 5. Implementation Details and Real-Time Constraints

Practical MIDI-DDSP systems have been deployed as DAW plug-ins (via JUCE) and MATLAB/C++ real-time engines:

- **Real-time inference**: Decoder runs in a dedicated low-latency inference thread (TensorFlow C API), while synthesis operates at audio block rates (≤4096 samples), with phase accumulation and buffer management to prevent discontinuities [2103.07220].
- **Block-wise DSP**: Additive, subtractive (filtered noise), and reverb modules run per audio block. MIDI and GUI controls are synchronized with DSP state.
- **GUI and Automation**: Modular single-page interfaces expose all synthesis and control parameters, allowing host automation of expressive macros (e.g., harmonic editor, noise color, MPE modulation). MIDI CC binding is implemented via JUCE parameter trees [2103.07220].
- **Latency**: Measured round-trip latencies at 512-sample buffer sizes (44.1 kHz) are ~11.6 ms, supporting live applications without dropouts.

For polyphonic models, as in guitar synthesis, string-wise MIDI input is transformed using bidirectional LSTMs with self-attention and decoded per-string to harmonic/noise/reverb parameters, summed at the output [2309.07658].

## 6. Evaluation, Limitations, and Future Directions

### Objective and Subjective Evaluation

- **Objective**: Multi-scale spectral loss (MSSL), CREPE-based pitch accuracy (typically $>0.89$ in classification/unified models), and note-level RMSE for expression controls.
- **Subjective**: Mean opinion scores (MOS) on realism, naturalness, and pitch appropriateness place the best MIDI-DDSP systems above alternative neural models and comparable to commercial sample-based libraries, though still lagging in absolute fidelity [2309.07658, 2112.09312].
- **User Interface Feedback**: GUI design is rated highly for appeal, although some features (e.g., harmonic editor) demand clearer affordances [2103.07220].

### Limitations

- Current MIDI-DDSP models are most robust for monophonic and string-wise polyphonic sources; full polyphonic instrument modeling requires accurate multi-pitch tracking.
- Expression controls are currently hand-specified; future work aims at learning these controls end-to-end (e.g., variational inference on DDSP parameters) [2112.09312].
- Adversarial training improves details but can add noise; spectral losses alone may smooth critical textures.

### Future Research Directions

- Extension to multi-instrument mixtures, singing voice, and more complex timbral modeling.
- Improved low-latency, multi-block context inference for better temporal coherence in real-time synthesis.
- End-to-end learning of expression, synthesis, and note representations within unified architectures.

## 7. Comparative Landscape and Significance

MIDI-DDSP bridges a key gap between black-box neural audio synthesis and traditional, highly controllable DSP-based synthesizers. The hierarchical separation and explicit mapping from MIDI through expressive controls to DDSP synthesis enables:

- Transparent, editable, and interpretable interfaces, allowing artistic or automatic intervention at every stage.
- State-of-the-art perceptual and objective audio quality with the capacity for nuanced expressive variation.
- Real-time, plug-in integration in digital audio production workflows, expanding the creative and research toolkit for musicians and engineers.

The approach has informed subsequent work in neural instrument modeling, expressive control, and DAW-integrable neural synthesis, fostering a new generation of musically meaningful machine learning audio tools [2112.09312, 2103.07220, 2309.07658].

Source: https://www.emergentmind.com/topics/midi-ddsp