Neural MIDI Synthesis: M2A Model

• Neural MIDI synthesis is a data-driven approach that converts MIDI and tokenized note sequences into high-fidelity audio using advanced architectures like Transformers and DDPMs.
• The methodology leverages a two-stage process where symbolic inputs are first mapped to spectrograms and then inverted to audio through GAN or diffusion-based decoders.
• Key challenges include maintaining long-range musical coherence and mitigating artifacts, while current models show improved harmonic structure and expressivity.

Neural MIDI synthesis, denoted throughout recent literature as "MIDI-to-Audio" (M2A), refers to data-driven methodologies that generate high-fidelity musical audio directly from symbolic music representations such as MIDI, piano-rolls, or tokenized note sequences. These models encode compositional (what to play), performance (how to play), and instrument-timbral (what it sounds like) features, synthesizing audio that ranges from raw waveforms to spectrograms and token-based codec representations. This article surveys the state of the art in M2A, with particular attention to the model family based on encoder-decoder Transformers and Denoising Diffusion Probabilistic Models (DDPMs) as exemplified by "Multi-instrument Music Synthesis with Spectrogram Diffusion" (Hawthorne et al., 2022), situating them within the broader landscape of M2A architectures, evaluation practices, and practical integration.

1. Foundations and Input Representations

Neural M2A systems universally condition on structured symbolic inputs encoding musical notes, events, and metadata over (potentially variable) time. The most common symbolic representations include:

• Event-based MIDI Token Sequences: Discrete tokens for note-on, note-off, time-shift, program change, etc.
• Piano-Roll or Framewise Velocity Grids: Fixed-rate matrices of active notes or per-note velocities or onsets.
• Multi-feature Encodings: Octuple or higher-order event types including pitch, velocity, articulation, bar, etc.

In (Hawthorne et al., 2022), the input is a variable-length sequence $x = (x_1, \dots, x_N)$ with each $x_i\in \{1,\dots,V\}$ ($V\approx 1000$), spanning instrument programs, note-on/off, time-shift, drum, segment tie, and EOS events. Token embeddings, along with sinusoidal positional encodings, are fed into a standard Transformer encoder; segment-level positional encodings support long-range structural coherence. Context-awareness is achieved via encoding the prior spectrogram segment to stabilize instrument/instrumentation and stylistic continuity across long musical pieces.

2. Model Architectures: Transformer and Diffusion Paradigms

M2A synthesis models follow a two-stage paradigm:

1. Symbolic-to-Spectrogram Synthesis:
   • Transformers dominate as encoder-decoder architectures, mapping symbolic event sequences to framewise spectrograms (typically 128-bin log-mel spectrograms at 16 kHz).
   • The decoder attends jointly to symbolic encodings and a context spectrogram encoder, producing $T$ sequential spectrogram frames.
2. Spectrogram-to-Audio Inversion:
   • Generative Adversarial Network (GAN) Inverters reconstruct audio from spectrograms. Architectures typically involve multiple upsampling convolutional blocks and multi-scale waveform/STFT discriminators, with a compound loss blending $L_1$ spectral distance, adversarial hinge loss, and feature matching.
   • Autoregressive and Diffusion-based Decoders: In (Hawthorne et al., 2022), the decoder is either
     • An autoregressive Transformer with causal masking, or
     • A DDPM-based Transformer with bidirectional attention, where the network is trained to denoise progressively noised versions of the spectrogram.

The DDPM variant learns a mapping from noisy spectrogram embeddings $x_t = \alpha_t x + \sigma_t \varepsilon$ to the noise $\varepsilon$, minimizing

$$\mathcal{L}_{\rm DDPM} = \mathbb{E}_{x,\varepsilon,t}\;\bigl[w(t)\,\|\varepsilon_\theta(x_t,c,t)-\varepsilon\|_1\bigr].$$

Inference samples $x_1\sim \mathcal{N}(0,I)$ and recursively applies the learned reverse denoising operator to yield a clean spectrogram.

3. Training Regimes, Datasets, and Hyperparameters

Large-scale, multi-instrument paired datasets underpin modern M2A systems. The models in (Hawthorne et al., 2022) are trained on

• MAESTRO v3 (piano, $\sim$100K+ clips)
• Slakh2100 (MIDI+audio for pop/jazz/EDM)
• Cerberus4, Guitarset, MusicNet, URMP for additional instrument coverage

Instrument classes are clustered (e.g., 34 + drums), and training uses

• Adafactor optimizer, constant learning rates ($x_i\in \{1,\dots,V\}$0)
• Batches of $x_i\in \{1,\dots,V\}$1 sequences, up to $x_i\in \{1,\dots,V\}$2K steps
• TPUv4 or similar high-memory accelerators for efficient scaling

For DDPM, a cosine noise-schedule is adopted and classifier-free guidance is implemented by randomly dropping conditioning signals at train time, then amplifying conditionality at sample time. GAN inverter training employs $x_i\in \{1,\dots,V\}$3M steps of Adam on 128-clip batches from $x_i\in \{1,\dots,V\}$4K hours of music, blending spectral, adversarial, and feature-matching objectives.

4. Comparative Performance and Evaluation Metrics

Multiple objective and perceptual metrics are standard:

Metric	Definition / Utility
VGGish/TRILL recon	Embedding-space $x_i\in \{1,\dots,V\}$5 between reference and generated audio
Fréchet Audio Distance (FAD)	Distributional discrepancy in embedding space
Transcription F1	Accuracy of note/instrument token recovery from generated audio
Real-Time Factor (RT)	Audio generation speed; RT $x_i\in \{1,\dots,V\}$6 indicates real-time capability

Key results (Hawthorne et al., 2022), mean over six sets:

• Base DDPM + Context: VGGish Recon = 3.13 (lowest), FAD = 1.00, Transcr. F1 = 0.36, RT = $x_i\in \{1,\dots,V\}$7 (real-time or better)
• Small AR or DDPM: F1 $x_i\in \{1,\dots,V\}$8 0.31–0.36; DDPM exhibits crisper harmonics and eliminates the over-smoothed artifacts of AR, especially with segment context.
• GAN inverter contributes to final waveform fidelity, and the diffusion-based symbol-to-spec decoder outperforms AR for harmonic structure reproduction and segment cohesion.

MOS and perceptual tests in the broader M2A literature (Shi et al., 2022, Tang et al., 11 Jul 2025) confirm that adversarial training and end-to-end joint optimization yield highest subjective quality, with DDPM-based models leading in reconstruction and expressivity.

5. Interactive, Expressive Control and Integration

Neural M2A architectures allow segment-level interactive rendering:

• Flexible event editing: Users can modify MIDI events (add/remove notes, change instruments) and immediately re-render at the segment level.
• Long-sequence consistency: Segment-context encoding in (Hawthorne et al., 2022) ensures style and instrumental coherence across arbitrarily long renders.
• Generalization: Training over disparate datasets and with variable instrumentation enables robust synthesis of unseen styles and instrument combinations.

Hybrid models apply these pipelines to practical symbolic-to-audio tasks in composition, arrangement, and digital instrument rendering, supporting multi-instrument, orchestral, and pop arrangements.

6. Challenges, Limitations, and Future Directions

Several limitations persist:

• Autoregressive/AR models suffer from framewise blurring and slow sampling.
• GAN inverters can introduce mild artifacts, but in combination with DDPM decoders they provide state-of-the-art audio quality.
• Context conditioning is crucial; without prior spectrogram segments, models may "jump" mid-sequence in timbre or loudness.

Future research aims to:

• Further scale context and memory, enabling longer dependencies and richer acoustic modeling.
• Explore fine-grained expressivity, articulation, and timbral control at the segment or note level.
• Integrate differentiable DSP and explicit performance models for more interpretable and editable control (Wu et al., 2021, Jonason et al., 2023).

7. Position in the Literature and Comparative Models

The Transformer+DDPM ("spectrogram diffusion") M2A paradigm (Hawthorne et al., 2022) sits within a constellation of approaches:

Model Category	Core Methodology	Key Works
WaveNet/WaveGlow conditioning	AR waveform models w/ local MIDI cond.	(Hawthorne et al., 2018, Kim et al., 2018)
TTS-inspired spectrogram+GAN	Tacotron/Transformer + HiFi-GAN/NSF-GAN	(Shi et al., 2022, Cooper et al., 2021)
Codec-based Transformer	Token-based generation + neural codec	(Tang et al., 11 Jul 2025, Kim et al., 13 Feb 2025)
DDSP hierarchical synthesis	Neural/diff-DSP parameter inference	(Wu et al., 2021, Jonason et al., 2023)
Diffusion AR/NAF spectrograms	Event-to-spectrogram with DDPM, GAN inv.	(Hawthorne et al., 2022)

The flexibility and quality of the diffusion-based approach, as well as its capacity for generalization across instrument classes and styles, make it a cornerstone for current and future neural M2A research.

---

References:

• (Hawthorne et al., 2022)
• (Shi et al., 2022)
• (Tang et al., 11 Jul 2025)
• (Hawthorne et al., 2018)
• (Kim et al., 2018)
• (Cooper et al., 2021)
• (Wu et al., 2021)
• (Jonason et al., 2023)
• (Kim et al., 13 Feb 2025)
• (Tang et al., 17 Jan 2025)