NES-MDB: NES Music Dataset

• NES-MDB is a comprehensive dataset for NES game music, capturing both compositional structure and low-level expressive parameters.
• It encodes four distinct monophonic voices from NES games, enabling isolated and blended score analyses for polyphonic modeling.
• The dataset supports benchmark evaluations using deep learning models, with clear metrics on negative log-likelihood and accuracy.

The Nintendo Entertainment System Music Database (NES-MDB) is a large-scale, multi-instrumental symbolic music dataset designed to support machine learning research on both polyphonic composition and expressive performance modeling. NES-MDB is distinguished by its preservation of both compositional structure and all low-level expressive parameters necessary to exactly reproduce original game soundtrack performances on authentic NES audio hardware. It serves as a canonical corpus for developing and benchmarking algorithms that address the unique compositional constraints and fine-grained expressive detail of NES-era video game music, offering unprecedented opportunities for symbolic music modeling and music information retrieval (Donahue et al., 2018).

1. Motivation and Design Rationale

Early machine learning approaches to music generation predominantly targeted composition—learning distributions over symbolic scores—while disregarding performance attributes such as velocity, articulation, and timbral nuance. Existing datasets like Piano-midi.de and Disklavier corpora capture some expressive details but are limited to solo piano, and multi-instrumental datasets with performance controls are rare. The NES’s Audio Processing Unit (APU), with its rigid hardware constraints (two pulse, one triangle, one noise voice; all monophonic), encodes exact playback semantics in game code as register writes, enabling recovery of not only the score but also the precise evolution of each voice’s dynamics and timbre. NES-MDB leverages these properties to enable systematically decoupled study of separated composition (each voice as an independent sequence) and expressive performance mapping (Donahue et al., 2018).

2. Corpus Structure and Representations

NES-MDB comprises 5,278 multi-instrumental pieces from 397 NES games (authored by 296 unique composers), accumulating ∼2.3 million note events and 46.1 hours of annotated music. Each piece encodes four monophonic voices with explicit score and expressive attributes:

• Pulse 1 & Pulse 2: MIDI pitch range {32,34,…,108}+rest, dynamics [0…15], timbre (duty cycle) [0…3]
• Triangle: MIDI pitch range {21,22,…,108}+rest, on/off dynamics, binary timbre
• Noise: 16 noise pitches+rest, dynamics [0…15], timbre [0,1]

For each timestep, an expressive score specifies $(N_{v,t}, V_{v,t}, T_{v,t})$ per voice. The separated score retains only the pitch matrix, and the blended score encodes a binary time–pitch activation regardless of voice assignment. Raw VGM (Video Game Music) logs at 44.1 kHz are translated into symbolic form via NES APU emulation, then downsampled to 24 Hz for computational tractability. The average ensemble polyphony is 2.79 (Donahue et al., 2018).

The dataset is stratified into train/validation/test sets (8:1:1 ratio) by game, ensuring no composer exists in multiple splits. All tooling and data are distributed under a permissive license (Donahue et al., 2018).

Statistic	Value	Notes
Total games	397	296 composers
Total pieces	5,278	3,513 longer than 10s
Note events	≈2.3 million	46.1 hours total
Temporal sampling	24 Hz	Downsampled from VGM logs
Polyphony avg.	2.79 voices @ timestep	Four total possible (Pulse1, Pulse2, Triangle, Noise)

3. Expressive Control and Symbolic-Audio Interface

Unlike standard MIDI datasets (e.g., Lakh MIDI), NES-MDB encodes all information required for acoustic fidelity via an emulation utility. This command-line tool can:

• Convert VGM files to expressive scores (vgm_to_nesmdb)
• Render generated symbolic music back into NES-style audio (nesmdb_render)
• Convert to standard MIDI for downstream analysis

The emulation reconstructs instantaneous APU states, parsing pitch, velocity, and timbre events into symbolic sequences and supporting full cycle (symbolic–audio–symbolic) research workflows (Donahue et al., 2018).

4. Supported Modeling Tasks and Mathematical Framework

NES-MDB explicitly separates three principal learning tasks:

A. Blended Composition: Learning the distribution $P(B_1,…,B_T)$ over blended scores via the auto-regressive model

$P(c) = \prod_{t=1}^T P(B_t | B_{<t})$

B. Separated Composition: Learning $P(S_1,…,S_T)$ for separated scores, factoring by voice and history,

$$P(c) = \prod_{t=1}^T \prod_{v=1}^V P(S_{v,t} | S_{v,<t}, S_{<v,∀·})$$

C. Expressive Performance: Mapping a separated score $c$ to expressive realizations $e$, modeled autoregressively over time,

$P(e|c) = \prod_{t=1}^T P(e_t | c, e_{<t})$

For dataset $\{(c^{(i)}, e^{(i)})\}$, models train to minimize negative log-likelihood:

$$\mathcal{L}_{comp} = -\sum_i \log P(c^{(i)}),\quad \mathcal{L}_{perf} = -\sum_i \log P(e^{(i)}|c^{(i)})$$

(Donahue et al., 2018).

5. Baseline Models and Benchmark Results

Benchmarking leverages both generative and discriminative models, evaluated on Negative Log-Likelihood (NLL), accuracy, and Points of Interest (POIs: timesteps where voice notes change):

• Separated Composition
  • DeepBach: bidirectional LSTM, NLL @ POI: 3.28, accuracy: 0.761
  • LSTM Quartet: single LSTM, all voices, NLL: 11.65, acc: 0.201
  • LSTM Soloists: independent LSTMs, NLL: 13.82, acc: 0.098
• Expressive Performance
  • LSTM Note+Auto: bidirectional over score plus forward LSTM on output, NLL: 9.93, acc: 0.337
  • LSTM Note: bidirectional on score only, NLL: 9.94, acc: 0.308
  • MultiReg Note: multinomial regression, NLL: 10.13, acc: 0.287

Metrics are reported both at all timesteps and POIs (Donahue et al., 2018).

6. Comparison With Related Datasets and Extensions

In contrast to General MIDI corpora, which have unbounded polyphony and no canonical score-to-audio path, or piano datasets focusing on a single instrument and set of expressive controls, NES-MDB packages fixed multi-instrumental scores with all APU parameterizations for rendering consistent NES audio. This facilitates polyphonic modeling and performance generation under rigid resource constraints, distinguishing NES-MDB for algorithmic research in symbolic music modeling (Donahue et al., 2018). Recent works such as LakhNES leverage cross-domain pretraining with Lakh MIDI to improve NES-MDB modeling using Transformer-XL architectures (Donahue et al., 2019).

The NES Video-Music Database (NES-VMDB) builds on NES-MDB by aligning 98,940 gameplay video snippets from 389 games with their original soundtracks in symbolic (MIDI) form. The video-to-MIDI alignment employs audio fingerprinting and enables research into multimodal music generation and genre conditioning (Cardoso et al., 2024).

7. Limitations and Open Research Directions

Known limitations include the exclusion of the fifth (sampler) APU channel due to its ambiguous musical role, lack of explicit tempo labels (using a fixed 24 Hz grid), and strong stylistic homogeneity imposed by NES hardware constraints. These factors may limit generalization beyond the NES music domain. Noted research directions include incorporating variable timing and tempo, learning subtle articulation, extending methodologies to platforms with richer polyphony or sampling capabilities, and developing joint end-to-end generative pipelines that address both composition and expressive performance at higher abstraction levels (Donahue et al., 2018).

A plausible implication is that the highly structured and instrumentally constrained nature of NES-MDB makes it especially suitable for benchmarking symbolic music models, particularly for research that requires an unambiguous and complete mapping between score, expressive controls, and rendered audio.

---

Principal References

• "The NES Music Database: A multi-instrumental dataset with expressive performance attributes" (Donahue et al., 2018)
• "LakhNES: Improving multi-instrumental music generation with cross-domain pre-training" (Donahue et al., 2019)
• "The NES Video-Music Database: A Dataset of Symbolic Video Game Music Paired with Gameplay Videos" (Cardoso et al., 2024)