MulTTiPop: A Multitrack Transcription Dataset for Pop Music

Abstract: We present MulTTiPop, a dataset of pop music segments and their associated multitrack MIDI recordings for the evaluation of automatic music transcription models. MulTTiPop contains 572 segments of popular music totaling 3.5 hours of audio, and contains songs from diverse genres and decades from the 1930s to 2000s. To collect this dataset, we perform metadata-based matching on song segments from the Lakh MIDI and TheoryTab datasets, manually identify an anchor beat between the audio and MIDI, then use beat tracking on the audio and warp the MIDI to match its tempo and timing. We evaluate state-of-the-art automatic music transcription models on MulTTiPop and find substantial room for improvement, with the best model achieving 38% Onset F1. More details and sound examples of MulTTiPop are available at https://gclef-cmu.org/multtipop.

• The paper introduces MulTTiPop, a dataset with 572 segments (3.5 hours) of commercial pop audio paired with detailed multitrack MIDI transcriptions.
• The dataset employs a semi-automatic, beat-aligned transcription workflow combining RNN-based beat tracking and human validation, achieving a 49.1% match rate.
• Empirical benchmarks reveal that state-of-the-art AMT models struggle with commercial audio, highlighting the gap between synthetic and real-world transcription performance.

MulTTiPop: A Multitrack Transcription Dataset for Pop Music

Motivation and Dataset Positioning

The landscape of automatic music transcription (AMT) has recently extended from well-curated domains such as solo piano and classical music (e.g., MAESTRO, MusicNet) to multitrack and multigenre targets. However, authentic, time-aligned multitrack labels for commercial pop music—a primary AMT use case—remain notably scarce. Existing benchmarks like Slakh2100 use synthesized audio and thus fail to capture the production complexity and timbral diversity inherent in popular commercial recordings. Other datasets such as TheoryTab provide melody/chord annotations but lack full multitrack alignment, while RWC-Pop suffers from limited genre diversity and non-commercial composition selection.

MulTTiPop directly addresses this deficit by furnishing 572 segments (3.5 hours) of commercial pop audio from YouTube, each tightly aligned with detailed multitrack MIDI transcriptions tracing real instrumentations found in popular music from the 1930s through the 2000s (Figure 1).

Figure 1: MulTTiPop contains segments of YouTube audio beat-aligned with multitrack MIDI transcriptions.

The resulting dataset exhibits a breadth in artist, genre, and temporal diversity rarely present in prior corpora, while standardizing time alignment and annotation rigor to enable robust and systematic AMT evaluation.

Dataset Construction and Alignment Procedure

MulTTiPop's assembly pipeline operationalizes a high-precision, semi-automatic workflow optimized for label fidelity (Figure 2). Audio segments and metadata are first sourced from TheoryTab, with corresponding multitrack MIDI files drawn from the Lakh MIDI Dataset (LMD-matched). Candidate pairs are filtered via title/artist Levenshtein metric matching to maximize exactness.

Figure 2: The methodology of creating MulTTiPop. Metadata matching, beat-based synchronization, and human-in-the-loop anchor beat selection yield aligned MIDI/audio pairs.

Key stages include:

• Beat-aligned time warping: RNN-based beat tracking (madmom) extracts audio beat grids, while MIDI beats are derived from strict quantization. Given variable tempo and arrangement differences, manual anchor beat selection is required to identify the correct mapping between audio and MIDI subsequences.
• Candidate alignment scoring: Hybrid chroma similarity (harmonic structure) and onset envelope matching (rhythmic detail) generate up to 4 candidate alignments for each audio/MIDI pair. Augmenting baseline harmonic methods with melody-aware metrics (derived from TheoryTab melody annotations) substantially boosts candidate quality.
• Human annotation interface: Expert annotators evaluate candidate alignments via a custom web interface, confirming or rejecting pairs using overlays of synthesized MIDI and commercial audio (Figure 3).

  Figure 3: Interface for annotators to select anchor beats among candidate options.

This annotation pipeline yields a high-quality benchmark, with segment pairs only included if annotators confirm an exact transcription match. Approximately 49.1% of attempted pairs meet the stringent criteria for inclusion; failures are primarily attributed to metadata mismatches or alignment ambiguity beyond automatic or human correction.

Statistical Properties and Diversity

MulTTiPop achieves remarkable diversity despite its focused duration:

• Genre spread: Encompasses 101 genres, vastly exceeding RWC-Pop and even the combined pop, jazz, and genre portions of the entire RWC database.
• Artistic and temporal range: 374 songs from 263 artists, dominantly from the 1970s–2000s; coverage includes outliers from the 1930s–1960s.
• Segment structure: Average segment duration is 22s, each with detailed multitrack MIDI covering 583 notes per segment.

Genre composition visualizations affirm robust coverage of canonical and subcultural pop styles (Figure 4).

Figure 4: Visualization of genres present in MulTTiPop.

The dataset's construction methodology and resultant breadth position it as a uniquely challenging and representative benchmark for the multitrack AMT task in commercial pop.

Empirical Benchmarking and Model Performance

Evaluation on two open-source SOTA AMT models, MT3 and YourMT3+, reveals substantial headroom for advancement:

• Onset F1 (exact instrumentation): Best model achieves only 28.42% (MT3), with YourMT3+ slightly lower.
• Onset F1 (harmonic-percussion reduction): Ceiling rises to 37.87% (YourMT3+), but still significantly lagging behind performance on synthetic or mono-instrument datasets.

Notably, the claimed improvement of YourMT3+ over MT3 on synthetic datasets does not transfer to MulTTiPop, indicating a potential disconnect between training on synthetic renderings and generalization to complex commercial audio.

Qualitative failure analysis highlights:

• Inability to maintain consistent instrument voicing in dense polyphonic contexts.
• Errors in melody/harmony assignment and frequent instrument program mismatches.
• Melody tracking is improved in YourMT3+, but rhythmic detail is often underspecified, and instrument timbre mapping is inconsistent.

These findings underscore the increased difficulty and domain transfer challenges represented by authentic, commercial pop music transcription.

Usage Restrictions and Dataset Scope

MulTTiPop is explicitly recommended for evaluation rather than model training. Its moderate size, Western genre bias (a reflection of its source datasets and annotation methodology), and reliance on copyrighted commercial recordings preclude use as a general-purpose training set for generative or discriminative models. The dataset is thus most appropriately leveraged for benchmarking multitrack pop AMT system performance and ablation studies focused on real-world generalization.

Implications and Future Trajectories

MulTTiPop underlines the critical gaps between current AMT evaluation landscapes and real-world transcription requirements. Its empirical findings demonstrate that the domain shift from synthetic or mono-instrument training corpora to commercial audio remains a major bottleneck for AMT model architectures, regardless of improvements in model capacity or training set scale.

For future work, several avenues are apparent:

• Domain adaptation: There is a pronounced need for models that bridge synthetic-real gaps via domain adaptation, adversarial alignment, or multimodal representation learning that effectively integrates commercial production characteristics.
• Annotation automation: Advances in robust alignment and matching algorithms could further automate the construction of even larger, more diverse datasets.
• Expansion to global music traditions: Addressing the Western-centric bias and extending coverage to non-Western traditions and instrumentations would afford broader applicability and fairer evaluations.
• Cross-modal understanding: Integration of music-theoretic structure with learned audio tokens may further boost alignment and semantic coherence in transcription outputs.

Conclusion

MulTTiPop introduces a critical benchmark of multitrack, time-aligned MIDI and commercial pop audio pairs, providing an empirical foundation for evaluating and advancing AMT models in realistic production settings. Its construction methodology, diversity, and challenging nature highlight substantial shortcomings in current model architectures and training paradigms, while simultaneously setting a new standard for rigor and representativity in multitrack pop transcription evaluation. These results support a shift in AMT research focus from synthetic benchmarks toward more ecologically valid, genre- and production-diverse, real-world evaluation scenarios.