MusicBERT: Transformer for Symbolic Music

Updated 7 November 2025

MusicBERT is a pre-trained transformer model designed for symbolic music, utilizing innovative OctupleMIDI encoding and bar-level masking strategies.
It achieves state-of-the-art results in melody completion, accompaniment suggestion, genre classification, and style classification across multiple benchmarks.
Its scalable architecture and universal representation scheme set new benchmarks for transfer learning in symbolic music, enabling efficient and robust downstream applications.

MusicBERT is a large-scale pre-trained transformer model specifically developed for symbolic music understanding, focusing on structured symbolic data formats such as MIDI rather than audio. Drawing core methodological inspiration from pre-trained LLMs (PLMs) in natural language processing, notably BERT, MusicBERT introduces a set of innovations to address the unique characteristics of symbolic music data, including its highly structured, hierarchical organization and redundant attribute distributions. Its primary contributions are a scalable, universal representation scheme (OctupleMIDI encoding), a bar-level masking pre-training strategy tailored to musical structure, and empirical validation on multiple downstream music understanding tasks. The model establishes a new standard for transfer learning in symbolic music by demonstrating superior performance over previous symbolic music representation methods in melody completion, accompaniment suggestion, genre classification, and style classification (Zeng et al., 2021).

1. Distinctions between Symbolic Music and Natural Language Processing

A defining challenge for symbolic music modeling is the divergence from natural language in both data structure and distributional properties:

Hierarchical Structure: Music data is organized bar-wise with a range of static or repetitive attributes (tempo, key signature, instrument) within bars—a contrast to mostly linear, context-dependent variations in natural language sequences.
Sequence Length: Canonical symbolic tokenizations (e.g., REMI) yield extremely long sequences—mean >15,000 tokens per song for REMI vs. ~3,600 for OctupleMIDI—dwarfing even the longest unsegmented documents in NLP.
Attribute Redundancy: Frequent repetition of attributes (e.g., tempo, instrument, bar-time) within bars creates a risk of information leakage with standard token-level masking, trivializing the proxy objective.
Data Scarcity: Supervised symbolic music labels are expensive to annotate; transfer learning and unsupervised representation learning are crucial for generalization.

Maintaining these structural characteristics is vital for any symbolic music PLM to be both efficient and semantically robust for downstream music information retrieval or semantic analysis.

2. Architecture and Encoding Strategy

MusicBERT adopts an encoder-only Transformer architecture analogous to BERT, but with a critical domain-specific adaptation: the OctupleMIDI encoding. Each note event is represented as an eight-element tuple, comprising the following:

Element	Value Range/Encoding
Time signature	254 tokenized values
Tempo	49 BPM bins (16–256), geom.
Bar number	0–255
Position	1/64 note granularity, max 128
Instrument	129 tokens (incl. percussion)
Pitch	128 (reg) + 128 (perc.)
Duration	128 bins, adaptively quantized
Velocity	32 steps (by 4, range 2–126)

The resulting octuple tokens are embedded (features concatenated, linearly projected), passed through 4 or 12-layer (small/base) transformer stacks, and subjected to multi-task masked element prediction with eight softmax output heads per time step.

Compression and Universality: OctupleMIDI achieves (i) ~4x reduction in sequence length compared to REMI, (ii) ~2x over CP. On the LMD dataset, mean tokens per song drop to 3,607 (OctupleMIDI) from 15,679 (REMI), directly enabling longer musical context modeling with fixed compute and facilitating transfer to complex symbolic music tasks.

3. Pre-Training Objective and Bar-Level Masking

The pre-training is grounded in multi-element masked language modeling. For each masked position $i$ :

$\mathcal{L} = -\sum_{i \in \mathcal{M}} \sum_{j=1}^8 \log P(x_{i,j} | \mathbf{x}_{\setminus \mathcal{M}})$

where $\mathcal{M}$ is the set of masked indices, $x_{i,j}$ the $j$ -th attribute in octuple $i$ , and $\mathbf{x}_{\setminus\mathcal{M}}$ the unmasked context. No next sentence prediction is used, aligning with RoBERTa’s simplification.

Bar-Level Masking replaces naive random token masking (which is prone to leakage via redundant intra-bar features) by masking all occurrences of an attribute type (e.g., pitch, instrument) within a bar simultaneously. This hardens the prediction task, requiring the model to capture cross-bar and global context rather than local repetition. The masking protocol (akin to BERT): 80% [MASK], 10% random token, 10% unchanged, dynamically sampled per batch/epoch.

4. Corpus Construction and Scale

MusicBERT is pre-trained on the Million MIDI Dataset (MMD), comprising 1.5 million deduplicated, genre-diverse songs (~2 billion octuple tokens), encoded in the OctupleMIDI format. This corpus is an order of magnitude larger than previous datasets (e.g., LMD with 148k songs), supporting robust pre-training and transfer across varied musical styles. The deduplication employs an instrument-pitch fingerprint hash, ensuring high data diversity across styles, periods, and genres (Rock, Electronic, Rap, Jazz, Latin, Classical, etc.).

5. Downstream Evaluation and Empirical Results

MusicBERT is empirically validated on four central symbolic music understanding tasks:

Task	Metric	MusicBERT $_{small}$	MusicBERT $_{base}$
Melody Completion	MAP	0.982	0.985
Accompaniment Suggestion	MAP	0.930	0.946
Genre Classification	F1-micro	0.761	0.784
Style Classification	F1-micro	0.626	0.645

MusicBERT surpasses all prior baselines (melody2vec, PiRhDy, etc.) across all four tasks. Notably, accuracy gains are tightly linked to its encoding and masking innovations: ablation reveals sizable drops when replacing OctupleMIDI with CP/REMI or reverting to random masking.

Ablation studies demonstrate:

OctupleMIDI encoding notably outperforms REMI-like and CP-like encodings for song-level tasks—at reduced computational cost (16× efficiency over REMI).
Bar-level masking yields higher accuracy than random masking or octuple-token masking.
No pre-training: Dramatically lower downstream accuracy (e.g., genre F1-micro drops from 0.730 to 0.662).

6. Comparative Assessment and Subsequent Research Directions

MusicBERT’s methodology has shaped subsequent symbolic music pre-training paradigms, establishing bar-level element masking and note-centric encoding as standard baselines. Subsequent models, such as Adversarial-MidiBERT (Zhao, 11 Jul 2024), introduce adversarial masking procedures to further address bias observed in masked language modeling; PianoBART (Liang et al., 26 Jun 2024) adopts encoder–decoder architectures and multi-level corruption strategies, showing improved performance on both music understanding and generation by expanding the diversity and granularity of corruption masks. Other frameworks (e.g., MMT-BERT (Zhu et al., 2 Sep 2024)) integrate MusicBERT as a symbolic-aware discriminator within GAN frameworks for multitrack music generation, leveraging its encoded symbolic music priors for improved realism and harmonicity.

The large-scale, domain-specific pre-training innovation of MusicBERT catalyzed the exploration of new representations, objective design (e.g., interval-aware or pianoroll-derived targets), and model architectures that move beyond direct NLP transfer, emphasizing the need for music-structure-aware pre-training regimes.

7. Impact and Limitations

MusicBERT demonstrates that symbolic music understanding requires explicit recognition of domain structure in both representation and pre-training objectives. Its efficient encoding, on-structure masking, and massive symbolic corpus have set benchmarks for melody/harmony understanding and style/genre inference, but its encoder-only design does not directly support generative tasks, and its symbolic focus limits applicability to audio domain MIR tasks.

Performance and resource requirements are typical of large transformer models: MusicBERT $_{base}$ (12 layers, 768-dim) can be pre-trained with contemporary GPU hardware, and inference for downstream tasks is efficient owing to the compression of input sequences (OctupleMIDI). Further practical extensions could involve adaptation to multi-track contexts, hierarchical musical forms, or alignment with score-level annotation, as well as interfacing with pre-trained audio models for end-to-end MIR systems.

MusicBERT establishes the critical importance of domain customization in symbolic music pre-training and remains foundational for subsequent developments in transformer-based symbolic music representation learning (Zeng et al., 2021).