MIDI Tokenization Techniques Overview

• MIDI tokenization is the process of converting raw MIDI data into discrete token sequences for deep-learning models, enhancing efficiency and musical expressiveness.
• Event-based and subword schemes, such as MIDI-Like, REMI, and BPE, reduce sequence length and improve embedding-space utilization while preserving musical structures.
• Advanced methods using multi-attribute, compound, and beat-based tokenizations optimize performance in expressive generation, style analysis, and real-time processing.

MIDI tokenization is the process of converting raw MIDI data, which consists of timestamped musical events, into a discrete sequence of tokens suitable for use by statistical and deep-learning models, particularly transformers and other sequence architectures. The optimal choice of tokenization scheme has substantial effects on musical modeling, sequence length, vocabulary size, model expressiveness, music-theoretic inductive bias, and downstream task performance. This article presents a comprehensive overview of the key methodologies, theoretical considerations, and empirical advances in MIDI tokenization as evidenced by contemporary research.

1. Event-based Schemes: MIDI-Like, REMI, and Their Variants

The canonical approach treats MIDI as a stream of atomic musical events. The principal token types are NOTE_ON(p), NOTE_OFF(p), VELOCITY(v), TIME_SHIFT(dt), plus optional tokens such as BAR, POS_i (position within the bar), DURATION_d, and control tokens. MIDI-Like tokenization adopts a serial order where each NOTE_ON is paired with its VELOCITY, followed by a TIME_SHIFT and a NOTE_OFF, with time quantized into regular units (e.g. 10 ms, 32 ticks, or musical subdivisions) (Fradet et al., 2023, Wiszenko et al., 2024).

REMI (REpresentation of Music with Intervals) introduces explicit BAR and POS tokens to mark metric structure, and replaces NOTE_OFF by explicit durations (DURATION_d), yielding event patterns such as [BAR, POS, PITCH, DURATION, VELOCITY] for each note onset (Fradet et al., 2023, Wiszenko et al., 2024). Other extensions—such as REMI+ and Structured—add further musicological controls, e.g. TIME_SIGNATURE or program changes.

Vocabulary size ranges from ≈150 (basic TSD/REMI) up to ≈300–600 (if including more fine-grained durations, dynamics, instruments), with sequence lengths 10–20 tokens per quarter note for polyphonic music.

2. Compression and Subword Tokenization

Atomic tokenizations, while general, incur long sequence lengths and under-utilize LLM embedding spaces. Byte-Pair Encoding (BPE) and Unigram models address this by greedily merging the most common adjacent token pairs to form higher-order "subword" music tokens, optionally within same-type event families (Fradet et al., 2023, Kumar et al., 2023, Tian et al., 2023).

• For BPE, the merge operation is:

$(a^*,b^*) = \arg\max_{(x,y)} f(x,y)$

where $f(x,y)$ is the frequency of token pair $(x,y)$ across the corpus.

• Vocabulary expands to 1–10k tokens; sequence length falls by up to 60–70%.
• Subword tokens capture common motifs (e.g., “Pitch_71 Duration_1080”) and larger musical phrases.
• Empirically, BPE improves generative structure, embedding-space isotropy, classifier accuracy, and reduces inference cost (Fradet et al., 2023, Kumar et al., 2023).

The "Unsupervised Compoundation and Feature Injection" strategy (UCW tokens) further builds "word-like" variable-length tokens by merging frequent patterns within event families and injects N-gram features at every transformer layer for enhanced motif modeling (Tian et al., 2023).

3. Advanced Multi-Attribute and Score-Transfer Tokenizations

Tokenizations targeting performance-to-score tasks, expressive generation, or symbolic-to-symbolic translation adopt more complex representations:

• Compound tokens: Each note is encoded as a bundle of parallel attributes (e.g. pitch, macro-timeshift, duration, velocity, microshift), optionally with MusicXML-specific features (hand, staff, articulations) (Beyer et al., 2024, Lenz et al., 2024). This massively reduces sequence length—by 3.5× or more—compared to linear key/value streams, while separating composition (pitch, rhythm) from expressive performance (timing and dynamics).
• Multi-stream vocabularies: Architectures like Octuple or CP-Word feed models with one-hot vectors for each attribute; chord, program, and meter tokens are handled by multi-head output layers (Fradet et al., 2023, Wiszenko et al., 2024).
• Beat-based quantization: Instead of absolute timesteps, onsets and durations are quantized on metric grids derived from beat/downbeat annotations or standard grids (16th, 8th triplets, 32-triplet grid) for rhythm quantization (Wachter et al., 18 Aug 2025, Wachter et al., 24 Apr 2026). This aligns input events for more robust transcription and reduces rhythm quantization errors.

4. Tokenization with Music-Theoretic Inductive Biases

Some schemes explicitly inject musical inductive bias, fostering better generalization and interpretability:

• Class-octave and interval-based pitch encoding: Splitting MIDI pitch into class ($c = p \bmod 12$) and octave ($o = \lfloor p/12 \rfloor$), or replacing absolute pitch with horizontal/vertical intervals, yields transposition-invariance and improves pitch-structure modeling (Li et al., 2023, Le et al., 8 Jan 2025). Empirically, class-octave and interval representations improve performance on phrase detection, chord inversion, and era classification tasks.
• Grammar-based hierarchical representations: Context-free grammars generate rhythmic trees encoding bar/beat/subdivision structure; terminals represent all note-on events at a given timestep. Linearizing this tree produces sequences reflecting rhythmic hierarchy, enabling compact parameterization and explicit modeling of higher-level rhythmic syntax (Géré et al., 2024).
• Domain-theoretic embeddings: Moonbeam introduces a 6-tuple event encoding (absolute onset, duration, octave, pitch-class, instrument, velocity), with "Fundamental Music Embedding" (FME) for smooth attribute generalization, and Multidimensional Relative Attention (MRA) that parameterizes self-attention by relative attribute differences (Guo et al., 21 May 2025).

5. Specialized Schemes for Performance, Expressivity, and Multitrack Music

Recent work emphasizes tokenizations suited for rich performance expressivity, multitrack/polyphonic modeling, and infilling:

• Microtiming and velocity tokens: PerTok (Performance Tokenizer) factors expressive timing into macro (grid-aligned) and micro (sub-tick) offsets, and separates velocity as its own token stream, cutting vocabulary size by up to 95% and sequence length by up to 59% on standard benchmarks (Lenz et al., 2024).
• Arrival-time tokenization: The "AMT-like" scheme used in MIDI-LLM emits, for each note, a fixed pattern of (onset, duration, instrument/pitch) tokens, with both "live" and "anticipated" blocks for forward and infilling tasks. This design ensures precise event localization and allows polyphonic multi-instrument encoding in a simple, vectorizable format (Wu et al., 6 Nov 2025).
• Parallelism and multitrack handling: Compound-token and multi-stream methods (e.g., in (Beyer et al., 2024), Octuple) exploit the natural parallelism of multitrack MIDI, representing longer time-slices with a small number of higher-dimensional tokens.

6. Quantization, Vocabulary, and Preprocessing Mechanisms

Quantization granularity for time and velocity is a tunable hyperparameter that directly impacts sequence length, model input/output matching, and quantization error:

• Time quantization: Choices include fixed grid (e.g., 10 ms, 32 ticks, or 1/24 quarter-note), log-scale buckets for duration, and metrical grids matched to beat/downbeat (Fradet et al., 2023, Wachter et al., 24 Apr 2026, Beyer et al., 2024).
• Velocity/dynamics quantization: Typically 8–32 bins for velocity, mapped linearly or logarithmically from MIDI 0–127.
• Canonicalization and augmentation: Preprocessing steps such as sorting events, simultaneous note grouping, or applying BPE over mapped tokens are standard (Kumar et al., 2023, Fradet et al., 2023).
• Rest handling: Rests are typically implicit (silent gaps between notes) rather than as explicit tokens; in some schemes, explicit rest tokens or special encodings can be introduced (Wu et al., 6 Nov 2025, Fradet et al., 2023).

"miditok" offers a configurable and extensible Python tokenization suite supporting most discussed schemes, with modular settings for pitch range, quantization, optional control tokens, and BPE integration (Fradet et al., 2023).

7. Empirical Outcomes and Recommendations

• Compression and inference efficiency: Subword and compound tokenizations achieve 2–3× speedups in inference, up to 70% reduction in sequence length, and full utilization of embedding spaces (isotropy, PCA-ID, IsoScore metrics) (Fradet et al., 2023, Kumar et al., 2023).
• Performance on downstream tasks: N-gram injection, class-octave, intervalized tokenizations, and grammar-based hierarchies consistently yield improvements in generative structure, MIR classification, rhythm quantization (measured by MUSTER, SI, F1, and other metrics) (Géré et al., 2024, Le et al., 8 Jan 2025, Wu et al., 6 Nov 2025, Wachter et al., 24 Apr 2026).
• Task adaptation: Beat- and grammar-based schemes excel in rhythm quantization and style analysis; PerTok and arrival-time are superior for expressive, performance-centric tasks. Interval and class-octave representations are optimal for analytical and explainability-focused applications (Li et al., 2023, Le et al., 8 Jan 2025).

The consensus is that compression via BPE/subword, explicit metric/beat-based time encoding, structure-aware grammar modeling, and domain-informed multi-attribute tokens represent the current best practices for MIDI tokenization, tailored to the demands of neural sequence modeling for music (Kumar et al., 2023, Beyer et al., 2024, Géré et al., 2024, Tian et al., 2023, Guo et al., 21 May 2025, Fradet et al., 2023).