---
title: MIDI Tokenization Techniques Overview
url: https://www.emergentmind.com/topics/midi-tokenization
type: topic
---

# MIDI Tokenization Techniques Overview

MIDI tokenization is the process of converting raw MIDI data, which consists of timestamped musical events, into a discrete sequence of tokens suitable for use by statistical and deep-learning models, particularly transformers and other sequence architectures. The optimal choice of tokenization scheme has substantial effects on musical modeling, sequence length, vocabulary size, model expressiveness, music-theoretic inductive bias, and downstream task performance. This article presents a comprehensive overview of the key methodologies, theoretical considerations, and empirical advances in MIDI tokenization as evidenced by contemporary research.

## 1. Event-based Schemes: MIDI-Like, REMI, and Their Variants

The canonical approach treats MIDI as a stream of atomic musical events. The principal token types are NOTE_ON(p), NOTE_OFF(p), VELOCITY(v), TIME_SHIFT(dt), plus optional tokens such as BAR, POS_i (position within the bar), DURATION_d, and control tokens. MIDI-Like tokenization adopts a serial order where each NOTE_ON is paired with its VELOCITY, followed by a TIME_SHIFT and a NOTE_OFF, with time quantized into regular units (e.g. 10 ms, 32 ticks, or musical subdivisions) [2310.17202][2410.20518].

REMI (REpresentation of Music with Intervals) introduces explicit BAR and POS tokens to mark metric structure, and replaces NOTE_OFF by explicit durations (DURATION_d), yielding event patterns such as [BAR, POS, PITCH, DURATION, VELOCITY] for each note onset [2310.17202][2410.20518]. Other extensions—such as REMI+ and Structured—add further musicological controls, e.g. TIME_SIGNATURE or program changes. 

Vocabulary size ranges from ≈150 (basic TSD/REMI) up to ≈300–600 (if including more fine-grained durations, dynamics, instruments), with sequence lengths 10–20 tokens per quarter note for polyphonic music.

## 2. Compression and Subword Tokenization

Atomic tokenizations, while general, incur long sequence lengths and under-utilize language model embedding spaces. Byte-Pair Encoding (BPE) and Unigram models address this by greedily merging the most common adjacent token pairs to form higher-order "subword" music tokens, optionally within same-type event families [2301.11975][2304.08953][2312.08931]. 

- For BPE, the merge operation is:
  $$(a^*,b^*) = \arg\max_{(x,y)} f(x,y)$$
  where $f(x,y)$ is the frequency of token pair $(x,y)$ across the corpus.
- Vocabulary expands to 1–10k tokens; sequence length falls by up to 60–70%.
- Subword tokens capture common motifs (e.g., “Pitch_71 Duration_1080”) and larger musical phrases.
- Empirically, BPE improves generative structure, embedding-space isotropy, classifier accuracy, and reduces inference cost [2301.11975][2304.08953].

The "Unsupervised Compoundation and Feature Injection" strategy (UCW tokens) further builds "word-like" variable-length tokens by merging frequent patterns within event families and injects N-gram features at every transformer layer for enhanced motif modeling [2312.08931].

## 3. Advanced Multi-Attribute and Score-Transfer Tokenizations

Tokenizations targeting performance-to-score tasks, expressive generation, or symbolic-to-symbolic translation adopt more complex representations:

- **Compound tokens**: Each note is encoded as a bundle of parallel attributes (e.g. pitch, macro-timeshift, duration, velocity, microshift), optionally with MusicXML-specific features (hand, staff, articulations) [2410.00210][2410.02060]. This massively reduces sequence length—by 3.5× or more—compared to linear key/value streams, while separating composition (pitch, rhythm) from expressive performance (timing and dynamics).
- **Multi-stream vocabularies**: Architectures like Octuple or CP-Word feed models with one-hot vectors for each attribute; chord, program, and meter tokens are handled by multi-head output layers [2310.17202][2410.20518].
- **Beat-based quantization**: Instead of absolute timesteps, onsets and durations are quantized on metric grids derived from beat/downbeat annotations or standard grids (16th, 8th triplets, 32-triplet grid) for rhythm quantization [2508.19262][2604.22290]. This aligns input events for more robust transcription and reduces rhythm quantization errors.

## 4. Tokenization with Music-Theoretic Inductive Biases

Some schemes explicitly inject musical inductive bias, fostering better generalization and interpretability:

- **Class-octave and interval-based pitch encoding**: Splitting MIDI pitch into class ($c = p \bmod 12$) and octave ($o = \lfloor p/12 \rfloor$), or replacing absolute pitch with horizontal/vertical intervals, yields transposition-invariance and improves pitch-structure modeling [2301.13383][2501.04630]. Empirically, class-octave and interval representations improve performance on phrase detection, chord inversion, and era classification tasks.
- **Grammar-based hierarchical representations**: Context-free grammars generate rhythmic trees encoding bar/beat/subdivision structure; terminals represent all note-on events at a given timestep. Linearizing this tree produces sequences reflecting rhythmic hierarchy, enabling compact parameterization and explicit modeling of higher-level rhythmic syntax [2407.17536].
- **Domain-theoretic embeddings**: Moonbeam introduces a 6-tuple event encoding (absolute onset, duration, octave, pitch-class, instrument, velocity), with "Fundamental Music Embedding" (FME) for smooth attribute generalization, and Multidimensional Relative Attention (MRA) that parameterizes self-attention by relative attribute differences [2505.15559].

## 5. Specialized Schemes for Performance, Expressivity, and Multitrack Music

Recent work emphasizes tokenizations suited for rich performance expressivity, multitrack/polyphonic modeling, and infilling:

- **Microtiming and velocity tokens**: PerTok (Performance Tokenizer) factors expressive timing into macro (grid-aligned) and micro (sub-tick) offsets, and separates velocity as its own token stream, cutting vocabulary size by up to 95% and sequence length by up to 59% on standard benchmarks [2410.02060].
- **Arrival-time tokenization**: The "AMT-like" scheme used in MIDI-LLM emits, for each note, a fixed pattern of (onset, duration, instrument/pitch) tokens, with both "live" and "anticipated" blocks for forward and infilling tasks. This design ensures precise event localization and allows polyphonic multi-instrument encoding in a simple, vectorizable format [2511.03942].
- **Parallelism and multitrack handling**: Compound-token and multi-stream methods (e.g., in [2410.00210], Octuple) exploit the natural parallelism of multitrack MIDI, representing longer time-slices with a small number of higher-dimensional tokens.

## 6. Quantization, Vocabulary, and Preprocessing Mechanisms

Quantization granularity for time and velocity is a tunable hyperparameter that directly impacts sequence length, model input/output matching, and quantization error:

- **Time quantization**: Choices include fixed grid (e.g., 10 ms, 32 ticks, or 1/24 quarter-note), log-scale buckets for duration, and metrical grids matched to beat/downbeat [2310.17202][2604.22290][2410.00210].
- **Velocity/dynamics quantization**: Typically 8–32 bins for velocity, mapped linearly or logarithmically from MIDI 0–127.
- **Canonicalization and augmentation**: Preprocessing steps such as sorting events, simultaneous note grouping, or applying BPE over mapped tokens are standard [2304.08953][2301.11975].
- **Rest handling**: Rests are typically implicit (silent gaps between notes) rather than as explicit tokens; in some schemes, explicit rest tokens or special encodings can be introduced [2511.03942][2310.17202].

"miditok" offers a configurable and extensible Python tokenization suite supporting most discussed schemes, with modular settings for pitch range, quantization, optional control tokens, and BPE integration [2310.17202].

## 7. Empirical Outcomes and Recommendations

- **Compression and inference efficiency**: Subword and compound tokenizations achieve 2–3× speedups in inference, up to 70% reduction in sequence length, and full utilization of embedding spaces (isotropy, PCA-ID, IsoScore metrics) [2301.11975][2304.08953].
- **Performance on downstream tasks**: N-gram injection, class-octave, intervalized tokenizations, and grammar-based hierarchies consistently yield improvements in generative structure, MIR classification, rhythm quantization (measured by MUSTER, SI, F1, and other metrics) [2407.17536][2501.04630][2511.03942][2604.22290].
- **Task adaptation**: Beat- and grammar-based schemes excel in rhythm quantization and style analysis; PerTok and arrival-time are superior for expressive, performance-centric tasks. Interval and class-octave representations are optimal for analytical and explainability-focused applications [2301.13383][2501.04630].

The consensus is that compression via BPE/subword, explicit metric/beat-based time encoding, structure-aware grammar modeling, and domain-informed multi-attribute tokens represent the current best practices for MIDI tokenization, tailored to the demands of neural sequence modeling for music [2304.08953][2410.00210][2407.17536][2312.08931][2505.15559][2310.17202].

Source: https://www.emergentmind.com/topics/midi-tokenization