MMT: Compact Multitrack Music Transformer

• MMT is a decoder-only autoregressive transformer that generates multitrack symbolic music using a compact 6-tuple event representation.
• It significantly reduces sequence length compared to models like MMM and REMI+, enabling faster decoding and longer music samples.
• The design co-optimizes event representation with transformer architecture to support real-time improvisation and extended orchestral generation.

Searching arXiv for the core MMT paper and closely related extensions to ground the article. Searching for the original "Multitrack Music Transformer". Multitrack Music Transformer (MMT) is a decoder-only autoregressive transformer for symbolic multitrack music generation built around a compact event representation in which one event corresponds to one note, together with a small number of structural event types. Each event is represented as a 6-tuple of type, beat, position, pitch, duration, and instrument, so the model generates one note per step rather than several sequential tokens per note. In the reported orchestral setting, this representation shortens sequences enough that a length-1024 sequence can represent 2.6× longer music than MMM and 3.5× longer music than REMI+/FIGARO, while enabling substantially faster decoding (Dong et al., 2022).

1. Origins and design objective

MMT was proposed to address a central bottleneck in transformer-based symbolic multitrack music generation: existing event representations make sequences too long, which in turn makes transformers expensive to train, memory-hungry, and slow at inference. The motivating problem is specifically multitrack symbolic music, where the model must capture polyphony within each track, coordination across many instruments, long-range temporal structure, meter and periodicity, arrangement decisions, and inter-voice dependencies across heterogeneous parts. Because self-attention scales poorly with sequence length, multitrack representations that require many events per note severely constrain usable context and practical generation speed (Dong et al., 2022).

The original paper positions MMT against earlier transformer-based multitrack systems such as MMM and REMI+/FIGARO. The critique is not that these systems are ineffective in principle, but that their representations require multiple autoregressive decoding steps per note for timing, pitch, duration, instrument, and related attributes. In the stated orchestral setting, this made prior systems unattractive for real-time improvisation, near-real-time co-creation, and longer orchestral generation. The central design response was therefore a representation-model co-design: keep a standard decoder-only transformer, but collapse note attributes into a single multi-field event so that the autoregressive process advances one note at a time rather than one attribute at a time (Dong et al., 2022).

This design also places MMT within a broader line of multitrack symbolic music modeling. MMM had already shown that track-contiguous representations can support conditioning, inpainting, and per-track controls through a GPT-2-style decoder (Ens et al., 2020). MMT instead retains a unified event stream, but compresses the per-note description into a tuple. A plausible implication is that MMT prioritizes sequence-length reduction and one-pass note generation, whereas MMM prioritizes explicit track contiguity for editing workflows.

2. Event representation and musical serialization

A music piece in MMT is represented as an event sequence

$\mathbf{x} = (\mathbf{x}_1, \dots, \mathbf{x}_n),$

where each event is a 6-tuple

$$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$

The field $x^\mathit{type}$ determines one of five event types: Start-of-song, Instrument, Start-of-notes, Note, and End-of-song. For any non-note event, the fields $$x^\mathit{beat}, x^\mathit{position}, x^\mathit{pitch}, x^\mathit{duration}, x^\mathit{instrument}$$ are set to $0$, which is reserved as an undefined value. The sequence structure is therefore: Start-of-song, several Instrument events, Start-of-notes, many Note events, and End-of-song (Dong et al., 2022).

Timing is encoded by splitting onset into beat and within-beat position. Following the paper’s notation, the actual onset is

$r \cdot x^\mathit{beat} + x^\mathit{position},$

where $r$ is the temporal resolution per beat. In the reported experiments, the resolution is 12 time steps per quarter note. Durations are quantized to a carefully chosen set of common note durations, following FIGARO, and any out-of-set duration is replaced by the nearest valid duration. Instrument identity is compressed by mapping the 128 MIDI programs to 64 merged instruments, grouping similar programs such as acoustic grand piano and bright acoustic piano into the same piano category (Dong et al., 2022).

A decisive property of the representation is that a single Note event carries onset, pitch, duration, and instrument in parallel fields. This makes the representation permutation invariant with respect to note-event ordering, because reordering note events does not change the decoded music. For autoregressive training, however, notes are sorted by beat, then position, then pitch, then duration, then instrument. The experiments on the Symbolic Orchestral Database (SOD) additionally discard tempo, velocity, and drum tracks because not all pieces contained that information (Dong et al., 2022).

Later work preserved this tuple representation unchanged when analyzing or extending MMT. MSAT explicitly adopts the MMT representation from Dong et al. and uses the same six token fields—type, beat, position, pitch, duration, and instrument—while constructing note-level, bar-level, and track-level serializations of the same underlying multitrack events (Wei et al., 2023). This continuity indicates that the 6-tuple is not only an implementation detail, but the canonical symbolic interface through which subsequent work has treated MMT.

3. Transformer architecture, objective, and decoding

Architecturally, MMT is based on a decoder-only transformer in the GPT-style autoregressive setting. For the reported orchestral experiments, the model uses 6 transformer decoder blocks, model dimension 512, 8 self-attention heads, embedding dimension 512 for all input embeddings, maximum sequence length 1024, and maximum beat 256. Positional information is provided by learnable absolute positional embeddings; the paper does not use relative positional attention or a custom attention mechanism in the model itself (Dong et al., 2022).

The factorization is autoregressive across events,

$$p(\mathbf{x}) = \prod_i p(\mathbf{x}_i \mid \mathbf{x}_{<i}),$$

but within each event the six fields are predicted nonautoregressively by separate output heads conditioned on the transformer hidden state for that event position. Training minimizes the sum of cross-entropy losses over the different fields under this autoregressive setting. This is the critical trade-off in MMT: it enables one inference pass per note, but it also means that the model cannot explicitly model dependencies within the same note event. The paper gives a concrete example: a token-by-token representation can predict pitch conditioned on the previously generated instrument token for that note, whereas MMT predicts instrument and pitch in parallel (Dong et al., 2022).

The generation interface supports three modes. In unconditioned generation, the prefix is only Start-of-song, after which the model generates the instrument list, Start-of-notes, and the note sequence. In instrument-informed generation, the prefix is Start-of-song followed by chosen Instrument events and Start-of-notes, after which the model generates notes conditioned on the specified instruments. In $N$-beat continuation, the prefix includes all instrument and note events from the first $N$ beats, and the model continues the sequence autoregressively (Dong et al., 2022).

Decoding uses top-$$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$0 sampling independently on each field, with

$$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$1

Because type and beat are sorted fields, the paper also imposes a monotonic constraint during decoding: when sampling $$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$2, outcomes smaller than $$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$3 are assigned zero probability, and similarly for beat ordering. This prevents invalid event orders such as generating a Note before an Instrument (Dong et al., 2022).

4. Dataset, efficiency, and empirical performance

The original experiments use the Symbolic Orchestral Database. After preprocessing, the corpus contains 5,743 songs and 357 hours. Temporal resolution is set to 12 time steps per quarter note; tempo, velocity, and drum tracks are discarded; preprocessing is performed with MusPy; and the split uses 10% validation and 10% test. Training applies a random pitch shift

$$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$4

in semitones, together with a random starting beat. Validation is performed every 1K steps, and training stops at 200K steps or after 20 validation rounds without improvement (Dong et al., 2022).

The principal empirical claim is efficiency under fixed model scale. MMT has 19.94M parameters, compared with 19.81M for MMM and 20.72M for REMI+, so the gains are not attributable to a much larger model. On the orchestral dataset, a sequence of length 1024 represents 2.6× longer samples than MMM and 3.5× longer samples than REMI+. In the reported practical measurements, MMT generates average sample length 100.42 seconds at 11.79 notes per second, whereas MMM reaches 38.69 seconds at 5.66 notes per second and REMI+ reaches 28.69 seconds at 3.58 notes per second. The paper explicitly attributes this to the fact that the baselines need several generated tokens to form a note, while MMT generates a whole note in one inference pass (Dong et al., 2022).

Quality is more mixed. In the subjective listening test, 9 music amateurs who could each play at least one musical instrument rated coherence, richness, arrangement, and overall quality. MMT beats MMM on all subjective criteria, but REMI+ gets the best quality scores overall. The reported overall MOS is 3.33 for MMT, 3.17 for MMM, and 3.77 for REMI+. The paper characterizes MMT as landing between MMM and REMI+ in subjective quality while strongly outperforming both in speed and effective generation length (Dong et al., 2022).

The objective evaluation for unconditioned generation measures pitch class entropy, scale consistency, and groove consistency. Ground truth values are 2.974 ± 0.018, 92.26 ± 1.25, and 93.05 ± 1.00. MMT obtains 2.802 ± 0.025, 94.74 ± 0.42, and 92.09 ± 0.49. REMI+ is generally closest to ground truth overall, while MMT is closer than MMM on groove consistency but not best overall. The original paper attributes part of this quality gap to the independent prediction of output fields within each event (Dong et al., 2022).

5. Self-attention analysis and mechanistic interpretability

A distinctive contribution of the original MMT paper is a quantitative analysis of musical self-attention. For a field $$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$5, the paper defines mean relative attention

$$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$6

and then defines mean relative attention gain

$$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$7

Using 100 test samples and the last attention layer, the paper reports that several heads place larger weight on beats that are $$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$8 beats away, that the model most strongly attends to notes with the same position within the beat, and that it attends more to pitches forming consonant intervals such as the 4th, 5th, and octave. The paper interprets these findings as evidence that, despite using standard absolute positional embeddings, MMT learns a kind of relative self-attention behavior with respect to beat, within-beat position, and pitch (Dong et al., 2022).

Subsequent work shifted from attention analysis to direct intervention in internal activations. One line of research studies a pretrained MMT on SOD as a fixed autoregressive symbolic music model and intervenes in the residual stream at the output of each transformer decoder block using forward hooks. Using Difference-in-Means (DiffMean), these papers isolate latent directions for average note pitch and average note duration, test the Linear Representation Hypothesis in symbolic music, and report that Pitch is most linearly separable around Layer 11 while Duration is most separable around Layer 2. They further introduce Dual Steering based on Gram–Schmidt orthogonalization to reduce interference when controlling the two attributes simultaneously, arguing that All-to-All layer injection gives the best trade-off overall (Prokopiou et al., 29 May 2026).

A later paper retains MMT as the underlying symbolic generator but adds sparse autoencoder-based steering and closed-loop PID control. In that formulation, the public SOD-pretrained checkpoint is described as a 6-block decoder-only transformer with 12 sublayers, hidden size 512, and 8 attention heads. The paper identifies Layer 10 as the best single intervention locus for sparse control, diagnoses a Top-K threshold problem in sparse activation steering, and proposes temporal PID control so that the steering coefficient is recomputed at every decoding step. It also quantifies entanglement between pitch and duration directions, reporting average absolute cosine similarity 0.49 across dense layers and active-feature overlap 51.8% in sparse SAE space. This line of work suggests that MMT’s internal states contain usable, partly linear representations of pitch register and note duration, but that these representations are not naturally disentangled (Prokopiou et al., 17 Jun 2026).

6. Extensions, alternatives, and limitations

MMT has functioned both as a standalone model and as a substrate for later multitrack architectures. MSAT is best viewed as a direct extension: it keeps the MMT event schema unchanged, replaces a single decoder with three MMT-style decoders over note-, bar-, and track-level serializations, and fuses their token-level representations with global or local attention. In the reported results, MSAT-LA is the strongest overall multiscale variant, and the paper frames the result as evidence that no single serialization order is universally best for multitrack symbolic music (Wei et al., 2023).

MMT-BERT is a different type of extension. It preserves the MMT-like generator but modifies the event inventory to add a chord event of the form

$$(x_i^\mathit{type}, x_i^\mathit{beat}, x_i^\mathit{position}, x_i^\mathit{pitch}, x_i^\mathit{duration}, x_i^\mathit{instrument}).$$9

inserted once per bar, and places the generator inside a GAN framework with a MusicBERT discriminator. Relative to base MMT, the reported quantitative gains are large on Pitch Class Entropy Similarity, Scale Consistency Similarity, and Groove Consistency Similarity, while average generated length remains close to the MMT baseline because chord events consume sequence budget but are not converted to audio during decoding (Zhu et al., 2024).

Other work has treated MMT more as a reference point than as a base architecture. MMM and MIDI-GPT both serialize each track as its own time-ordered event sequence and then concatenate tracks, making track-level and bar-level editing straightforward. MIDI-GPT is explicitly described as MMT-adjacent rather than a standard MMT formulation: both are autoregressive Transformer models for symbolic multitrack MIDI, but MIDI-GPT serializes by track first and by time within track second, whereas canonical multitrack event streams often serialize by global time first and only secondarily encode track identity. This suggests a persistent design split in multitrack symbolic generation between compact unified event streams and track-contiguous serializations optimized for controllability and editing (Ens et al., 2020); (Pasquier et al., 28 Jan 2025).

The limitations of canonical MMT are also well defined. The most important is the independent prediction of the six fields within each event, which weakens modeling of pitch–instrument compatibility and other same-note dependencies. Empirically, REMI+ still performs better in subjective quality and objective metrics. The demonstrated system additionally discards velocity, tempo, and drums, and it does not provide an explicit mechanism for long-form structure beyond the context that the standard decoder-only transformer can retain. Later interpretability papers add that pitch and duration are substantially entangled in both dense residual space and sparse feature space, so reliable multi-attribute control requires explicit geometric decoupling or feedback control rather than naive steering (Dong et al., 2022); (Prokopiou et al., 29 May 2026); (Prokopiou et al., 17 Jun 2026).

Taken together, these developments position MMT as a canonical compact multitrack event model rather than merely a single paper. Its defining idea is that symbolic multitrack generation can be made substantially faster and more scalable by making one autoregressive step correspond to one musical event. Later work has used that compactness as a foundation for multiscale fusion, chord-aware adversarial training, and mechanistic intervention in internal activations, while adjacent research has explored alternative serializations for infilling, editing, and stronger track-local control (Dong et al., 2022).