Sheet Music Transformer Overview

• Sheet Music Transformers are transformer-based neural systems that generate, convert, and transcribe notated music using domain-specific tokenization.
• They utilize advanced encoder-decoder and image-transformer architectures to achieve high fidelity in notation and robust polyphonic modeling.
• State-of-the-art implementations demonstrate significant error reduction, sequence length compression, and cross-domain adaptability in tasks like OMR and orchestration.

A sheet music transformer is a transformer-based neural system for the generation, conversion, or transcription of notated music, characterized by domain-specific tokenization strategies, autoregressive/self-attention architectures, and end-to-end mapping between symbolic (MIDI-like), audio, or visual (score image) inputs and standard notation outputs. Recent sheet music transformers are deployed in a number of tasks: MIDI-to-score conversion, optical music recognition (OMR), orchestration of traditional music, and full-score generation, with state-of-the-art results in notation fidelity, robustness, and polyphonic complexity.

1. Tokenization, Representation, and Serialization

Sheet music transformers rely on highly systematic tokenization to encode both input and output domains. For note-level symbolic data, the Score Transformer adopts a one-token-per-symbol (plus voice-tag) strategy, including explicit representations for staff assignment (R/L), barlines, clefs, key signatures, time signatures, pitch, durations, stem direction, beaming, tie, and rests (Suzuki, 2021). Chords are serialized as repeated note tokens preceding a single length token, and staff context is provided by interleaved staff tokens. For compression, regular token forms (average 455 tokens per system) are optionally replaced by concatenated tokens (average length reduction to 301 tokens), merging duration, stem, and beam into compound tokens without accuracy loss.

For performance-to-score mapping, end-to-end systems construct "compound" token streams encoding pitch (128 tokens), onset/duration (log-bucketed), velocity, musical onset, duration, staff, voice, stem direction, grace/staccato/trill flags, accidentals, measure length, and alignment "space" tokens for temporal synchronization (Beyer et al., 30 Sep 2024). Compound tokenization reduces sequence length by a factor of up to 3.5 compared to non-streamed MusicXML tokenizers.

For OMR, both TrOMR and Sheet Music Transformer utilize parallel discrete symbol streams over rhythm, pitch, and accidental axes, either through a strict alignment (TrOMR, with additional note/non-note classification) (Li et al., 2023), or through direct encoding in the Humdrum **kern format (SMT), where polyphony is represented as simultaneous spines and tokens correspond to music symbols and events (Ríos-Vila et al., 12 Feb 2024). Jeongganbo-based transformers demonstrate historical notation support using position-duration tokens and ornament markers (Han et al., 2 Aug 2024).

2. Transformer Model Architecture

The canonical backbone is encoder–decoder, with architectural variants depending on input modality. Note-symbolic models (Score Transformer, PM2S) deploy vanilla transformers with layered attention (Score Transformer: 3 encoder + 3 decoder layers, $d_{model} = 256$, $d_{ff} = 512$, $h = 4$; PM2S: 4 layers per side, $d = 512$, $h = 8$, pre-norm, rotary embeddings) (Suzuki, 2021, Beyer et al., 30 Sep 2024). Multi-stream encoding is achieved by parallel embedding tables and summed inputs per time step.

For image-based OMR, model architectures combine a patchwise transformer encoder (ViT or CNN-based), producing feature embeddings from image patches, followed by a transformer decoder. TrOMR uses 4 encoder + 4 decoder layers, $d_{model} = 256$, with multiple output branches (rhythm, pitch, accidental, note probability), integrating a novel consistency loss to enforce semantic agreement (Li et al., 2023). SMT incorporates a 2D-positional encoding over extracted patch features, feeding into an 8-layer decoder with masked self-attention and explicit cross-attention to encode image-to-sequence translation (Ríos-Vila et al., 12 Feb 2024).

For Jeongganbo arrangement tasks, both BERT-style masked LLMs (monophonic infilling, 12-layer encoder, $d = 128$) and encoder–decoder architectures (6 layers per side, $d = 128$) are used, with auxiliary beat-counter embeddings to facilitate rhythmic coherence (Han et al., 2 Aug 2024).

3. Training Strategies and Loss Functions

Training employs standard autoregressive cross-entropy minimization, typically with teacher-forcing: at each decoding step the ground-truth prefix is supplied for target prediction. Loss is computed as the negative log-likelihood over each output token stream, summed over system sequence length (Suzuki, 2021, Beyer et al., 30 Sep 2024, Ríos-Vila et al., 12 Feb 2024, Han et al., 2 Aug 2024). Label smoothing (0.1) and dropout (0.1–0.2) are used for regularization.

PM2S employs sequence-to-sequence translation, relaxing strict note-wise alignment; space tokens are inserted during alignment, and beat-level binning aligns performed notes with notated events. Exposure bias is mitigated by randomly masking up to 75% of previous tokens during training (Beyer et al., 30 Sep 2024).

TrOMR introduces a multi-branch consistency loss:

$L_{TrOMR} = \lambda L_{ce} + \beta L_{con},$

where $L_{ce}$ is cross-entropy (over rhythm, pitch, accidentals) and $L_{con}$ penalizes branch-to-note deviations, reducing semantic mismatches. For Jeongganbo models, mixed masking schedules in BERT-style models augment robustness, and orchestration models train over multi-instrument input streams (Han et al., 2 Aug 2024).

4. Datasets, Benchmarks, and Evaluation Metrics

Sheet music transformers are evaluated on diverse corpora:

• Score Transformer: popular and classical piano data (7,161 and 354 pieces), quantized at 24 ticks/beat; metrics over 12 musical aspects—note preservation, segregation, clef, key/time signature, barline, voice tagging, stem direction, beaming, ties/rests—yield state-of-the-art average error rates (4.1%) and near-perfect note/duration recovery (0.27%) (Suzuki, 2021).
• PM2S: ASAP performance-to-score testset, reporting MUSTER error rates—$\mathcal{E}_{onset}$ 15.6% (vs. 22.6% for HMM+heuristics), staff/voice errors under 7%, F1-trill 54.6% (first direct trill prediction) (Beyer et al., 30 Sep 2024).
• OMR: TrOMR trains on ≈400K MuseScore synthetic images, splits into rendered (MSD) and real-world (CMSD) sets; Symbol Error Rate (SER) is reduced to 0.025 and 0.024 for polyphonic rendered and camera images, outperforming CNN+RNN baselines by factors of 5–10 (Li et al., 2023). SMT evaluates on GrandStaff and Quartets datasets, showing up to 92–94% error reductions over previous systems; over 98% renderable outputs (Ríos-Vila et al., 12 Feb 2024).
• Jeongganbo arrangement: 85-piece corpus (28,010 jeonggans, 141,820 part-jeonggans); F1 for pitch-onset correctness up to 0.679; subjective expert ratings confirm ensemble coherence and ornamentation fidelity (Han et al., 2 Aug 2024).

Metrics emphasize symbol/character/line error rates (edit distances), F1 for micro-notations, and renderability in standard software.

5. Robustness, Polyphony, and Cross-Domain Extensions

Sheet music transformers demonstrate high robustness to timing noise and cross-genre adaptation: Score Transformer retains low error rates under random onset/duration perturbations (average error 3.1%), and generalizes from "Popular" to "Classical" genres with fine-tuning (Suzuki, 2021). OMR systems maintain SER under photographic distortions (blur, lighting, moiré), though extreme degradation remains challenging (Li et al., 2023, Ríos-Vila et al., 12 Feb 2024).

Crucially, architectural and tokenization choices enable intrinsic polyphonic modeling. Multi-stream/kern formats naturally encode simultaneous events; multi-head attention captures voice separation, simultaneous notes, and cross-staff relationships. Humdrum, ABC, and LilyPond tokenizations are less stable than dedicated notation-symbol vocabularies; the inclusion of explicit context tokens (staff tags, beat positions, beat-counter embeddings) is essential for rhythmically coherent polyphony (Suzuki, 2021, Han et al., 2 Aug 2024).

Cross-domain extension has been demonstrated for non-western notation (Jeongganbo) and multi-instrument orchestration (ensemble arrangement for 6-part Korean court music), confirming flexibility beyond conventional European staved notation (Han et al., 2 Aug 2024).

6. Comparative Performance and Open Directions

Sheet music transformers outperform both heuristic-driven and CNN+RNN systems in accuracy, sequence compactness, and musicological renderability. For instance, PM2S's compound tokenization yields 3.5× sequence length reduction for MusicXML format, boosting efficiency (Beyer et al., 30 Sep 2024). TrOMR and SMT are the first single-model solutions to polyphonic OMR; rendered outputs are immediately processable in standard music engraving software.

Open technical avenues include: direct prediction of key/time signatures/tempo, expansion to multi-instrument arrangements, adaptation to further notation formats (MEI, extended MusicXML), and improved domain-specific data augmentation for degraded or rare notation types. The streaming mode for real-time score following and integration with unpaired data extend transformer versatility.

The sheet music transformer paradigm thus unifies model-driven music notation generation, image transcription, and performance-to-score mapping, establishing a foundation for universal music information processing (Suzuki, 2021, Li et al., 2023, Beyer et al., 30 Sep 2024, Ríos-Vila et al., 12 Feb 2024, Han et al., 2 Aug 2024).