Transcoda: Synthetic Training in Zero-Shot OMR

• Transcoda is an end-to-end OMR system that converts sheet music images into normalized **kern notation using a compact encoder-decoder model.
• It employs a robust synthetic data generation pipeline and a 21-stage normalization process to resolve one-to-many transcription ambiguities.
• The system demonstrates significant improvements in zero-shot accuracy on historical scans, reducing OMR-NED from over 80% to around 64%.

Transcoda is an end-to-end optical music recognition (OMR) system for transcribing sheet-music images into normalized Humdrum **kern, designed for the practically important regime in which training is performed mostly or entirely on synthetic data and evaluation is performed zero-shot on real scans. Its central design combines a compact encoder-decoder model, a large synthetic data generation pipeline, a normalization of **kern into a near-unique normal form, and optional grammar-based constrained decoding for formal validity. In the reported evaluation, a roughly 59M-parameter model trained in 6 hours on one NVIDIA RTX 5090 achieves 18.46% OMR-NED on a clean synthetic benchmark and 63.97% OMR-NED on historical Polish scans (Dratschuk et al., 11 May 2026).

1. Problem formulation and scope

Transcoda frames OMR as image-to-sequence generation: given an image of a score, generate a structured textual transcription in Humdrum **kern. The paper emphasizes that this setting differs materially from OCR because the source is a dense two-dimensional notation system in which noteheads, stems, beams, clefs, barlines, accidentals, voices, and rhythmic alignment interact spatially and temporally. It also argues that zero-shot OMR is limited by three coupled factors: the lack of large-scale annotated real scans, the visual and structural sim-to-real gap, and the one-to-many mapping between sheet music and **kern strings (Dratschuk et al., 11 May 2026).

The one-to-many issue is treated as a core learning bottleneck rather than a formatting nuisance. In **kern, multiple syntactically different strings may render to the same visible score. The paper gives examples including chord note ordering ambiguity, token syntax ordering ambiguity such as [8fJ versus 8fJ[, contradictory accidental artifacts such as 4c#n, redundant metadata such as *met(c) then *M4/4, and self-canceling ties such as 4G[]. Transcoda addresses this by canonicalizing **kern into a normalized form intended to create a “near-deterministic mapping” from score image to transcription.

This design choice is also the paper’s main substantive claim. Rather than attributing OMR limits primarily to insufficient model scale, it argues that the dominant bottleneck is data design: synthetic training pipelines are often too simplistic, and target strings are often ill-posed because equivalent notations remain unresolved in the supervision signal. The ablation in which target normalization is removed supports that claim directly, with synthetic OMR-NED worsening from 18.71% to 82.51% (Dratschuk et al., 11 May 2026).

2. Model architecture and target representation

Transcoda takes a fixed-resolution image of $1485 \times 1050$ pixels and generates a **kern token sequence up to length 2048. The total parameter count is approximately 58.8M, described in the abstract and discussion as a 59M-parameter model. The architecture has three components: a visual encoder, a projection bridge, and an autoregressive decoder (Dratschuk et al., 11 May 2026).

The visual encoder is a pretrained facebook/convnextv2-tiny-22k-224 ConvNeXt-V2 backbone with 27.9M parameters. Its output visual grid has shape $47 \times 33 \times 768$. Before flattening, the model appends 2D sinusoidal positional encodings to that grid. The projection bridge has 2.6M parameters and is a 2-layer MLP with Linear, GELU, Linear; after flattening, it produces a sequence of shape $1551 \times 512$. The decoder has 28.3M parameters and is an 8-layer Pre-LN Transformer decoder with $d_{\mathrm{model}} = 512$, $d_{\mathrm{ff}} = 1024$, 8 attention heads, GELU feed-forward blocks, and RoPE for self-attention.

The tokenizer is a 3000-token BPE model trained on the normalized **kern corpus, with the explicit constraint that there are no merges across spaces. The paper motivates this as a way to prevent memorization of whole chord strings as single tokens and to preserve a compositional representation capable of generating novel chord combinations. This suggests that tokenization is treated არა only as compression, but as part of the inductive bias for structured music notation modeling.

The paper also explores optional in-domain visual pretraining on about 200,000 unlabeled historical score images from IMSLP using a Fully Convolutional Masked Autoencoder style setup, modified into a dense SimMIM-style variant. Learned mask tokens are injected after patch embedding, the full dense ConvNeXt-V2 encoder runs on masked features, and masking can be biased toward ink-heavy notation regions. This pretraining was only lightly explored due to compute limits, but the reported ablation indicates promising gains on real scans (Dratschuk et al., 11 May 2026).

3. Synthetic corpus construction and normalization

The data engine is one of Transcoda’s central contributions. It converts symbolic corpora to normalized **kern, filters malformed examples, renders synthetic pages, and degrades them to resemble scanned historical media. The source corpora are PDMX, Grandstaff, MuseTrainer, OpenScore Lieder, and OpenScore Quartets. After conversion, filtering, and normalization, the training split yields 216,298 normalized examples from 277,717 raw files, while validation yields 7,583 normalized examples; the real-scan benchmark contains 102 Polish historical scores (Dratschuk et al., 11 May 2026).

The conversion step uses a patched fork of musicxml2hum. The paper notes that the standard implementation had memory safety issues and segfaulted on more than 10% of the PDMX corpus, whereas the patched version reduces failure rate to under 0.07%. Filtering removes files with broken UTF-8, missing spine terminators, missing clefs, severe conversion artifacts, impossible accidental runs, corrupted octave spellings, and invalid measure mathematics.

Normalization is a 21-stage pipeline grouped into four classes. Extraneous spine removal eliminates non-notation parallel spines such as lyrics and dynamics. Visual-semantic alignment removes non-visual symbols such as playback-only grace rests and terminal string markers. Syntactic token sorting enforces a fixed character-level hierarchy within tokens and sorts chord notes in ascending pitch order. Structural repair resolves contradictions introduced upstream, including incompatible accidental combinations. The paper treats this normalization not as a superficial cleanup stage but as the mechanism that collapses a one-to-many transcription space into an approximately one-to-one mapping.

Rendering uses Verovio 6.0.1, and all outputs are scaled to $1485 \times 1050$ pixels. Training-time rendering parameters are randomized; the appendix specifies, among others, scale in $[52,84]$, pageWidth = image_width \cdot U(1.80, 2.40) with image_width = 1050, randomized staff and spacing parameters, and one of five fonts: Leipzig, Bravura, Gootville, Leland, or Petaluma. If rendering fails or the layout is invalid, a retry path tightens the page by lowering scale, reducing spacing, reducing measureMinWidth, possibly enlarging page width, and shrinking margins.

The raster-stage augmentation is explicitly multi-stage. It applies geometric distortions such as affine translations, perspective warping, and spatial stretching via OpenCV; paper compositing such as synthetic textures, lighting gradients, and color casts; and document degradations through Augraphy, including ink bleed, mottling, dirty scanner rollers, shadow casting, and JPEG compression. In addition, Transcoda injects target-irrelevant notation such as dynamics and tempo text into the image without adding them to the **kern target. The paper describes this as training selective transcription: predict structural notation while ignoring distractors.

A further structural step is stochastic score concatenation. Although the normalized source pool has 216,298 examples, a training run uses 310,554 training examples because multiple short samples are concatenated into denser full-page examples containing between one and six systems, subject to a maximum context length of 2048 tokens. The paper reports that removing score concatenation strongly harms transfer, which it interprets as evidence that real pages differ from simplistic synthetic snippets not only visually but also in sequence length and structural density (Dratschuk et al., 11 May 2026).

4. Training protocol, decoding, and validity constraints

Training uses PyTorch Lightning with bfloat16 precision. The paper reports 6 hours of training on one NVIDIA RTX 5090 GPU with 32 GB memory, using AdamW, effective batch size 72, $(\beta_1,\beta_2) = (0.9, 0.999)$, encoder learning rate $3 \times 10^{-4}$, projector and decoder learning rate $1 \times 10^{-3}$, 500 warmup steps, cosine decay to $47 \times 33 \times 768$0, weight decay 0.01 on weight matrices, gradient clipping 1.0, and label smoothing 0.1. It does not print a full loss formula, but states that the model is trained autoregressively and uses label smoothing (Dratschuk et al., 11 May 2026).

Inference supports greedy decoding, beam search with width 3, and constrained decoding. Standard Transcoda decoding uses maximum generation length 2048 and beam width 3. Baselines are evaluated with fixed settings: Legato uses beam width 3, repetition penalty 1.1, and maximum length 2048, while SMT++ uses greedy decoding with maximum length 2048.

The optional constrained decoder is grammar-based. A GBNF grammar is compiled with xgrammar, and at each step the grammar masks locally invalid next tokens. A Python-side logits processor tracks global state that the local grammar cannot express, specifically the active spine count and consistent line width. It masks tabs, newlines, spine split tokens, and spine merge tokens when these would violate structural constraints. The paper states that invalid continuations receive $47 \times 33 \times 768$1 logit mass, so if $47 \times 33 \times 768$2 are the pre-softmax logits at step $47 \times 33 \times 768$3, invalid candidates are assigned $47 \times 33 \times 768$4 before token selection.

This constraint stack guarantees formal validity but not semantic correctness. A syntactically valid **kern output can still be musically wrong if the model is visually misaligned. The paper also notes a practical limit: some runaway generations remain grammatically valid because they repeat valid multiline **kern patterns, so grammar constraints alone cannot eliminate all failure modes.

Evaluation uses three metrics. OMR-NED is the main metric; it is format-agnostic, computes set edit distance between constituent musical symbols, enforces strict temporal offset matching, and only directly compares notes, rests, and non-note directions when they occur at the exact same temporal position within a measure. TEDn is Tree Edit Distance with Note Flattening, computed on MusicXML. CER is character error rate via Levenshtein distance on text strings. The paper does not provide a closed-form formula for OMR-NED, citing Martinez-Sevilla et al. (2025) instead (Dratschuk et al., 11 May 2026).

5. Empirical performance and ablation evidence

Transcoda is evaluated against SMT++ and Legato on both a clean synthetic benchmark of Verovio renderings and zero-shot historical Polish scans. Lower OMR-NED is better. The reported main results are as follows (Dratschuk et al., 11 May 2026).

Benchmark	SMT++	Legato	Transcoda
Clean synthetic benchmark	92.23%	43.91%	18.46%
Historical Polish scans	80.16%	86.73%	63.97%

The headline comparison is that a compact model trained entirely on synthetic data outperforms larger evaluated baselines on both the synthetic benchmark and the real-scan benchmark. The paper’s interpretation is that improved data realism and canonicalized targets can outweigh sheer model scale in zero-shot OMR.

The ablations are unusually decisive. On the synthetic benchmark, removing target normalization worsens OMR-NED from 18.71% to 82.51%, which is the paper’s strongest single piece of evidence that **kern non-uniqueness is a central source of learning instability. On real scans, removing asymmetric semantic augmentation worsens OMR-NED from 65.76% to 78.99%, removing visual degradation worsens it to 76.95%, and removing score concatenation worsens it to 80.10%. These numbers support the paper’s broader claim that data realism is both visual and structural, not merely a matter of adding scanner noise.

Beam search helps, but only modestly. On the synthetic benchmark, OMR-NED improves from 18.71% under greedy decoding to 18.46% with beam search. On the Polish scans, it improves from 65.76% to 63.97%. Grammar-based constrained decoding changes the distance metrics little—18.71% to 18.74% on synthetic data and 65.76% to 63.91% on Polish scans—but the paper does not present it as a metric-optimization device. Its role is to guarantee formal validity for downstream tools that may fail on malformed **kern.

Optional visual pretraining appears promising. The ablation labeled “FCMAE + CNX-V2-Base” improves Polish transfer to 60.70% OMR-NED. The paper also notes a small textual inconsistency: in the narrative, in-domain pretraining is described as improving zero-shot transfer from 63.97% to 61.11% OMR-NED, while the ablation table gives 60.70%. The table is presented as the more precise source in the provided material.

6. Failure modes, design trade-offs, and nomenclature

The paper is explicit about several remaining weaknesses. The principal real-scan failure mode is structural matrix misalignment in dense pianoform notation. The model may hallucinate spine split tokens *^ and spine merge tokens *v; once a wrong split occurs, line widths drift and horizontal alignment can be lost for the rest of the page. A second failure mode is rare synthetic runaway looping: about 1.18% of synthetic cases exhibit catastrophic generation loops in which the prediction becomes up to four times longer than the target by repeating the same valid multiline pattern. Because these loops are syntactically valid **kern, grammar constraints do not remove them (Dratschuk et al., 11 May 2026).

There is also an intentional trade-off between musical semantics and strict visual fidelity. By normalizing away semantically redundant notation, Transcoda often predicts the musically correct pitch while omitting visible courtesy accidentals. The qualitative example on Bach’s Duetto No. 1 in E minor (BWV 802) is used to illustrate this behavior: Transcoda captures internal beam subdivisions better than the baselines, yet exhibits a normalization-induced bias toward omission of courtesy accidentals. This suggests that the system is optimized for stable symbolic transcription rather than pixel-faithful reproduction of every visible mark.

The name “Transcoda” can also be confused with several unrelated research lines. It is distinct from “TransCoder,” the unsupervised neural transcompiler for C++, Java, and Python (Lachaux et al., 2020), and from a different 2023 “TransCoder” that denotes a transferable fine-tuning framework for code pre-trained models rather than a translation system (Sun et al., 2023). It is equally distinct from “Transcoders” in mechanistic interpretability, where the term refers to sparse function-centric approximations of MLP sublayers in vision-LLMs (Damianos et al., 21 May 2026). A plausible implication is that the similarity of names reflects convergent terminology rather than shared method or domain.

Within OMR specifically, Transcoda’s broader significance lies in its data-centric thesis. The paper argues that canonical targets, realistic synthetic rendering, and structurally matched full-page training examples are the dominant levers for zero-shot transfer. Its reported results support that claim, but the remaining failure modes indicate that dense historical scans, archival degradation, and structurally unstable decoding remain open problems (Dratschuk et al., 11 May 2026).