---
title: 'LadderSym: Multimodal Music Error Detection'
url: https://www.emergentmind.com/topics/laddersym
type: topic
---

# LadderSym: Multimodal Music Error Detection

LadderSym is a multimodal encoder–decoder Transformer for music practice error detection that compares a learner’s performance audio against a reference score and generates note-level error annotations. It was introduced as a response to two limitations attributed to prior comparison models: late fusion, which “limits inter-stream alignment and cross-modality comparison capability,” and exclusive reliance on score audio, which “introduces ambiguity in the frequency spectrum” for music with concurrent notes. The model addresses these issues with a two-stream interleaved encoder called “Ladder” and a multimodal decoding strategy that uses symbolic score prompts in addition to score audio [2510.08580].

## 1. Task formulation and conceptual basis

LadderSym is designed for **music practice error detection**, a sequence-comparison problem in which a performed recording is evaluated against a reference score. The output space is organized around three note categories used in the reported evaluation: **Correct notes**, **Missed notes**, and **Extra notes**. The paper also discusses **wrong notes** conceptually, treating a wrong note as a combination of a missed note and an extra note occurring together, while noting that the evaluation tables focus on Correct, Missed, and Extra categories [2510.08580].

The work is situated against two earlier methodological families. One uses explicit alignment, typically by transcribing score and performance and then aligning them with procedures such as Dynamic Time Warping. The other, represented by **Polytune**, uses latent alignment: the score is synthesized into audio and compared with performance audio in a learned latent space. LadderSym is motivated by the claim that explicit alignment becomes brittle when students omit notes, insert notes, or vary timing substantially, while latent alignment with late fusion underuses cross-stream comparison opportunities [2510.08580].

The paper’s motivating critique of score-audio-only comparison is especially tied to polyphony. When several notes occur concurrently, spectral overlap makes it difficult to infer the exact symbolic content from audio alone. This motivates the model’s combination of score audio with a symbolic score representation. The authors further position the architecture as a general comparison model whose lessons may inform “sequence evaluation tasks for reinforcement Learning, human skill assessment, and model evaluation” [2510.08580].

## 2. Interleaved architecture and ladder mechanism

LadderSym uses an encoder–decoder Transformer architecture. Its encoder, named **Ladder**, has two audio streams: a **reference stream** for synthesized score audio and a **practice stream** for performance audio. Rather than processing the two streams independently until a late fusion stage, the encoder inserts **cross-attention alignment modules before each transformer block**, allowing repeated cross-stream interaction throughout the depth of the network [2510.08580].

The encoder updates are defined as
\[
\mathbf{P}_\text{ref}^{(i+1)} = \text{ViT}_\text{ref}\Big( \mathbf{P}_\text{ref}^{(i)} + \text{CA}\big(\mathbf{P}_\text{prac}^{(i)}, \mathbf{P}_\text{ref}^{(i)}\big) \Big),
\]
\[
\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).
\]
Here, each stream is first aligned to the other by cross-attention and then updated by a stream-specific ViT block. This makes the inter-stream exchange alternating and layerwise, rather than centralized in one shared block near the top of the encoder [2510.08580].

After the final encoder layer, the two hidden-state sequences are concatenated,
\[
\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),
\]
and used as context for an autoregressive **T5 decoder**. The decoder is also conditioned by a **symbolic score prompt** prepended before the start-of-sequence token. This multimodal prompting strategy is intended to reduce ambiguity that remains when the score is represented only acoustically [2510.08580].

The output representation is a sequence of MIDI-like event tokens encoding event time, pitch, note-on or note-off status, and an error label such as Correct, Missed, or Extra. The paper describes this representation as following Polytune/MT3-style event tokenization with instrument tokens removed because the setting assumes a single instrument [2510.08580].

## 3. Representations, preprocessing, and training procedure

Audio preprocessing follows the MT3 and Polytune pipeline. Each recording is segmented into **2.145-second non-overlapping segments**. Spectrograms are computed with **2048-point FFT**, **128-sample hop**, and **512 mel bins**. Each spectrogram is then divided into **\(16 \times 16\)** patches, producing **512 tokens per segment per stream** [2510.08580].

The symbolic modality is derived from a MIDI version of the score and tokenized as a decoder prompt. On the output side, the model predicts token sequences corresponding to labeled musical note events. The architecture uses **12 transformer encoder layers** and **8 decoder layers**; the encoder hidden size is **768**, projected to **512** before entering the T5 decoder. The resulting model has **172M parameters** [2510.08580].

Training uses **weighted cross-entropy** to address class imbalance, with the **error loss weight** set to **10**. Optimization uses **AdamW** with cosine annealing of the learning rate from
\[
2 \times 10^{-4} \rightarrow 1 \times 10^{-4},
\]
for **300 epochs** in **bf16-mixed** precision. Batch size is **48 spectrograms** for MAESTRO-E and **96 spectrograms** for CocoChorales-E. The implementation uses **PyTorch 2.3.0**, **Transformers 4.40.1**, and **NVIDIA A100-80GB GPU**. The training recipe also includes **token shuffling**, following MR-MT3, to improve generalization [2510.08580].

## 4. Datasets, synthetic error generation, and evaluation protocol

LadderSym is evaluated on two synthetic practice-error benchmarks. **MAESTRO-E** is derived from MAESTRO and contains **200+ hours** of piano audio, **1k+ tracks**, and **200k+ annotated errors**. **CocoChorales-E** is derived from CocoChorales and contains **300+ hours** of audio, **40k+ tracks**, **13 instruments**, and **25k+ annotated errors**. The paper emphasizes that MAESTRO-E has **high note concurrency**, whereas CocoChorales-E has **no overlapping notes** [2510.08580].

Because large corpora of real annotated practice errors are unavailable, the datasets are created by injecting errors into clean MIDI. The error rate is sampled as
\[
\lambda \sim U(0.1, 0.4),
\]
with pitch offsets drawn from a truncated normal distribution centered at zero with standard deviation \(1\), and timing offsets drawn from a truncated normal distribution centered at zero with standard deviation \(0.02\). The generation procedure can apply missed notes, pitch changes, timing shifts, and extra notes, after which the annotated MIDI is split into separate labeled tracks for Correct, Missed, and Extra events [2510.08580].

Evaluation uses note-level **Error Detection F1** with **mir\_eval** matching conventions, including a **50 ms tolerance** and pitch agreement. The paper states that all reported results are evaluated over a **combined test set of 4401 tracks** [2510.08580].

## 5. Empirical performance and comparison with prior systems

LadderSym is compared against a classical score-informed baseline and against **Polytune**, described as the previous state of the art. On **MAESTRO-E**, the paper reports the following F1 scores: Correct **94.4\%**, Missed **54.7\%**, and Extra **86.4\%** for LadderSym, versus Correct **90.1\%**, Missed **26.8\%**, and Extra **72.0\%** for Polytune. On **CocoChorales-E**, LadderSym achieves Correct **97.7\%**, Missed **61.7\%**, and Extra **61.4\%**, versus Correct **95.4\%**, Missed **51.3\%**, and Extra **46.8\%** for Polytune [2510.08580].

The abstract highlights especially large gains on MAESTRO-E, stating that LadderSym “more than doubles F1 for missed notes on MAESTRO-E (26.8\% \(\rightarrow\) 56.3\%) and improves extra note detection by 14.4 points (72.0\% \(\rightarrow\) 86.4\%).” The detailed result table, however, lists **56.3\%** as the **Missed-note recall** and **54.7\%** as the **Missed-note F1**. This establishes a discrepancy between the abstract-level summary and the main table for that metric [2510.08580].

The model is also reported to be somewhat smaller and faster than Polytune under the paper’s setup: **172M** parameters and **124 ms/token** for LadderSym, versus **192M** parameters and **147 ms/token** for Polytune [2510.08580].

The reported gains are largest on MAESTRO-E, which the paper attributes to the dataset’s high note concurrency. This is consistent with the paper’s central claim that symbolic prompting is most helpful when the acoustic score representation is ambiguous because multiple notes overlap in time [2510.08580].

## 6. Ablation studies, probing analyses, and limitations

The ablation studies separate the two main design claims. For input modality, the paper compares **Prompt Only**, **Audio Only**, and **Prompt + Audio**. On MAESTRO-E, Missed-note F1 rises from **24.3\%** with Prompt Only and **26.8\%** with Audio Only to **46.7\%** with Prompt + Audio; Extra-note F1 rises from **62.5\%** and **72.0\%** to **81.7\%**. On CocoChorales-E, Missed-note F1 rises from **44.6\%** and **46.8\%** to **56.1\%**, and Extra-note F1 from **45.8\%** and **51.3\%** to **58.1\%**. This indicates that symbolic prompting is most effective as a complement to, rather than a replacement for, acoustic comparison [2510.08580].

A second ablation varies how much of the encoder is jointly fused. With 12 total layers on CocoChorales-E, the paper reports that \(L_{\text{joint}}=1\) gives Missed **51.26\%** and Extra **46.40\%**, \(L_{\text{joint}}=2\) gives Missed **59.58\%** and Extra **57.38\%**, \(L_{\text{joint}}=3\) gives Missed **56.81\%** and Extra **59.61\%**, \(L_{\text{joint}}=4\) gives Missed **59.51\%** and Extra **58.11\%**, and \(L_{\text{joint}}=12\) gives Missed **54.60\%** and Extra **56.20\%**. The paper interprets this as evidence that some earlier fusion helps, but too much fully shared fusion reduces stream specialization [2510.08580].

Encoder-level ablations compare **3 Joint Encoders**, **Self-Attention**, **Ladder**, and **LadderSym**. On MAESTRO-E, Missed-note F1 improves from **36.1\%** and **33.8\%** to **46.0\%** with Ladder and **54.7\%** with LadderSym; Extra-note F1 improves from **75.3\%** and **74.6\%** to **82.0\%** and **86.4\%**. On CocoChorales-E, LadderSym does not dominate Ladder in every category, which the paper relates to the fact that CocoChorales-E has no overlapping notes and therefore less spectral ambiguity in the score-audio channel [2510.08580].

The probing analysis measures locality, globality, and cross-stream correspondence in frozen encoder representations. For LadderSym, the paper reports Local\(_{\text{score}}\) **0.197**, Local\(_{\text{practice}}\) **0.681**, Global\(_{\text{score}}\) **0.162**, Global\(_{\text{practice}}\) **0.252**, cross-stream practice \(\rightarrow\) score **0.158**, and cross-stream score \(\rightarrow\) practice **0.300**. The authors interpret these values as showing an asymmetric division of labor: the practice stream retains local detail, while the score stream becomes more cross-stream informed [2510.08580].

The reported gains over prior methods are supported by paired \(t\)-tests and Wilcoxon signed-rank tests with Bonferroni-corrected \(\alpha=0.017\). For Polytune versus LadderSym, the paper reports \(t=-21.85\), \(p_t=4.98\times 10^{-103}\), \(W=1.79\times 10^6\), \(p_w=1.47\times 10^{-170}\) on CocoChorales-E, and \(t=-20.53\), \(p_t=1.43\times 10^{-85}\), \(W=6.25\times 10^5\), \(p_w=1.03\times 10^{-67}\) on MAESTRO-E [2510.08580].

The paper also identifies several limitations. The training audio is relatively homogeneous and synthesized, so performance may degrade under real-world shifts in timbre, acoustics, recording conditions, or microphones. The method requires **score audio**, **symbolic score**, and **performance audio**, which increases modality dependence. The datasets are synthetic and may not span the full distribution of real student errors. The paper suggests future robustness work via pitch shifting, reverberation, noise injection, more diverse synthesis sources, and retraining MIDI-DDSP on target timbres [2510.08580].

Source: https://www.emergentmind.com/topics/laddersym