LadderSym: Multimodal Music Error Detection

• LadderSym is a multimodal encoder–decoder Transformer designed for note-level music error detection by comparing performance audio to both synthesized and symbolic score representations.
• It employs a two-stream interleaved encoder with cross-attention layers before each transformer block, enhancing alignment and addressing limitations of late fusion methods.
• The model achieves significant gains in detecting Correct, Missed, and Extra notes using synthetic benchmarks, detailed ablations, and optimized training protocols.

LadderSym is a multimodal encoder–decoder Transformer for music practice error detection that compares a learner’s performance audio against a reference score and generates note-level error annotations. It was introduced as a response to two limitations attributed to prior comparison models: late fusion, which “limits inter-stream alignment and cross-modality comparison capability,” and exclusive reliance on score audio, which “introduces ambiguity in the frequency spectrum” for music with concurrent notes. The model addresses these issues with a two-stream interleaved encoder called “Ladder” and a multimodal decoding strategy that uses symbolic score prompts in addition to score audio (Chou et al., 16 Sep 2025).

1. Task formulation and conceptual basis

LadderSym is designed for music practice error detection, a sequence-comparison problem in which a performed recording is evaluated against a reference score. The output space is organized around three note categories used in the reported evaluation: Correct notes, Missed notes, and Extra notes. The paper also discusses wrong notes conceptually, treating a wrong note as a combination of a missed note and an extra note occurring together, while noting that the evaluation tables focus on Correct, Missed, and Extra categories (Chou et al., 16 Sep 2025).

The work is situated against two earlier methodological families. One uses explicit alignment, typically by transcribing score and performance and then aligning them with procedures such as Dynamic Time Warping. The other, represented by Polytune, uses latent alignment: the score is synthesized into audio and compared with performance audio in a learned latent space. LadderSym is motivated by the claim that explicit alignment becomes brittle when students omit notes, insert notes, or vary timing substantially, while latent alignment with late fusion underuses cross-stream comparison opportunities (Chou et al., 16 Sep 2025).

The paper’s motivating critique of score-audio-only comparison is especially tied to polyphony. When several notes occur concurrently, spectral overlap makes it difficult to infer the exact symbolic content from audio alone. This motivates the model’s combination of score audio with a symbolic score representation. The authors further position the architecture as a general comparison model whose lessons may inform “sequence evaluation tasks for reinforcement Learning, human skill assessment, and model evaluation” (Chou et al., 16 Sep 2025).

2. Interleaved architecture and ladder mechanism

LadderSym uses an encoder–decoder Transformer architecture. Its encoder, named Ladder, has two audio streams: a reference stream for synthesized score audio and a practice stream for performance audio. Rather than processing the two streams independently until a late fusion stage, the encoder inserts cross-attention alignment modules before each transformer block, allowing repeated cross-stream interaction throughout the depth of the network (Chou et al., 16 Sep 2025).

The encoder updates are defined as

$$\mathbf{P}_\text{ref}^{(i+1)} = \text{ViT}_\text{ref}\Big( \mathbf{P}_\text{ref}^{(i)} + \text{CA}\big(\mathbf{P}_\text{prac}^{(i)}, \mathbf{P}_\text{ref}^{(i)}\big) \Big),$$

$$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$

Here, each stream is first aligned to the other by cross-attention and then updated by a stream-specific ViT block. This makes the inter-stream exchange alternating and layerwise, rather than centralized in one shared block near the top of the encoder (Chou et al., 16 Sep 2025).

After the final encoder layer, the two hidden-state sequences are concatenated,

$$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$

and used as context for an autoregressive T5 decoder. The decoder is also conditioned by a symbolic score prompt prepended before the start-of-sequence token. This multimodal prompting strategy is intended to reduce ambiguity that remains when the score is represented only acoustically (Chou et al., 16 Sep 2025).

The output representation is a sequence of MIDI-like event tokens encoding event time, pitch, note-on or note-off status, and an error label such as Correct, Missed, or Extra. The paper describes this representation as following Polytune/MT3-style event tokenization with instrument tokens removed because the setting assumes a single instrument (Chou et al., 16 Sep 2025).

3. Representations, preprocessing, and training procedure

Audio preprocessing follows the MT3 and Polytune pipeline. Each recording is segmented into 2.145-second non-overlapping segments. Spectrograms are computed with 2048-point FFT, 128-sample hop, and 512 mel bins. Each spectrogram is then divided into $16 \times 16$ patches, producing 512 tokens per segment per stream (Chou et al., 16 Sep 2025).

The symbolic modality is derived from a MIDI version of the score and tokenized as a decoder prompt. On the output side, the model predicts token sequences corresponding to labeled musical note events. The architecture uses 12 transformer encoder layers and 8 decoder layers; the encoder hidden size is 768, projected to 512 before entering the T5 decoder. The resulting model has 172M parameters (Chou et al., 16 Sep 2025).

Training uses weighted cross-entropy to address class imbalance, with the error loss weight set to 10. Optimization uses AdamW with cosine annealing of the learning rate from

$2 \times 10^{-4} \rightarrow 1 \times 10^{-4},$

for 300 epochs in bf16-mixed precision. Batch size is 48 spectrograms for MAESTRO-E and 96 spectrograms for CocoChorales-E. The implementation uses PyTorch 2.3.0, Transformers 4.40.1, and NVIDIA A100-80GB GPU. The training recipe also includes token shuffling, following MR-MT3, to improve generalization (Chou et al., 16 Sep 2025).

4. Datasets, synthetic error generation, and evaluation protocol

LadderSym is evaluated on two synthetic practice-error benchmarks. MAESTRO-E is derived from MAESTRO and contains 200+ hours of piano audio, 1k+ tracks, and 200k+ annotated errors. CocoChorales-E is derived from CocoChorales and contains 300+ hours of audio, 40k+ tracks, 13 instruments, and 25k+ annotated errors. The paper emphasizes that MAESTRO-E has high note concurrency, whereas CocoChorales-E has no overlapping notes (Chou et al., 16 Sep 2025).

Because large corpora of real annotated practice errors are unavailable, the datasets are created by injecting errors into clean MIDI. The error rate is sampled as

$\lambda \sim U(0.1, 0.4),$

with pitch offsets drawn from a truncated normal distribution centered at zero with standard deviation $1$, and timing offsets drawn from a truncated normal distribution centered at zero with standard deviation $0.02$. The generation procedure can apply missed notes, pitch changes, timing shifts, and extra notes, after which the annotated MIDI is split into separate labeled tracks for Correct, Missed, and Extra events (Chou et al., 16 Sep 2025).

Evaluation uses note-level Error Detection F1 with mir_eval matching conventions, including a 50 ms tolerance and pitch agreement. The paper states that all reported results are evaluated over a combined test set of 4401 tracks (Chou et al., 16 Sep 2025).

5. Empirical performance and comparison with prior systems

LadderSym is compared against a classical score-informed baseline and against Polytune, described as the previous state of the art. On MAESTRO-E, the paper reports the following F1 scores: Correct 94.4\%, Missed 54.7\%, and Extra 86.4\% for LadderSym, versus Correct 90.1\%, Missed 26.8\%, and Extra 72.0\% for Polytune. On CocoChorales-E, LadderSym achieves Correct 97.7\%, Missed 61.7\%, and Extra 61.4\%, versus Correct 95.4\%, Missed 51.3\%, and Extra 46.8\% for Polytune (Chou et al., 16 Sep 2025).

The abstract highlights especially large gains on MAESTRO-E, stating that LadderSym “more than doubles F1 for missed notes on MAESTRO-E (26.8\% $\rightarrow$ 56.3\%) and improves extra note detection by 14.4 points (72.0\% $\rightarrow$ 86.4\%).” The detailed result table, however, lists 56.3\% as the Missed-note recall and 54.7\% as the Missed-note F1. This establishes a discrepancy between the abstract-level summary and the main table for that metric (Chou et al., 16 Sep 2025).

The model is also reported to be somewhat smaller and faster than Polytune under the paper’s setup: 172M parameters and 124 ms/token for LadderSym, versus 192M parameters and 147 ms/token for Polytune (Chou et al., 16 Sep 2025).

The reported gains are largest on MAESTRO-E, which the paper attributes to the dataset’s high note concurrency. This is consistent with the paper’s central claim that symbolic prompting is most helpful when the acoustic score representation is ambiguous because multiple notes overlap in time (Chou et al., 16 Sep 2025).

6. Ablation studies, probing analyses, and limitations

The ablation studies separate the two main design claims. For input modality, the paper compares Prompt Only, Audio Only, and Prompt + Audio. On MAESTRO-E, Missed-note F1 rises from 24.3\% with Prompt Only and 26.8\% with Audio Only to 46.7\% with Prompt + Audio; Extra-note F1 rises from 62.5\% and 72.0\% to 81.7\%. On CocoChorales-E, Missed-note F1 rises from 44.6\% and 46.8\% to 56.1\%, and Extra-note F1 from 45.8\% and 51.3\% to 58.1\%. This indicates that symbolic prompting is most effective as a complement to, rather than a replacement for, acoustic comparison (Chou et al., 16 Sep 2025).

A second ablation varies how much of the encoder is jointly fused. With 12 total layers on CocoChorales-E, the paper reports that $$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$0 gives Missed 51.26\% and Extra 46.40\%, $$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$1 gives Missed 59.58\% and Extra 57.38\%, $$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$2 gives Missed 56.81\% and Extra 59.61\%, $$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$3 gives Missed 59.51\% and Extra 58.11\%, and $$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$4 gives Missed 54.60\% and Extra 56.20\%. The paper interprets this as evidence that some earlier fusion helps, but too much fully shared fusion reduces stream specialization (Chou et al., 16 Sep 2025).

Encoder-level ablations compare 3 Joint Encoders, Self-Attention, Ladder, and LadderSym. On MAESTRO-E, Missed-note F1 improves from 36.1\% and 33.8\% to 46.0\% with Ladder and 54.7\% with LadderSym; Extra-note F1 improves from 75.3\% and 74.6\% to 82.0\% and 86.4\%. On CocoChorales-E, LadderSym does not dominate Ladder in every category, which the paper relates to the fact that CocoChorales-E has no overlapping notes and therefore less spectral ambiguity in the score-audio channel (Chou et al., 16 Sep 2025).

The probing analysis measures locality, globality, and cross-stream correspondence in frozen encoder representations. For LadderSym, the paper reports Local$$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$5 0.197, Local$$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$6 0.681, Global$$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$7 0.162, Global$$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$8 0.252, cross-stream practice $$\mathbf{P}_\text{prac}^{(i+1)} = \text{ViT}_\text{prac}\Big( \mathbf{P}_\text{prac}^{(i)} + \text{CA}\big(\mathbf{P}_\text{ref}^{(i+1)}, \mathbf{P}_\text{prac}^{(i)}\big) \Big).$$9 score 0.158, and cross-stream score $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$0 practice 0.300. The authors interpret these values as showing an asymmetric division of labor: the practice stream retains local detail, while the score stream becomes more cross-stream informed (Chou et al., 16 Sep 2025).

The reported gains over prior methods are supported by paired $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$1-tests and Wilcoxon signed-rank tests with Bonferroni-corrected $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$2. For Polytune versus LadderSym, the paper reports $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$3, $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$4, $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$5, $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$6 on CocoChorales-E, and $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$7, $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$8, $$\mathbf{H}_\text{fused} = \text{Concat}(\mathbf{P}_\text{ref}^{(\text{final})}, \mathbf{P}_\text{prac}^{(\text{final})}),$$9, $16 \times 16$0 on MAESTRO-E (Chou et al., 16 Sep 2025).

The paper also identifies several limitations. The training audio is relatively homogeneous and synthesized, so performance may degrade under real-world shifts in timbre, acoustics, recording conditions, or microphones. The method requires score audio, symbolic score, and performance audio, which increases modality dependence. The datasets are synthetic and may not span the full distribution of real student errors. The paper suggests future robustness work via pitch shifting, reverberation, noise injection, more diverse synthesis sources, and retraining MIDI-DDSP on target timbres (Chou et al., 16 Sep 2025).