---
title: Synthetic Sheet Music Reasoning Benchmark
url: https://www.emergentmind.com/topics/synthetic-sheet-music-reasoning-benchmark-ssmr-bench
type: topic
---

# Synthetic Sheet Music Reasoning Benchmark

Searching arXiv for the cited benchmark and related papers.
arXiv search: "2509.04059 Synthetic Sheet Music Reasoning Benchmark SSMR-Bench"
Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench) is a synthetic, music-theory-grounded benchmark for evaluating and improving the capacity of LLMs and MLLMs to interpret Western staff notation in both textual and visual forms [2509.04059]. It was introduced together with a data synthesis framework that programmatically generates sheet-music question-answer pairs with automatically verifiable answers, thereby supporting both benchmark evaluation and Reinforcement Learning with Verifiable Rewards (RLVR) [2509.04059]. Within the emerging literature on machine reasoning over symbolic and visual music representations, SSMR-Bench occupies a distinct position: it targets theory-grounded compositional reasoning rather than only symbol recognition, and it is explicitly designed to supply objective reward signals for reinforcement learning [2509.04059].

## 1. Definition, scope, and research context

SSMR-Bench (Synthetic Sheet Music Reasoning Benchmark) is a large-scale, rule-grounded evaluation suite for testing and improving the ability of LLMs and MLLMs to read and reason about Western staff notation [2509.04059]. Its central motivation is to address two limitations identified in prior work: the lack of standardized, scalable evaluation data for sheet-music QA in both textual and visual formats, and the absence of verifiable training corpora suitable for RLVR aimed at music-theoretic reasoning [2509.04059].

The benchmark is positioned against prior resources including MusicTheoryBench, MMMU, and MusiXQA. In this framing, those earlier benchmarks either operated at smaller scale or emphasized recognition of individual symbols rather than theory-grounded, compositional reasoning, and they were not designed for reinforcement learning with objective reward signals [2509.04059]. MusiXQA, for example, introduced a comprehensive synthetic dataset for music sheet understanding with structured annotations and tasks spanning OCR, OMR, layout understanding, and chord estimation, but its emphasis was on music-sheet understanding rather than on a benchmark specifically structured for verifiable reasoning-driven RL training [2506.23009].

This suggests that SSMR-Bench should be understood not merely as another notation-reading benchmark, but as a benchmark-training dual-use artifact: its evaluation set measures reasoning over notation, while its companion training set instantiates a controlled environment for policy optimization under exact reward supervision [2509.04059].

## 2. Rule-grounded synthesis framework

The synthesis framework underlying SSMR-Bench leverages formal music-theory rules to programmatically generate infinite QA pairs with automatically verifiable answers [2509.04059]. The formalization begins with a pitch-class encoding in which each note’s pitch is mapped to an integer modulo 12, written as \(n \in \mathbb{N}_{12}\) [2509.04059]. Directed intervals are defined as \(\Delta = (p_2 - p_1) \bmod 12\), and chord construction is expressed through explicit set-theoretic rules, such as the major triad example
\[
\mathrm{MajorTriad}(r) = \{ r, r + 4, r + 7 \} \bmod 12
\]
[2509.04059].

The generation pipeline has three stages [2509.04059]. First, it samples a musical snippet in ABC notation, for example `L:1/8 K:C | c2 e2 g2 c'2 | …`. Second, it applies a rule-based template to generate question text, multiple-choice options, and the correct answer. Third, it uses `abcm2ps + ImageMagick` to render the ABC snippet into a staff-notation image. The result is a paired item consisting of either textual input (ABC + question) or visual input (PNG + question), together with a provably correct answer [2509.04059].

Nine question template classes are distributed across four domains: Rhythm, Interval, Chord, and Scale [2509.04059]. Rhythm includes time-signature inference and bar-placement; Interval includes interval naming and note completion by interval; Chord includes chord identification, root identification, and chord completion; Scale includes scale identification and membership testing [2509.04059]. Because the answers are generated from explicit music-theoretic rules, correctness can be checked algorithmically rather than heuristically [2509.04059].

A related but distinct construction recipe was later described in work on MusICA-MetaBench, which specialized an end-to-end pipeline from MusicXML to rendered sheet images and multiple-choice items [2607.06015]. That recipe uses MusicXML parsing, rendering via MuseScore, LilyPond, or Verovio, feature extraction with `music21`, ontology dictionaries for pitches, intervals, durations, chord types, and keys, and multiple-choice instantiation with distractors and occasional “None of the above” options [2607.06015]. Although this recipe is presented as a detailed pathway for constructing SSMR-Bench end-to-end, it differs from the original SSMR-Bench paper in its reliance on MusicXML rather than ABC notation [2607.06015]. A plausible implication is that SSMR-Bench admits a broader implementation family than the original paper’s concrete instantiation.

## 3. Modalities, task structure, and benchmark composition

SSMR-Bench is explicitly bimodal. It contains both textual QA items and visual QA items, enabling comparison between reasoning over symbolic text representations and reasoning over rendered notation images [2509.04059]. The evaluation set comprises 1,600 textual QA items and 1,600 visual QA items, while the RLVR training set contains 8,000 textual and 8,000 visual QA items [2509.04059]. Per modality, the domain distribution is balanced: 25% Scale, 25% Rhythm, 25% Interval, and 25% Chord [2509.04059].

The benchmark’s structure is therefore symmetric across modalities and domains.

| Component | Textual | Visual |
|---|---:|---:|
| Evaluation set | 1,600 | 1,600 |
| Training set for RLVR | 8,000 | 8,000 |
| Domain breakdown | 25% each across 4 domains | 25% each across 4 domains |

In the original SSMR-Bench formulation, the item types are verifiable QA problems grounded in theory rules [2509.04059]. In the MusicXML-based construction recipe associated with MusICA-MetaBench, the question interface is multiple-choice closed QA with 5 options per question: 1 correct answer, 3 distractors, and 1 NOTA option, with NOTA correct in 20% of items to match a uniform prior of \(1/5\) [2607.06015]. The prompt template for visual evaluation is likewise specified: a sheet image is embedded through a Vision-LLM API and the model is instructed to answer with a letter from A to E [2607.06015].

The predefined question templates in that construction recipe cover pitch recognition, interval identification, chord type recognition, key signature detection, rhythmic duration, and temporal proportion [2607.06015]. Three worked examples are given: “What is the 2nd interval in the bass line?” with answer “Minor Third”; “What chord occurs on the downbeat of measure 5?” with answer “Diminished triad”; and “What is the rhythmic notation of the 3rd soprano note?” with answer “Eighth note (\(\tfrac{1}{8}\))” [2607.06015].

This suggests that the benchmark family spans a spectrum from theory-grounded free-answer or structured-answer tasks to tightly controlled multiple-choice perception-reasoning tasks, with modality parity as a central design principle.

## 4. Evaluation protocol and statistical design

In the original SSMR-Bench paper, accuracy is the primary metric, defined as the percentage of correctly answered items [2509.04059]. This is consistent with the benchmark’s verifiable structure: because every item has a programmatically determined ground truth, exact matching suffices for evaluation [2509.04059].

The associated RLVR setting uses a binary reward function,
\[
R(a,q) = 1 \text{ if model’s answer } a = \text{ ground truth } a^*; \text{ otherwise } 0,
\]
and optimizes the policy objective
\[
\max \mathbb{E}_{q \sim D,\; a \sim \pi_\theta(\cdot|q)}[R(a,q)]
\]
via Group Relative Policy Optimization (GRPO) without KL or entropy penalties, optimizing solely for accuracy [2509.04059]. A simplified policy-gradient step is also specified:
\[
\nabla_\theta J \approx \mathbb{E}_{q,a}\left[\nabla_\theta \log \pi_\theta(a|q) \cdot (R(a,q)-b)\right]
\]
[2509.04059].

The MusicXML-based SSMR-Bench recipe adds a detailed statistical framework for benchmark sizing and significance testing [2607.06015]. It defines a minimum meaningful accuracy difference \(E\), proposes the standard two-proportion sample-size formula with significance level \(\alpha\) and power \(1-\beta\), and recommends bootstrap calibration by repeated balanced subsampling [2607.06015]. For candidate benchmark sizes \(s \in \{60,120,300,\dots\}\), it measures the standard deviation of model accuracy across seeds and identifies the smallest \(s\) for which paired t-tests, Bonferroni-corrected, detect a difference of \(E\)% in all pairs [2607.06015]. For a typical \(E=5\)% effect size and 5 categories \(\times\) 4 skills, it reports that \(s \approx 300\) total items, or 60 per category, is sufficient [2607.06015].

The same protocol also specifies cross-modal ablations designed to verify that performance depends on actual perception of notation rather than prompt priors [2607.06015]. These include a no-input condition, where the sheet image is omitted, and a noise-input condition, where the image is replaced by white or Gaussian noise [2607.06015]. The purpose is to compare accuracy drops and thereby confirm that the model must “look at” the sheet [2607.06015].

## 5. Empirical results and model behavior

The benchmark reveals a marked disparity between textual and visual performance [2509.04059]. On textual QA, Deepseek-R1 achieved 93.63%, Qwen3-32B 76.69%, Deepseek-V3 74.94%, QwQ-32B 72.38%, GPT-4.1-mini 71.69%, Qwen3-8B 57.88%, and the random baseline 25.00% [2509.04059]. On visual QA, Gemini 2.5 Pro achieved 55.44%, InternVL3-78B 45.75%, Qwen2.5-VL-72B-Instruct 45.56%, and InternVL2.5-78B 40.44% [2509.04059].

These results support the paper’s claim that reasoning is important for interpreting sheet music, while also showing that current MLLMs still face substantial difficulty when notation is presented visually [2509.04059]. In particular, the poor performance of Gemini 2.5 Pro on visual QA is highlighted as evidence of the challenge of sheet-music interpretation in image form [2509.04059].

The paper further compares against MusicTheoryBench, a set of 367 human-written ABC questions [2509.04059]. On that benchmark, GPT-4-0shot achieved 41.90%, GPT-4-CoT 52.55%, GPT-4-RolePlay 52.45%, and baseline Qwen3-8B 28.65% [2509.04059]. This comparison is important because it places SSMR-Bench-trained models against a human-authored out-of-distribution evaluation set rather than only against synthetic held-out data [2509.04059].

MusiXQA provides additional context for the difficulty of visual score understanding. Its evaluations on synthetic sheet music showed strong OCR gains but weak OMR and chord reasoning for zero-shot MLLMs, with GPT-4o scoring 68.9/85.3 on OCR but only 4.0/42.0 on OMR and 5.5/74.6 on chord tasks in G-Acc/PNLS, while the fine-tuned Phi-3-MusiX (kern+) improved OMR to 68.4/99.2 and chord estimation to 84.9/96.5 [2506.23009]. The MusiXQA error analysis argues that high PNLS can reflect format mimicry rather than actual recognition, and that `kern+` output forces focus on pitch and duration content tokens [2506.23009]. This provides a useful adjacent result: both SSMR-Bench and MusiXQA indicate that superficial output conformity can mask weak symbolic understanding unless the task design and metric tightly couple answer form to musical content.

## 6. RLVR, transfer, and implications for music generation

A defining feature of SSMR-Bench is its use as training data for RLVR [2509.04059]. Models fine-tuned via GRPO on 8,000 synthetic QA items showed substantial gains. Qwen3-8B-Base on textual QA improved from 23.18% pre-RLVR to 70.94% post-RLVR, a gain of +47.76, and exceeded the original Qwen3-8B by 13.06 points [2509.04059]. Qwen2.5-VL-7B-Instruct on visual QA improved from 41.81% to 70.56%, a gain of +28.75, and surpassed Gemini 2.5 Pro’s 55.44% [2509.04059]. Training dynamics are described as showing cumulative reward steadily rising, while response length stabilizes under the system prompt [2509.04059].

The improvements transfer beyond the synthetic benchmark. On MusicTheoryBench, Qwen3-8B-Base + Music-RLVR reached 49.97%, an improvement of +21.32 over its baseline 28.65%, and surpassed GPT-4-0shot [2509.04059]. On mathematics benchmarks, the same model improved on AIME24 to 13.75% (+5.00), AMC to 52.11% (+12.95), MATH-500 to 79.40% (+13.40), Minerva to 34.19% (+8.82), and Olympiad to 40.44% (+10.22) [2509.04059]. The paper interprets this as indicating that domain-specific RLVR yields the largest gains on target tasks, with moderate cross-domain spill-over [2509.04059].

The work also links improved sheet-music reasoning to symbolic composition. In a four-measure continuation task, Qwen3-8B-Base + Music-RLVR improved Rhythmic Consistency (RC) from 66.67% to 99.33% and Novel Rhythmic Coherence (NRC) from 50.00% to 88.89% [2509.04059]. The paper accordingly states that enhanced reasoning ability can facilitate music composition and opens the door to agentic music creation workflows, automated score editing, and educational tools for music theory [2509.04059].

This suggests a broader interpretation of SSMR-Bench: it is not only a diagnostic benchmark for notation understanding, but also an instrument for shaping symbolic reasoning policies whose effects can propagate into generative music tasks.

## 7. Limitations, misconceptions, and future directions

A common misconception would be to equate SSMR-Bench with generic OMR evaluation. The available evidence does not support that reduction. The benchmark is framed around music-theoretic reasoning over notation rather than around transcription alone [2509.04059]. By contrast, MusiXQA explicitly includes OCR- and OMR-based tasks such as title extraction, note durations, note pitches, and clef or bar counting, with outputs in free text, JSON, or `kern+` formats [2506.23009]. The two resources therefore overlap in musical input domain but differ materially in epistemic target: SSMR-Bench emphasizes verifiable reasoning tasks, whereas MusiXQA more broadly benchmarks sheet understanding and recognition [2506.23009; 2509.04059].

Another misconception would be to treat strong textual performance as evidence of mature visual sheet-music competence. The reported results contradict this: textual QA results are substantially higher than visual QA results, and the authors explicitly note that advanced MLLMs still struggle on visual notation [2509.04059]. The no-input and noise-input ablations proposed in the later construction recipe reinforce the importance of verifying actual visual dependence in any claimed score-reading ability [2607.06015].

Several limitations are explicitly documented in the MusicXML-based recipe [2607.06015]. Synthetic sheet images may lack OMR noise present in real scans, so pure synthetic evaluation may overestimate performance. Current templates focus on monophonic queries per staff, while multi-voice queries such as intervals between soprano and alto require future extension. The repertoire is currently Western tonal only; modal and microtonal systems would require adaptation of both ontology and templates [2607.06015]. These caveats are consistent with the original benchmark’s emphasis on Western staff notation and formal music-theory rules [2509.04059].

Future directions are therefore already implicit in the benchmark family. One path is expansion from ABC-based synthesis to broader MusicXML-driven or user-provided corpora, as exemplified by the MusICA-MetaBench paradigm of deriving on-demand benchmarks directly from structured symbolic data [2607.06015]. Another is richer polyphonic and cross-staff reasoning. A further direction is closer coupling between evaluation, RLVR, and downstream composition systems, since the existing results already indicate that gains in notation reasoning can translate into improved compositional behavior [2509.04059].

Source: https://www.emergentmind.com/topics/synthetic-sheet-music-reasoning-benchmark-ssmr-bench