Synthetic Sheet Music Reasoning Benchmark

• The paper introduces SSMR-Bench, a benchmark that employs rule-grounded synthesis to generate infinite, verifiable QA pairs for evaluating theory-driven reasoning over Western staff notation.
• It leverages formal music-theory rules and an ABC notation pipeline to produce dual-modality data that supports both traditional evaluation and reinforcement learning with verifiable rewards.
• Empirical results reveal a significant performance gap between textual and visual modalities, underscoring challenges in visual sheet music interpretation and its impact on music generation.

Searching arXiv for the cited benchmark and related papers. arXiv search: "(Wang et al., 4 Sep 2025) Synthetic Sheet Music Reasoning Benchmark SSMR-Bench" Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench) is a synthetic, music-theory-grounded benchmark for evaluating and improving the capacity of LLMs and MLLMs to interpret Western staff notation in both textual and visual forms (Wang et al., 4 Sep 2025). It was introduced together with a data synthesis framework that programmatically generates sheet-music question-answer pairs with automatically verifiable answers, thereby supporting both benchmark evaluation and Reinforcement Learning with Verifiable Rewards (RLVR) (Wang et al., 4 Sep 2025). Within the emerging literature on machine reasoning over symbolic and visual music representations, SSMR-Bench occupies a distinct position: it targets theory-grounded compositional reasoning rather than only symbol recognition, and it is explicitly designed to supply objective reward signals for reinforcement learning (Wang et al., 4 Sep 2025).

1. Definition, scope, and research context

SSMR-Bench (Synthetic Sheet Music Reasoning Benchmark) is a large-scale, rule-grounded evaluation suite for testing and improving the ability of LLMs and MLLMs to read and reason about Western staff notation (Wang et al., 4 Sep 2025). Its central motivation is to address two limitations identified in prior work: the lack of standardized, scalable evaluation data for sheet-music QA in both textual and visual formats, and the absence of verifiable training corpora suitable for RLVR aimed at music-theoretic reasoning (Wang et al., 4 Sep 2025).

The benchmark is positioned against prior resources including MusicTheoryBench, MMMU, and MusiXQA. In this framing, those earlier benchmarks either operated at smaller scale or emphasized recognition of individual symbols rather than theory-grounded, compositional reasoning, and they were not designed for reinforcement learning with objective reward signals (Wang et al., 4 Sep 2025). MusiXQA, for example, introduced a comprehensive synthetic dataset for music sheet understanding with structured annotations and tasks spanning OCR, OMR, layout understanding, and chord estimation, but its emphasis was on music-sheet understanding rather than on a benchmark specifically structured for verifiable reasoning-driven RL training (Chen et al., 28 Jun 2025).

This suggests that SSMR-Bench should be understood not merely as another notation-reading benchmark, but as a benchmark-training dual-use artifact: its evaluation set measures reasoning over notation, while its companion training set instantiates a controlled environment for policy optimization under exact reward supervision (Wang et al., 4 Sep 2025).

2. Rule-grounded synthesis framework

The synthesis framework underlying SSMR-Bench leverages formal music-theory rules to programmatically generate infinite QA pairs with automatically verifiable answers (Wang et al., 4 Sep 2025). The formalization begins with a pitch-class encoding in which each note’s pitch is mapped to an integer modulo 12, written as $n \in \mathbb{N}_{12}$ (Wang et al., 4 Sep 2025). Directed intervals are defined as $\Delta = (p_2 - p_1) \bmod 12$, and chord construction is expressed through explicit set-theoretic rules, such as the major triad example

$$\mathrm{MajorTriad}(r) = \{ r, r + 4, r + 7 \} \bmod 12$$

(Wang et al., 4 Sep 2025).

The generation pipeline has three stages (Wang et al., 4 Sep 2025). First, it samples a musical snippet in ABC notation, for example L:1/8 K:C | c2 e2 g2 c'2 | …. Second, it applies a rule-based template to generate question text, multiple-choice options, and the correct answer. Third, it uses abcm2ps + ImageMagick to render the ABC snippet into a staff-notation image. The result is a paired item consisting of either textual input (ABC + question) or visual input (PNG + question), together with a provably correct answer (Wang et al., 4 Sep 2025).

Nine question template classes are distributed across four domains: Rhythm, Interval, Chord, and Scale (Wang et al., 4 Sep 2025). Rhythm includes time-signature inference and bar-placement; Interval includes interval naming and note completion by interval; Chord includes chord identification, root identification, and chord completion; Scale includes scale identification and membership testing (Wang et al., 4 Sep 2025). Because the answers are generated from explicit music-theoretic rules, correctness can be checked algorithmically rather than heuristically (Wang et al., 4 Sep 2025).

A related but distinct construction recipe was later described in work on MusICA-MetaBench, which specialized an end-to-end pipeline from MusicXML to rendered sheet images and multiple-choice items (Sourada et al., 7 Jul 2026). That recipe uses MusicXML parsing, rendering via MuseScore, LilyPond, or Verovio, feature extraction with music21, ontology dictionaries for pitches, intervals, durations, chord types, and keys, and multiple-choice instantiation with distractors and occasional “None of the above” options (Sourada et al., 7 Jul 2026). Although this recipe is presented as a detailed pathway for constructing SSMR-Bench end-to-end, it differs from the original SSMR-Bench paper in its reliance on MusicXML rather than ABC notation (Sourada et al., 7 Jul 2026). A plausible implication is that SSMR-Bench admits a broader implementation family than the original paper’s concrete instantiation.

3. Modalities, task structure, and benchmark composition

SSMR-Bench is explicitly bimodal. It contains both textual QA items and visual QA items, enabling comparison between reasoning over symbolic text representations and reasoning over rendered notation images (Wang et al., 4 Sep 2025). The evaluation set comprises 1,600 textual QA items and 1,600 visual QA items, while the RLVR training set contains 8,000 textual and 8,000 visual QA items (Wang et al., 4 Sep 2025). Per modality, the domain distribution is balanced: 25% Scale, 25% Rhythm, 25% Interval, and 25% Chord (Wang et al., 4 Sep 2025).

The benchmark’s structure is therefore symmetric across modalities and domains.

Component	Textual	Visual
Evaluation set	1,600	1,600
Training set for RLVR	8,000	8,000
Domain breakdown	25% each across 4 domains	25% each across 4 domains

In the original SSMR-Bench formulation, the item types are verifiable QA problems grounded in theory rules (Wang et al., 4 Sep 2025). In the MusicXML-based construction recipe associated with MusICA-MetaBench, the question interface is multiple-choice closed QA with 5 options per question: 1 correct answer, 3 distractors, and 1 NOTA option, with NOTA correct in 20% of items to match a uniform prior of $1/5$ (Sourada et al., 7 Jul 2026). The prompt template for visual evaluation is likewise specified: a sheet image is embedded through a Vision-LLM API and the model is instructed to answer with a letter from A to E (Sourada et al., 7 Jul 2026).

The predefined question templates in that construction recipe cover pitch recognition, interval identification, chord type recognition, key signature detection, rhythmic duration, and temporal proportion (Sourada et al., 7 Jul 2026). Three worked examples are given: “What is the 2nd interval in the bass line?” with answer “Minor Third”; “What chord occurs on the downbeat of measure 5?” with answer “Diminished triad”; and “What is the rhythmic notation of the 3rd soprano note?” with answer “Eighth note ($\tfrac{1}{8}$)” (Sourada et al., 7 Jul 2026).

This suggests that the benchmark family spans a spectrum from theory-grounded free-answer or structured-answer tasks to tightly controlled multiple-choice perception-reasoning tasks, with modality parity as a central design principle.

4. Evaluation protocol and statistical design

In the original SSMR-Bench paper, accuracy is the primary metric, defined as the percentage of correctly answered items (Wang et al., 4 Sep 2025). This is consistent with the benchmark’s verifiable structure: because every item has a programmatically determined ground truth, exact matching suffices for evaluation (Wang et al., 4 Sep 2025).

The associated RLVR setting uses a binary reward function,

$$R(a,q) = 1 \text{ if model’s answer } a = \text{ ground truth } a^*; \text{ otherwise } 0,$$

and optimizes the policy objective

$$\max \mathbb{E}_{q \sim D,\; a \sim \pi_\theta(\cdot|q)}[R(a,q)]$$

via Group Relative Policy Optimization (GRPO) without KL or entropy penalties, optimizing solely for accuracy (Wang et al., 4 Sep 2025). A simplified policy-gradient step is also specified: $$\nabla_\theta J \approx \mathbb{E}_{q,a}\left[\nabla_\theta \log \pi_\theta(a|q) \cdot (R(a,q)-b)\right]$$ (Wang et al., 4 Sep 2025).

The MusicXML-based SSMR-Bench recipe adds a detailed statistical framework for benchmark sizing and significance testing (Sourada et al., 7 Jul 2026). It defines a minimum meaningful accuracy difference $E$, proposes the standard two-proportion sample-size formula with significance level $\alpha$ and power $\Delta = (p_2 - p_1) \bmod 12$0, and recommends bootstrap calibration by repeated balanced subsampling (Sourada et al., 7 Jul 2026). For candidate benchmark sizes $\Delta = (p_2 - p_1) \bmod 12$1, it measures the standard deviation of model accuracy across seeds and identifies the smallest $\Delta = (p_2 - p_1) \bmod 12$2 for which paired t-tests, Bonferroni-corrected, detect a difference of $\Delta = (p_2 - p_1) \bmod 12$3% in all pairs (Sourada et al., 7 Jul 2026). For a typical $\Delta = (p_2 - p_1) \bmod 12$4% effect size and 5 categories $\Delta = (p_2 - p_1) \bmod 12$5 4 skills, it reports that $\Delta = (p_2 - p_1) \bmod 12$6 total items, or 60 per category, is sufficient (Sourada et al., 7 Jul 2026).

The same protocol also specifies cross-modal ablations designed to verify that performance depends on actual perception of notation rather than prompt priors (Sourada et al., 7 Jul 2026). These include a no-input condition, where the sheet image is omitted, and a noise-input condition, where the image is replaced by white or Gaussian noise (Sourada et al., 7 Jul 2026). The purpose is to compare accuracy drops and thereby confirm that the model must “look at” the sheet (Sourada et al., 7 Jul 2026).

5. Empirical results and model behavior

The benchmark reveals a marked disparity between textual and visual performance (Wang et al., 4 Sep 2025). On textual QA, Deepseek-R1 achieved 93.63%, Qwen3-32B 76.69%, Deepseek-V3 74.94%, QwQ-32B 72.38%, GPT-4.1-mini 71.69%, Qwen3-8B 57.88%, and the random baseline 25.00% (Wang et al., 4 Sep 2025). On visual QA, Gemini 2.5 Pro achieved 55.44%, InternVL3-78B 45.75%, Qwen2.5-VL-72B-Instruct 45.56%, and InternVL2.5-78B 40.44% (Wang et al., 4 Sep 2025).

These results support the paper’s claim that reasoning is important for interpreting sheet music, while also showing that current MLLMs still face substantial difficulty when notation is presented visually (Wang et al., 4 Sep 2025). In particular, the poor performance of Gemini 2.5 Pro on visual QA is highlighted as evidence of the challenge of sheet-music interpretation in image form (Wang et al., 4 Sep 2025).

The paper further compares against MusicTheoryBench, a set of 367 human-written ABC questions (Wang et al., 4 Sep 2025). On that benchmark, GPT-4-0shot achieved 41.90%, GPT-4-CoT 52.55%, GPT-4-RolePlay 52.45%, and baseline Qwen3-8B 28.65% (Wang et al., 4 Sep 2025). This comparison is important because it places SSMR-Bench-trained models against a human-authored out-of-distribution evaluation set rather than only against synthetic held-out data (Wang et al., 4 Sep 2025).

MusiXQA provides additional context for the difficulty of visual score understanding. Its evaluations on synthetic sheet music showed strong OCR gains but weak OMR and chord reasoning for zero-shot MLLMs, with GPT-4o scoring 68.9/85.3 on OCR but only 4.0/42.0 on OMR and 5.5/74.6 on chord tasks in G-Acc/PNLS, while the fine-tuned Phi-3-MusiX (kern+) improved OMR to 68.4/99.2 and chord estimation to 84.9/96.5 (Chen et al., 28 Jun 2025). The MusiXQA error analysis argues that high PNLS can reflect format mimicry rather than actual recognition, and that kern+ output forces focus on pitch and duration content tokens (Chen et al., 28 Jun 2025). This provides a useful adjacent result: both SSMR-Bench and MusiXQA indicate that superficial output conformity can mask weak symbolic understanding unless the task design and metric tightly couple answer form to musical content.

6. RLVR, transfer, and implications for music generation

A defining feature of SSMR-Bench is its use as training data for RLVR (Wang et al., 4 Sep 2025). Models fine-tuned via GRPO on 8,000 synthetic QA items showed substantial gains. Qwen3-8B-Base on textual QA improved from 23.18% pre-RLVR to 70.94% post-RLVR, a gain of +47.76, and exceeded the original Qwen3-8B by 13.06 points (Wang et al., 4 Sep 2025). Qwen2.5-VL-7B-Instruct on visual QA improved from 41.81% to 70.56%, a gain of +28.75, and surpassed Gemini 2.5 Pro’s 55.44% (Wang et al., 4 Sep 2025). Training dynamics are described as showing cumulative reward steadily rising, while response length stabilizes under the system prompt (Wang et al., 4 Sep 2025).

The improvements transfer beyond the synthetic benchmark. On MusicTheoryBench, Qwen3-8B-Base + Music-RLVR reached 49.97%, an improvement of +21.32 over its baseline 28.65%, and surpassed GPT-4-0shot (Wang et al., 4 Sep 2025). On mathematics benchmarks, the same model improved on AIME24 to 13.75% (+5.00), AMC to 52.11% (+12.95), MATH-500 to 79.40% (+13.40), Minerva to 34.19% (+8.82), and Olympiad to 40.44% (+10.22) (Wang et al., 4 Sep 2025). The paper interprets this as indicating that domain-specific RLVR yields the largest gains on target tasks, with moderate cross-domain spill-over (Wang et al., 4 Sep 2025).

The work also links improved sheet-music reasoning to symbolic composition. In a four-measure continuation task, Qwen3-8B-Base + Music-RLVR improved Rhythmic Consistency (RC) from 66.67% to 99.33% and Novel Rhythmic Coherence (NRC) from 50.00% to 88.89% (Wang et al., 4 Sep 2025). The paper accordingly states that enhanced reasoning ability can facilitate music composition and opens the door to agentic music creation workflows, automated score editing, and educational tools for music theory (Wang et al., 4 Sep 2025).

This suggests a broader interpretation of SSMR-Bench: it is not only a diagnostic benchmark for notation understanding, but also an instrument for shaping symbolic reasoning policies whose effects can propagate into generative music tasks.

7. Limitations, misconceptions, and future directions

A common misconception would be to equate SSMR-Bench with generic OMR evaluation. The available evidence does not support that reduction. The benchmark is framed around music-theoretic reasoning over notation rather than around transcription alone (Wang et al., 4 Sep 2025). By contrast, MusiXQA explicitly includes OCR- and OMR-based tasks such as title extraction, note durations, note pitches, and clef or bar counting, with outputs in free text, JSON, or kern+ formats (Chen et al., 28 Jun 2025). The two resources therefore overlap in musical input domain but differ materially in epistemic target: SSMR-Bench emphasizes verifiable reasoning tasks, whereas MusiXQA more broadly benchmarks sheet understanding and recognition (Chen et al., 28 Jun 2025, Wang et al., 4 Sep 2025).

Another misconception would be to treat strong textual performance as evidence of mature visual sheet-music competence. The reported results contradict this: textual QA results are substantially higher than visual QA results, and the authors explicitly note that advanced MLLMs still struggle on visual notation (Wang et al., 4 Sep 2025). The no-input and noise-input ablations proposed in the later construction recipe reinforce the importance of verifying actual visual dependence in any claimed score-reading ability (Sourada et al., 7 Jul 2026).

Several limitations are explicitly documented in the MusicXML-based recipe (Sourada et al., 7 Jul 2026). Synthetic sheet images may lack OMR noise present in real scans, so pure synthetic evaluation may overestimate performance. Current templates focus on monophonic queries per staff, while multi-voice queries such as intervals between soprano and alto require future extension. The repertoire is currently Western tonal only; modal and microtonal systems would require adaptation of both ontology and templates (Sourada et al., 7 Jul 2026). These caveats are consistent with the original benchmark’s emphasis on Western staff notation and formal music-theory rules (Wang et al., 4 Sep 2025).

Future directions are therefore already implicit in the benchmark family. One path is expansion from ABC-based synthesis to broader MusicXML-driven or user-provided corpora, as exemplified by the MusICA-MetaBench paradigm of deriving on-demand benchmarks directly from structured symbolic data (Sourada et al., 7 Jul 2026). Another is richer polyphonic and cross-staff reasoning. A further direction is closer coupling between evaluation, RLVR, and downstream composition systems, since the existing results already indicate that gains in notation reasoning can translate into improved compositional behavior (Wang et al., 4 Sep 2025).