ABC-Eval: Symbolic Music Benchmark

• ABC-Eval is an open-source benchmark for evaluating large language models' ability to comprehend and process text-based symbolic music in ABC notation.
• It is structured into three task categories—Basic Syntax, Segment-level, and Sequence-level—with 1,086 test samples across 10 sub-tasks.
• The benchmark reveals that while LLMs excel at low-level parsing, they struggle with higher-level musical semantics and complex reasoning tasks.

ABC-Eval is an open-source benchmark for evaluating LLMs on text-based symbolic music understanding and instruction following in ABC notation. It was introduced to fill a gap left by task-specific music datasets, music QA benchmarks, and multimodal or audio-language benchmarks that do not test whether text LLMs can directly read and reason over symbolic scores as text. The benchmark contains 1,086 test samples spanning 10 sub-tasks organized into three categories, ranging from basic syntax understanding to segment-level and sequence-level reasoning (Zhao et al., 27 Sep 2025).

1. Scope, representation, and motivation

ABC-Eval is presented as the first open-source benchmark specifically designed to evaluate LLMs on text-based symbolic music understanding and instruction following, with ABC notation as the sole score representation (Zhao et al., 27 Sep 2025). Its design target is not audio understanding, optical music recognition, or multimodal music QA. Instead, it isolates a narrower question: whether a text model can parse, interpret, and reason over symbolic music strings.

This focus is a response to a specific methodological gap. Prior work had emphasized task-specific music datasets, broader music QA benchmarks such as ZIQI-Eval, multimodal or audio-language benchmarks such as MuChoMusic and CMI-Bench, and image-based music reasoning settings such as WildScore. The benchmark’s premise is that these settings do not directly evaluate pure text-based symbolic music comprehension. ABC notation is therefore used because it is described as lightweight, human-readable, and friendly to LLMs.

Within the benchmark, ABC notation includes metadata fields and symbolic score content. The examples in the benchmark description include M: for meter, K: for key signature, note-length-related metadata, and bar-delimited note sequences. The benchmark’s error cases make the formalism concrete: invalid meter M:6/7, invalid key signature K: Rmin, and invalid content such as Z8 are all treated as objects of symbolic understanding rather than mere string anomalies.

A potential source of confusion is nomenclature. The name ABC-Eval is not globally unique in the literature: an unrelated dialogue-evaluation framework uses ABC-Eval as shorthand for “Annotation of Behaviors in Chat Evaluation” in open-domain dialogue-system assessment (Finch et al., 2022). In current music-LLM usage, however, ABC-Eval refers to the ABC-notation benchmark introduced in 2025.

2. Task structure and benchmark composition

The benchmark is partitioned into Basic Syntax Understanding, Segment-level Understanding, and Sequence-level Understanding. This organization is intended to span a progression from fundamental symbolic parsing to local structural reasoning and then to longer-range musical semantics (Zhao et al., 27 Sep 2025).

Category	Task	Samples
Basic Syntax Understanding	Bar Count Estimation	100
Basic Syntax Understanding	Metadata QA	60
Segment-level Understanding	Next-Bar Prediction	119
Segment-level Understanding	Bar Sequencing	119
Segment-level Understanding	Error Detection	220
Sequence-level Understanding	Music Captioning	60
Sequence-level Understanding	Metadata Prediction	60
Sequence-level Understanding	Emotion Recognition	120
Sequence-level Understanding	Composer Recognition	96
Sequence-level Understanding	Genre Recognition	132

The Basic Syntax Understanding tasks probe explicit symbolic literacy. Metadata QA is a four-choice task in which the model reads header information such as key signature, meter, or note length field. Bar Count Estimation instead requires direct output of the number of bars, so it couples symbolic parsing with strict structured output.

The Segment-level Understanding tasks focus on local or mid-range score structure. Next-Bar Prediction provides the first few bars and asks for the most likely continuation from four options; distractors are random bars sampled from the remaining part of the same score. Bar Sequencing presents several bars in random order and requires the model to output the correct order using indices such as 0312. Error Detection is the most structurally demanding task in the benchmark: the model receives a complete score with multiple inserted errors and must output all bar numbers containing errors, with the metadata section counted as part of the first bar.

The Sequence-level Understanding tasks target global reasoning. Metadata Prediction differs from Metadata QA because metadata are masked and must be inferred from musical content rather than copied from the header. Music Captioning is title selection from four candidates. Emotion Recognition, Composer Recognition, and Genre Recognition are all four-way classification tasks on complete scores. Emotion recognition uses the four quadrants of Russell’s circumplex model.

The benchmark mixes multiple-choice tasks with structured-output tasks. That design is central: the benchmark is intended to evaluate not only symbolic music understanding, but also strict instruction following under output constraints.

3. Data sources, notation content, and benchmark construction

ABC-Eval draws from several corpora, with different sources used for different tasks (Zhao et al., 27 Sep 2025). ABC scores from the Nottingham dataset are used for many tasks, and its title field serves as the ground truth for Music Captioning. Irish Massive ABC Notation (IrishMAN) provides ABC scores and bar-count ground truth. EMOPIA supplies emotion labels, ADL-piano MIDI supplies genre labels, and Pianist8 supplies composer labels.

Not all source datasets were originally in ABC notation. For datasets available only as MIDI, the construction pipeline is explicit: the authors used a state-of-the-art MIDI-to-score model to transcribe MIDI to MusicXML, and then used the EasyABC library to convert MusicXML into ABC notation. This implies that some sequence-level tasks inherit possible transcription or conversion artifacts. The benchmark is also stated to be for non-commercial research use only, with compliance to original licenses.

The benchmark’s most curated component is Error Detection. Because naturally occurring symbolic scores may already contain ambiguities or mistakes, the authors manually selected 22 high-quality scores judged by music experts, then inserted random errors at random positions. The five error categories are explicitly defined:

1. Invalid metadata
2. Invalid content
3. Invalid bar durations
4. Unreasonable melodic leaps
5. Accidental errors outside the key signature

The benchmark gives concrete examples for these categories. Invalid metadata includes M:6/7 and K: Rmin. Invalid content includes Z8. Invalid bar duration is illustrated by a score marked M:4/4 whose bar totals 3/4. Unreasonable melodic leaps are created by modifying a note to produce a sudden interval jump $> 10$ degrees. Accidental errors include _C in G major, where flat C is not part of the key signature. A construction constraint is also stated: error type 1 does not co-occur with types 3, 4, and 5.

This construction strategy suggests a benchmark that separates several competence types that are often conflated: formal syntax checking, rhythmic consistency, tonal constraints, local melodic plausibility, and sequence-level semantic inference.

4. Prompt format, evaluation protocol, and scoring

All evaluated models use the same prompt templates, with temperature = 0 and top_p = 1, and without roleplay prompting or chain-of-thought prompting (Zhao et al., 27 Sep 2025). For multiple-choice tasks, the prompt format is:

Input: {input_content} Task: {task_instruction} Options: 0. {options[0]} 1. {options[1]} 2. {options[2]} 3. {options[3]} Please only output the index of the correct option (0, 1, 2, or 3), do not output any additional content.

For structured-output tasks, the template is:

Input: {input_content} Task: {task_instruction} Template: {structured_output_template} Please directly output the answer of the given task, without any explanation or additional content.

The benchmark evaluates seven LLMs through official APIs: DeepSeek-chat, DeepSeek-reasoner, Gemini-2.5-pro, Gemini-2.5-flash, GPT-5, GPT-5-mini, and GPT-5-nano. These are described as spanning three major model families and including lightweight, ultra-large, and MoE-style systems.

Most tasks are scored by accuracy. Two tasks use specialized metrics. Bar Sequencing uses the Kendall tau coefficient, which is mapped from $[-1,1]$ to $[0,1]$, then multiplied by a completeness penalty factor; outputs that are too long or violate rules receive 0. Error Detection uses Macro-F1. The paper does not give a closed-form formula for either the Kendall-tau mapping or the completeness penalty.

The evaluation protocol also includes task-specific exclusions. Contrasts with fewer than three contexts are excluded where relevant, and tasks whose solution depends on missing articulators are excluded in the articulatory case of other literature; in ABC-Eval’s music setting, the analogous principle appears in task construction rather than modality coverage. Here the important point is stricter: benchmark scores are meaningful only because all models face exactly the same textual symbolic inputs, prompt format, and decoding settings.

5. Empirical findings

The central empirical result is that current LLMs remain limited on symbolic music understanding, especially beyond explicit notation parsing (Zhao et al., 27 Sep 2025). The overall benchmark averages are:

Model	Overall average
GPT-5	55.02
GPT-5-mini	53.51
DeepSeek-reasoner	52.01
Gemini-2.5-pro	51.25
GPT-5-nano	48.98
Gemini-2.5-flash	44.14
DeepSeek-chat	40.17

The Basic Syntax Understanding category is comparatively easy for strong models. The category averages are 99.50 for GPT-5-mini, 96.67 for GPT-5-nano, 96.5 for GPT-5, 94.67 for DeepSeek-reasoner, 91.17 for Gemini-flash, 82.17 for Gemini-pro, and 65.00 for DeepSeek-chat. The paper notes that most foundation models exceed 90% on these tasks. Yet performance can still be fragile: DeepSeek-chat gets only 30.00 on Bar Count Estimation, despite scoring 100 on Metadata QA. The authors interpret this as prompt sensitivity and lack of robustness even on simple symbolic tasks.

The Segment-level Understanding tasks are substantially harder. Category averages are 44.73 for GPT-5, 43.31 for Gemini-pro, 41.88 for GPT-5-mini, 40.65 for DeepSeek-reasoner, 35.11 for GPT-5-nano, 32.53 for Gemini-flash, and 28.49 for DeepSeek-chat. On Next-Bar Prediction, the best result is 50.42 by GPT-5. On Bar Sequencing, GPT-5 reaches 59.66, closely followed by Gemini-pro at 59.01. On Error Detection, the best score is only 27.22 by Gemini-pro, with GPT-5 at 24.12 and DeepSeek-chat at 6.29. The paper describes these tasks as challenging and states that all foundation models failed to tackle them effectively.

The Sequence-level Understanding tasks are similarly difficult. Category averages are 44.60 for GPT-5, 43.65 for Gemini-pro, 42.10 for GPT-5-mini, 41.76 for DeepSeek-reasoner, 38.24 for GPT-5-nano, 37.25 for DeepSeek-chat, and 32.29 for Gemini-flash. One exception is Metadata Prediction, where GPT-5 reaches 95.00, GPT-5-mini and Gemini-pro each reach 93.33, and GPT-5-nano reaches 90.00. The authors interpret this as evidence that models can perform some basic musical inference from symbolic content.

The remaining sequence-level tasks remain weak. Music Captioning peaks at 40.00 with GPT-5-mini. Emotion Recognition peaks at 34.17 with GPT-5, while many models are near the 25% random baseline for a four-way task. Composer Recognition and Genre Recognition are best for DeepSeek-reasoner at 36.46 and 34.85, respectively. The benchmark therefore suggests a marked gap between explicit symbolic parsing and higher-level musical semantics such as style, affect, and attribution.

6. Reliability, limitations, and significance

The paper’s explicit reliability claim is modest but important: the consistent performance of individual baselines across different sub-tasks supports the reliability of the benchmark (Zhao et al., 27 Sep 2025). Stronger or more reasoning-capable models generally outperform lighter variants, and the authors argue that this shows the benchmark can effectively discriminate symbolic music understanding capabilities between models.

Several features support that interpretation. The benchmark spans multiple levels of abstraction, from syntax to segment structure to sequence semantics. This suggests broad construct coverage for symbolic music understanding. It also mixes multiple-choice and structured-output formats, so it probes both recognition and instruction-following precision.

At the same time, the benchmark has clear limits. It is specific to ABC notation, so conclusions do not automatically transfer to MusicXML, MIDI-event tokenizations, or piano-roll representations. Some tasks depend on MIDI-to-score transcription and MusicXML-to-ABC conversion, which may introduce artifacts. The paper description discusses only the test benchmark and does not specify training or development splits. The prompt style is deliberately minimal and excludes chain-of-thought, so the reported numbers characterize direct-answer behavior rather than prompt-engineered upper bounds. Music Captioning may also partly reflect tune-title memorization rather than only structural musical understanding.

The benchmark’s substantive significance lies in the failure modes it exposes. Current LLMs are already strong at low-level symbolic parsing and can infer some metadata, but they remain weak at segment-level coherence, multi-error localization, and high-level musical semantics. In that sense, ABC-Eval measures a gap between general language competence and robust symbolic music reasoning. The benchmark therefore functions less as a test of generic music knowledge than as a probe of whether an LLM can read a score-like symbolic sequence, maintain formal constraints, and follow tightly specified output instructions over that sequence.