---
title: 'ABC-Eval: Symbolic Music Benchmark'
url: https://www.emergentmind.com/topics/abc-eval
type: topic
---

# ABC-Eval: Symbolic Music Benchmark

ABC-Eval is an open-source benchmark for evaluating large language models on text-based symbolic music understanding and instruction following in **ABC notation**. It was introduced to fill a gap left by task-specific music datasets, music QA benchmarks, and multimodal or audio-language benchmarks that do not test whether text LLMs can directly read and reason over symbolic scores as text. The benchmark contains **1,086 test samples** spanning **10 sub-tasks** organized into **three categories**, ranging from basic syntax understanding to segment-level and sequence-level reasoning [2509.23350].

## 1. Scope, representation, and motivation

ABC-Eval is presented as the **first open-source benchmark specifically designed to evaluate large language models on text-based symbolic music understanding and instruction following**, with **ABC notation** as the sole score representation [2509.23350]. Its design target is not audio understanding, optical music recognition, or multimodal music QA. Instead, it isolates a narrower question: whether a text model can parse, interpret, and reason over symbolic music strings.

This focus is a response to a specific methodological gap. Prior work had emphasized task-specific music datasets, broader music QA benchmarks such as ZIQI-Eval, multimodal or audio-language benchmarks such as MuChoMusic and CMI-Bench, and image-based music reasoning settings such as WildScore. The benchmark’s premise is that these settings do not directly evaluate **pure text-based symbolic music comprehension**. ABC notation is therefore used because it is described as lightweight, human-readable, and friendly to LLMs.

Within the benchmark, ABC notation includes metadata fields and symbolic score content. The examples in the benchmark description include `M:` for meter, `K:` for key signature, note-length-related metadata, and bar-delimited note sequences. The benchmark’s error cases make the formalism concrete: invalid meter `M:6/7`, invalid key signature `K: Rmin`, and invalid content such as `Z8` are all treated as objects of symbolic understanding rather than mere string anomalies.

A potential source of confusion is nomenclature. The name **ABC-Eval** is not globally unique in the literature: an unrelated dialogue-evaluation framework uses ABC-Eval as shorthand for **“Annotation of Behaviors in Chat Evaluation”** in open-domain dialogue-system assessment [2212.09180]. In current music-LLM usage, however, ABC-Eval refers to the ABC-notation benchmark introduced in 2025.

## 2. Task structure and benchmark composition

The benchmark is partitioned into **Basic Syntax Understanding**, **Segment-level Understanding**, and **Sequence-level Understanding**. This organization is intended to span a progression from fundamental symbolic parsing to local structural reasoning and then to longer-range musical semantics [2509.23350].

| Category | Task | Samples |
|---|---|---:|
| Basic Syntax Understanding | Bar Count Estimation | 100 |
| Basic Syntax Understanding | Metadata QA | 60 |
| Segment-level Understanding | Next-Bar Prediction | 119 |
| Segment-level Understanding | Bar Sequencing | 119 |
| Segment-level Understanding | Error Detection | 220 |
| Sequence-level Understanding | Music Captioning | 60 |
| Sequence-level Understanding | Metadata Prediction | 60 |
| Sequence-level Understanding | Emotion Recognition | 120 |
| Sequence-level Understanding | Composer Recognition | 96 |
| Sequence-level Understanding | Genre Recognition | 132 |

The **Basic Syntax Understanding** tasks probe explicit symbolic literacy. **Metadata QA** is a four-choice task in which the model reads header information such as key signature, meter, or note length field. **Bar Count Estimation** instead requires direct output of the number of bars, so it couples symbolic parsing with strict structured output.

The **Segment-level Understanding** tasks focus on local or mid-range score structure. **Next-Bar Prediction** provides the first few bars and asks for the most likely continuation from four options; distractors are random bars sampled from the remaining part of the same score. **Bar Sequencing** presents several bars in random order and requires the model to output the correct order using indices such as `0312`. **Error Detection** is the most structurally demanding task in the benchmark: the model receives a complete score with multiple inserted errors and must output all bar numbers containing errors, with the metadata section counted as part of the first bar.

The **Sequence-level Understanding** tasks target global reasoning. **Metadata Prediction** differs from Metadata QA because metadata are masked and must be inferred from musical content rather than copied from the header. **Music Captioning** is title selection from four candidates. **Emotion Recognition**, **Composer Recognition**, and **Genre Recognition** are all four-way classification tasks on complete scores. Emotion recognition uses the four quadrants of **Russell’s circumplex model**.

The benchmark mixes **multiple-choice tasks** with **structured-output tasks**. That design is central: the benchmark is intended to evaluate not only symbolic music understanding, but also strict instruction following under output constraints.

## 3. Data sources, notation content, and benchmark construction

ABC-Eval draws from several corpora, with different sources used for different tasks [2509.23350]. ABC scores from the **Nottingham dataset** are used for many tasks, and its title field serves as the ground truth for Music Captioning. **Irish Massive ABC Notation (IrishMAN)** provides ABC scores and bar-count ground truth. **EMOPIA** supplies emotion labels, **ADL-piano MIDI** supplies genre labels, and **Pianist8** supplies composer labels.

Not all source datasets were originally in ABC notation. For datasets available only as MIDI, the construction pipeline is explicit: the authors used a **state-of-the-art MIDI-to-score model** to transcribe MIDI to **MusicXML**, and then used the **EasyABC** library to convert MusicXML into **ABC notation**. This implies that some sequence-level tasks inherit possible transcription or conversion artifacts. The benchmark is also stated to be for **non-commercial research use only**, with compliance to original licenses.

The benchmark’s most curated component is **Error Detection**. Because naturally occurring symbolic scores may already contain ambiguities or mistakes, the authors **manually selected 22 high-quality scores judged by music experts**, then **inserted random errors at random positions**. The five error categories are explicitly defined:

1. **Invalid metadata**
2. **Invalid content**
3. **Invalid bar durations**
4. **Unreasonable melodic leaps**
5. **Accidental errors outside the key signature**

The benchmark gives concrete examples for these categories. Invalid metadata includes `M:6/7` and `K: Rmin`. Invalid content includes `Z8`. Invalid bar duration is illustrated by a score marked `M:4/4` whose bar totals `3/4`. Unreasonable melodic leaps are created by modifying a note to produce a sudden interval jump \(> 10\) degrees. Accidental errors include `_C` in G major, where flat C is not part of the key signature. A construction constraint is also stated: **error type 1 does not co-occur with types 3, 4, and 5**.

This construction strategy suggests a benchmark that separates several competence types that are often conflated: formal syntax checking, rhythmic consistency, tonal constraints, local melodic plausibility, and sequence-level semantic inference.

## 4. Prompt format, evaluation protocol, and scoring

All evaluated models use the same prompt templates, with **`temperature = 0`** and **`top_p = 1`**, and without roleplay prompting or chain-of-thought prompting [2509.23350]. For multiple-choice tasks, the prompt format is:

```text
Input: {input_content}
Task: {task_instruction}
Options:
0. {options[0]}
1. {options[1]}
2. {options[2]}
3. {options[3]}

Please only output the index of the correct option (0, 1, 2, or 3), do not output any additional content.
```

For structured-output tasks, the template is:

```text
Input: {input_content}
Task: {task_instruction}
Template: {structured_output_template}

Please directly output the answer of the given task, without any explanation or additional content.
```

The benchmark evaluates **seven LLMs** through official APIs: **DeepSeek-chat**, **DeepSeek-reasoner**, **Gemini-2.5-pro**, **Gemini-2.5-flash**, **GPT-5**, **GPT-5-mini**, and **GPT-5-nano**. These are described as spanning three major model families and including lightweight, ultra-large, and MoE-style systems.

Most tasks are scored by **accuracy**. Two tasks use specialized metrics. **Bar Sequencing** uses the **Kendall tau coefficient**, which is **mapped from \([-1,1]\) to \([0,1]\)**, then multiplied by a **completeness penalty factor**; outputs that are too long or violate rules receive **0**. **Error Detection** uses **Macro-F1**. The paper does not give a closed-form formula for either the Kendall-tau mapping or the completeness penalty.

The evaluation protocol also includes task-specific exclusions. Contrasts with fewer than three contexts are excluded where relevant, and tasks whose solution depends on missing articulators are excluded in the articulatory case of other literature; in ABC-Eval’s music setting, the analogous principle appears in task construction rather than modality coverage. Here the important point is stricter: benchmark scores are meaningful only because all models face exactly the same textual symbolic inputs, prompt format, and decoding settings.

## 5. Empirical findings

The central empirical result is that current LLMs remain limited on symbolic music understanding, especially beyond explicit notation parsing [2509.23350]. The overall benchmark averages are:

| Model | Overall average |
|---|---:|
| GPT-5 | 55.02 |
| GPT-5-mini | 53.51 |
| DeepSeek-reasoner | 52.01 |
| Gemini-2.5-pro | 51.25 |
| GPT-5-nano | 48.98 |
| Gemini-2.5-flash | 44.14 |
| DeepSeek-chat | 40.17 |

The **Basic Syntax Understanding** category is comparatively easy for strong models. The category averages are **99.50** for GPT-5-mini, **96.67** for GPT-5-nano, **96.5** for GPT-5, **94.67** for DeepSeek-reasoner, **91.17** for Gemini-flash, **82.17** for Gemini-pro, and **65.00** for DeepSeek-chat. The paper notes that most foundation models exceed **90%** on these tasks. Yet performance can still be fragile: **DeepSeek-chat gets only 30.00 on Bar Count Estimation**, despite scoring **100** on Metadata QA. The authors interpret this as prompt sensitivity and lack of robustness even on simple symbolic tasks.

The **Segment-level Understanding** tasks are substantially harder. Category averages are **44.73** for GPT-5, **43.31** for Gemini-pro, **41.88** for GPT-5-mini, **40.65** for DeepSeek-reasoner, **35.11** for GPT-5-nano, **32.53** for Gemini-flash, and **28.49** for DeepSeek-chat. On **Next-Bar Prediction**, the best result is **50.42** by GPT-5. On **Bar Sequencing**, GPT-5 reaches **59.66**, closely followed by Gemini-pro at **59.01**. On **Error Detection**, the best score is only **27.22** by Gemini-pro, with GPT-5 at **24.12** and DeepSeek-chat at **6.29**. The paper describes these tasks as challenging and states that **all foundation models failed to tackle them effectively**.

The **Sequence-level Understanding** tasks are similarly difficult. Category averages are **44.60** for GPT-5, **43.65** for Gemini-pro, **42.10** for GPT-5-mini, **41.76** for DeepSeek-reasoner, **38.24** for GPT-5-nano, **37.25** for DeepSeek-chat, and **32.29** for Gemini-flash. One exception is **Metadata Prediction**, where GPT-5 reaches **95.00**, GPT-5-mini and Gemini-pro each reach **93.33**, and GPT-5-nano reaches **90.00**. The authors interpret this as evidence that models can perform some basic musical inference from symbolic content.

The remaining sequence-level tasks remain weak. **Music Captioning** peaks at **40.00** with GPT-5-mini. **Emotion Recognition** peaks at **34.17** with GPT-5, while many models are near the **25%** random baseline for a four-way task. **Composer Recognition** and **Genre Recognition** are best for DeepSeek-reasoner at **36.46** and **34.85**, respectively. The benchmark therefore suggests a marked gap between explicit symbolic parsing and higher-level musical semantics such as style, affect, and attribution.

## 6. Reliability, limitations, and significance

The paper’s explicit reliability claim is modest but important: the **consistent performance of individual baselines across different sub-tasks supports the reliability of the benchmark** [2509.23350]. Stronger or more reasoning-capable models generally outperform lighter variants, and the authors argue that this shows the benchmark can **effectively discriminate symbolic music understanding capabilities between models**.

Several features support that interpretation. The benchmark spans multiple levels of abstraction, from syntax to segment structure to sequence semantics. This suggests broad construct coverage for symbolic music understanding. It also mixes multiple-choice and structured-output formats, so it probes both recognition and instruction-following precision.

At the same time, the benchmark has clear limits. It is specific to **ABC notation**, so conclusions do not automatically transfer to **MusicXML**, **MIDI-event tokenizations**, or piano-roll representations. Some tasks depend on **MIDI-to-score transcription** and **MusicXML-to-ABC conversion**, which may introduce artifacts. The paper description discusses only the test benchmark and does not specify training or development splits. The prompt style is deliberately minimal and excludes chain-of-thought, so the reported numbers characterize direct-answer behavior rather than prompt-engineered upper bounds. Music Captioning may also partly reflect tune-title memorization rather than only structural musical understanding.

The benchmark’s substantive significance lies in the failure modes it exposes. Current LLMs are already strong at low-level symbolic parsing and can infer some metadata, but they remain weak at segment-level coherence, multi-error localization, and high-level musical semantics. In that sense, ABC-Eval measures a gap between general language competence and robust symbolic music reasoning. The benchmark therefore functions less as a test of generic music knowledge than as a probe of whether an LLM can read a score-like symbolic sequence, maintain formal constraints, and follow tightly specified output instructions over that sequence.

Source: https://www.emergentmind.com/topics/abc-eval