SongFormBench: Music Structure Analysis Benchmark

• SongFormBench is a benchmark for flat music segmentation, featuring 300 expert-verified songs and a unified functional labeling scheme.
• It addresses previous dataset biases by incorporating both 200 Western and 100 Chinese songs to ensure a fair evaluation of music structure analysis systems.
• Evaluation metrics such as HR.5F, HR3F, and frame-wise ACC provide robust assessments of both boundary precision and functional labeling accuracy.

SongFormBench is a 300-song, expert-verified benchmark for music structure analysis (MSA) introduced in the SongFormer framework as a standardized evaluation set for segment boundary detection and functional labeling (Hao et al., 3 Oct 2025). It consists of 200 refined HarmonixSet songs and 100 Chinese songs, uses a unified flat segmentation scheme with functional labels such as intro, verse, chorus, pre-chorus, bridge, and instrumental, and was designed to address small corpora, inconsistent labeling schemes, and the Western and English-centric bias of earlier evaluation practice. Within the SongFormer study, SongFormBench functions as the primary testbed for comparing models on strict boundary detection, relaxed boundary detection, and frame-wise functional label accuracy (Hao et al., 3 Oct 2025).

1. Origin and research role

SongFormBench was created to provide a fair and rigorous evaluation for MSA systems that must jointly detect structural boundaries and assign functional labels (Hao et al., 3 Oct 2025). The motivating problem is that existing corpora are small, often inconsistent in labeling schemes, and under-represent non-English songs, especially Chinese songs. The benchmark therefore standardizes evaluation across both boundary detection and functional labeling, while also introducing a dedicated Chinese subset to reduce linguistic and cultural bias in evaluation (Hao et al., 3 Oct 2025).

In the SongFormer paper, SongFormBench is tightly coupled to, but distinct from, SongFormDB, the larger heterogeneous training corpus. SongFormDB supplies large-scale supervision under heterogeneous labeling conditions, whereas SongFormBench is the held-out evaluation suite with expert-refined annotations and fixed protocols (Hao et al., 3 Oct 2025). This division makes SongFormBench the clean reference point for measuring whether models trained on noisy, partial, or schema-mismatched data can generalize to a unified structural taxonomy.

A common misconception is to treat SongFormBench as a general benchmark for any notion of “musical form.” More precisely, it is a benchmark for flat functional segmentation of songs. It does not annotate a multi-level hierarchical tree, nor does it directly benchmark generation, symbolic formal templates, or score-level question answering. Those neighboring problems are addressed by other lines of work, including prompt-sequence control for long-form text-to-music audio generation (Atassi, 2023), explicit section–phrase template generation for melody (Lu et al., 2022), song-form-aware lyrics generation with syllable control (Chae et al., 2024), and score-level Texture & Form question answering (Dai et al., 24 Nov 2025).

2. Corpus composition and annotation process

SongFormBench contains two test-only subsets: SongFormBench-HarmonixSet (BHX) and SongFormBench-CN (BC) (Hao et al., 3 Oct 2025).

Subset	Source	Role
BHX	200 songs drawn from the existing HarmonixSet	Test only
BC	100 Chinese songs	Test only

The total is 300 songs, and these tracks are strictly held out for evaluation (Hao et al., 3 Oct 2025). BHX is derived from the public HarmonixSet dataset and reflects Western popular music typical of that corpus, while BC was explicitly introduced to address the scarcity of Chinese data in MSA and to evaluate models across languages (Hao et al., 3 Oct 2025).

The annotation workflow is a defining property of the benchmark. SongFormBench is described as a “high-quality benchmark” whose annotations were carefully revised by expert annotators (Hao et al., 3 Oct 2025). Annotators cross-checked the audio, original labels such as those from HarmonixSet, Gemini 2.5 Pro outputs, and lyrics from MusixMatch and Genius. This multi-source validation procedure was intended to ensure that segment boundaries line up with the musical structure as heard and that functional labels are consistent with lyrics and song-writing conventions (Hao et al., 3 Oct 2025).

The paper does not report explicit inter-annotator agreement figures such as Cohen’s $\kappa$. The basis of the benchmark’s “expert-verified” status is instead manual expert revision and cross-checking across multiple sources (Hao et al., 3 Oct 2025). This suggests a benchmark philosophy centered on curation fidelity rather than on multiple-reference statistical agreement.

3. Structural representation and label normalization

Each song in SongFormBench is represented as a flat sequence of labeled segments rather than as a hierarchical structure (Hao et al., 3 Oct 2025). The benchmark annotation format is

$$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$

where $t_i$ is the start time in seconds of segment $i$, $l_i$ is its functional label, and end denotes the track end (Hao et al., 3 Oct 2025). This representation supports evaluation of contiguous sectioning and categorical function assignment, but it does not encode higher-order grouping relations such as repeated supersections or nested formal levels.

For evaluation, SongFormBench adopts the seven-category functional scheme from Wang et al. (2022), with an important normalization rule: during evaluation, pre-chorus in SongFormer is mapped to verse (Hao et al., 3 Oct 2025). Training may preserve pre-chorus as a distinct label to capture transitions better, but the benchmark itself uses a seven-label evaluation space so that heterogeneous systems can be compared under a unified taxonomy (Hao et al., 3 Oct 2025).

This normalization extends to external baselines. All-In-One maps start/end to silence and break/solo to inst, while LinkSeg-7Labels is already directly compatible with the seven-label scheme (Hao et al., 3 Oct 2025). SongFormBench therefore functions not only as a dataset but also as a label-space reconciliation mechanism for systems trained under different annotation conventions.

The benchmark’s use of flat segmentation is methodologically consequential. It makes the task well-defined for models that output timestamped section labels, but it excludes explicit evaluation of hierarchical form. In this respect, SongFormBench differs from symbolic generation systems such as MeloForm, which uses hierarchical section–phrase templates like $A(a_1, a_1)B(b_1, b_2)$ (Lu et al., 2022), and from score-level reasoning benchmarks such as MSU-Bench, whose highest level targets “Texture & Form” questions over complete scores (Dai et al., 24 Nov 2025).

4. Evaluation protocol and metrics

SongFormBench uses a fixed evaluation protocol with three metrics: HR.5F, HR3F, and ACC (Hao et al., 3 Oct 2025). The first two are F-measure variants for boundary detection under strict and relaxed temporal tolerances; the third is frame-wise functional label accuracy.

HR.5F is defined as “The F-measure of boundary hit rate within 0.5 seconds,” and HR3F as “The F-measure of boundary hit rate within 3 seconds” (Hao et al., 3 Oct 2025). If $H$ denotes the number of hits, $N_p$ the number of predicted boundaries, and $N_g$ the number of ground-truth boundaries, then precision and recall are

$P = \frac{H}{N_p}, \quad R = \frac{H}{N_g},$

and the F-measure is

$$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$0

HR.5F applies this computation with a $$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$1 s tolerance, while HR3F uses a $$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$2 s tolerance (Hao et al., 3 Oct 2025). The distinction is not merely cosmetic. HR.5F tests whether systems can place boundaries sharply enough for detailed structural analysis, whereas HR3F measures coarse alignment.

ACC is defined as “Frame-wise accuracy comparing the predicted function to the ground truth” (Hao et al., 3 Oct 2025). After downsampling, the effective frame rate is approximately $$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$3 Hz. If $$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$4 is the predicted functional label at frame $$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$5, $$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$6 is the ground-truth label, and $$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$7 is the number of frames, then

$$\{(t_0, l_0), (t_1, l_1), \dots, (t_{N-1}, l_{N-1}), (t_N, end)\},$$8

This metric emphasizes segment-internal label correctness over mere boundary placement (Hao et al., 3 Oct 2025).

The coexistence of HR.5F and HR3F is one of the benchmark’s most informative design choices. It exposes systems that approximately locate sections but fail to mark precise transitions. The SongFormer results make this distinction explicit: Gemini 2.5 Pro can achieve high HR3F while performing much worse on HR.5F, indicating coarse structural alignment without sharp boundary localization (Hao et al., 3 Oct 2025).

5. Comparative results and benchmark behavior

On SongFormBench-HarmonixSet, SongFormer establishes the strongest reported performance on strict boundary detection and functional label accuracy (Hao et al., 3 Oct 2025). SongFormer (HX) reaches 0.795 ACC, 0.703 HR.5F, and 0.784 HR3F, while SongFormer (HX+P+H+G) reaches the best ACC at 0.807, with 0.696 HR.5F and 0.780 HR3F (Hao et al., 3 Oct 2025). Among the listed baselines on BHX, LinkSeg-7Labels attains 0.780 ACC and 0.630 HR.5F, TA reaches 0.787 ACC and 0.610 HR.5F, and Gemini 2.5 Pro reaches 0.748 ACC, 0.423 HR.5F, and 0.813 HR3F (Hao et al., 3 Oct 2025).

These numbers are diagnostically important. SongFormer has the best strict boundary detection on BHX, while Gemini 2.5 Pro has the highest relaxed HR3F. The benchmark therefore reveals that high relaxed tolerance does not imply state-of-the-art structural segmentation in the strict sense (Hao et al., 3 Oct 2025).

On SongFormBench-CN, SongFormer’s gains are even larger relative to the baselines (Hao et al., 3 Oct 2025). SongFormer (HX+P+H+G) reaches 0.891 ACC, 0.688 HR.5F, and 0.851 HR3F, while SongFormer (HX+P+H) reaches the highest HR.5F at 0.690 (Hao et al., 3 Oct 2025). The principal baselines on BC are All-In-One at 0.834 ACC, 0.563 HR.5F, and 0.771 HR3F; LinkSeg-7Labels at 0.828 ACC, 0.518 HR.5F, and 0.757 HR3F; and Gemini 2.5 Pro at 0.806 ACC, 0.412 HR.5F, and 0.833 HR3F (Hao et al., 3 Oct 2025).

The BC results show that the Chinese subset is not simply an auxiliary add-on. It materially differentiates systems, and it demonstrates that training with heterogeneous multilingual supervision can transfer to Chinese songs when evaluation is performed on carefully annotated ground truth (Hao et al., 3 Oct 2025).

SongFormBench also supports ablation analysis. On the combined 300-song benchmark, removing the data source embedding reduces ACC from 0.848 to 0.825 and HR3F from 0.816 to 0.801, indicating that heterogeneous supervision and source conditioning affect benchmark performance in measurable ways (Hao et al., 3 Oct 2025). Multi-resolution self-supervised representations combining 30 s and 420 s windows also produce the best ACC and HR.5F on SongFormBench, showing that the benchmark is sensitive to representation design rather than only to classifier choice (Hao et al., 3 Oct 2025).

6. Reproducibility, experimental ecosystem, and relation to adjacent work

SongFormBench is distributed as part of a reproducible research package: “Code, datasets, and model are publicly available,” with the SongFormer repository listed at https://github.com/ASLP-lab/SongFormer (Hao et al., 3 Oct 2025). The released resources include evaluation scripts for ACC, HR.5F, and HR3F, as well as label mappings for external models such as All-In-One and LinkSeg (Hao et al., 3 Oct 2025). This makes SongFormBench not only a dataset but also a standardized protocol.

Its relation to SongFormDB is structurally clear. The 912 HarmonixSet songs in SongForm-HX are split into 512 for training, 200 for validation, and 200 held out; those held-out 200 are manually refined to become BHX, while BC is an additional 100-song evaluation-only subset (Hao et al., 3 Oct 2025). The benchmark is therefore deliberately separated from training data, even though it shares the same harmonized label scheme.

The benchmark also sits within a broader research landscape on musical form, though adjacent works define “form” differently. “Musical Form Generation” operationalizes form as a sequence of text-conditioned segments and transitions, with high-level prompt planning separated from low-level audio generation (Atassi, 2023). “MeloForm” uses an explicit symbolic hierarchy of sections, phrases, and motifs and reports 97.79% accuracy for musical form control in melody generation without labeled musical form data (Lu et al., 2022). “Song Form-aware Full-Song Text-to-Lyrics Generation with Multi-Level Granularity Syllable Count Control” treats form as paragraph-level section labels such as verse and chorus and evaluates form consistency alongside syllable accuracy (Chae et al., 2024). MSU-Bench, by contrast, embeds form inside a score-understanding question-answering hierarchy whose top level is “Texture & Form” over complete scores (Dai et al., 24 Nov 2025).

This comparison clarifies SongFormBench’s niche. It is not a generative benchmark and not a full-score reasoning benchmark. It is a focused evaluation set for flat song segmentation and function labeling under a standardized annotation scheme. A plausible implication is that SongFormBench can serve as a reference point for models that later need to connect analysis with generation, lyrics conditioning, or multimodal score understanding, but those extensions are outside its current formal scope.

7. Limitations and prospective extensions

The benchmark has several explicit limitations (Hao et al., 3 Oct 2025). First, although SongFormBench is more curated and more diverse than earlier test sets, 300 songs remain moderate in scale relative to the over-10k-track SongFormDB training corpus. Second, MSA remains partly subjective: boundaries and labels such as pre-chorus versus verse or instrumental break versus bridge can be ambiguous, and the paper does not provide inter-annotator agreement scores or multiple reference annotations (Hao et al., 3 Oct 2025).

Third, SongFormBench is restricted to flat, single-level segmentation with functional labels. Hierarchical structure is not annotated, so models designed for multi-level formal analysis require other resources or extensions to the benchmark (Hao et al., 3 Oct 2025). Fourth, the SongFormer preprocessing truncates inputs at 420 s, which can affect the representation of very long pieces, even though the benchmark itself is defined at the song level (Hao et al., 3 Oct 2025).

The natural future directions identified in the paper include adding more languages beyond English and Chinese, incorporating more genres, providing hierarchical annotations, and collecting multiple independent annotations per track (Hao et al., 3 Oct 2025). These extensions would move SongFormBench toward a richer account of musical form. Related work suggests additional possibilities: prompt-indexed segment plans for audio generation (Atassi, 2023), explicit structural templates and form-control metrics for melody (Lu et al., 2022), section-aware lyric generation with line- and phrase-level constraints (Chae et al., 2024), and QA-based form reasoning over symbolic and visual scores (Dai et al., 24 Nov 2025). As presently defined, however, SongFormBench remains a benchmark for expert-verified flat structural segmentation, valuable precisely because it narrows the task to a reproducible and comparable core.