Music Structure Analysis (MSA)

• Music Structure Analysis is the computational task that segments musical works into interpretable units, such as verses and choruses, based on rhythmic and harmonic patterns.
• It employs diverse methodologies including barwise self-similarity, graph-based segmentation, and deep learning approaches to reveal multi-level musical hierarchies.
• Modern MSA techniques drive applications in music information retrieval, generative modeling, and computational musicology by providing robust, quantifiable structure analysis.

Music Structure Analysis (MSA) is the computational task of partitioning a musical work into interpretable, musically meaningful structural units—typically sections such as “verse,” “chorus,” “bridge”—and, in advanced settings, providing a hierarchical, functional, or statistical characterization of these structures. MSA undergirds diverse applications including music information retrieval, content-based navigation, computational musicology, and structure-aware generation. Recent research spans audio-based, symbolic, and graph-theoretical paradigms, with particular focus on barwise representations, self-similarity, multi-level segmentation, and the interplay between learned representations and human annotation schemes.

1. Formal Models and Definitions of Musical Structure

At the core of MSA are rigorous definitions of musical units and their hierarchical relationships. Dai et al. define “repetition” as any pair of non-overlapping, measure-aligned segments whose chord, melodic, and rhythmic similarities exceed pre-set thresholds. A “phrase” is the maximal set of mutually repeated segments, computable as maximal cliques in an undirected repetition graph, while a “section” is a higher-level grouping of consecutive phrases delimited by runs of non-melodic or non-repeating material longer than two measures. This yields a two-level hierarchy: phrases (level 1), sections (level 2), and the song as the root (level 3) (Dai et al., 2020).

The complexity of structure is quantified using the Structure Description Length (SDL):

$$SDL(\Omega) = h\cdot|\Omega| + g\cdot\sum_{p\in P} avglen(p)$$

where $\Omega$ is the sequence of phrase labels, $|Ω|$ the total number of phrase instances, $P$ the set of distinct phrase types, and $avglen(p)$ the average length of phrase $p$ (in measures).

Beyond segmental models, graph-theoretical approaches represent music as time-evolving networks, with events or note-pixels as nodes and edges encoding temporal, harmonic, or rhythmic relations. Centrality measures, community structure, entropy, and motif statistics are then used to analyze and visualize structure (Alcalá-Alvarez et al., 1 Apr 2024, Tsai et al., 2023).

2. Methodological Approaches

MSA methods can be categorized as repetition-based, self-similarity driven, supervised/semi-supervised learning, and graph/network-analytic. Key paradigms include:

• Barwise Self-Similarity and Compression: Audio is segmented into bars via downbeat detection, then each bar is represented as a high-dimensional feature (e.g., log-mel, chroma), and a self-similarity matrix (SSM) is constructed using cosine, centered cosine, or RBF similarity. Dynamic programming algorithms, such as the Correlation Block-Matching (CBM) algorithm, segment this SSM to maximize within-segment homogeneity and regularity, subject to penalties favoring 4/8-bar lengths (Marmoret et al., 2023, Marmoret et al., 2022). Linear (PCA, NMF) and non-linear (autoencoder) compression schemes provide latent codes that highlight structural regularities (Marmoret et al., 2022, Marmoret et al., 2021).
• Symbolic and Graph-Based Models: On symbolic data (e.g., MIDI scores), graph construction yields adjacency matrices reflecting pitch, chord, and rhythmic relations; novelty functions or changepoint detection (e.g., PELT) on these matrices identify structural boundaries. Methods such as G-PELT and G-Window allow multi-level segmentation with parametric control over granularity (Hernandez-Olivan et al., 2023). CNNs operating on overtone-augmented piano-rolls achieve state-of-the-art section boundary detection on MIDI corpora (Eldeeb et al., 20 Sep 2025).
• Supervised Representation Learning: Supervised metric learning approaches train neural encoders such that embeddings for two segments are proximal if their annotated functions are identical (“verse” vs. “chorus”), ultra-metricizing the SSM and improving the sharpness of homogeneity and repetition cues for downstream segmentation (Wang et al., 2021). SSM-Net trains encoders directly to reconstruct a ground-truth SSM via differentiable loss, bypassing triplet losses and yielding segment-contrastive features (Peeters et al., 2022, Peeters, 2023).
• Multi-task Deep Learning for Functionality: Jointly predicting boundaries and section functions (e.g., “verseness,” “chorusness”) is performed using architectures such as Spectral-Temporal Transformers (SpecTNT), trained with BCE and sequence-level CTL loss, and proven to set new benchmarks in frame-level and section-level accuracy on multiple datasets (Wang et al., 2022, Wang et al., 2022). SongFormer fuses local and global self-supervised features with a Transformer and a learned source embedding, robustly aggregating supervision from large, heterogeneous corpora for both segmentation and labeling (Hao et al., 3 Oct 2025).
• Foundational Audio Encoders (FAEs): Large-scale, self-supervised music models trained with masked language modeling (MLM), such as MusicFM and MERT, have emerged as the most effective FAEs for MSA, outperforming models trained on contrastive or codec objectives. Pretraining with longer context windows and exclusively musical data further boosts segmentation and labeling performance (Toyama et al., 19 Dec 2025).

3. Evaluation Metrics, Datasets, and Experimental Protocols

Evaluation of MSA systems uses application-standard boundary F-measures at strict (0.5 s) and relaxed (3 s) tolerances (HR.5F, HR3F). Segment labeling is evaluated via frame-level accuracy and pairwise frame clustering F1 (PWF). Annotation corpora include Harmonix Set, SALAMI, RWC-Pop, Isophonics/Beatles, and in symbolic domains, the SLMS (Segmented Lakh MIDI Subset) and Baroque Allemandes corpus (Eldeeb et al., 20 Sep 2025, Carnovalini et al., 2022). Cross-validation and cross-dataset protocols are standard, with parameter grids and ablation studies reporting the robustness of results.

Recent systems such as SongFormer report HR.5F=0.703 and ACC=0.807 on the SongFormBench-HarmonixSet (Hao et al., 3 Oct 2025), with SOTA models achieving >0.8 functional label accuracy on Chinese song subsets. Symbolic boundary detection attains F1=0.767 on large MIDI datasets, exceeding audio-based baselines by wide margins (Eldeeb et al., 20 Sep 2025). Unsupervised barwise CBM approaches are competitive with fully supervised CNNs (F3≈0.80 on RWC-Pop) (Marmoret et al., 2023).

4. Feature Interaction and Statistical Characterization

Detailed statistical analyses confirm that harmony, melody, and rhythm are intricately modulated by structure:

• Harmony: Chord I (tonic) is significantly over-represented at section endings, V (dominant) at phrase endings; V–I cadence probability peaks at section ends (94% vs. 47% elsewhere); these findings are established by one-tailed unpaired t-tests with p < 10⁻⁴ (Dai et al., 2020).
• Melody/Rhythm: Long notes (≥quarter) dominate at section boundaries (72% vs. 6.4% at phrase boundaries within sections). Pitch-class and note-length distributions, computed per structure locus, reveal significant interaction as measured by multinomial mutual information.
• Temporal trends: Cross-phrase similarity has trended downward and phrase complexity (chord entropy) upward over time (P < 0.01 and P < 0.002), evidencing increased structural contrast and harmonic diversity.

Graph and network analyses allow time-varying investigation of centrality, entropy, and community structure—peaks in entropy or community count time series often align with formal structural boundaries (Alcalá-Alvarez et al., 1 Apr 2024, Tsai et al., 2023).

5. Implications, Limitations, and Future Directions

The confluence of accurate, data-driven boundary detection with principled functional labeling undergirds several application domains:

• Generative Models: Conditioning neural architectures (LSTM, Transformer) on two-level structure plans (section and phrase lengths, labels) yields more authentic structure-aware generation, with the ability to statistically match chord and melody distributions at appropriate structural loci (Dai et al., 2020).
• Section-aware Evaluation: Automated frameworks now test whether new material respects learned norms for harmonic, melodic, and rhythmic distributions within and at the boundaries of sections.
• Symbolic-to-Audio Bridging: With symbolic CNNs proving superior in strict boundary placement, future work will likely involve transfer and fusion between symbolic (score/MIDI) and audio (waveform/spectrogram) paradigms (Eldeeb et al., 20 Sep 2025).
• Scalability and Robustness: SongFormer demonstrates that fusing strong pre-trained self-supervised features with learned supervision source embeddings permits scaling to noisy, partial, and schema-mismatched labels while retaining SOTA performance (Hao et al., 3 Oct 2025).
• Hierarchical and Multi-modal Integration: Ongoing work includes expanding MSA to multi-level, multi-modal (audio, score, lyric) inputs, leveraging segmentation kernels, self-similarity-aware losses, graph neural networks, and hierarchical sequence models.

Limitations remain in boundary detection under misestimated downbeats/bar positions, generalization to non-Western and non-popular music, and hierarchical functional annotation. Richer, multi-layered or continuous similarity measures, adaptive penalties, and the fusion of explanatory graph-theoretic embeddings with deep metric learning are active research directions.

6. Comparative Summary Table

Methodology	Input Type	Segmentation	Labeling	Notable Performance
Barwise CBM/Compression	Audio (barscale)	Unsupervised DP on SSM	None/posthoc	F₃≈0.80 on RWC-Pop (Marmoret et al., 2023)
Symbolic CNN (MobileNetV3)	MIDI (overtones)	Supervised; per-bar window	None	F1=0.7675 (SLMS) (Eldeeb et al., 20 Sep 2025)
SpecTNT/CTL	Audio	Joint boundary and function	Multi-task	HR.5F=.623, ACC=.675 (Wang et al., 2022)
SongFormer	Audio	Transformer over SSL features	Multi-task	HR.5F=0.703, ACC=0.807 (Hao et al., 3 Oct 2025)
FAE Linear Probe	Audio (encoder)	SSM + linear probe	Linear probe	HR.5F=0.54 (MusicFM) (Toyama et al., 19 Dec 2025)
Graph-based (G-PELT)	Symbolic (graph)	Unsupervised changepoint (adj.)	None	F1=0.564 (1-bar SWD) (Hernandez-Olivan et al., 2023)
Supervised Metric Learning	Audio (mel)	SSM clustering, spectral/NMF	None/posthoc	HR.5F↑ (0.497→0.684) (Wang et al., 2021)

All findings point to the criticality of hierarchical, statistically informed representations, tight bar/beat alignment, and the integration of learned or symbolic music priors for high-fidelity music structure analysis.