MuSFA: Music Structural Function Analysis

• MuSFA is a supervised framework that directly predicts semantic section labels from audio, moving beyond the abstract segmentation of traditional methods.
• It employs a multi-task deep learning architecture with SpecTNT and CTL loss, achieving improved boundary detection and function labeling across datasets.
• MuSFA adapts to incomplete annotations through partially labeled training and extends its approach to hierarchical, symbolic, and domain-specific music analyses.

Music Structural Function Analysis (MuSFA) is presented as a supervised framework for predicting musically meaningful section labels such as “verse” and “chorus” directly from audio, without requiring the full context of a song. Relative to conventional music structure analysis (MSA), which typically divides a song into segments and groups them with abstract labels such as “A,” “B,” and “C,” MuSFA models structural semantics as time-varying activations and couples boundary detection with function prediction rather than treating functional labeling as a post-processing step (Wang et al., 2022, Wang et al., 2022). In the surrounding literature, this perspective is extended through partially labeled training, supervised metric-learning front ends, foundation-model adaptation, symbolic and hierarchical form analysis, and domain-specific applications ranging from Western popular music to Mozart sonata form and Korean folk song (Wang et al., 2021, Zhang et al., 17 Jul 2025, Zhao et al., 18 May 2026, Han et al., 14 Aug 2025).

1. Conceptual scope and relation to music structure analysis

Conventional MSA methods traditionally search for musically meaningful patterns in audio: homogeneity, repetition, novelty, and segment-length regularity. Hand-crafted audio features such as MFCCs or chromagrams are commonly used to elicit these patterns, and structure labels are often abstract rather than semantic. MuSFA reorients the task toward direct estimation of structural function, so that “verseness,” “chorusness,” and related activations are inferred from the audio signal itself rather than reconstructed from a full-song self-similarity analysis (Wang et al., 2021, Wang et al., 2022).

A core formulation defines a fixed 7-class semantic taxonomy targeted at Western popular music: intro, verse, chorus, bridge, instrumental (“inst”), outro, and silence. Each class is treated as a binary activation curve over time during training and prediction. In parallel, later partially labeled training work reports a $K=7$ class setting used to map HookTheory labels into MuSFA classes, with chorus, verse, inst, bridge, intro, and outro explicitly enumerated; that formulation centers on learning from incomplete annotation rather than on redefining the semantic premise (Wang et al., 2022, Wang et al., 2022).

The distinction from earlier pipelines is methodological as well as terminological. Traditional systems generally require the full song to determine whether two spans repeat and frequently assign abstract labels before any semantic interpretation. MuSFA is instead described as a fully supervised, context-free system that assigns explicit semantic section labels from raw audio. A plausible implication is that MuSFA should be understood less as a single architecture than as a research program that prioritizes structural function as the primary target of analysis (Wang et al., 2022).

2. Core supervised formulation from audio

The foundational MuSFA architecture is a multi-task deep learning framework built on SpecTNT, a spectral-temporal Transformer model. Audio is sampled at 16 kHz, transformed with STFT using window $=1024$ and hop $=512$, projected to a trainable harmonic filterbank, and passed through a small 2D-ResNet front end before a stack of 5 SpecTNT blocks. Each block contains a spectral encoder operating per time frame over Frequency Class Tokens and frequency-bin embeddings, followed by a temporal encoder operating across frames; the reported configuration uses 4 heads for the spectral encoder, 8 heads for the temporal encoder, and model dimension $96$ (Wang et al., 2022).

The linear output layer predicts 1 boundary activation and 7 function activations per frame, all interpreted as sigmoid activation curves. Post-processing converts peaks in the boundary activation to estimated boundaries and labels each inter-boundary segment by the most active function. In addition to frame-wise binary cross-entropy for boundary and function curves, the system can include an auxiliary connectionist temporal localization (CTL) loss,

$$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$

which is described as a sequence-level criterion akin to CTC. The overall training objective is reported as

$$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$

The cited experiments use 24 s chunks, with a 36 s ablation, 3 s chunk hop, data augmentation via random additive noise, random gain, and random high/low pass filtering, and Adam with learning rate $5\times10^{-4}$, weight decay $0.9$, and early stopping patience of 2 epochs (Wang et al., 2022).

Quantitatively, SpecTNT with 24 s chunks and CTL achieved on Harmonix Set 4-fold cross-validation HR.5F $=0.570$, ACC $=0.701$, CHR.5F $=1024$0, and CF1 $=1024$1. In cross-dataset evaluation on RWC-Pop, the same configuration reached HR.5F $=1024$2 and CF1 $=1024$3. The same study reports that multi-point prediction over 24 s is over $=1024$4 more efficient at inference than the instant per-frame Harmonic-CNN baseline, while 36 s chunks improve ACC but can slightly reduce boundary localization, indicating a trade-off between longer-context function modeling and boundary precision (Wang et al., 2022).

3. Data harmonization and partially labeled training

A central practical constraint for MuSFA is the scarcity of fully annotated songs. One response is cross-corpus label consolidation: Harmonix Set, SALAMI-pop, RWC-Pop, and Isophonics are unified by a substring-matching procedure that maps disparate free-form labels into the MuSFA taxonomy. The reported mapping gives priority by list order, so that “pre-chorus” maps to verse rather than chorus, and 99.3% of raw labels collapse cleanly, with the remainder forced into “inst” (Wang et al., 2022).

A second response is the incorporation of partially labeled data through the HookTheory Lead Sheet Dataset (HLSD). HLSD contains 18,843 user-annotated lead-sheet sections with timestamps in full recordings. In the MuSFA extension, each excerpt is trimmed to lie within the original recording but padded randomly by 8–12 s front and back, yielding approximately 30–42 s excerpts. Because each excerpt carries only a single annotated section span, the remainder is treated as “unknown,” and the function-label loss is masked outside the labeled span. The training pool then combines the fully labeled public datasets—Harmonix Set (912 songs), SALAMI-pop (274 songs), RWC-Pop (100 songs), and Isophonics (277 songs), for a total of approximately 1,563 full-song annotations—with the HLSD excerpts (Wang et al., 2022).

The partially labeled formulation introduces a frame-wise mask $=1024$5 and trains with

$=1024$6

where $=1024$7 on fully labeled material and only within the annotated section for HLSD excerpts. In reported cross-dataset evaluations, including HLSD improves state-of-the-art boundary detection and section labeling by approximately 3% and 1% respectively. On Harmonix 4-fold evaluation, HR.5F rises from .570 to .595 and ACC from .701 to .714; on SALAMI-pop cross-dataset testing, HR.5F rises from .490 to .532 and CHR.5F from .357 to .399 (Wang et al., 2022).

These results are significant because they show that MuSFA can scale with incomplete supervision. At the same time, the same study notes that function-label gains are modest, attributing this to the fact that neighboring sections in HLSD excerpts are not labeled, so the added data benefit boundary estimation more strongly than fine-grained section discrimination (Wang et al., 2022).

4. Learned representations, metric learning, and foundation-model adaptation

Before direct structural-function prediction was established, supervised metric learning had already shown that section annotations can be used to learn structure-sensitive embeddings. In that approach, a deep neural network maps 8-second log-mel spectrogram windows, beat- or downbeat-centered, to 100-dimensional L2-normalized embeddings such that same-section fragments from the same song are near each other and different-section fragments are far apart. The model replaces classification layers with a Linear(256→100), Leaky ReLU, BatchNorm, Linear(100→100), and L2-normalization head, and is trained with Multi-Similarity loss together with a same-song batch sampling scheme so that “positive” means “same section of the same piece” and “negative” means “different section of that piece” (Wang et al., 2021).

The resulting embeddings serve as drop-in features for existing MSA algorithms. Self-similarity is computed from Euclidean distances and converted by $=1024$8. These learned self-similarity matrices are reported to exhibit much clearer block patterns than MFCC or chroma, and they improve spectral clustering, convex NMF, and Foote-style novelty detection plus agglomerative labeling without redesigning post-processing. On intra-dataset Harmonix evaluation, baseline scluster on MFCC + chroma gives HR.5F $=1024$9 and PWF $=512$0, while Harmonic-CNN + Multi-Similarity yields HR.5F $=512$1 and PWF $=512$2; all three MSA algorithms show significant boosts with $=512$3, and cross-dataset Harmonix→SALAMI-pop testing improves HR.5F from 0.31 to 0.38 with $=512$4 (Wang et al., 2021).

Later work investigates pre-trained foundational audio encoders (FAEs) for MuSFA tasks. Across 11 evaluated encoder types, the reported trend is MLM $=512$5 contrastive $=512$6 codec $=512$7 supervised, with longer pretraining context improving both boundary detection and function prediction. MusicFM, AudioMAE trained on private music, and MERT-large are identified as particularly effective, and pooling to 2 Hz improves HR3F and PWF at the cost of slightly lower HR.5F. A stated practical recommendation is to prefer Transformer-based encoders pretrained with masked language modeling on long-form music, and to combine FAE embeddings with traditional self-similarity matrices or use boundary logits as priors in peak-picking (Toyama et al., 19 Dec 2025).

Temporal adaptation extends this line by modifying the input-processing stage rather than re-engineering the pre-trained encoder. A 12-layer Conformer pre-trained on large-scale music audio is adapted by increasing the input window from 30 s to as much as approximately 270 s, coarsening temporal resolution through low-resolution adaptation, and appending four lightweight Transformer layers with separate boundary and function heads. The reported objective is $=512$8 with $=512$9 and $96$0, and the structural labels follow the Harmonix Set schema $96$1intro, verse, pre-chorus, chorus, bridge, outro$96$2 (Zhang et al., 17 Jul 2025).

The main empirical pattern is a trade-off between global structural-function accuracy and fine boundary precision. On Harmonix Set 8-fold cross-validation, the baseline MusicFM 30 s window reaches ACC $96$3, HR.5F $96$4, and HR3F $96$5, whereas the temporally adapted model with $96$6 s and $96$7 obtains ACC $96$8 and HR3F $96$9 but HR.5F $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$0. Cross-dataset evaluation from RWC-Pop to Harmonix improves ACC from $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$1 to $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$2 and HR3F from $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$3 to $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$4 for a 180 s, $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$5 configuration. The same study explicitly characterizes the principal trade-off as one between boundary precision and global-label accuracy under coarse temporal pooling (Zhang et al., 17 Jul 2025).

5. Hierarchical and domain-specific extensions

MuSFA-oriented work increasingly treats structural function as hierarchical rather than flat. In symbolic popular music, automatic detection of phrase- and section-level repetition structures shows that the two levels interact differently with harmony, melody, and rhythm. The reported findings include high incidence of half cadences V at phrase ends in-section (approximately 58%), authentic cadence V–I at section ends 94% of the time in major mode versus 47% baseline, phrase-start tonic I around 35%, and long notes at 72% of section-end phrase endings versus 6.4% at mid-section phrase ends. These results are presented as evidence that structural hierarchy carries distinct functional roles beyond simple repetition or similarity (Dai et al., 2020).

In baroque and classical domains, the same tendency appears as a move toward explicit trees, thematic functions, and upper-level form labels. The following examples show how the term is extended beyond verse–chorus annotation.

Work	Musical domain	Reported MuSFA-relevant contribution
(Carnovalini et al., 2022)	Strongly structured baroque allemandes	Schenkerian-inspired Sk_trees, four-feature Diff_tree, top-down shared-structure tree
(Zhao et al., 18 May 2026)	Mozart sonata-form movements	SoSA-Moz dataset; upper level E/D/R and lower level S₁/TR/S₂/C; Sonalyzer-Moz baseline
(Han et al., 14 Aug 2025)	Korean folk song	Automatic motive segmentation with Whisper-small; motif count and duration entropy as structural features

Carnovalini et al. model each segment as a rooted, ordered, labeled tree $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$6, where labels include MIDI pitch, duration, reduction level, and optional motif identifiers. Pairwise comparison of two trees yields a Diff_tree whose nodes carry four feature-values: Sk, Ch, Dir, and Int. The abstract emphasizes a top-down approach that abstracts shared structure from pairwise differences between Schenkerian-inspired analyses, thereby providing a rich hierarchical description of the corpus. In the provided MuSFA-oriented interpretation, the resulting shared-structure tree yields a corpus-wide hierarchy of motifs, repeats, transpositions, and local contour-changes (Carnovalini et al., 2022).

For classical audio, SoSA-Moz introduces 582 piano/string recordings of 35 unique Mozart sonata-form movements, grouped by movement to avoid train/test leakage, with hierarchical annotations at an upper structure level—Exposition, Development, Recapitulation—and a lower thematic-function level within Exposition and Recapitulation—First Subject, Transition, Second Subject, and Coda. Sonalyzer-Moz combines mel-spectrogram embedding, feature aggregation at approximately 1 s granularity, a dynamic self-similarity matrix, positional encoding, and a 5-layer LSTM with hidden size 1024. With $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$7, $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$8, and $$L_{\mathrm{ctl}} = - \log \sum_{\pi\in B^{-1}(y)} \prod_{t=1}^T P(\pi_t \mid x),$$9, the best reported performance is HR3F $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$0, described as demonstrating upper-level analysis of sonata form (Zhao et al., 18 May 2026).

In Korean folk song, MuSFA appears at the level of motive statistics rather than section labels. Fine-tuning Whisper-small on 150 annotated songs (186.4 min) with a special $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$1 token yields mean boundary F1 $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$2 at $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$3 s tolerance under 10-fold cross-validation. The system is then applied to 856 songs to compute motif count per minute and duration entropy, with MANOVA reporting Wilks’ $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$4, $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$5, $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$6. The reported category means differentiate agricultural, artisan, lament, entertainment, minstrel, and lullaby repertoires, indicating that structural features vary systematically according to social function (Han et al., 14 Aug 2025).

6. Evaluation practice, recurring limitations, and research directions

Evaluation across MuSFA-related work is heterogeneous but internally consistent. Boundary detection commonly uses HR.5F and HR3F, corresponding to hit-rate $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$7 within $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$8 s and $$L = 0.9\,L_{\mathrm{boundary}} + 0.1\,L_{\mathrm{function}} + \lambda\,L_{\mathrm{ctl}}.$$9 s tolerance. Function prediction is evaluated with frame-wise accuracy (ACC), pairwise frame-level clustering $5\times10^{-4}$0 (PWF), pairwise chorus/non-chorus $5\times10^{-4}$1 (CF1), chorus-boundary hit-rate (CHR.5F), and normalized entropy segmentation score $5\times10^{-4}$2 depending on the task and benchmark (Wang et al., 2022, Wang et al., 2022, Wang et al., 2021, Zhao et al., 18 May 2026).

A recurring clarification in the literature is that MuSFA does not simply eliminate earlier MSA machinery. One branch remains compatible with self-similarity matrices, novelty curves, spectral clustering, convex NMF, graph novelty, or hierarchical repetition analysis, but uses supervision to make structural cues clearer; another branch predicts boundary likelihoods and function activations directly from audio without full-song self-similarity. This suggests that MuSFA is best viewed as a shift in target representation—from abstract segment identity to musically meaningful function—rather than as a single canonical decoder (Wang et al., 2021, Wang et al., 2022).

The main limitations are also recurrent. Fully annotated structural data remain scarce; HLSD introduces domain mismatch, label noise, and imbalance; gains from partial labels are stronger for boundaries than for function labeling; Isophonics remains difficult because of atypical or experimental structures and few labeled choruses; and fade-out choruses can be predicted as “outro,” which the cited work treats as an arguably reasonable alternative. For foundation-model pipelines, public replication is hindered by the use of private corpora, while temporal adaptation introduces an explicit trade-off in which coarse pooling improves ACC and HR3F but can reduce HR.5F. In sonata-form analysis, the annotation bottleneck is described as a major reason the field remains in an early stage (Wang et al., 2022, Wang et al., 2022, Toyama et al., 19 Dec 2025, Zhang et al., 17 Jul 2025, Zhao et al., 18 May 2026).

The future directions reported across these papers converge on several themes: integrating the MSA decoder into the neural network for end-to-end training, using self-attention to model long-range repetition explicitly, extending metric learning to hierarchical or multi-label section structures, leveraging semi-supervised or self-supervised pre-training, allowing multiple labels per segment, incorporating symbolic data such as MIDI or score, and broadening coverage to more granular functions and non-pop repertoires. A plausible implication is that MuSFA is evolving from verse–chorus recognition into a broader agenda for function-aware, multi-scale music analysis across audio and symbolic modalities (Wang et al., 2021, Wang et al., 2022, Wang et al., 2022, Zhao et al., 18 May 2026, Han et al., 14 Aug 2025).