SongFormer: Dual Music AI Models

• SongFormer is a dual-use music AI framework that combines music structure analysis and full-song text-to-lyrics generation under a shared focus on song form.
• The music structure analysis system employs multi-resolution SSL audio fusion and a learned source embedding to accurately segment and label song parts.
• The lyrics-generation model utilizes a GPT-2 backbone with explicit song-form markers and multi-level syllable-count control to produce form-consistent lyrics.

SongFormer is a model name used for two distinct systems in music AI. In "SongFormer: Scaling Music Structure Analysis with Heterogeneous Supervision," it denotes a framework for music structure analysis (MSA) that segments songs into functionally meaningful parts and assigns functional labels under heterogeneous supervision (Hao et al., 3 Oct 2025). In "Song Form-aware Full-Song Text-to-Lyrics Generation with Multi-Level Granularity Syllable Count Control," it denotes a GPT-2–based conditional LLM for full-song text-to-lyrics generation with explicit song-form markers and multi-level syllable-count control (Chae et al., 2024). The shared name reflects a common emphasis on song form, but the two systems address different tasks, operate on different inputs, and are evaluated by different criteria.

1. Terminological scope and disambiguation

A common source of confusion is that SongFormer does not denote a single canonical architecture. The name is attached to two separate research contributions with different objectives and technical stacks: one is an audio-based MSA framework, and the other is a text generation model conditioned on song form and syllable budgets (Hao et al., 3 Oct 2025, Chae et al., 2024).

Paper	Primary task	Defining mechanism
"SongFormer: Scaling Music Structure Analysis with Heterogeneous Supervision" (Hao et al., 3 Oct 2025)	Music structure analysis	Multi-resolution SSL audio fusion plus learned source embedding
"Song Form-aware Full-Song Text-to-Lyrics Generation with Multi-Level Granularity Syllable Count Control" (Chae et al., 2024)	Full-song text-to-lyrics generation	GPT-2 with song-form markers and multi-level syllable count tokens

The distinction is substantive. The MSA system is designed for boundary detection and functional labeling in songs, using raw audio and heterogeneous annotations. The lyrics-generation system is designed for autoregressive text generation, using an input text embedding together with a pre-specified generation plan. This suggests that "SongFormer" is best understood as a reused model name centered on structural form rather than as a single research lineage.

2. SongFormer for music structure analysis

In the MSA setting, music structure analysis aims to segment a song into functionally meaningful parts such as intro, verse, and chorus, and to label each segment. Precise boundary detection and accurate functional labeling underpin music understanding, recommendation, and controllable generation. The motivating problem is that existing MSA has been hampered by small, heterogeneous corpora with varying annotation schemas; the paper gives HarmonixSet as an example with only 912 songs. It also identifies a second bottleneck: models trained from scratch or relying on heavy preprocessing such as beat-tracking and source separation, which limit scalability and generalization. General-purpose LLMs such as Gemini 2.5 Pro are described as offering coarse-grained structure annotations but lacking temporal precision (Hao et al., 3 Oct 2025).

The framework’s stated contributions are fourfold. First, it uses multi-resolution feature fusion, combining short-window and long-window self-supervised audio representations to capture both fine-grained details and long-range context without fine-tuning the SSL backbones. Second, it introduces heterogeneous supervision via a learned source embedding that conditions the model on dataset provenance, allowing training on partial, noisy, and schema-mismatched annotations. Third, it releases SongFormDB, described as the largest MSA corpus to date, with over 10 k tracks spanning languages and genres, and SongFormBench, a 300-song expert-verified benchmark. Fourth, it reports state of the art on strict boundary detection and functional label accuracy while remaining computationally efficient.

The framing of the task is explicitly audio-structural rather than symbolic. Input is raw audio truncated to at most 420 s, and the model is designed to recover section boundaries and a unified 7-class scheme consisting of intro, verse, chorus, pre-chorus, bridge, outro, and inst. The preservation of pre-chorus in the unified schema is notable because schema harmonization often collapses such distinctions; here, the data explicitly states that pre-chorus is preserved.

3. Architecture and heterogeneous supervision in the MSA model

The MSA SongFormer uses two pre-trained SSL backbones, MuQ and MusicFM, each producing frame-level embeddings at 25 Hz. The local path splits the input into consecutive, non-overlapping 30 s chunks, yielding $x_s \in \mathbb{R}^{T \times d}$; concatenating 14 chunks aligns local context with the 420 s global window. The global path processes the entire 420 s window in one shot to obtain $x_\ell \in \mathbb{R}^{T \times d}$. For each backbone, the local and global streams are fused framewise by

$$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$

The fused MuQ and MusicFM features are then stacked along the feature dimension to form $H \in \mathbb{R}^{T \times 2D}$ (Hao et al., 3 Oct 2025).

Temporal reduction is handled by a dedicated downsampling module with two parallel branches that reduce the 25 Hz sequence to approximately $8.33$ Hz: a depthwise-pointwise convolution with kernel $=3$, stride $=3$, and an average-pooling branch with factor $=3$ followed by pointwise convolution. The branch outputs are summed element-wise. Heterogeneous supervision enters through a learned source embedding $s \in \mathbb{R}^{2D}$, indexed by dataset origin $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$, which is added after downsampling: $x_\ell \in \mathbb{R}^{T \times d}$0 At inference, $x_\ell \in \mathbb{R}^{T \times d}$1 is fixed to the HarmonixSet embedding to favor high-quality annotation patterns.

Sequence modeling is performed by a 4-layer Transformer with hidden size 512 and RoPE positional encoding. Two parallel heads are attached: a boundary head that projects to a scalar per frame with sigmoid activation, and a function head that projects to 7 classes per frame with softmax. The supervision regime mirrors the heterogeneity of the corpus. SongFormDB aggregates four sources: HX, reconstructed from HarmonixSet with 512 train and 200 val songs; Private with 4,314 lyric-aligned songs; Hook with 5,933 songs with partial annotations; and Gem with 4,387 tracks annotated by Gemini 2.5 Pro after cleaning. All labels are mapped to the unified 7-class scheme.

Training combines boundary detection and functional labeling objectives. Boundary detection uses binary cross-entropy plus boundary-aware total variation smoothing, with the TV term downweighted near true boundaries by setting $x_\ell \in \mathbb{R}^{T \times d}$2 if $x_\ell \in \mathbb{R}^{T \times d}$3 is near a true boundary and $x_\ell \in \mathbb{R}^{T \times d}$4 otherwise. Functional labeling uses frame-wise cross-entropy with focal weighting. The total loss interpolates between the boundary and function terms with $x_\ell \in \mathbb{R}^{T \times d}$5, $x_\ell \in \mathbb{R}^{T \times d}$6, and $x_\ell \in \mathbb{R}^{T \times d}$7. Dataset-specific masks are used when labels are incomplete; for example, in SongForm-Hook the function loss is computed only within $x_\ell \in \mathbb{R}^{T \times d}$8 s of annotated segments. This training design is central to the claim that the framework can learn from partial, noisy, and schema-mismatched labels.

4. Benchmark construction, metrics, and empirical results for MSA

The MSA evaluation protocol is built around SongFormBench, a 300-song benchmark consisting of 200 Western pop songs and 100 Chinese modern songs. The annotations are expert-verified via cross-checking audio, original labels, LLM outputs, and lyrics, and the released splits are intended to ensure reproducibility. The evaluation metrics are strict boundary hit-rate $x_\ell \in \mathbb{R}^{T \times d}$9, defined as $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$0 of predicted versus ground-truth boundaries within $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$1 s; relaxed boundary hit-rate $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$2, defined within $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$3 s; and frame-wise functional label accuracy $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$4 (Hao et al., 3 Oct 2025).

On SongFormBench with HX+P+H+G training, All-In-One reports $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$5, $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$6, $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$7; LinkSeg-7Labels reports $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$8, $$h = W_s x_s + W_\ell x_\ell + b,\quad h \in \mathbb{R}^D.$$9, $H \in \mathbb{R}^{T \times 2D}$0; Gemini 2.5 Pro reports $H \in \mathbb{R}^{T \times 2D}$1, $H \in \mathbb{R}^{T \times 2D}$2, $H \in \mathbb{R}^{T \times 2D}$3; and SongFormer reports $H \in \mathbb{R}^{T \times 2D}$4, $H \in \mathbb{R}^{T \times 2D}$5, $H \in \mathbb{R}^{T \times 2D}$6. On the Chinese subset, All-In-One reports $H \in \mathbb{R}^{T \times 2D}$7, $H \in \mathbb{R}^{T \times 2D}$8, $H \in \mathbb{R}^{T \times 2D}$9; LinkSeg-7Labels reports $8.33$0, $8.33$1, $8.33$2; Gemini 2.5 Pro reports $8.33$3, $8.33$4, $8.33$5; and SongFormer reports $8.33$6, $8.33$7, $8.33$8. The paper’s summary states that SongFormer sets a new state-of-the-art ACC $8.33$9 and strict boundary $=3$0 $=3$1 on HX alone$=3$2, improves boundary precision by more than 60 points relative to Gemini 2.5 Pro at $=3$3 s tolerance, and remains competitive under relaxed tolerance.

The ablation results attribute gains to three specific components. Removing the source embedding drops ACC from $=3$4 to $=3$5 and $=3$6 from $=3$7 to $=3$8. Replacing the Transformer backend with a linear layer yields $=3$9. Omitting multi-resolution drops ACC to $=3$0. A dedicated comparison between windowing strategies reports $=3$1 s only at $=3$2, $=3$3; $=3$4 s only at $=3$5, $=3$6; and $=3$7 at $=3$8, $=3$9, which is the best setting.

The efficiency profile is also reported explicitly. Downsampling from 25 Hz to 8.33 Hz reduces Transformer compute by $=3$0. Task-specific layers comprise approximately 4 M parameters, with frozen SSL backbones. Inference on NVIDIA L40 is approximately 0.2 real-time factor per 3 min song, including SSL feature extraction. The paper also notes a limitation: a slight boundary-sharpness trade-off when incorporating lower-quality labels, together with reliance on fixed-window SSL models. Its stated future directions are dynamic temporal adaptation of SSL features beyond fixed 30 s and 420 s windows, incorporation of symbolic or multimodal cues, semi-supervised refinement of low-quality annotations using the trained SongFormer as a teacher, and extension to hierarchical structure with sub-sections and nested forms.

5. SongFormer for song form-aware full-song text-to-lyrics generation

The second system named SongFormer addresses a different problem: full-song text-to-lyrics generation conditioned on both input semantics and song form. Its backbone is GPT-2, implemented as a decoder-only Transformer with 12 layers, hidden size $=3$1, 12 attention heads, and maximum length $=3$2. It can be initialized from pretrained GPT-2 (117M) or trained from scratch. Self-attention is standard: $=3$3

$=3$4

where $=3$5 and $=3$6 (Chae et al., 2024).

Conditioning is split into semantic and structural channels. A pretrained SentenceTransformer, "all-mpnet-base-v2," maps arbitrary input text into a continuous embedding $=3$7. During generation or fine-tuning, $=3$8 is prepended to the token stream by bypassing GPT-2’s embedding layer, so that the first "token" is the 768-dimensional semantic vector. Structural conditioning is expressed through special tokens. Song-form markers include tokens such as <VERSE>, <CHORUS>, and <BRIDGE>. A syllable count token <SYL:s> specifies the total syllable count for the upcoming segment. Generation-directive tokens encode granularity: <GEN_P> ... <END_P> for paragraph-level, <GEN_L> ... <END_L> for whole-line, <GEN_L_NW> ... <END_L> for line broken into phrases or words, <GEN_N> ... <END_NW> for phrase-level, and <GEN_W> ... <END_NW> for word-level. These tokens are interleaved with lyric tokens so that the model knows the structural unit and syllable budget at each step.

The training objective is autoregressive cross-entropy,

$=3$9

where the plan is the sequence of special tokens. The paper explicitly states that there is no explicit syllable-penalty term in training; syllable adherence is learned from the forced-choice conditioning on syllable tokens. For evaluation, it defines Syllable Count Distance: $s \in \mathbb{R}^{2D}$0

The dataset is drawn from Genius Song Lyrics, with approximately 5.1 M songs and 3.3 M English songs. Training retains only songs with explicit paragraph-level form annotations, removes non-lyric tokens such as "guitar solo" and repetition markers like $s \in \mathbb{R}^{2D}$1, converts numerals to words, and applies a toxicity filter with Detoxify score $s \in \mathbb{R}^{2D}$2. Syllable counts are computed per word via an off-the-shelf syllable-counter library. The split is approximately 340 K training examples, 18 K validation, and 10 K test. Because no text-to-plan paired data exists, each lyric sample is converted into a random plan by pre-order tree traversal of its paragraph-to-lines-to-phrases-or-words hierarchy. At each subtree, with probability $s \in \mathbb{R}^{2D}$3 the subtree is selected and replaced by its special tokens and aggregated syllable count; if a leaf is selected, there is a 50% chance of selecting just that word or a random phrase of up to 8 words. Optimization uses AdamW with weight decay, learning rate $s \in \mathbb{R}^{2D}$4, 500 warm-up steps, batch size 8, and 10 epochs on one 3090Ti GPU, with byte-level BPE vocabulary size 50,257 and maximum length $s \in \mathbb{R}^{2D}$5.

6. Decoding strategy, evaluation, and reported behavior in lyrics generation

Inference in the lyrics-generation SongFormer begins with a pre-specified full plan, for example a sequence of song-form markers, syllable-count tokens, and generation directives. Decoding is autoregressive but constrained by the plan: each plan-directive token is fed into the model rather than generated, the corresponding <GEN_*> block is opened, lyric tokens are sampled until the matching <END_*> token appears, and the process then advances to the next directive. The reported sampling hyperparameters are top-$s \in \mathbb{R}^{2D}$6, top-$s \in \mathbb{R}^{2D}$7, temperature $s \in \mathbb{R}^{2D}$8, and repetition_penalty $s \in \mathbb{R}^{2D}$9. If the matching end token never appears or the output is structurally invalid, generation is discarded and retried up to 10 times (Chae et al., 2024).

Evaluation uses Perplexity on text tokens only, trimmed at the 99th percentile; Syllable Count Distance and Syllable Count Error Rate; BERT-Score between the input text embedding and generated lyrics; and Normalized Levenshtein Distance to assess song-form consistency across paragraphs. The comparison includes Front-P/S, in which all plan tokens are prepended before <LYR_START> for single-shot generation; Back-P/S, in which plan tokens are fed step-by-step during decoding; and Both-P/S, a hybrid in which plan tokens are both prepended and fed stepwise.

The key findings are specific. Back-S achieves the best SCD, approximately $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$0 at the full-song level, SCErr of approximately $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$1, and the top BERT-Score of approximately $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$2. Pretrained models lower Perplexity but yield slightly worse syllable-control than scratch. Paragraph-level and line-level control are harder than phrase-level or word-level control. For song-form consistency, BERT-Score is highest and NLD lowest when comparing paragraphs of the same form, such as verse-to-verse, rather than different form, such as verse-to-chorus.

The paper also includes a large-LLM comparison on a simple four-line verse with fixed syllable counts. ChatGPT-3.5 and ChatGPT-4.0 are reported at approximately $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$3, $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$4, and $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$5, whereas the proposed model is reported at approximately $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$6, $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$7, and $\text{HX}, \text{Private}, \text{Hook}, \text{Gem}$8 on the same subset. The contrast is not framed as a general superiority claim over LLMs in all lyric-generation dimensions; rather, it is specifically tied to fine-grained syllable adherence under explicit song-form constraints.

Taken together, the two SongFormer systems instantiate different technical responses to song form. One uses multi-resolution SSL audio representations, a learned source embedding, and heterogeneous supervision to improve strict boundary detection and functional label accuracy in MSA. The other uses explicit structural tokens and multi-level syllable-count control to generate lyrics that conform closely to a prescribed form. A plausible implication is that "song form" functions as an organizing prior across both analysis and generation, but the current literature treats that prior through task-specific architectures rather than through a unified model family.