SegTune: Structured Song Generation

• The paper introduces SegTune, a framework that distinctly separates global and segment-level prompts to enable structured and controllable song synthesis.
• It utilizes a Diffusion Transformer with Conditional Flow Matching and an LLM-based duration predictor to achieve precise lyric-to-music alignment in temporal segments.
• Empirical evaluations show that SegTune enhances musical coherence and controllability, although trade-offs in attribute accuracy may arise with preference optimization.

Searching arXiv for the specified SegTune paper and closely related versions to ground the article in current records. {"^^^^^^^^3query3 OR abs:\3^^^^^^^^"Structured and Fine-Grained Control for Song Generation\"","max_results":^^^^3ti:\3query3^^^^} SegTune is a non-autoregressive framework for structured and controllable song generation that conditions full-song synthesis on lyrics, a global text prompt, and temporally aligned segment-level prompts. Its central contribution is to make the distinction between song-wide attributes and section-specific attributes explicit: global prompts govern properties intended to hold across the whole song, while local prompts govern temporally varying properties such as section identity, instrumentation, emotional intensity, rhythmic feel, and arrangement changes. The framework couples this hierarchical conditioning with an LLM-based duration predictor that generates sentence-level timestamped lyrics in .lrc format, thereby supporting lyric-to-music alignment and temporal placement of segment prompts (&&&^^^^3query3^^^^&&&). A later arXiv version with the same title preserves this overall formulation while expanding several implementation and evaluation details (&&&^^^^3ti:\3^^^^&&&).

^^^^3ti:\3^^^^. Problem setting and conceptual scope

SegTune was proposed to address a limitation of prior end-to-end song generators: most systems could follow only a single global prompt, even though songs are structurally nonuniform over time. The motivating examples are explicitly sectional. An intro may be sparse and atmospheric, a verse restrained, a chorus more energetic, and a bridge more dramatic; instrumentation and emotional intensity may also evolve section by section. In this setting, a single prompt such as “Mandarin pop ballad with female singer and warm mood” can impose broad style, but it cannot directly specify local instructions such as “keep the intro sparse with piano only,” “make the chorus emotionally intense with layered guitars,” or “switch to a brighter, more uplifting feeling in the second chorus” (&&&^^^^3query3^^^^&&&).

The framework distinguishes global prompts from segment-level prompts. Global prompts describe properties that should persist across the whole song, including genre, overall mood, vocal timbre, singer gender, and related stylistic attributes. Segment prompts describe section-dependent properties such as structure labels, instrumentation, emotional change, rhythmic feel, intensity, and local performance character. These local descriptions may be authored manually or generated by a LLM during prompt engineering. This hierarchical prompt design is the basis of what the paper calls structured and fine-grained control (&&&^^^^3query3^^^^&&&).

SegTune is positioned against both autoregressive and non-autoregressive song generators. The work emphasizes that song generation is more demanding than singing voice synthesis because it must jointly compose and render vocals and accompaniment end to end. It also frames the non-autoregressive choice as a practical design for latent-space generation under conditional flow matching, rather than token-by-token audio language modeling. This suggests that SegTune should be understood not merely as a caption-conditioned music model, but as a temporally organized conditional generation system whose control surface is aligned to song structure itself (&&&^^^^3query3^^^^&&&).

^^^^3 OR abs:\3^^^^. Architecture and hierarchical conditioning mechanism

SegTune is built on a Diffusion Transformer backbone trained with Conditional Flow Matching. The backbone has ^^^^3ti:\3^^^^6 LLaMA-style Transformer blocks and about ^^^^3ti:\3^^^^.^^^^3ti:\3 parameters. Audio is modeled in latent space rather than waveform space: a ^^^^3ti:\3^^^^D VAE compresses raw 44 kHz audio into a latent sequence at ^^^^3 OR abs:\3ti:\3^^^^.5 Hz, and that latent sequence is the supervision target for the flow model. Conditioning consists of lyrics embeddings, global prompt embeddings, segment prompt embeddings, and diffusion timestep embeddings, which are concatenated channel-wise before entering the DiT backbone (&&&^^^^3query3^^^^&&&).

The segment-control mechanism is the core architectural feature. A global prompt PRESERVED_PLACEHOLDER_^^^^3query3^^^^ is encoded once as

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^

then broadcast across all PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^ latent frames: $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$ For each segment prompt $x_l^i$ associated with a temporal interval $(t_s^i, t_e^i)$, the prompt is encoded as

$$\mathbf{e}_l^i \leftarrow f_l(x_l^i) \in \mathbb{R}^{1 \times d_l}.$$

The temporal boundaries are mapped to latent indices using audio sampling rate $r$ and downsampling rate $r_d$: $$j_s^i \leftarrow \lfloor t_s^i \cdot r / r_d \rfloor,\qquad j_e^i \leftarrow \lfloor t_e^i \cdot r / r_d \rfloor.$$ The segment embedding is then written into the corresponding slice of a framewise local-conditioning tensor: PRESERVED_PLACEHOLDER_^^^^3ti:\3query3^^^^ After all segments are filled, global and local conditioning are concatenated,

PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:\3^^^^

and a 3-layer MLP produces the fused text-conditioning sequence,

PRESERVED_PLACEHOLDER_^^^^3ti:\3 OR abs:\3^^^^

with PRESERVED_PLACEHOLDER_^^^^3ti:\33^^^^ (&&&^^^^3query3^^^^&&&).

This construction preserves a sharp semantic distinction between prompt types. The global prompt affects every frame and supplies stylistic coherence; segment prompts act only in local windows and enable temporal variation in arrangement, mood, instrumentation, and sectional role. The paper compares this concatenate-based design with a “mixed” strategy that linearly blends global and local embeddings, and reports that mixing harms both musicality and controllability because it blurs the distinction between persistent and local instructions. For text encoding, SegTune uses Qwen3-Embedding-^^^^3query3^^^^.6B as both global and local prompt encoder, and the authors report that it preserves fine-grained semantic attributes in long natural-language prompts better than alternatives such as MuQ-Mulan, especially for singer-related attributes like gender and age (&&&^^^^3query3^^^^&&&).

The song generator itself is trained with the Conditional Flow Matching objective

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^4

where

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^5

Here PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^6, PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^7, PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^8, and PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^9 denotes the conditioning input. At inference time, SegTune uses an Euler ODE solver and a classifier-free guidance variant

PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3query3^^^^

with empirically chosen coefficients PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3ti:\3^^^^ and PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3 OR abs:\3^^^^. During training, ^^^^3 OR abs:\3query3^^^^% dropout is independently applied to global and segment-level conditions to support this guidance scheme (&&&^^^^3query3^^^^&&&).

3. Duration prediction and lyric-to-music alignment

A separate duration prediction module is one of SegTune’s major components. Rather than requiring the user to provide total duration or sentence/word timestamps manually, the framework fine-tunes Qwen3-4B-Base as a “composer” that autoregressively generates sentence-level timestamps in .lrc format. The model takes lyrics together with global and local prompts as input and outputs a complete timestamped lyric file. The paper emphasizes that duration prediction matters for three reasons: it determines the total song duration and therefore latent sequence length, provides sentence-level lyric/audio alignment, and supplies segment durations for local-prompt broadcasting (&&&^^^^3query3^^^^&&&).

The prompt template for this module instructs the model to analyze lyrics and song description, estimate a reasonable singing duration for each line based on line characteristics, overall song attributes, and structural flow including instrumental breaks and transitions, and then return a complete .lrc list with timestamps. The later arXiv version characterizes this as LyRiCs-format output and describes the predictor as operating autoregressively over the formatted timestamp sequence (&&&^^^^3ti:\3^^^^&&&). The duration predictor is fine-tuned for 8 epochs using LoRA rank 3^^^^3 OR abs:\3^^^^, batch size 8, gradient accumulation 4, max new tokens 4^^^^3query3^^^^96, and learning rate PRESERVED_PLACEHOLDER_^^^^3 OR abs:\33^^^^ (&&&^^^^3query3^^^^&&&).

Once timestamps are available, lyrics are aligned to the latent sequence at sentence level. The lyrics encoder produces a frame-aligned sequence PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^4, and this sequence is concatenated with text conditioning, current audio latent, and timestep embedding before entering the Diffusion Transformer. Segment windows are derived from predicted lyric boundaries; for instrumental sections such as intro, bridge, and outro, temporal extents are inferred from neighboring lyric-containing segments. The later version further states that lyrics are converted to phonemes using phonemizer for English and jieba plus pypinyin for Chinese, then inserted into a placeholder sequence according to sentence start frames (&&&^^^^3ti:\3^^^^&&&).

The duration-prediction ablation indicates that the fine-tuned model is close to using ground-truth timestamps. On held-out real songs, the mean absolute error is ^^^^3query3^^^^.99 seconds for Qwen3-SFT and 3.^^^^3 OR abs:\3^^^^4 seconds for zero-shot GPT-4o. Downstream generation quality with predicted timestamps is nearly identical to generation with ground-truth timings and consistently better than generation with GPT-4o timings. For SegTune-DPO, overall musicality is 4.^^^^3query3^^^^6 using Qwen3-SFT timestamps, 4.^^^^3query3ti:\3^^^^ with ground truth, and 3.86 with GPT-4o timestamps. Global MuLan is ^^^^3query3^^^^.453 for both ground truth and Qwen3-SFT, but ^^^^3query3^^^^.4^^^^3 OR abs:\3query3^^^^ for GPT-4o; segment MuLan is slightly lower for predicted timings than for ground truth, but the reported difference is small. This establishes duration prediction as more than a convenience layer: it is part of the control and alignment mechanism of the model (&&&^^^^3query3^^^^&&&).

4. Data pipeline, annotation strategy, and training regime

SegTune is trained on a curated in-house corpus composed primarily of Chinese or Mandarin pop songs. The raw corpus is filtered through a multi-stage pipeline. Metadata filtering removes non-musical material using a sound event detection module and applies constraints on duration, sampling rate, channel count, compression rate, and energy. Automatic quality assessment is then performed with Audiobox Aesthetics and SongEval. For lyrics, songs without annotations are transcribed after source separation with Demucs v4; FireRedASR is used for Mandarin and Whisper-Large-v3 for other languages. If LRC files already exist, an LLM-based cleaner removes metadata, and the cleaned lyrics are compared against ASR transcripts using edit distance, with high-discrepancy samples discarded. Structural segmentation labels such as intro, verse, chorus, and outro are extracted using an all-in-one music understanding model (&&&^^^^3query3^^^^&&&).

Prompt annotation is also automated. Global and segment-level descriptions are generated using Audio Flamingo 3. The global caption template asks for genre, mood, ambience, and singer vocal characteristics including gender, age range, timbre, and pitch range. The segment caption template asks for instrumentation, rhythm and melody style, mood, emotional impact, intensity and change, and notable singing or playing techniques. Structural segment labels are prepended to segment captions, and the first and last ^^^^3query3^^^^.5 seconds of each song are assigned fixed prompts—“This piece is the start/end of the song.”—to mark boundaries (&&&^^^^3query3^^^^&&&).

The dataset scale is substantial. For pretraining, after filtering out songs below 3^^^^3 OR abs:\3^^^^ kHz, outside 3^^^^3query3^^^^ seconds to 6 minutes, and the lowest 5% by quality score, the pipeline retains about 37^^^^3query3^^^^,^^^^3query3query3query3^^^^ songs totaling around ^^^^3 OR abs:\3^^^^7,^^^^3query3query3query3^^^^ hours. For fine-tuning, stricter requirements—44 kHz, stereo, and top 5^^^^3query3^^^^% on all automatic quality metrics—leave about 5^^^^3query3^^^^,^^^^3query3query3query3^^^^ songs or roughly 4,^^^^3query3query3query3^^^^ hours. The corpus is over 9^^^^3query3^^^^% Mandarin pop. The duration predictor is separately fine-tuned on about ^^^^3ti:\3query3query3^^^^,^^^^3query3query3query3^^^^ LRC-format lyrics (&&&^^^^3query3^^^^&&&).

Training is described as a three-stage procedure: diffusion-model pretraining for ^^^^3 OR abs:\3query3^^^^ epochs with batch size 3^^^^3 OR abs:\3^^^^ and learning rate PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^5, supervised fine-tuning for 8 epochs under the same batch size and learning rate, and preference alignment using iterative DPO. The ^^^^3 OR abs:\3query3 OR abs:\3^^^^5 record states that the main text refers to 3 rounds of DPO, that one table caption mentions “^^^^3 OR abs:\3^^^^ iterations” for SegTune-DPO, and that the appendix clarifies results for DPO-^^^^3ti:\3^^^^, DPO-^^^^3 OR abs:\3^^^^, and DPO-3; it therefore interprets the paper as having conducted three rounds overall, with the main comparison table likely highlighting a chosen checkpoint, probably DPO-^^^^3 OR abs:\3^^^^. By contrast, the later arXiv record summarizes the preference-alignment stage as ^^^^3 OR abs:\3^^^^ rounds of DPO, 4 epochs each (&&&^^^^3query3^^^^&&&, &&&^^^^3ti:\3^^^^&&&). This discrepancy is part of the publication record. In the more detailed description, each DPO round uses ^^^^3ti:\3^^^^6 generated songs per lyric sample, win–loss selection based on SongEval score differences and a third-quartile threshold, around ^^^^3 OR abs:\3query3^^^^,^^^^3query3query3query3^^^^ win–loss pairs, batch size 8, gradient accumulation 4, and learning rate PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^6 (&&&^^^^3query3^^^^&&&).

5. Evaluation protocol and empirical findings

SegTune is evaluated against YuE, LeVo, DiffRhythm+, and ACE-Step. YuE and LeVo are autoregressive, whereas DiffRhythm+ and ACE-Step are diffusion-based. All baselines support lyrics and global textual tags, but not the same segmental prompt mechanism. The main generated-song evaluation uses ^^^^3ti:\3^^^^5 Mandarin pop lyrics generated by ChatGPT; the later version specifies ^^^^3ti:\3query3^^^^ unique samples per prompt, giving ^^^^3ti:\3^^^^5^^^^3query3^^^^ generated tracks per system, and reports human evaluation on 9 songs rated by 5 listeners (&&&^^^^3query3^^^^&&&, &&&^^^^3ti:\3^^^^&&&).

The metric suite is deliberately broader than generic music-quality scoring. Audiobox-aesthetic provides production quality, production complexity, content enjoyment, and content usefulness. SongEval provides coherence, memorability, naturalness of vocal breathing and phrasing, clarity of song structure, and overall musicality. For controllability, the paper uses global MuLan for full-song prompt alignment and segment MuLan for prompt alignment averaged over aligned song segments. Because MuLan does not capture singer-related attributes well, the evaluation also includes vocal-attribute consistency tests: gender control is measured by modifying prompts and using Qwen3-Omni-3^^^^3query3^^^^B-A3B-Captioner to classify singer gender; age control is measured with A/B comparisons over prompts specifying ages such as teenager, ^^^^3 OR abs:\3query3^^^^s, or 4^^^^3query3^^^^s (&&&^^^^3query3^^^^&&&).

The main quantitative picture is twofold. First, SegTune-DPO is competitive or superior on music-quality metrics. In the baseline comparison table, SegTune-DPO outperforms the baselines on most SongEval metrics, reaching coherence PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^7, memorability PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^8, NVBP PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^9, CSS $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$^^^^3query3^^^^, and overall musicality $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$^^^^3ti:\3^^^^, compared with DiffRhythm+ at $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$^^^^3 OR abs:\3^^^^ and ACE-Step at $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$3. On Audiobox-aesthetic it also scores strongly, with $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$4, $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$5, and $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$6 (&&&^^^^3query3^^^^&&&). Second, the later version adds lyric-fidelity and subjective results: SegTune-SFT attains the best PER at ^^^^3ti:\3^^^^4.5%, while SegTune-DPO reaches a musicality MOS of $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$7, significantly higher than all baselines by Wilcoxon signed-rank test with $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$8; its quality MOS is $$E_g \leftarrow repeat(\mathbf{e}_g, T) \in \mathbb{R}^{T \times d_g}.$$9, statistically tied with LeVo in quality (&&&^^^^3ti:\3^^^^&&&).

Controllability results show both the gains and the trade-offs of the method. SegTune-SFT reaches global MuLan ^^^^3query3^^^^.47, segment-level improvements over global-only baselines, and the highest reported gender control accuracy at 96.67%, with age accuracy 57%. SegTune-DPO retains high global MuLan at ^^^^3query3^^^^.46, but gender control drops to 8^^^^3query3^^^^.95%. The authors attribute this decline to preference construction based on SongEval, which does not explicitly preserve instruction-following constraints and may bias the model toward preferred vocal types, particularly female vocals (&&&^^^^3query3^^^^&&&). This trade-off is reinforced by the DPO-round analysis: from SFT to DPO-^^^^3ti:\3^^^^ to DPO-^^^^3 OR abs:\3^^^^ to DPO-3, SongEval coherence rises from 3.54 to 4.^^^^3query3query3^^^^ to 4.^^^^3 OR abs:\3^^^^5 to 4.4^^^^3ti:\3^^^^, while gender accuracy falls from 96.7% to 79.^^^^3ti:\3^^^^%, 8^^^^3ti:\3^^^^.^^^^3query3 and 74.3%; segment MuLan remains relatively stable around ^^^^3query3^^^^.36–^^^^3query3 (&&&^^^^3query3^^^^&&&).

The ablations isolate the hierarchical conditioning design. In a global-only setting, Qwen3-Embedding as global encoder outperforms MuQ-Mulan in both music quality and control: global-only Qwen3 achieves Global MuLan ^^^^3query3^^^^.4^^^^3query3ti:\3 Segment MuLan ^^^^3query3^^^^.3^^^^3 OR abs:\3^^^^8, and gender accuracy 9^^^^3 OR abs:\3^^^^.^^^^3 OR abs:\3^^^^%, whereas global-only MuQ reaches ^^^^3query3^^^^.389, ^^^^3query3^^^^.3^^^^3query3query3 and 47.6%. In the full concatenate setting with Qwen3 as both global and segment encoder, performance rises to Global MuLan ^^^^3query3^^^^.465, Segment MuLan ^^^^3query3^^^^.378, gender 96.7%, and age 57%. The mixed strategy that linearly blends global and segment embeddings performs worse than concatenation. The authors interpret this as evidence that keeping prompt channels structurally distinct is beneficial for both controllability and musicality (&&&^^^^3query3^^^^&&&).

6. Limitations, misconceptions, and broader significance

Several limitations are explicit. SegTune is trained primarily on Mandarin pop, so its demonstrated domain coverage is narrower than that of some general-purpose baselines. The authors also identify richer local-control scenarios—especially duet singing and transitions between multiple singers—as difficult because of data scarcity. The later record adds that the model assumes clear song structure and does not support finer intra-segment control such as gradual crescendo, continuously changing arrangement density, or detailed ornamentation within a segment; the conditioning mechanism uses hard segment assignments and does not describe overlap handling or transition smoothing (&&&^^^^3query3^^^^&&&, &&&^^^^3ti:\3^^^^&&&).

A common misconception would be to treat SegTune as merely a lyric-conditioned diffusion model with more tags. The method is more specific than that. Its distinctive move is the explicit temporal alignment of global and local textual instructions with the latent audio time axis, using predicted lyric timestamps as the shared temporal scaffold. Another misconception would be to view the duration predictor as an ancillary preprocessing step. In SegTune it determines total song length, lyric alignment, and the time windows into which local prompts are broadcast, so it is part of the control interface rather than a detachable utility (&&&^^^^3query3^^^^&&&).

The most consequential internal tension in the work concerns preference optimization. DPO improves perceptual and structural quality, but it can reduce control fidelity, especially for singer demographic attributes. This is not presented as a contradiction of the framework itself; rather, it indicates that preference-learning objectives based on generic quality raters do not automatically preserve all user-specified attributes. A plausible implication is that future controllability-aware preference objectives would need to include explicit instruction-following terms, not only perceptual preference scores (&&&^^^^3query3^^^^&&&).

In the broader history of song generation, SegTune can be understood as a structured conditioning framework that moves from one-prompt song generation toward section-aware generation. Its central idea is technically simple but consequential: represent control at two timescales—global and segmental—align segmental descriptions to actual temporal windows through predicted lyric timestamps, and inject both forms of conditioning directly into a latent diffusion or flow-based generator. The reported results suggest that this improves not only controllability in the narrow sense of prompt following, but also musical coherence, likely because the generator receives a temporally organized plan rather than a single undifferentiated textual description (&&&^^^^3query3^^^^&&&).