SegTune: Structured and Fine-Grained Control for Song Generation

Abstract: Recent advances in neural song generation have enabled high-quality synthesis from lyrics and global textual prompts. However, most systems fail to model temporally varying attributes of songs, severely limiting fine-grained control over musical structure and dynamics. To address this, we propose SegTune, a Diffusion Transformer-based framework enabling structured and fine-grained controllability by allowing users or LLMs to specify local musical descriptions aligned to song segments. These segment prompts are temporally broadcast to corresponding time windows, while global prompts ensure stylistic coherence. To support precise lyric-to-music alignment, we introduce an LLM-based duration predictor that autoregressively generates sentence-level timestamps in LyRiCs format. We further construct a large-scale data pipeline for high-quality song collection with aligned lyrics and prompts, and propose new metrics to evaluate segment alignment and vocal consistency. Experiments demonstrate that SegTune outperforms existing baselines in both musicality and controllability. Visit our project page (https://github.com/KlingAIResearch/SegTune) for codes and more generated songs.

• The paper introduces a novel hierarchical conditioning scheme separating global and segment prompts to overcome limitations of AR and NAR systems.
• It details a diffusion transformer architecture that utilizes a fine-tuned Qwen3-4B duration predictor for accurate lyric-to-music alignment.
• Experimental results show significant improvements in musicality, prompt adherence, and control consistency compared to existing methods.

SegTune: Structured and Fine-Grained Control for Song Generation

Introduction and Motivation

Song generation remains a complex task at the intersection of music and language, requiring models to produce coherent vocals and accompaniment conditioned on lyrics and high-level prompts. While prior autoregressive (AR) and non-autoregressive (NAR) architectures have made substantive progress, both suffer critical limitations regarding the inability to control temporally varying musical attributes at a fine resolution. AR approaches such as SongCreator and MusiCoT exhibit slow inference and expressive bottlenecks due to shared token vocabularies, while NAR models, despite improved efficiency and audio fidelity, compress all composition and rendering tasks into a single latent diffusion process, thereby limiting granularity in temporal and structural control.

SegTune directly addresses these restrictions by introducing a hierarchical control scheme supporting segment-level prompting and precise lyric-to-music alignment, thus providing a platform for both structured and fine-grained control in song synthesis.

SegTune Architecture and Methodology

SegTune is implemented on top of a Diffusion Transformer (DiT) backbone operating in a latent audio space at 44 kHz, leveraging both global and segment-level textual signals to condition the entire generative trajectory. The model comprises several key components: a lyrics encoder for semantic and phoneme-level lyric embedding, a hierarchical prompt encoder for independently processing global and segment prompts, and an LLM-based duration predictor to estimate sentence-level lyric timestamps in LRC format. This enables accurate temporal partitioning and supports frame-aligned prompt broadcasting at either the segment or global level. All conditional embeddings are fused channel-wise and serve as context vectors for the DiT, facilitating scalable and temporally sensitive generative control.

Figure 1: Overview of the SegTune architecture enabling broadcast of segment and global control signals to dedicated temporal windows.

SegTune's hierarchical conditioning mechanism is distinctive. Global prompts modulate song-level attributes (genre, style, vocalist characteristics), while segment prompts—inserted at temporally determined intervals—define local, time-varying features such as instrumentation, emotion, and rhythm. This separation avoids the semantic conflation observed in prior work and provides explicit, fine-grained guidance to the generation pipeline.

The duration predictor itself is instantiated as a fine-tuned Qwen3-4B LLM, trained to predict sentence/segment start times based on input lyrics and prompt context. This eliminates reliance on error-prone manual annotation or zero-shot LLM estimations, and it acts as the primary mechanism for aligning hierarchical text input to temporal audio structures during both training and inference.

Data Pipeline and Prompt Engineering

High-quality and robust alignment between lyric, prompt, and audio segments is realized through an industrial-scale data pipeline. Initial filtering proceeds via metadata and aesthetic scoring (e.g., Audiobox Aesthetics, SongEval) to curate a musically plausible corpus. Lyrics are processed and aligned using ASR tools (FireRedASR, Whisper), and LRC validation steps ensure timestamp reliability. Hierarchical prompts are constructed via Audio Flamingo 3, producing dense, context-sensitive descriptions for both global and segment-level conditioning. Segment labels (intro, verse, chorus, etc.) are extracted for precise musical structure control.

Figure 3: Overview of the data pipeline of SegTune, illustrating filtering, lyrics processing, and hierarchical caption/segment alignment.

Experimental Results and Analysis

SegTune is rigorously benchmarked against current AR and NAR baselines (YuE, LeVo, DiffRhythm+, ACE-Step) using a suite of objective and subjective metrics. Objective metrics include Phoneme Error Rate (PER), Audiobox-Aesthetics (PQ, PC, CE, CU), SongEval (coherence, memorability, vocal phrasing, song structure, overall musicality), and instruction-following via Muq-MuLan scores (global/segment alignment), alongside vocalist attribute (gender, age) control. Subjective quality is assessed by 5-point MOS listening tests, with results demonstrating statistically significant gains for SegTune, especially in musicality, vocalist intelligibility, and prompt adherence.

SegTune-SFT (supervised fine-tuning) achieves the lowest PER and highest gender control accuracy, while SegTune-DPO (direct preference optimization) further improves musicality and production metrics, surpassing all baselines in 8 out of 9 objective measures. Notably, SegTune exhibits superior consistency across samples, evidenced by the smallest standard deviations in MOS ratings.

Figure 2: Violin plots of MOS results for musicality and quality evaluation across competing models.

Ablation studies confirm that hierarchical prompt concatenation using Qwen3-Embedding for both global and segment prompts maximizes both musicality and attribute control. Qwen3-Embedding provides much finer separation in singer gender embeddings compared to Muq-MuLan, illustrated by t-SNE visualizations demonstrating clear clustering by gender.

Figure 4: t-SNE visualization of Muq-MuLan embeddings on singer gender control—showing overlapping gender representations.

Figure 5: t-SNE visualization of Qwen3-Embedding on singer gender control—showing clear separation between female and male clusters.

Duration prediction ablations reveal that the Qwen3-based duration predictor achieves near-ground-truth alignment (MAE = 0.99s), outperforming zero-shot GPT-4o-based approaches on all timing and generation quality metrics.

Implications, Limitations, and Speculative Future Directions

SegTune demonstrates the value and technical feasibility of decoupling global and segment-level control for song generation, significantly advancing the controllability, fidelity, and alignment of generated music to structured, user- or LLM-supplied textual prompts. The segment prompt paradigm enables dynamic emotional and instrumental transitions, and the LLM-based duration predictor streamlines user experience by obviating the need for laborious manual timestamping.

Practically, SegTune enables a new spectrum of interactive and personalized music creation tools, supporting both professional and amateur workflows. Theoretically, it sets a template for future NAR generative frameworks where complex temporal dynamics can be effectively managed through hierarchical conditional architectures.

Nevertheless, limitations remain. The model's reliance on well-structured segment prompts means ambiguous user input can degrade output quality; further research on robust prompt enhancement and template induction is needed. Intra-segment dynamism (e.g., gradual crescendos, elaborate ornamentation) is not explicitly modeled by the current segment-wise framework. Multi-objective alignment strategies—that jointly optimize for musicality and fine-grained attribute adherence—are also identified as a direction for future work, especially to address the bias propagation observed during preference-based fine-tuning.

Conclusion

SegTune substantially advances the state of conditional song generation by introducing structured, segment-level textual control and reliable lyric-to-music alignment within a scalable Diffusion Transformer system. The empirical gains over established baselines are robust across both objective and subjective metrics, with validated improvements in musical quality, fidelity, and controllability. The approach is extendable to multi-lingual and multi-domain music generation given adequate training data and interface engineering. Future research should address automatic segment structure inference, intra-segment dynamism, and integration with conversational assistants for fully interactive song generation workflows.