Qwen-Music: Full Song Generation with Vocals

• Qwen-Music is a large-scale music generation system that produces full songs with complete vocals by separating semantic composition from acoustic rendering.
• It employs a three-part architecture—Tokenizer, LLM with Melody-CoT, and high-fidelity Render—to enable precise control over melody, lyrics, and arrangement.
• The system achieves strong performance in melody preservation and musicality while addressing challenges in long-range structure and nuanced expressive control.

Qwen-Music is a large-scale music generation system introduced as a unified model for producing full songs with complete vocal singing. It supports two core tasks: Text to Music Generation, which creates new songs from text descriptions, lyrics, and musical attributes, and Cover Song Generation, which reinterprets existing songs while preserving reference melody and altering style, arrangement, and vocal characteristics. The report frames its central design decision as a separation between semantic composition and acoustic rendering, implemented through three modules—Qwen-Music-Tokenizer, Qwen-Music-LLM, and Qwen-Music-Render—and trained on more than 5 million hours of multilingual music data covering hundreds of languages (Xu et al., 13 Jul 2026).

1. Scope and position within the Qwen music line

Qwen-Music is a music generation system rather than an audio understanding, captioning, or question-answering model. Earlier Qwen-family work established explicit music scope inside a broader universal audio understanding framework: Qwen-Audio covered music and songs alongside speech and natural sounds, with music-related tasks including music caption, music instruments classification, music note analysis such as pitch, velocity, music genre recognition, music recognition, and music question answering, but it remained an audio-to-text model rather than a music generator (Chu et al., 2023). A separate line of work, Jamendo-QA, used Qwen-Omni to generate captions and question-answer pairs from freely licensed Jamendo tracks, thereby contributing benchmark infrastructure rather than a dedicated music model (Koh et al., 19 Sep 2025).

Within that broader trajectory, Qwen-Music is the first entry in the provided record that is explicitly organized around full-song generation with vocals, long-horizon planning, and high-fidelity waveform synthesis (Xu et al., 13 Jul 2026). Its problem formulation differs not only from music QA and captioning but also from co-creative assistance systems such as MixAssist, which fine-tuned Qwen-Audio-Instruct-7B to generate contextually relevant mixing advice instead of songs (Clemens et al., 8 Jul 2025). It also differs from evaluator architectures such as WhisQ, where Qwen 3 0.6B functions as a frozen text encoder for text-to-music MOS prediction rather than generation (Emon et al., 6 Jun 2025).

This positioning is significant because the Qwen music-related literature in the provided corpus spans several distinct tasks: audio understanding, dataset construction, evaluation, music-production dialogue, and full generative synthesis. Qwen-Music occupies the last category, and the report repeatedly treats its novelty as a decomposition of song generation into semantic-token planning followed by high-fidelity rendering (Xu et al., 13 Jul 2026).

2. Three-part architecture and end-to-end pipeline

The system is organized into three principal components.

Component	Primary role	Key specifications
Qwen-Music-Tokenizer	Compresses audio into Music Semantic Tokens	25 Hz, single codebook, size 32768
Qwen-Music-LLM	Autoregressive semantic music modeling	initialized from a 3B dense variant of Qwen3.5-Omni
Qwen-Music-Render	Generates final waveform from semantic plan	48 kHz stereo, semantic-conditioned DiT + Spec-VAE + Band-Mode Refiner

The end-to-end workflow begins with a user request that may contain a natural-language prompt, lyrics, style attributes, singer properties, or a reference song. A prompt rewriter converts this request into a structured textual condition containing musical tags, singer characteristics, instrumental arrangement, and lyrics. Qwen-Music-LLM then generates a sequence of Music Semantic Tokens, optionally conditioned on melody tokens extracted from a reference song or on an internally generated melody plan. Qwen-Music-Render consumes both the semantic-token sequence and the rewritten textual condition and synthesizes the final 48 kHz stereo waveform (Xu et al., 13 Jul 2026).

The report states that the tokenizer emits one token every 40 ms, so the semantic stream operates at 25 Hz. With a codebook size of $2^{15}=32768$, the resulting semantic bitrate is 375 bit/s. The tokenizer’s full-song supervised fine-tuning supports songs up to 300 s, corresponding to roughly 7.5k latent frames at 25 Hz, while renderer training supports tracks up to 6 minutes, or 9000 frames at the same rate (Xu et al., 13 Jul 2026).

A plausible implication is that Qwen-Music treats long-form composition and waveform realization as different optimization problems. The report’s architectural split is expressly motivated by the claim that long-horizon song planning and sample-level synthesis have mismatched requirements, and the three-module design is presented as the practical answer to that mismatch (Xu et al., 13 Jul 2026).

3. Qwen-Music-Tokenizer and Music Semantic Tokens

Qwen-Music-Tokenizer converts mono 24 kHz music audio into a compact stream of Music Semantic Tokens intended to preserve lyrical content, melody, and harmony or spectral semantics while discarding the acoustic detail later reconstructed by the renderer. The frontend is a lightweight convolutional stack that reduces frame rate to 25 Hz, followed by a 24-layer Conformer with 0.6B parameters, hidden width 1024, and 16 attention heads. From Stage 2 onward, attention is made causal (Xu et al., 13 Jul 2026).

The tokenizer is trained in four stages.

Stage 1: bidirectional BestRQ pretraining. BestRQ uses masked prediction with frozen random quantizer targets. The waveform is partitioned into 0.4 s spans, each masked independently with probability 0.3, with at least one span masked per sample and replaced by low-variance Gaussian noise. The objective is written as

$$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$

where $\mathcal{M}$ denotes masked frames, $h_t$ the encoder output, $q(\cdot)$ the frozen random-projection quantizer, and $p(\cdot)$ the prediction head (Xu et al., 13 Jul 2026).

Stage 2: causal adaptation. The Stage 1 bidirectional encoder is converted into a causal encoder by changing attention masking while keeping the same BestRQ objective. The report describes this as a cheaper alternative to causal pretraining from scratch (Xu et al., 13 Jul 2026).

Stage 3: multi-task supervised fine-tuning. Three heads are added to the 25 Hz encoder outputs: a multilingual CTC lyric transcription head, a ConvNeXt Mel head reconstructing a 100 Hz, 128-bin Mel spectrogram, and a ConvNeXt chroma head predicting 12-bin chroma. The loss is

$$\mathcal{L}_{\text{SFT}} = \lambda_{\text{ctc}} \mathcal{L}_{\text{CTC}} + \lambda_{\text{mel}} \mathcal{L}_{\text{spec}^{\text{mel}}} + \lambda_{\text{chr}} \mathcal{L}_{\text{spec}^{\text{chroma}}}, \qquad \lambda_{\text{ctc}}=\lambda_{\text{mel}}=\lambda_{\text{chr}}=1.$$

At this stage, input masking is almost disabled with $p_{\text{mask}}=0.01$, and training uses full songs up to 300 s (Xu et al., 13 Jul 2026).

Stage 4: vector quantization. A single VQ bottleneck is inserted at an intermediate layer, using one codebook of size 32768, cosine distance, a straight-through estimator, and commitment loss. The report states 99%+ codebook utilization and that dead codes are “virtually eliminated.” To stabilize insertion, the system uses

$h_{\text{out}} = h + \alpha (h_q - h),$

with $\alpha$ ramped from 0 to 1 during warmup. Early in this stage, the frontend and lower layers up to the VQ insertion point are frozen, while the quantizer, upper layers, and heads are trained first; the whole model is unfrozen afterward. The CTC weight is lowered to 0.5, while Mel, chroma, and VQ weights remain 1.0 (Xu et al., 13 Jul 2026).

At inference, only the frontend, Conformer layers up to the VQ insertion point, and the quantizer are kept. Each 40 ms frame becomes one integer in $$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$0. This yields the semantic stream consumed by Qwen-Music-LLM (Xu et al., 13 Jul 2026).

4. Qwen-Music-LLM and the Melody-CoT mechanism

Qwen-Music-LLM is the semantic composition model. It is initialized from a 3B dense variant of Qwen3.5-Omni and trained autoregressively over sequences that combine text tokens, optional melody tokens, and final Music Semantic Tokens (Xu et al., 13 Jul 2026). The report describes several sequence patterns, including plain text-to-music

$$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$1

and melody-planning variants

$$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$2

and

$$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$3

Loss is applied both to the Melody-CoT region and to the final Music Semantic Token region (Xu et al., 13 Jul 2026).

The report’s main novelty at this stage is Melody-CoT, a melody-token-based chain-of-thought mechanism that plans melody before full-song generation. Rather than directly mapping text to full-mixture song tokens, the model first produces or conditions on a coarse vocal melody plan and then generates the full semantic-token sequence. The report states that this improves creativity, musicality, structural coherence, and reference-audio-based melody cloning (Xu et al., 13 Jul 2026).

For cover generation, reference audio is not passed to the LLM as raw waveform. Instead, vocal pitch is extracted and converted into melody tokens. The procedure begins with a 50 Hz vocal pitch curve from RMVPE, then downsamples it by 8× median pooling to obtain a 6.25 Hz melody sequence: $$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$4 The sequence is converted from Hz to MIDI, centered by subtracting the median voiced MIDI pitch, clipped to $$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$5, and mapped to a 256-entry vocabulary with one token reserved for unvoiced frames: $$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$6

$$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$7

The report emphasizes that this preserves melody shape while discarding absolute key, absolute pitch range, and singer-dependent timbral information (Xu et al., 13 Jul 2026).

Two Melody-CoT variants are used. In section-level melody, only lyric-bearing vocal sections are represented, each prefixed by labels such as [verse] or [chorus], while non-vocal sections are omitted. In unique-section-level melody, repeated sections are grouped by label and one representative segment is sampled for each label during training. The report presents these as a trade-off: section-level conditioning produces stronger direct melody cloning, whereas unique-section-level conditioning improves tag following and can reduce over-conditioning (Xu et al., 13 Jul 2026).

The system’s conditioning context includes global music tags, structured lyrics, singer attributes, and optional melody tokens from a reference song. A plausible implication is that Qwen-Music-LLM is optimized less as a raw waveform model than as a controllable planner operating in a compressed semantic space. That interpretation is consistent with the report’s repeated emphasis on long-horizon composition, structure, and melody control (Xu et al., 13 Jul 2026).

5. Qwen-Music-Render and high-fidelity stereo synthesis

Qwen-Music-Render is the acoustic synthesis stage. The report states that discrete semantic tokens are sufficient for global structure, section ordering, and melody and arrangement planning, but insufficient for timbral richness, phase detail, stereo placement, and high-frequency detail. The renderer addresses these deficits with a three-stage pipeline: a semantic-conditioned Diffusion Transformer (DiT), a Spec-VAE, and a Band-Mode Refiner, producing the final 48 kHz stereo waveform (Xu et al., 13 Jul 2026).

The DiT is a 1.3B parameter diffusion transformer with 32 Transformer blocks, hidden dimension 1024, 24 attention heads, FFN expansion ratio 8, RoPE on queries and keys, self-attention, cross-attention to condition context, feed-forward layers, and AdaLN modulation from the diffusion timestep. It predicts continuous acoustic latents $$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$8 with $$\mathcal{L}_{\mathrm{BestRQ}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),$$9, conditioned on frame-aligned Music Semantic Tokens at 25 Hz and on rewritten textual context (Xu et al., 13 Jul 2026).

The text branch uses a frozen Qwen3-Embedding-0.6B encoder for song-description tokens, while lyric tokens are processed by a jointly trained 6-layer RoPE Transformer encoder. Description and lyric features are concatenated into the cross-attention context $\mathcal{M}$0. For classifier-free guidance, the report distinguishes a learnable null text context $\mathcal{M}$1, and emphasizes that the preferred strategy retains semantic tokens while dropping only the text branch—described as Text-drop CFG—because semantic tokens are the primary scaffold (Xu et al., 13 Jul 2026).

The Spec-VAE is based on a SpectroStream-style 2D convolutional architecture with continuous latents. Stereo 48 kHz audio is transformed to a complex STFT

$\mathcal{M}$2

encoded into 128-dimensional latents at 25 Hz, and decoded back to a coarse complex spectrogram. The total compression ratio is 192×. Stereo is handled through delayed fusion in the encoder and early splitting in the decoder (Xu et al., 13 Jul 2026).

The decoder uses Spec-SnakeBeta, a frequency-aware activation defined as

$\mathcal{M}$3

with initialization

$\mathcal{M}$4

The stated effect is stronger periodic modulation at higher frequencies (Xu et al., 13 Jul 2026).

The final Band-Mode Refiner is a lightweight ConvNeXt-1D module that applies different correction modes by frequency band: phase-only in the low band, magnitude + phase in the mid band, and magnitude-only in the high band. All refiner weights are initialized to zero so that the refined output initially matches the decoder output exactly, which the report presents as a stabilization mechanism (Xu et al., 13 Jul 2026).

Renderer training proceeds in three stages: reconstruction-only Spec-VAE pretraining, then adding waveform-domain adversarial training, then freezing the Spec-VAE encoder-decoder and training the Band-Mode Refiner with waveform and spectral discriminators. The loss stack includes multi-resolution STFT loss, IF/GD phase loss, LSGAN adversarial loss, feature matching, and KL regularization, with additional techniques such as K-weighting, MSLR stereo decomposition, adaptive log-magnitude normalization, and mixed-scale spectral losses (Xu et al., 13 Jul 2026).

6. Training regime, evaluation, and empirical results

The report states that Qwen-Music-LLM is trained on more than 5 million hours of multilingual music data spanning hundreds of languages. The training corpus includes 20% instrumental music during Stage 1 LLM training to strengthen accompaniment modeling. Data are quality-ranked by an internal MOS-based reward model trained from professional human ratings; within each genre, samples are partitioned into Q1 through Q7 using percentile thresholds at 90%, 75%, 50%, 25%, 5%, and 1%, and Q7 is removed (Xu et al., 13 Jul 2026).

Pretraining is organized as a quality-aware curriculum. Stage 1 trains on Q3–Q6 data with dynamic quality sampling, language balancing, and the instrumental fraction. Stage 2 anneals on Q2 data with learning-rate decay. Stage 3 refines on Q1 plus carefully selected high-quality samples from the full corpus, while preserving balanced genre and language distributions (Xu et al., 13 Jul 2026).

Post-training is then described as progressive: a supervised cold start on carefully curated data with reliable control annotations, iterative offline preference alignment using DPO, and final on-policy optimization with GSPO. Musicality rewards come from the internal MOS predictor, while instruction-following rewards for genre control and lyric following are computed by Qwen3.5-Omni and Qwen3-ASR (Xu et al., 13 Jul 2026).

The main text-to-music benchmark uses 600 prompts total, evenly divided into 300 Chinese and 300 English, with AI-generated lyrics and musical tags and balanced genre coverage. The report defines 16 objective musicality and audio-quality metrics as the union of SongBench (7), SongEval (5), and AudioBox-Aesthetic (4). Across these 16, Qwen-Music is reported to achieve the best score on 13: 6/7 SongBench dimensions, 5/5 SongEval dimensions, and 2/4 AudioBox-Aesthetic dimensions (Xu et al., 13 Jul 2026).

On SongBench, Qwen-Music is best on Melody 7.03, Arrangement 7.35, Musicality 6.22, Vocal 7.44, Instrumental 7.24, and Mixing 7.13, but not on Structure 6.94, where Mureka V8 reaches 7.02. On SongEval, it is best on all five metrics: Coherence 4.55, Musicality 4.42, Memorability 4.51 (tied numerically with Mureka V8), Clarity 4.45, and Naturalness 4.37 (also tied numerically with Mureka V8). On AudioBox-Aesthetic, it is best on Content Enjoyment 7.47 and Content Usefulness 7.86, but not on Production Complexity 6.64 or Production Quality 8.15 (Xu et al., 13 Jul 2026).

The report also evaluates tag following across five categories—Genre, Moods, Instruments, Vocal Gender, and Vocal Timbre—using Gemini 3.1 Pro, and reports an average score of 8.44, within 0.04 of the best system average. The per-category scores are Genre 8.08, Moods 8.94, Instruments 8.65, Vocal Gender 7.85 (best), and Vocal Timbre 8.68 (second). For lyric intelligibility, Qwen-Music attains PER 6.10, second only to Suno V5.5 at 4.19 (Xu et al., 13 Jul 2026).

Human evaluation uses 50 professional raters, each with music creation or production experience, in blind A/B comparisons with anonymized randomized presentation, with each pair judged by 3 experts. Qwen-Music’s preference win rates are 59.1% versus MiniMax Music 2.5+, 66.7% versus MiniMax Music 2.6, 58.3% versus Mureka V8, 55.4% versus Suno V5, and 50.3% versus Suno V5.5. The report also states that Qwen-Music appeared as JazzCat on the Artificial Analysis Music with Vocals Leaderboard and ranked third among leading English vocal music generation systems (Xu et al., 13 Jul 2026).

For cover song generation, the report evaluates two reference sets: an AI-generated reference set with 100 English + 100 Chinese items, and a real-world popular-song reference set also with 100 English + 100 Chinese items. Melody preservation is measured by Melody MAE, computed by converting both generated cover and reference song to relative MIDI melody representations, aligning them with dynamic time warping, and computing mean absolute semitone error. On the AI-generated reference set, Qwen-Music (section-level) reaches Melody MAE 1.48, outperforming Qwen-Music (unique-section-level) 1.80, Suno V5.5 2.00, Suno V5 1.87, and MiniMax Cover 1.89. On the real-world popular-song reference set, the section-level model again gives the lowest Melody MAE at 1.44, while the unique-section-level model scores 1.80 and MiniMax Cover scores 1.76 (Xu et al., 13 Jul 2026).

These results establish a recurring trade-off. Section-level conditioning yields tighter melody preservation, whereas unique-section-level conditioning improves tag following and intelligibility in many cases. The report treats this as evidence that melody conditioning can be too strong, and that partial abstraction over repeated sections improves style controllability (Xu et al., 13 Jul 2026).

7. Limitations, interpretation, and broader research context

The report explicitly points to future work on more flexible long-context structure modeling, finer-grained expressive control over singing and performance, and more efficient rendering architectures. These priorities imply current limitations in long-range structure flexibility, expressive nuance control, and computational efficiency. The report does not provide a dedicated failure-case section, and it does not describe concrete safeguards for voice imitation, style imitation, copyright issues, watermarking, or release restrictions, even though these concerns are plainly relevant to a system capable of cover-song generation and vocal-characteristic control (Xu et al., 13 Jul 2026).

Qwen-Music should also be distinguished from broader claims about the Qwen family’s music perception ability. In the MUSE benchmark, which probes music perception and auditory relational reasoning, Qwen2.5-Omni performed at or near chance on several tasks requiring invariant musical representations, including Melody Shape Identification, Pitch Shift Detection, Syncopation Comparison, and Meter Identification. The benchmark authors interpreted this as evidence of severe weaknesses in relative pitch, contour, and temporal abstraction (Carone et al., 21 Oct 2025). This does not directly invalidate Qwen-Music’s generative results, but it shows that strong music generation and strong audio-language relational reasoning are not interchangeable capabilities.

A plausible implication is that the Qwen music line has evolved through partly disjoint research threads: universal audio understanding in Qwen-Audio (Chu et al., 2023), music-audio annotation and QA dataset construction in Jamendo-QA (Koh et al., 19 Sep 2025), music-mixing dialogue adaptation in MixAssist (Clemens et al., 8 Jul 2025), text-to-music evaluation in WhisQ (Emon et al., 6 Jun 2025), general audio latent compression for generation in Qwen-Audio-VAE (Jiang et al., 13 Jul 2026), and finally a dedicated full-song generative system in Qwen-Music (Xu et al., 13 Jul 2026). Qwen-Audio-VAE is especially relevant in this regard because it introduced a general audio generation backbone trained on 5 million hours and evaluated on music reconstruction, but it remained a general audio latent model rather than the semantic-token-plus-renderer design used by Qwen-Music (Jiang et al., 13 Jul 2026).

Taken together, the available record supports a precise characterization. Qwen-Music is a specialized semantic-composition plus acoustic-rendering system for multilingual song generation with vocals. Its reported strengths lie in melody, arrangement, musicality, controllability, cover-song melody preservation, and stereo rendering quality. Its reported weaknesses or open problems lie in structure, production-complexity leadership, cover-song intelligibility relative to text-to-music, safety under-specification, and the broader question of whether Qwen-family systems possess equally strong perceptual music understanding outside the specific generative pipeline (Xu et al., 13 Jul 2026).