---
title: 'Qwen-Music: Full Song Generation with Vocals'
url: https://www.emergentmind.com/topics/qwen-music
type: topic
---

# Qwen-Music: Full Song Generation with Vocals

Qwen-Music is a large-scale music generation system introduced as a unified model for producing full songs with complete vocal singing. It supports two core tasks: **Text to Music Generation**, which creates new songs from text descriptions, lyrics, and musical attributes, and **Cover Song Generation**, which reinterprets existing songs while preserving reference melody and altering style, arrangement, and vocal characteristics. The report frames its central design decision as a separation between **semantic composition** and **acoustic rendering**, implemented through three modules—**Qwen-Music-Tokenizer**, **Qwen-Music-LLM**, and **Qwen-Music-Render**—and trained on more than **5 million hours** of multilingual music data covering **hundreds of languages** [2607.11699].

## 1. Scope and position within the Qwen music line

Qwen-Music is a **music generation** system rather than an audio understanding, captioning, or question-answering model. Earlier Qwen-family work established explicit music scope inside a broader **universal audio understanding** framework: Qwen-Audio covered music and songs alongside speech and natural sounds, with music-related tasks including **music caption**, **music instruments classification**, **music note analysis such as pitch, velocity**, **music genre recognition**, **music recognition**, and **music question answering**, but it remained an audio-to-text model rather than a music generator [2311.07919]. A separate line of work, Jamendo-QA, used **Qwen-Omni** to generate captions and question-answer pairs from freely licensed Jamendo tracks, thereby contributing benchmark infrastructure rather than a dedicated music model [2509.15662].

Within that broader trajectory, Qwen-Music is the first entry in the provided record that is explicitly organized around **full-song generation with vocals**, long-horizon planning, and high-fidelity waveform synthesis [2607.11699]. Its problem formulation differs not only from music QA and captioning but also from co-creative assistance systems such as MixAssist, which fine-tuned **Qwen-Audio-Instruct-7B** to generate contextually relevant mixing advice instead of songs [2507.06329]. It also differs from evaluator architectures such as WhisQ, where **Qwen 3 0.6B** functions as a frozen text encoder for text-to-music MOS prediction rather than generation [2506.05899].

This positioning is significant because the Qwen music-related literature in the provided corpus spans several distinct tasks: **audio understanding**, **dataset construction**, **evaluation**, **music-production dialogue**, and **full generative synthesis**. Qwen-Music occupies the last category, and the report repeatedly treats its novelty as a decomposition of song generation into semantic-token planning followed by high-fidelity rendering [2607.11699].

## 2. Three-part architecture and end-to-end pipeline

The system is organized into three principal components.

| Component | Primary role | Key specifications |
|---|---|---|
| **Qwen-Music-Tokenizer** | Compresses audio into Music Semantic Tokens | **25 Hz**, **single codebook**, size **32768** |
| **Qwen-Music-LLM** | Autoregressive semantic music modeling | initialized from a **3B dense variant of Qwen3.5-Omni** |
| **Qwen-Music-Render** | Generates final waveform from semantic plan | **48 kHz stereo**, semantic-conditioned **DiT + Spec-VAE + Band-Mode Refiner** |

The end-to-end workflow begins with a user request that may contain a natural-language prompt, lyrics, style attributes, singer properties, or a reference song. A prompt rewriter converts this request into a **structured textual condition** containing musical tags, singer characteristics, instrumental arrangement, and lyrics. Qwen-Music-LLM then generates a sequence of **Music Semantic Tokens**, optionally conditioned on melody tokens extracted from a reference song or on an internally generated melody plan. Qwen-Music-Render consumes both the semantic-token sequence and the rewritten textual condition and synthesizes the final **48 kHz stereo waveform** [2607.11699].

The report states that the tokenizer emits one token every **40 ms**, so the semantic stream operates at **25 Hz**. With a codebook size of **\(2^{15}=32768\)**, the resulting semantic bitrate is **375 bit/s**. The tokenizer’s full-song supervised fine-tuning supports songs up to **300 s**, corresponding to roughly **7.5k latent frames** at 25 Hz, while renderer training supports tracks up to **6 minutes**, or **9000 frames** at the same rate [2607.11699].

A plausible implication is that Qwen-Music treats long-form composition and waveform realization as different optimization problems. The report’s architectural split is expressly motivated by the claim that long-horizon song planning and sample-level synthesis have mismatched requirements, and the three-module design is presented as the practical answer to that mismatch [2607.11699].

## 3. Qwen-Music-Tokenizer and Music Semantic Tokens

Qwen-Music-Tokenizer converts **mono 24 kHz** music audio into a compact stream of **Music Semantic Tokens** intended to preserve lyrical content, melody, and harmony or spectral semantics while discarding the acoustic detail later reconstructed by the renderer. The frontend is a lightweight convolutional stack that reduces frame rate to **25 Hz**, followed by a **24-layer Conformer** with **0.6B parameters**, hidden width **1024**, and **16 attention heads**. From Stage 2 onward, attention is made causal [2607.11699].

The tokenizer is trained in **four stages**.

**Stage 1: bidirectional BestRQ pretraining.** BestRQ uses masked prediction with frozen random quantizer targets. The waveform is partitioned into **0.4 s spans**, each masked independently with probability **0.3**, with at least one span masked per sample and replaced by low-variance Gaussian noise. The objective is written as
\[
\mathcal{L}_{\mathrm{BestRQ}}
=
\frac{1}{|\mathcal{M}|}
\sum_{t \in \mathcal{M}}
\operatorname{CE}\bigl(p(h_t), q(x_t)\bigr),
\]
where \(\mathcal{M}\) denotes masked frames, \(h_t\) the encoder output, \(q(\cdot)\) the frozen random-projection quantizer, and \(p(\cdot)\) the prediction head [2607.11699].

**Stage 2: causal adaptation.** The Stage 1 bidirectional encoder is converted into a causal encoder by changing attention masking while keeping the same BestRQ objective. The report describes this as a cheaper alternative to causal pretraining from scratch [2607.11699].

**Stage 3: multi-task supervised fine-tuning.** Three heads are added to the **25 Hz** encoder outputs: a multilingual **CTC lyric transcription head**, a **ConvNeXt Mel head** reconstructing a **100 Hz, 128-bin Mel spectrogram**, and a **ConvNeXt chroma head** predicting **12-bin chroma**. The loss is
\[
\mathcal{L}_{\text{SFT}}
=
\lambda_{\text{ctc}} \mathcal{L}_{\text{CTC}}
+
\lambda_{\text{mel}} \mathcal{L}_{\text{spec}^{\text{mel}}}
+
\lambda_{\text{chr}} \mathcal{L}_{\text{spec}^{\text{chroma}}},
\qquad
\lambda_{\text{ctc}}=\lambda_{\text{mel}}=\lambda_{\text{chr}}=1.
\]
At this stage, input masking is almost disabled with \(p_{\text{mask}}=0.01\), and training uses full songs up to **300 s** [2607.11699].

**Stage 4: vector quantization.** A **single VQ bottleneck** is inserted at an intermediate layer, using one codebook of size **32768**, cosine distance, a straight-through estimator, and commitment loss. The report states **99%+** codebook utilization and that dead codes are “virtually eliminated.” To stabilize insertion, the system uses
\[
h_{\text{out}} = h + \alpha (h_q - h),
\]
with \(\alpha\) ramped from 0 to 1 during warmup. Early in this stage, the frontend and lower layers up to the VQ insertion point are frozen, while the quantizer, upper layers, and heads are trained first; the whole model is unfrozen afterward. The CTC weight is lowered to **0.5**, while Mel, chroma, and VQ weights remain **1.0** [2607.11699].

At inference, only the frontend, Conformer layers up to the VQ insertion point, and the quantizer are kept. Each **40 ms** frame becomes one integer in \([0,32768)\). This yields the semantic stream consumed by Qwen-Music-LLM [2607.11699].

## 4. Qwen-Music-LLM and the Melody-CoT mechanism

Qwen-Music-LLM is the semantic composition model. It is initialized from a **3B dense variant of Qwen3.5-Omni** and trained autoregressively over sequences that combine text tokens, optional melody tokens, and final **Music Semantic Tokens** [2607.11699]. The report describes several sequence patterns, including plain text-to-music
\[
[\mathrm{text},\ \mathrm{music\ semantic\ tokens}],
\]
and melody-planning variants
\[
[\mathrm{text},\ \mathrm{section\text{-}level\ melody},\ \mathrm{music\ semantic\ tokens}]
\]
and
\[
[\mathrm{text},\ \mathrm{unique\text{-}section\text{-}level\ melody},\ \mathrm{music\ semantic\ tokens}].
\]
Loss is applied both to the Melody-CoT region and to the final Music Semantic Token region [2607.11699].

The report’s main novelty at this stage is **Melody-CoT**, a melody-token-based chain-of-thought mechanism that plans melody before full-song generation. Rather than directly mapping text to full-mixture song tokens, the model first produces or conditions on a **coarse vocal melody plan** and then generates the full semantic-token sequence. The report states that this improves **creativity**, **musicality**, **structural coherence**, and **reference-audio-based melody cloning** [2607.11699].

For cover generation, reference audio is not passed to the LLM as raw waveform. Instead, vocal pitch is extracted and converted into melody tokens. The procedure begins with a **50 Hz** vocal pitch curve from RMVPE, then downsamples it by **8× median pooling** to obtain a **6.25 Hz** melody sequence:
\[
\tilde{\mathbf{f}}=\operatorname{MedianPool}_{8}(\mathbf{f}).
\]
The sequence is converted from Hz to MIDI, centered by subtracting the median voiced MIDI pitch, clipped to \([-127,127]\), and mapped to a **256-entry vocabulary** with one token reserved for unvoiced frames:
\[
\bar m = \operatorname{median}\{ \operatorname{round}(\operatorname{hz2midi}(\tilde f_i))\mid \tilde f_i>0\},
\]
\[
z_i =
\begin{cases}
\operatorname{clip}(\operatorname{round}(\operatorname{hz2midi}(\tilde f_i))-\bar m,\,-127,\,127)+127, & \tilde f_i>0,\\
255, & \tilde f_i=0.
\end{cases}
\]
The report emphasizes that this preserves **melody shape** while discarding **absolute key**, **absolute pitch range**, and singer-dependent timbral information [2607.11699].

Two Melody-CoT variants are used. In **section-level melody**, only lyric-bearing vocal sections are represented, each prefixed by labels such as **[verse]** or **[chorus]**, while non-vocal sections are omitted. In **unique-section-level melody**, repeated sections are grouped by label and one representative segment is sampled for each label during training. The report presents these as a trade-off: section-level conditioning produces stronger direct melody cloning, whereas unique-section-level conditioning improves **tag following** and can reduce over-conditioning [2607.11699].

The system’s conditioning context includes global music tags, structured lyrics, singer attributes, and optional melody tokens from a reference song. A plausible implication is that Qwen-Music-LLM is optimized less as a raw waveform model than as a controllable planner operating in a compressed semantic space. That interpretation is consistent with the report’s repeated emphasis on long-horizon composition, structure, and melody control [2607.11699].

## 5. Qwen-Music-Render and high-fidelity stereo synthesis

Qwen-Music-Render is the acoustic synthesis stage. The report states that discrete semantic tokens are sufficient for **global structure**, **section ordering**, and **melody and arrangement planning**, but insufficient for **timbral richness**, **phase detail**, **stereo placement**, and **high-frequency detail**. The renderer addresses these deficits with a three-stage pipeline: a semantic-conditioned **Diffusion Transformer (DiT)**, a **Spec-VAE**, and a **Band-Mode Refiner**, producing the final **48 kHz stereo waveform** [2607.11699].

The **DiT** is a **1.3B parameter** diffusion transformer with **32 Transformer blocks**, hidden dimension **1024**, **24 attention heads**, FFN expansion ratio **8**, RoPE on queries and keys, self-attention, cross-attention to condition context, feed-forward layers, and AdaLN modulation from the diffusion timestep. It predicts continuous acoustic latents \(\phi_t \in \mathbb{R}^{T \times m}\) with \(m=128\), conditioned on frame-aligned Music Semantic Tokens at **25 Hz** and on rewritten textual context [2607.11699].

The text branch uses a frozen **Qwen3-Embedding-0.6B** encoder for song-description tokens, while lyric tokens are processed by a jointly trained **6-layer RoPE Transformer encoder**. Description and lyric features are concatenated into the cross-attention context \(\mathcal{C}\). For classifier-free guidance, the report distinguishes a learnable null text context \(\mathcal{C}_{\emptyset}\), and emphasizes that the preferred strategy retains semantic tokens while dropping only the text branch—described as **Text-drop CFG**—because semantic tokens are the primary scaffold [2607.11699].

The **Spec-VAE** is based on a SpectroStream-style 2D convolutional architecture with continuous latents. Stereo **48 kHz** audio is transformed to a complex STFT
\[
\mathbf{X} \in \mathbb{C}^{2 \times 480 \times T},
\]
encoded into **128-dimensional latents at 25 Hz**, and decoded back to a coarse complex spectrogram. The total compression ratio is **192×**. Stereo is handled through **delayed fusion** in the encoder and **early splitting** in the decoder [2607.11699].

The decoder uses **Spec-SnakeBeta**, a frequency-aware activation defined as
\[
\text{Spec\mbox{-}SnakeBeta}(\mathbf{x})_{b,c,t,f}
=
x_{b,c,t,f}
+
\frac{1}{\exp(\beta_f)+\epsilon}
\sin^2\left(x_{b,c,t,f}\cdot \exp(\alpha_f)\right),
\]
with initialization
\[
\alpha_f = \log\left(\frac{\mathrm{freq}_f}{\bar{f}+\epsilon}\right),
\qquad
\bar{f} = \frac{1}{F}\sum_f \mathrm{freq}_f.
\]
The stated effect is stronger periodic modulation at higher frequencies [2607.11699].

The final **Band-Mode Refiner** is a lightweight **ConvNeXt-1D** module that applies different correction modes by frequency band: **phase-only** in the low band, **magnitude + phase** in the mid band, and **magnitude-only** in the high band. All refiner weights are initialized to zero so that the refined output initially matches the decoder output exactly, which the report presents as a stabilization mechanism [2607.11699].

Renderer training proceeds in three stages: **reconstruction-only Spec-VAE pretraining**, then adding waveform-domain adversarial training, then freezing the Spec-VAE encoder-decoder and training the Band-Mode Refiner with waveform and spectral discriminators. The loss stack includes **multi-resolution STFT loss**, **IF/GD phase loss**, **LSGAN adversarial loss**, **feature matching**, and **KL regularization**, with additional techniques such as **K-weighting**, **MSLR stereo decomposition**, adaptive log-magnitude normalization, and mixed-scale spectral losses [2607.11699].

## 6. Training regime, evaluation, and empirical results

The report states that **Qwen-Music-LLM** is trained on **more than 5 million hours** of multilingual music data spanning **hundreds of languages**. The training corpus includes **20% instrumental music** during Stage 1 LLM training to strengthen accompaniment modeling. Data are quality-ranked by an internal **MOS-based reward model** trained from professional human ratings; within each genre, samples are partitioned into **Q1** through **Q7** using percentile thresholds at **90%**, **75%**, **50%**, **25%**, **5%**, and **1%**, and **Q7** is removed [2607.11699].

Pretraining is organized as a **quality-aware curriculum**. **Stage 1** trains on **Q3–Q6** data with dynamic quality sampling, language balancing, and the instrumental fraction. **Stage 2** anneals on **Q2** data with learning-rate decay. **Stage 3** refines on **Q1** plus carefully selected high-quality samples from the full corpus, while preserving balanced genre and language distributions [2607.11699].

Post-training is then described as **progressive**: a supervised cold start on carefully curated data with reliable control annotations, iterative offline preference alignment using **DPO**, and final on-policy optimization with **GSPO**. Musicality rewards come from the internal MOS predictor, while instruction-following rewards for genre control and lyric following are computed by **Qwen3.5-Omni** and **Qwen3-ASR** [2607.11699].

The main text-to-music benchmark uses **600 prompts total**, evenly divided into **300 Chinese** and **300 English**, with **AI-generated lyrics and musical tags** and balanced genre coverage. The report defines **16 objective musicality and audio-quality metrics** as the union of **SongBench (7)**, **SongEval (5)**, and **AudioBox-Aesthetic (4)**. Across these 16, Qwen-Music is reported to achieve the best score on **13**: **6/7** SongBench dimensions, **5/5** SongEval dimensions, and **2/4** AudioBox-Aesthetic dimensions [2607.11699].

On **SongBench**, Qwen-Music is best on **Melody 7.03**, **Arrangement 7.35**, **Musicality 6.22**, **Vocal 7.44**, **Instrumental 7.24**, and **Mixing 7.13**, but not on **Structure 6.94**, where **Mureka V8** reaches **7.02**. On **SongEval**, it is best on all five metrics: **Coherence 4.55**, **Musicality 4.42**, **Memorability 4.51** (tied numerically with Mureka V8), **Clarity 4.45**, and **Naturalness 4.37** (also tied numerically with Mureka V8). On **AudioBox-Aesthetic**, it is best on **Content Enjoyment 7.47** and **Content Usefulness 7.86**, but not on **Production Complexity 6.64** or **Production Quality 8.15** [2607.11699].

The report also evaluates **tag following** across five categories—Genre, Moods, Instruments, Vocal Gender, and Vocal Timbre—using **Gemini 3.1 Pro**, and reports an average score of **8.44**, within **0.04** of the best system average. The per-category scores are **Genre 8.08**, **Moods 8.94**, **Instruments 8.65**, **Vocal Gender 7.85** (best), and **Vocal Timbre 8.68** (second). For lyric intelligibility, Qwen-Music attains **PER 6.10**, second only to **Suno V5.5** at **4.19** [2607.11699].

Human evaluation uses **50 professional raters**, each with music creation or production experience, in blind A/B comparisons with anonymized randomized presentation, with each pair judged by **3 experts**. Qwen-Music’s preference win rates are **59.1%** versus **MiniMax Music 2.5+**, **66.7%** versus **MiniMax Music 2.6**, **58.3%** versus **Mureka V8**, **55.4%** versus **Suno V5**, and **50.3%** versus **Suno V5.5**. The report also states that Qwen-Music appeared as **JazzCat** on the Artificial Analysis Music with Vocals Leaderboard and ranked **third** among leading English vocal music generation systems [2607.11699].

For **cover song generation**, the report evaluates two reference sets: an **AI-generated reference set** with **100 English + 100 Chinese** items, and a **real-world popular-song reference set** also with **100 English + 100 Chinese** items. Melody preservation is measured by **Melody MAE**, computed by converting both generated cover and reference song to relative MIDI melody representations, aligning them with dynamic time warping, and computing mean absolute semitone error. On the AI-generated reference set, **Qwen-Music (section-level)** reaches **Melody MAE 1.48**, outperforming **Qwen-Music (unique-section-level) 1.80**, **Suno V5.5 2.00**, **Suno V5 1.87**, and **MiniMax Cover 1.89**. On the real-world popular-song reference set, the section-level model again gives the lowest Melody MAE at **1.44**, while the unique-section-level model scores **1.80** and **MiniMax Cover** scores **1.76** [2607.11699].

These results establish a recurring trade-off. Section-level conditioning yields tighter melody preservation, whereas unique-section-level conditioning improves tag following and intelligibility in many cases. The report treats this as evidence that melody conditioning can be too strong, and that partial abstraction over repeated sections improves style controllability [2607.11699].

## 7. Limitations, interpretation, and broader research context

The report explicitly points to future work on **more flexible long-context structure modeling**, **finer-grained expressive control over singing and performance**, and **more efficient rendering architectures**. These priorities imply current limitations in long-range structure flexibility, expressive nuance control, and computational efficiency. The report does not provide a dedicated failure-case section, and it does not describe concrete safeguards for **voice imitation**, **style imitation**, **copyright issues**, watermarking, or release restrictions, even though these concerns are plainly relevant to a system capable of cover-song generation and vocal-characteristic control [2607.11699].

Qwen-Music should also be distinguished from broader claims about the Qwen family’s music perception ability. In the **MUSE** benchmark, which probes music perception and auditory relational reasoning, **Qwen2.5-Omni** performed at or near chance on several tasks requiring invariant musical representations, including **Melody Shape Identification**, **Pitch Shift Detection**, **Syncopation Comparison**, and **Meter Identification**. The benchmark authors interpreted this as evidence of severe weaknesses in relative pitch, contour, and temporal abstraction [2510.19055]. This does not directly invalidate Qwen-Music’s generative results, but it shows that strong music generation and strong audio-language relational reasoning are not interchangeable capabilities.

A plausible implication is that the Qwen music line has evolved through partly disjoint research threads: **universal audio understanding** in Qwen-Audio [2311.07919], **music-audio annotation and QA dataset construction** in Jamendo-QA [2509.15662], **music-mixing dialogue adaptation** in MixAssist [2507.06329], **text-to-music evaluation** in WhisQ [2506.05899], **general audio latent compression for generation** in Qwen-Audio-VAE [2607.11738], and finally a dedicated **full-song generative system** in Qwen-Music [2607.11699]. Qwen-Audio-VAE is especially relevant in this regard because it introduced a general audio generation backbone trained on **5 million hours** and evaluated on music reconstruction, but it remained a general audio latent model rather than the semantic-token-plus-renderer design used by Qwen-Music [2607.11738].

Taken together, the available record supports a precise characterization. Qwen-Music is a specialized **semantic-composition plus acoustic-rendering** system for multilingual song generation with vocals. Its reported strengths lie in melody, arrangement, musicality, controllability, cover-song melody preservation, and stereo rendering quality. Its reported weaknesses or open problems lie in structure, production-complexity leadership, cover-song intelligibility relative to text-to-music, safety under-specification, and the broader question of whether Qwen-family systems possess equally strong **perceptual** music understanding outside the specific generative pipeline [2607.11699].

Source: https://www.emergentmind.com/topics/qwen-music