---
title: 'DiffRhythm 2: Song Generation Framework'
url: https://www.emergentmind.com/topics/diffrhythm-2
type: topic
---

# DiffRhythm 2: Song Generation Framework

DiffRhythm 2 is an end-to-end song generation framework for producing full-length mixed-track songs—singing vocals plus accompaniment—from lyrics and style prompts. It is introduced as a response to the difficulty of maintaining long-term coherence across text and music modalities, preserving lyric–vocal alignment, supporting controllable prompting, and adapting generation to human preferences across multiple quality dimensions [2510.22950]. The framework combines a semi-autoregressive latent generator based on block flow matching, a Music VAE with a 5 Hz latent rate, a stochastic block representation alignment loss, and cross-pair preference optimization. Within the broader DiffRhythm lineage, it departs from the original DiffRhythm’s sentence-level alignment with sentence-start annotations and from fully non-autoregressive generation over the whole latent sequence, while retaining the efficiency benefits of latent-space flow-based synthesis [2503.01183].

## 1. Position within the DiffRhythm lineage

The original DiffRhythm established the baseline problem formulation: given lyrics and a style prompt, generate a complete stereo song containing both vocals and accompaniment, using a latent diffusion-based architecture with a 44.1 kHz VAE, a 21.5 Hz latent frame rate, and a sentence-level alignment mechanism that required sentence-start annotations [2503.01183]. DiffRhythm+ then reframed the line around three upgrades: a substantially expanded and balanced training dataset, multi-modal style conditioning via MuLan for both text and audio prompts, and diffusion-adapted Direct Preference Optimization for listener-aligned generation [2507.12890].

DiffRhythm 2 formalizes a different successor path. Instead of retaining fully non-autoregressive generation, it introduces a semi-autoregressive architecture in which generation is non-autoregressive within each latent block and autoregressive across blocks [2510.22950]. The paper explicitly positions this as a remedy for lyric–vocal alignment problems in existing non-autoregressive frameworks. Unlike DiffRhythm, it does not require sentence-level timestamps; unlike ACE-Step, it does not rely on semantic alignment constraints such as mHuBERT REPA for lyric alignment itself [2510.22950].

This makes DiffRhythm 2 distinct from both earlier systems. Relative to DiffRhythm, the principal architectural novelty is block-wise sequential conditioning. Relative to DiffRhythm+, the principal novelty is not merely stronger conditioning or DPO-style alignment, but a change in the generative factorization itself. A plausible implication is that the paper treats alignment as a modeling problem at the sequence-structure level rather than only as a conditioning or post-training problem.

## 2. Core architecture and latent representation

The system takes as input lyrics \(L\) and a style prompt \(S\), where style can be provided either as text \(S_t\) or audio \(S_a\). These conditioning inputs are fed into a Diffusion Transformer backbone trained with flow matching in a latent space learned by a Music VAE. The Diffusion Transformer predicts latent blocks sequentially; once the complete latent sequence is generated, the Music VAE decoder reconstructs the final waveform [2510.22950].

The full inference pipeline is specified as follows: a user provides lyrics and a style prompt; the conditioning encoder supplies style and lyric information to the generative backbone; the latent song representation is generated block by block using block flow matching; generation stops automatically when an end-of-prediction block or frame is emitted; and the full latent sequence is decoded by the Music VAE decoder into 48 kHz audio [2510.22950].

The Music VAE is central to tractability. It processes 24 kHz input audio and reconstructs at 48 kHz, with reported compression ratios of **4800× during encoding** and **9600× during decoding**. Architecturally, it contains an encoder using the same architecture as Stable Audio 2 VAE, a transformer block inserted before the decoder, and a BigVGAN decoder used as the waveform decoder or vocoder for high-quality output [2510.22950]. The latent rate is **5 Hz**, so each frame represents 200 ms. The block size used for DiffRhythm 2 training is **10 latent frames**, corresponding to **2 seconds per block** [2510.22950].

The low latent rate is one of the paper’s defining design choices. At 5 Hz, a 210-second song corresponds to roughly 1050 latent frames before blocking, which the paper uses to justify minute-scale Transformer modeling. This differs sharply from the original DiffRhythm, whose VAE used a compression factor of \(f=2048\), **64 latent channels**, and a latent frame rate of approximately **21.5 Hz** [2503.01183]. The contrast suggests that DiffRhythm 2 trades finer temporal resolution for substantially longer-sequence tractability.

## 3. Block flow matching and lyric–vocal alignment

DiffRhythm 2 starts from standard flow matching over a latent target sequence \(Z\). A Gaussian noise sample \(Z_0 \sim \mathcal{N}(0, I)\) is linearly interpolated with \(Z\) at time \(t \sim \mathcal{U}[0,1]\):
\[
Z_t = (1-t) Z_0 + t Z.
\]
The model predicts
\[
\hat{v} = f_\theta(S, L, Z_t, t),
\]
with ground-truth velocity
\[
v = Z - Z_0.
\]
The standard objective is
\[
\mathcal{L}_{fm}=
\mathbb{E}\big[\|\hat{v} - v\|_2^2\big] =
\mathbb{E}\big[\|f_\theta(S, L, Z_t, t) - (Z - Z_0) \|_2^2\big].
\]
[2510.22950]

The paper’s main contribution is to partition \(Z\) into blocks:
\[
Z = \{z^1, z^2, \cdots z^k\}, \qquad
k = \lceil l / b \rceil .
\]
For block \(i\), the model predicts
\[
\hat{v}^i = f_\theta(S, L, z^{<i}, z_t^i, t^i),
\]
where \(z_0^i \sim \mathcal{N}(0,I)\), \(z_t^i = (1-t^i) z_0^i + t^i z^i\), and \(v^i = z^i - z_0^i\). The block flow-matching objective is
\[
\mathcal{L}_{bfm}=
\frac{1}{k}\sum^{k}_{i=1} \mathbb{E} \big[\|\hat{v}^i - v^i\|_2^2\big] =
\frac{1}{k}\sum^{k}_{i=1}\mathbb{E}\big[\|f_\theta(S, L, z^{<i}, z_t^i, t^i) - (z^i - z_0^i) \|_2^2\big].
\]
[2510.22950]

The alignment mechanism is implemented through the dependency structure rather than explicit timestamp supervision. During training, the model concatenates clean and noisy sequences into a joint input \((S, L, Z, Z_t)\) and applies an attention mask with three rules: style prompt \(S\) and lyrics \(L\) are visible to all blocks; in the clean sequence, block \(i\) can attend only to clean blocks \(1,\dots,i\); and in the noisy sequence, block \(i\) can attend only to clean blocks \(1,\dots,i-1\) and its own noisy block [2510.22950]. To distinguish sequence types without delimiter tokens, style prompt and lyrics get timestep \(-1\), the clean latent sequence gets timestep \(1\), and noisy latent blocks get independently sampled \(t^i \sim \mathcal{U}[0,1]\).

This factorization underwrites the paper’s central claim that DiffRhythm 2 achieves faithful lyric-to-vocal alignment without external labels or handcrafted constraints. Local dependencies are preserved because all positions within a block are generated jointly with bidirectional self-attention; long-range dependencies are preserved because each new block conditions on the entire previously generated clean prefix \(z^{<i}\) [2510.22950]. The original DiffRhythm instead addressed alignment through phoneme token placement at sentence-start latent frames, a sparse supervision strategy that the paper itself showed to be essential but externally annotated [2503.01183].

## 4. Variable-length generation, reconstruction, and stochastic block REPA

Inference proceeds block by block. The paper describes the following procedure: initialize an empty latent sequence \(Z\); obtain a KV cache from a prompt-only pass; for each block, sample \(z_0^i\), generate \(z^i \leftarrow SP(m_\theta, z_0^i, KV)\), update the cache with the clean block, append the block to \(Z\), and stop when an end-of-prediction frame is detected [2510.22950]. In the speed appendix, **32 sampling steps** are used for DiffRhythm 2.

Variable-length generation is handled through an end-of-prediction frame. Because generation is blockwise, the model pads the target with repeated EOP frames to complete the final block; the paper gives
\[
n = l^k \bmod b .
\]
The printed concatenation formula is corrupted, but the intended operation is that the original sequence is extended with \(n\) copies of the EOP frame [2510.22950]. The EOP frame itself is chosen as a constant vector of ones, written in the paper as \(\mathcal N(1,0)\), because it is well separated from latent features while remaining easy to learn. The paper reports that other choices either produced noisy endings or made stopping difficult to detect.

The Music VAE is trained with a combination of reconstruction and adversarial losses: multi-scale mel loss from BigVGAN, multi-scale STFT loss from Stable Audio 2 VAE, and adversarial training with a multi-period discriminator, multi-scale discriminator, and CQT discriminator [2510.22950]. The paper explicitly notes that no KL constraint is applied to the latent. In appendix reconstruction tests on **50 songs**, the Music VAE at **5 Hz** achieved **PESQ 2.477**, **STOI 0.683**, and **PER 0.121**, while Stable Audio 2 VAE at **21.5 Hz** achieved **PESQ 1.981**, **STOI 0.634**, and **PER 0.148** [2510.22950].

To improve musicality and structural coherence, DiffRhythm 2 adds stochastic block representation alignment loss using **MuQ**, a self-supervised music representation model [2510.22950]. The paper does not provide an explicit formula for this loss. It states, however, that REPA is computed on the combination of the clean history and the current noisy block rather than on isolated corresponding blocks, and that for each noisy sequence the system randomly samples **10 blocks** for REPA computation, weighting reused clean blocks so that each block’s total weight sums to 1 [2510.22950]. Removing REPA reduces SongEval metrics and leads to qualitative failures such as structural misalignment and complete structure failure.

## 5. Preference optimization and controllability

DiffRhythm 2 uses post-training preference optimization to align outputs with human judgments, but it rejects the common strategy of training separate models for different preference dimensions and then merging weights. The paper argues that such multi-preference model merging causes significant degradation as the number of preferences grows [2510.22950]. It therefore proposes **cross-pair preference optimization (CPPO)**.

The four preference dimensions are **musicality**, **style similarity**, **lyric alignment accuracy**, and **audio quality**. The paper reports pairwise interactions: improving lyric alignment can hurt musicality; improving audio quality may help alignment but hurt style similarity; and improving style similarity tends to help musicality [2510.22950]. CPPO groups the dimensions into two pairs:
- \((\text{musicality}, \text{lyric alignment accuracy})\)
- \((\text{style similarity}, \text{audio quality})\)

Within each pair, DPO-style training is organized so that the winning sample satisfies both preferences, the losing sample satisfies at least one of the two preferences, and the three possible losing-case combinations are kept in equal proportion for balanced training [2510.22950]. The paper does not provide an explicit CPPO objective equation.

This places DiffRhythm 2 in a broader line of preference-aligned song generation. DiffRhythm+ had already integrated **\(\mathcal{L}_{\text{DPO-Diff}}\)** into a diffusion-based full-length song generator, using MuLan-based multi-modal style conditioning and automatic preference construction from SongEval and Audiobox-aesthetic [2507.12890]. A later survey characterizes DiffRhythm+ as a representative case of multi-preference alignment in diffusion-based architectures, emphasizing MuLan conditioning and the use of SongEval and Audiobox-aesthetic as preference criteria [2511.15038]. DiffRhythm 2’s contribution is narrower and more specific: it replaces multi-model merging with cross-pair optimization over interacting preference dimensions [2510.22950].

Controllability in DiffRhythm 2 is explicitly defined through conditioning by **lyrics**, **style prompt as text**, and **style prompt as audio** [2510.22950]. The dataset also includes annotations for structure, style, instrumentation, and emotion, but the paper does not present a formal user-facing control interface for those attributes. Variable duration is supported implicitly through EOP-based stopping rather than fixed-length generation [2510.22950].

## 6. Training regime, empirical results, and limitations

The main song dataset contains about **1.4 million songs** and about **70,000 hours**, with a **Chinese : English : instrumental** ratio of approximately **4:5:1**. Preprocessing uses **Audiobox-Aesthetics** for audio quality filtering, **Whisper** and **FireRedASR** to transcribe vocal tracks, cross-validation of transcriptions against original lyrics, and **All-in-One** plus **Qwen2.5-omni** to annotate structure, style, instrumentation, and emotion [2510.22950]. DiffRhythm 2 does not require explicit lyric alignment labels such as timestamps for its core alignment mechanism.

Training proceeds in stages. The Music VAE is trained on a **70,000-hour music dataset + 100,000-hour speech dataset** for **1,500,000** steps with global batch size **128** and total duration of about **7 days** [2510.22950]. The main DiffRhythm 2 model, at about **1B** parameters, is trained on the **70,000-hour music dataset** with batch size **64**, block size **10** latent frames, **AdamW**, weight decay **1e-2**, betas **(0.8, 0.9)**, gradient clipping max norm **0.5**, learning rate **1e-4**, warm-up over the first **10,000 steps**, and uniform timestep sampling \(t \sim \mathcal U[0,1]\). It is then finetuned on a **20,000-hour high-quality subset** at learning rate **1e-5**, followed by preference optimization using **40,000 pairs for DPO** [2510.22950].

On objective evaluation against open-source baselines DiffRhythm+, ACE-Step, and LeVo, DiffRhythm 2 reports **PER 0.13**, **Mulan-T 0.40**, and **Mulan-A 0.75**. Its SongEval metrics are **CO 4.09**, **MU 3.93**, **ME 4.01**, **CL 3.89**, and **NA 3.78**, which the paper describes as best among open-source models across the board [2510.22950]. Subjective scores are **MUS 3.57**, **HAR 3.81**, **VOC 3.31**, **ACC 3.64**, and **OVP 3.77**. For a fixed **2-minute** generation on **RTX 4090**, the reported runtime is **25.6 s** with **RTF 0.213**, compared with **18.3 s / 0.153** for DiffRhythm+, **15.2 s / 0.127** for ACE-Step, and **147 s / 1.225** for LeVo [2510.22950].

The ablations isolate the contributions of post-training and structure losses. Without DPO, **PER** worsens from **0.13** to **0.27**, **Mulan-T** falls from **0.40** to **0.34**, and **SongEval MU** drops from **3.93** to **3.46**. Replacing CPPO with separate DPO models plus merging yields **PER 0.18**, **Mulan-T 0.37**, **Mulan-A 0.69**, and **SongEval MU 3.57**. Removing REPA gives **PER 0.15**, **Mulan-T 0.30**, and **SongEval MU 3.73** [2510.22950]. These results support the paper’s claim that alignment quality depends not only on the block generator itself but also on the post-training preference formulation and the structural auxiliary loss.

The paper is explicit about several limitations. The **5 Hz VAE limits fidelity**; vocal modeling remains difficult; lower **Mulan-A** suggests that global audio style representations do not capture all stylistic nuance; open-source systems still trail commercial systems; and the system carries misuse risks, including generation of disinformation or deepfake-like content [2510.22950]. Related work points to several adjacent directions. JAM, for example, pursues a competing flow-based design centered on word-level timing and duration control, synthetic DPO from SongEval, and a **530M**-parameter model, outperforming DiffRhythm on many lyric-specific and music-specific metrics in its own benchmark setting [2507.20880]. This suggests that the next stage of research may not be defined by scale alone, but by the interaction among temporal control, latent compression, preference optimization, and evaluation design.

Source: https://www.emergentmind.com/topics/diffrhythm-2