DiffRhythm 2: Song Generation Framework

• The paper introduces a semi-autoregressive, block-wise diffusion framework that directly addresses lyric–vocal alignment challenges without external annotations.
• It employs a low latent rate Music VAE and latent flow matching to generate high-fidelity 48 kHz audio with efficient block processing.
• The approach integrates cross-pair preference optimization with stochastic block REPA to balance musicality, style alignment, and audio quality.

DiffRhythm 2 is an end-to-end song generation framework for producing full-length mixed-track songs—singing vocals plus accompaniment—from lyrics and style prompts. It is introduced as a response to the difficulty of maintaining long-term coherence across text and music modalities, preserving lyric–vocal alignment, supporting controllable prompting, and adapting generation to human preferences across multiple quality dimensions (Jiang et al., 27 Oct 2025). The framework combines a semi-autoregressive latent generator based on block flow matching, a Music VAE with a 5 Hz latent rate, a stochastic block representation alignment loss, and cross-pair preference optimization. Within the broader DiffRhythm lineage, it departs from the original DiffRhythm’s sentence-level alignment with sentence-start annotations and from fully non-autoregressive generation over the whole latent sequence, while retaining the efficiency benefits of latent-space flow-based synthesis (Ning et al., 3 Mar 2025).

1. Position within the DiffRhythm lineage

The original DiffRhythm established the baseline problem formulation: given lyrics and a style prompt, generate a complete stereo song containing both vocals and accompaniment, using a latent diffusion-based architecture with a 44.1 kHz VAE, a 21.5 Hz latent frame rate, and a sentence-level alignment mechanism that required sentence-start annotations (Ning et al., 3 Mar 2025). DiffRhythm+ then reframed the line around three upgrades: a substantially expanded and balanced training dataset, multi-modal style conditioning via MuLan for both text and audio prompts, and diffusion-adapted Direct Preference Optimization for listener-aligned generation (Chen et al., 17 Jul 2025).

DiffRhythm 2 formalizes a different successor path. Instead of retaining fully non-autoregressive generation, it introduces a semi-autoregressive architecture in which generation is non-autoregressive within each latent block and autoregressive across blocks (Jiang et al., 27 Oct 2025). The paper explicitly positions this as a remedy for lyric–vocal alignment problems in existing non-autoregressive frameworks. Unlike DiffRhythm, it does not require sentence-level timestamps; unlike ACE-Step, it does not rely on semantic alignment constraints such as mHuBERT REPA for lyric alignment itself (Jiang et al., 27 Oct 2025).

This makes DiffRhythm 2 distinct from both earlier systems. Relative to DiffRhythm, the principal architectural novelty is block-wise sequential conditioning. Relative to DiffRhythm+, the principal novelty is not merely stronger conditioning or DPO-style alignment, but a change in the generative factorization itself. A plausible implication is that the paper treats alignment as a modeling problem at the sequence-structure level rather than only as a conditioning or post-training problem.

2. Core architecture and latent representation

The system takes as input lyrics $L$ and a style prompt $S$, where style can be provided either as text $S_t$ or audio $S_a$. These conditioning inputs are fed into a Diffusion Transformer backbone trained with flow matching in a latent space learned by a Music VAE. The Diffusion Transformer predicts latent blocks sequentially; once the complete latent sequence is generated, the Music VAE decoder reconstructs the final waveform (Jiang et al., 27 Oct 2025).

The full inference pipeline is specified as follows: a user provides lyrics and a style prompt; the conditioning encoder supplies style and lyric information to the generative backbone; the latent song representation is generated block by block using block flow matching; generation stops automatically when an end-of-prediction block or frame is emitted; and the full latent sequence is decoded by the Music VAE decoder into 48 kHz audio (Jiang et al., 27 Oct 2025).

The Music VAE is central to tractability. It processes 24 kHz input audio and reconstructs at 48 kHz, with reported compression ratios of 4800× during encoding and 9600× during decoding. Architecturally, it contains an encoder using the same architecture as Stable Audio 2 VAE, a transformer block inserted before the decoder, and a BigVGAN decoder used as the waveform decoder or vocoder for high-quality output (Jiang et al., 27 Oct 2025). The latent rate is 5 Hz, so each frame represents 200 ms. The block size used for DiffRhythm 2 training is 10 latent frames, corresponding to 2 seconds per block (Jiang et al., 27 Oct 2025).

The low latent rate is one of the paper’s defining design choices. At 5 Hz, a 210-second song corresponds to roughly 1050 latent frames before blocking, which the paper uses to justify minute-scale Transformer modeling. This differs sharply from the original DiffRhythm, whose VAE used a compression factor of $f=2048$, 64 latent channels, and a latent frame rate of approximately 21.5 Hz (Ning et al., 3 Mar 2025). The contrast suggests that DiffRhythm 2 trades finer temporal resolution for substantially longer-sequence tractability.

3. Block flow matching and lyric–vocal alignment

DiffRhythm 2 starts from standard flow matching over a latent target sequence $Z$. A Gaussian noise sample $Z_0 \sim \mathcal{N}(0, I)$ is linearly interpolated with $Z$ at time $t \sim \mathcal{U}[0,1]$: $Z_t = (1-t) Z_0 + t Z.$ The model predicts

$S$0

with ground-truth velocity

$S$1

The standard objective is

$S$2

(Jiang et al., 27 Oct 2025)

The paper’s main contribution is to partition $S$3 into blocks: $S$4 For block $S$5, the model predicts

$S$6

where $S$7, $S$8, and $S$9. The block flow-matching objective is

$S_t$0

(Jiang et al., 27 Oct 2025)

The alignment mechanism is implemented through the dependency structure rather than explicit timestamp supervision. During training, the model concatenates clean and noisy sequences into a joint input $S_t$1 and applies an attention mask with three rules: style prompt $S_t$2 and lyrics $S_t$3 are visible to all blocks; in the clean sequence, block $S_t$4 can attend only to clean blocks $S_t$5; and in the noisy sequence, block $S_t$6 can attend only to clean blocks $S_t$7 and its own noisy block (Jiang et al., 27 Oct 2025). To distinguish sequence types without delimiter tokens, style prompt and lyrics get timestep $S_t$8, the clean latent sequence gets timestep $S_t$9, and noisy latent blocks get independently sampled $S_a$0.

This factorization underwrites the paper’s central claim that DiffRhythm 2 achieves faithful lyric-to-vocal alignment without external labels or handcrafted constraints. Local dependencies are preserved because all positions within a block are generated jointly with bidirectional self-attention; long-range dependencies are preserved because each new block conditions on the entire previously generated clean prefix $S_a$1 (Jiang et al., 27 Oct 2025). The original DiffRhythm instead addressed alignment through phoneme token placement at sentence-start latent frames, a sparse supervision strategy that the paper itself showed to be essential but externally annotated (Ning et al., 3 Mar 2025).

4. Variable-length generation, reconstruction, and stochastic block REPA

Inference proceeds block by block. The paper describes the following procedure: initialize an empty latent sequence $S_a$2; obtain a KV cache from a prompt-only pass; for each block, sample $S_a$3, generate $S_a$4, update the cache with the clean block, append the block to $S_a$5, and stop when an end-of-prediction frame is detected (Jiang et al., 27 Oct 2025). In the speed appendix, 32 sampling steps are used for DiffRhythm 2.

Variable-length generation is handled through an end-of-prediction frame. Because generation is blockwise, the model pads the target with repeated EOP frames to complete the final block; the paper gives

$S_a$6

The printed concatenation formula is corrupted, but the intended operation is that the original sequence is extended with $S_a$7 copies of the EOP frame (Jiang et al., 27 Oct 2025). The EOP frame itself is chosen as a constant vector of ones, written in the paper as $S_a$8, because it is well separated from latent features while remaining easy to learn. The paper reports that other choices either produced noisy endings or made stopping difficult to detect.

The Music VAE is trained with a combination of reconstruction and adversarial losses: multi-scale mel loss from BigVGAN, multi-scale STFT loss from Stable Audio 2 VAE, and adversarial training with a multi-period discriminator, multi-scale discriminator, and CQT discriminator (Jiang et al., 27 Oct 2025). The paper explicitly notes that no KL constraint is applied to the latent. In appendix reconstruction tests on 50 songs, the Music VAE at 5 Hz achieved PESQ 2.477, STOI 0.683, and PER 0.121, while Stable Audio 2 VAE at 21.5 Hz achieved PESQ 1.981, STOI 0.634, and PER 0.148 (Jiang et al., 27 Oct 2025).

To improve musicality and structural coherence, DiffRhythm 2 adds stochastic block representation alignment loss using MuQ, a self-supervised music representation model (Jiang et al., 27 Oct 2025). The paper does not provide an explicit formula for this loss. It states, however, that REPA is computed on the combination of the clean history and the current noisy block rather than on isolated corresponding blocks, and that for each noisy sequence the system randomly samples 10 blocks for REPA computation, weighting reused clean blocks so that each block’s total weight sums to 1 (Jiang et al., 27 Oct 2025). Removing REPA reduces SongEval metrics and leads to qualitative failures such as structural misalignment and complete structure failure.

5. Preference optimization and controllability

DiffRhythm 2 uses post-training preference optimization to align outputs with human judgments, but it rejects the common strategy of training separate models for different preference dimensions and then merging weights. The paper argues that such multi-preference model merging causes significant degradation as the number of preferences grows (Jiang et al., 27 Oct 2025). It therefore proposes cross-pair preference optimization (CPPO).

The four preference dimensions are musicality, style similarity, lyric alignment accuracy, and audio quality. The paper reports pairwise interactions: improving lyric alignment can hurt musicality; improving audio quality may help alignment but hurt style similarity; and improving style similarity tends to help musicality (Jiang et al., 27 Oct 2025). CPPO groups the dimensions into two pairs:

• $S_a$9
• $f=2048$0

Within each pair, DPO-style training is organized so that the winning sample satisfies both preferences, the losing sample satisfies at least one of the two preferences, and the three possible losing-case combinations are kept in equal proportion for balanced training (Jiang et al., 27 Oct 2025). The paper does not provide an explicit CPPO objective equation.

This places DiffRhythm 2 in a broader line of preference-aligned song generation. DiffRhythm+ had already integrated $f=2048$1 into a diffusion-based full-length song generator, using MuLan-based multi-modal style conditioning and automatic preference construction from SongEval and Audiobox-aesthetic (Chen et al., 17 Jul 2025). A later survey characterizes DiffRhythm+ as a representative case of multi-preference alignment in diffusion-based architectures, emphasizing MuLan conditioning and the use of SongEval and Audiobox-aesthetic as preference criteria (Herremans et al., 19 Nov 2025). DiffRhythm 2’s contribution is narrower and more specific: it replaces multi-model merging with cross-pair optimization over interacting preference dimensions (Jiang et al., 27 Oct 2025).

Controllability in DiffRhythm 2 is explicitly defined through conditioning by lyrics, style prompt as text, and style prompt as audio (Jiang et al., 27 Oct 2025). The dataset also includes annotations for structure, style, instrumentation, and emotion, but the paper does not present a formal user-facing control interface for those attributes. Variable duration is supported implicitly through EOP-based stopping rather than fixed-length generation (Jiang et al., 27 Oct 2025).

6. Training regime, empirical results, and limitations

The main song dataset contains about 1.4 million songs and about 70,000 hours, with a Chinese : English : instrumental ratio of approximately 4:5:1. Preprocessing uses Audiobox-Aesthetics for audio quality filtering, Whisper and FireRedASR to transcribe vocal tracks, cross-validation of transcriptions against original lyrics, and All-in-One plus Qwen2.5-omni to annotate structure, style, instrumentation, and emotion (Jiang et al., 27 Oct 2025). DiffRhythm 2 does not require explicit lyric alignment labels such as timestamps for its core alignment mechanism.

Training proceeds in stages. The Music VAE is trained on a 70,000-hour music dataset + 100,000-hour speech dataset for 1,500,000 steps with global batch size 128 and total duration of about 7 days (Jiang et al., 27 Oct 2025). The main DiffRhythm 2 model, at about 1B parameters, is trained on the 70,000-hour music dataset with batch size 64, block size 10 latent frames, AdamW, weight decay 1e-2, betas (0.8, 0.9), gradient clipping max norm 0.5, learning rate 1e-4, warm-up over the first 10,000 steps, and uniform timestep sampling $f=2048$2. It is then finetuned on a 20,000-hour high-quality subset at learning rate 1e-5, followed by preference optimization using 40,000 pairs for DPO (Jiang et al., 27 Oct 2025).

On objective evaluation against open-source baselines DiffRhythm+, ACE-Step, and LeVo, DiffRhythm 2 reports PER 0.13, Mulan-T 0.40, and Mulan-A 0.75. Its SongEval metrics are CO 4.09, MU 3.93, ME 4.01, CL 3.89, and NA 3.78, which the paper describes as best among open-source models across the board (Jiang et al., 27 Oct 2025). Subjective scores are MUS 3.57, HAR 3.81, VOC 3.31, ACC 3.64, and OVP 3.77. For a fixed 2-minute generation on RTX 4090, the reported runtime is 25.6 s with RTF 0.213, compared with 18.3 s / 0.153 for DiffRhythm+, 15.2 s / 0.127 for ACE-Step, and 147 s / 1.225 for LeVo (Jiang et al., 27 Oct 2025).

The ablations isolate the contributions of post-training and structure losses. Without DPO, PER worsens from 0.13 to 0.27, Mulan-T falls from 0.40 to 0.34, and SongEval MU drops from 3.93 to 3.46. Replacing CPPO with separate DPO models plus merging yields PER 0.18, Mulan-T 0.37, Mulan-A 0.69, and SongEval MU 3.57. Removing REPA gives PER 0.15, Mulan-T 0.30, and SongEval MU 3.73 (Jiang et al., 27 Oct 2025). These results support the paper’s claim that alignment quality depends not only on the block generator itself but also on the post-training preference formulation and the structural auxiliary loss.

The paper is explicit about several limitations. The 5 Hz VAE limits fidelity; vocal modeling remains difficult; lower Mulan-A suggests that global audio style representations do not capture all stylistic nuance; open-source systems still trail commercial systems; and the system carries misuse risks, including generation of disinformation or deepfake-like content (Jiang et al., 27 Oct 2025). Related work points to several adjacent directions. JAM, for example, pursues a competing flow-based design centered on word-level timing and duration control, synthetic DPO from SongEval, and a 530M-parameter model, outperforming DiffRhythm on many lyric-specific and music-specific metrics in its own benchmark setting (Liu et al., 28 Jul 2025). This suggests that the next stage of research may not be defined by scale alone, but by the interaction among temporal control, latent compression, preference optimization, and evaluation design.