DiffRhythm+: Controllable Song Generation

• The paper presents DiffRhythm+ as a system that combines a balanced, expanded training dataset with multi-modal style conditioning and direct preference optimization to enhance full-length song generation.
• It leverages a VAE encoder/decoder combined with a diffusion transformer backbone that integrates text and audio cues to improve lyric intelligibility, vocal coordination, and rhythmic coherence.
• Empirical results indicate that DiffRhythm+ achieves lower phoneme error rates and higher style alignment, leading to improved subjective ratings compared to its predecessor.

DiffRhythm+ is an enhanced diffusion-based framework for controllable and flexible full-length song generation. It was introduced to address major challenges in full-length song synthesis, including data imbalance, insufficient controllability, and inconsistent musical quality. Relative to DiffRhythm, a pioneering diffusion-based model for generating full-length songs with expressive vocals and accompaniment, DiffRhythm+ combines a substantially expanded and balanced training dataset, a multi-modal style conditioning strategy based on descriptive text and reference audio, and direct performance optimization aligned with user preferences (Chen et al., 17 Jul 2025).

1. Research context and design goals

Full-length song generation differs from short audio continuation or instrumental music generation because it must jointly maintain lyric intelligibility, vocal-accompaniment coordination, stylistic consistency, and long-range structural coherence. The motivating claim behind DiffRhythm+ is that prior systems still exhibited noticeable quality disparities and restricted creative flexibility, especially when training data were imbalanced across languages and when style control was limited to narrower conditioning channels (Chen et al., 17 Jul 2025).

The immediate precursor was DiffRhythm, which trained on approximately $50\mathrm{k}$ hours of music in a $3{:}6{:}1$ Chinese:English:instrumental ratio. That imbalance led to frequent lyric repetition and omission, especially on Chinese tracks. DiffRhythm+ was therefore framed not merely as a larger model, but as a reworked full-stack system in which dataset composition, conditioning design, and post-pretraining optimization were all modified together. The resulting emphasis was on naturalness, arrangement complexity, listener satisfaction, and flexible user control over musical style (Chen et al., 17 Jul 2025).

A later overview characterizes DiffRhythm+ as, to its knowledge, the first system to integrate direct preference optimization into a continuous diffusion-based music generator. That secondary framing is important because it places DiffRhythm+ at the intersection of long-form music generation and preference alignment, rather than treating it as only a scaling exercise (Herremans et al., 19 Nov 2025).

2. Latent architecture and flow-matching formulation

DiffRhythm+ consists of two modules: a VAE encoder/decoder that maps raw waveforms $x_0$ to and from a lower-dimensional latent $z_0$, and a conditional flow-matching backbone based on a diffusion transformer (DiT) that models the distribution $p_1(z_0)$ of clean musical latents by non-autoregressively transforming simple “noise” latents $p_0$ into music latents $p_1$ (Chen et al., 17 Jul 2025).

In practice, the system borrows the same VAE architecture and pre-trained weights from DiffRhythm, with approximately $1.1$B total parameters. The forward “diffusion” or flow-matching process is defined by coupling

$y^- \sim p_0,\qquad y^+ \sim p_1$

and mixing them along a straight path,

$$y_t = (1-t)\cdot y^- + t\cdot y^+,\qquad u_t(y|z)=\frac{d}{dt}y_t = y^+-y^-.$$

The model then learns a vector field $3{:}6{:}1$0 so that integrating

$3{:}6{:}1$1

drives $3{:}6{:}1$2. Given multi-modal condition $3{:}6{:}1$3, the training objective is the conditional flow-matching loss

$3{:}6{:}1$4

At inference time, the model solves

$3{:}6{:}1$5

from $3{:}6{:}1$6 to $3{:}6{:}1$7 with an Euler solver in $3{:}6{:}1$8 steps, and the resulting $3{:}6{:}1$9 is passed through the VAE decoder to produce a full-length waveform (Chen et al., 17 Jul 2025).

The DiT backbone uses $x_0$0 LLaMA-style transformer decoder layers with hidden size $x_0$1 and $x_0$2 heads. Lyrics, style, timestep, and noisy latent all enter as conditioning tokens concatenated along the channel dimension. Classifier-free guidance with scale $x_0$3 is applied by stochastically dropping out lyrics or style embeddings, with $x_0$4 dropout each during training. This architecture makes controllability an explicit part of the denoising dynamics rather than a post hoc reranking mechanism (Chen et al., 17 Jul 2025).

3. Dataset expansion, balancing, and supervised refinement

A central intervention in DiffRhythm+ is dataset reconstruction. The system collected $x_0$5 hours of raw recordings and filtered them via a rule-based pipeline to approximately $x_0$6 hours balanced $x_0$7 in Chinese:English:instrumental. A further audio-quality filtering stage using Audiobox and SongEval yielded a $x_0$8-hour high-quality subset for supervised fine-tuning (SFT) (Chen et al., 17 Jul 2025).

The paper attributes two specific effects to this restructuring. First, balanced Chinese and English reduces language-specific artifacts such as repeated or missing phonemes. Second, larger scale fosters richer arrangements, more complex instrumentation, and fewer collapsed sections in long sequences. These claims are not presented as generic scaling laws; they are tied to the observed failure modes of the original DiffRhythm dataset composition (Chen et al., 17 Jul 2025).

The ablation results reinforce that the balance decision mattered independently of raw scale. In Table 4, $x_0$9 hours at $z_0$0 Chinese:English outperforms $z_0$1 hours and $z_0$2 hours at $z_0$3. This suggests that multilingual balance was treated as a structural variable for lyric-conditioned song generation, not simply as a corpus hygiene measure (Chen et al., 17 Jul 2025).

4. Multi-modal style conditioning and controllability

DiffRhythm+ leverages MuLan, and specifically its MuQ-MuLan variant, as a cross-modal style extractor. MuLan maps either a reference audio clip to a style embedding $z_0$4 or a text description prompt to a style embedding $z_0$5, with both placed in the same shared latent space. This shared embedding space is the basis for the system’s claim of precise style specification through both descriptive text and reference audio (Chen et al., 17 Jul 2025).

Lyrics are first converted to phonemes via a G2P tokenizer and then embedded into $z_0$6. The timestep $z_0$7 is embedded through a standard sinusoidal or learned embedding $z_0$8. At each diffusion step, the conditioning vector is formed as

$z_0$9

which is linearly projected and summed with the DiT’s key/value and feed-forward inputs. During training, the style embedding is randomly replaced with zero with $p_1(z_0)$0 chance and the lyric embedding is randomly replaced with zero with $p_1(z_0)$1 chance. At inference, the model blends the predicted vector fields from the full condition and from conditions with style or lyric dropped in order to sharpen stylization (Chen et al., 17 Jul 2025).

The style-conditioning ablation is also specific. Mixed MuLan, defined as training on both audio and text prompts, yields the best generalization to text prompts. A plausible implication is that DiffRhythm+ treats text-prompt and reference-audio control not as disjoint modes, but as two access paths into a common style geometry (Chen et al., 17 Jul 2025).

5. Preference-driven performance optimization

Beyond reconstruction loss, DiffRhythm+ introduces Direct Preference Optimization for diffusion, described in the paper as aligning outputs to human aesthetic metrics. The reward model $p_1(z_0)$2 is defined by two off-the-shelf evaluators: SongEval, on a $p_1(z_0)$3-to-$p_1(z_0)$4 scale for full songs, and Audiobox, mapped to $p_1(z_0)$5-to-$p_1(z_0)$6 for instrumentals. From each batch, the system selects a “winner” $p_1(z_0)$7 and “loser” $p_1(z_0)$8 whose score gap is at least $p_1(z_0)$9 and whose winner’s score is at least $p_0$0, forming $p_0$1 preference tuples (Chen et al., 17 Jul 2025).

The diffusion-DPO objective is derived from a KL-constrained reward maximization over the entire diffusion path and is implemented as an $p_0$2-based preference loss between noise predictions: $p_0$3 Here, $p_0$4 is the model’s noise estimate at step $p_0$5, $p_0$6 is a frozen reference model’s estimate, $p_0$7 is a scheduling weight, $p_0$8, $p_0$9 is the number of diffusion steps, and $p_1$0 is the logistic sigmoid. DPO training runs for $p_1$1 epochs on the generated preference dataset, with per-GPU batch size $p_1$2 and EMA of weights with decay $p_1$3 (Chen et al., 17 Jul 2025).

The training-stage ablation gives the clearest summary of its contribution: Pre-training $p_1$4 SFT $p_1$5 DPO progressively boosts mean subjective score from $p_1$6. This suggests that preference optimization is not redundant with corpus filtering or supervised refinement, but instead changes the ranking of generated outputs in ways that are audible to listeners (Chen et al., 17 Jul 2025).

6. Empirical performance and ablation profile

The principal objective metrics reported for DiffRhythm+ compare the original DiffRhythm baseline with DiffRhythm+ in base and full variants (Chen et al., 17 Jul 2025).

Metric	DiffRhythm	DiffRhythm+ (base / full)
KL$p_1$7	.723	.488 / .512
FAD$p_1$8	2.113	1.835 / 1.872
CLaMP 3$p_1$9 (style alignment)	.103	.152 / .165
PER$1.1$0 (phoneme error)	17.47%	14.85% / 14.96%
RTF$1.1$1 (real-time factor)	0.034	0.036 / 0.039

On aesthetic metrics, DiffRhythm+ wins among baselines on AudioBox (CE, CU, PC, PQ), approaching or exceeding YuE in clarity and complexity. On SongEval (Coh, Mem, NVBP, CSS, OM), DiffRhythm+ is approximately equal to YuE and significantly above DiffRhythm. In subjective MOS evaluation, listeners $1.1$2 rated intelligibility, musicality, and audio quality, and DiffRhythm+ (full) achieved median MOS around $1.1$3–$1.1$4 versus DiffRhythm around $1.1$5–$1.1$6. YuE remained slightly higher at around $1.1$7, but DiffRhythm+ ran more than $1.1$8 faster at inference (Chen et al., 17 Jul 2025).

The ablations identify three major levers. Data scale and balance matter: $1.1$9 hours at $y^- \sim p_0,\qquad y^+ \sim p_1$0 Chinese:English outperforms $y^- \sim p_0,\qquad y^+ \sim p_1$1 hours and $y^- \sim p_0,\qquad y^+ \sim p_1$2 hours at $y^- \sim p_0,\qquad y^+ \sim p_1$3. Style conditioning matters: Mixed MuLan yields the best generalization to text prompts. Preference stages matter: Pre-training, SFT, and DPO each contribute measurable gains in subjective quality (Chen et al., 17 Jul 2025).

7. Subsequent interpretations, related work, and points of ambiguity

DiffRhythm+ has been read by later work as a template for preference-aligned music generation. The perspective paper “Aligning Generative Music AI with Human Preferences: Methods and Challenges” cites DiffRhythm+ as a diffusion-based preference optimization framework and states that it is, to its knowledge, the first system to integrate direct preference optimization into a continuous diffusion-based music generator (Herremans et al., 19 Nov 2025). That same source also highlights challenges that remain relevant to the DiffRhythm+ line: scalability to long-form compositions, reliability of automated scorers, multi-objective trade-offs, inference cost, and preference drift (Herremans et al., 19 Nov 2025).

There is, however, a source-level ambiguity in how the system is described. The primary DiffRhythm+ paper specifies a conditional flow-matching DiT backbone, whereas the later overview summarizes DiffRhythm+ using DDPM-style notation and LDM-style UNet terminology. This indicates a divergence in exposition across sources rather than a settled alternative canonical architecture (Chen et al., 17 Jul 2025).

A second source of confusion arises from later nomenclature. “DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching” is a distinct successor system, but its details note that it is sometimes referred to as “DiffRhythm+” in those experiments. DiffRhythm 2 introduces a semi-autoregressive architecture based on block flow matching, a $y^- \sim p_0,\qquad y^+ \sim p_1$4 Hz music VAE, cross-pair preference optimization, and stochastic block representation alignment loss, and it reports improvements over DiffRhythm+ rather than being identical to it (Jiang et al., 27 Oct 2025).

Finally, interpretability work on audio latent spaces has connected DiffRhythm-derived VAEs to sparse autoencoder analysis. Paek et al. analyze DiffRhythm’s latent space and sketch extensions to DiffRhythm+ involving additional probes and controls for rhythmic features, harmonic features, and instrument identity, as well as the insertion of SAE-derived control vectors as conditioning tokens in a cross-attention block. This suggests one concrete future direction for DiffRhythm+-style systems: richer controllability grounded in interpretable latent coordinates rather than only text or reference-audio style embeddings (Paek et al., 27 Oct 2025).