MR-FlowDPO: Fine-Tuning Flow Matching for Music

• MR-FlowDPO is a framework that enhances text-to-music generation by fine-tuning flow-matching models with multi-reward signals in a single DPO stage.
• It integrates automated reward functions for text alignment, production quality, and semantic consistency to steer generated music towards improved rhythmic stability and aesthetic quality.
• The approach leverages multi-reward direct preference optimization to achieve superior audio quality and human-aligned musicality, validated through objective metrics and human evaluations.

MR-FlowDPO is a framework for improving text-to-music generation by directly fine-tuning flow-matching generative models with multiple model-based reward signals in a single Direct Preference Optimization (DPO) stage. MR-FlowDPO advances the alignment of music generation models with subjective human preferences by simultaneously targeting text alignment, audio production quality, and semantic consistency, applied to flow-matching architectures capable of high-fidelity waveform synthesis (Ziv et al., 11 Dec 2025).

1. Flow-Matching Model Foundation

The core architecture underlying MR-FlowDPO is the flow-matching generative model, based on continuous normalizing flows parameterized via neural ODEs. The model learns a time-indexed vector field $u_t(z)$ transporting a standard Gaussian base distribution $\mathcal N(0,I)$ toward the data distribution in latent audio space. The training objective is

$$L_{\rm FM}(\theta) = \mathbb{E}_{t, z_1, z \sim p_t(\cdot \mid z_1)} \bigl\| u_t(z; \theta) - (z_1 - z_0) \bigr\|^2,$$

where $p_t(z \mid z_1)$ is an analytically defined perturbation kernel, and $(z_0, z_1)$ are sampled from the base and data distributions respectively. Inference proceeds by solving the learned ODE from $z_0$ to $z_1$ in the latent space.

Flow-matching offers advantages over discrete-time diffusion (DDPM) or autoregressive models by avoiding large token vocabularies, enabling non-autoregressive synthesis, and supporting efficient high-fidelity waveform generation (Ziv et al., 11 Dec 2025).

2. Multi-Reward Direct Preference Optimization

Direct Preference Optimization (DPO) is adapted in MR-FlowDPO from its original application to text and diffusion models for alignment. DPO fine-tunes the model such that for each context $x$ and paired candidate outputs $y_+, y_-$, the score function $s_\theta(y|x)$ satisfies $s_\theta(y_+) > s_\theta(y_-)$, as operationalized by a canonical binary cross-entropy loss. For flow-matching models, the DPO loss is applied on the vector field prediction error differences: $$L_{\rm FM\text{-}DPO}(\theta) =\,-\mathbb{E}_{t,x^w,z^l}\Bigl[\log\sigma\bigl(-\beta\bigl[\,\ell_t(z^w;\theta)-\ell_t(z^l;\theta)\;-\,\bigl(\ell(\theta_{\rm ref});\cdot)_{\rm same}\bigr]\bigr)\Bigr],$$ with

$$\ell_t(z; \theta) = \bigl\|u_t(z; \theta) - v_t \bigr\|^2$$

and $\theta_{\rm ref}$ denoting the frozen reference model. The loss scaling hyperparameter is set to $\beta=2000$ for stabilization (Ziv et al., 11 Dec 2025).

3. Automatic Multi-Axis Reward Construction

Three fully automatic model-based reward functions are used to assess music sample quality:

• Text Alignment ($r_{\rm text}$): Cosine similarity between CLAP embeddings of the prompt and generated audio; $$r_{\rm text}(x, y) = \frac{\langle e_x, e_y \rangle}{\|e_x\|\|e_y\|} \in [-1,1]$$.
• Production Quality ($r_{\rm Aes}$): Output of the A4 aesthetic predictor, a regression Transformer trained for human-annotated technical proficiency, clarity, and dynamics (rescaled to $[0,1]$ or $[1,10]$ as used).
• Semantic Consistency ($r_{\rm sem}$): Novel metric based on Music HuBERT Large. Layer-12 activations are clustered into 1024 centroids by $k$-means; $r_{\rm sem}(y)$ averages the maximum log-probability over time of token assignments in this space. This rewards rhythmic and melodic stability, representing a scalable, differentiable proxy for musicality.

Rewards are not only used to construct preference pairs but also integrated into text prompts during training ("reward prompting") (Ziv et al., 11 Dec 2025).

4. Multi-Reward Strong Domination (MRSD) and Reward Prompting

Preference data for DPO is obtained via MRSD: for each axis (text, production, semantic), all candidate pairs are scored, and dominant pairs are selected such that the positive example strongly dominates on a primary axis (at the 95th percentile of the reward difference) and weakly dominates on the other axes (at the 50th percentile). For each axis, 30K pairs are sampled, yielding 90K triplets in total from 20K prompts and 16 candidate generations per prompt.

During fine-tuning, positive reward values are prefixed to the model text prompt (e.g., "Text alignment is 0.73, Audio quality is 8.42, Semantic consistency is –2.15."). At inference, the prompt includes the 99th percentile reward values to steer the model toward exceptional quality across all axes (Ziv et al., 11 Dec 2025).

5. Training and Implementation

MR-FlowDPO applies to two main model variants:

• Flow-400M: 400M-parameter flow-matching model trained on 20K hr of music at 32 kHz.
• MelodyFlow-1B: Public 1B-parameter flow-matching text-to-music model.

Reward models (CLAP, A4, Music HuBERT Large) are either off-the-shelf or retrained on the identical 20K hr corpus. DPO fine-tuning uses AdamW optimizer, batch size 32, peak learning rate $1 \times 10^{-6}$, and 10–20 epochs. The reference model is frozen throughout DPO (Ziv et al., 11 Dec 2025).

6. Empirical Results and Ablations

Objective Metrics

Music generation is evaluated on MusicCaps using:

• Aesthetic score (Aes): Output of A4.
• Content enjoyment (EA): Secondary head of A4.
• CLAP alignment: Text–audio embedding similarity.
• BPM-std: Standard deviation of beats-per-minute estimates across 3.33s windows, lower is more stable rhythm.
• FAD (Fréchet Audio Distance): Lower indicates better match to real music distribution.

Method	Aes	EA	CLAP	BPM-std	FAD
MelodyFlow-1B	7.13	6.69	0.29	8.01	4.96
Flow-400M	7.08	6.50	0.29	9.09	2.70
MR-FlowDPO-400M	8.10	7.18	0.28	7.57	6.47
MR-FlowDPO-1B	8.26	7.72	0.27	6.11	11.26

MR-FlowDPO raises aesthetic metrics and reduces BPM-std by up to 32%, though yields higher FAD, attributed to over-optimization for aligned proxy rewards (Ziv et al., 11 Dec 2025).

Human Evaluation

Net win rates (win%-loss%) compare MR-FlowDPO to baselines in overall quality, audio, text relevance, and musicality. MR-FlowDPO-400M outperforms Flow-400M and MelodyFlow-1B by +25–+41% overall, +12–+71% audio, +13–+24% text, and +20–+38% musicality. MR-FlowDPO-1B leads MelodyFlow-1B by +16.7% overall and +43.3% in quality, with statistical significance at 95% confidence (Ziv et al., 11 Dec 2025).

Ablations

• DPO with only text or text+quality rewards fails to reduce BPM-std; all three axes are necessary for rhythmic stability.
• Reward prompting further improves Aes and BPM-std.
• MRSD is critical for maximizing both Aes and BPM-std.
• Choice of semantic reward (HuBERT unmasked scoring) outperforms span-masked variants; larger margin percentiles benefit Aes/CLAP but may harm BPM-std.

7. Significance, Limitations, and Future Directions

MR-FlowDPO represents the first application of multi-reward DPO fine-tuning for flow-matching text-to-music models, leveraging multi-dimensional rewards in a single supervised stage. The combination of tri-axis reward construction, MRSD pairing, and reward prompting yields improved alignment with human musical preference, higher audio quality, and enhanced rhythmic stability while maintaining efficient sampling inherent to flow-matching models.

Future work may include development of richer semantic or structural rewards (e.g., chord progression adherence), hierarchical or staged DPO approaches, interactive adjustment of reward weighting, and extensions to multi-track or conditional music editing (Ziv et al., 11 Dec 2025). The framework establishes an extensible baseline for preference-aligned generative modeling in complex, subjectively evaluated domains such as music. Public code and sample generations are available at https://github.com/lonzi/mrflow_dpo.