FlowTTS-GRPO: GRPO for Flow-Based TTS

• The paper introduces FlowTTS-GRPO, a novel RL framework that converts ODE trajectories into SDE paths for fine-tuning flow-based TTS models.
• It details multi-objective reward designs combining ASR, speaker similarity, and perceptual quality to drive improvements in speech generation.
• Empirical results demonstrate significant gains in metrics like CER, WER, and MOS across various flow-based and autoregressive TTS systems.

FlowTTS-GRPO denotes the application of Group Relative Policy Optimization (GRPO) to flow-based or flow-matching text-to-speech systems. In its most explicit formulation, it is an online reinforcement-learning framework for flow-matching (FM) TTS that converts ordinary differential equation trajectories into stochastic differential equation paths, thereby enabling direct fine-tuning of open-source FM models without auxiliary models (Wang et al., 22 Jun 2026). In the broader literature, the same label also covers closely related adaptations in semantic-token-plus-flow architectures, probabilistically reformulated FM backbones, few-step flow maps, and autoregressive TTS systems with downstream flow refinement (Liu et al., 23 Sep 2025, Sun et al., 3 Apr 2025, Li et al., 1 Jul 2026). Across these variants, the central idea is stable post-training of speech generation policies by sampling groups of candidate utterances, computing relative advantages within each group, and updating the policy toward candidates that improve intelligibility, speaker similarity, perceptual quality, or prosodic structure.

1. Historical development and conceptual scope

The immediate background for FlowTTS-GRPO is the observation that prior RL work in TTS concentrated on LLM-based systems, whereas flow-matching models remained comparatively under-explored. This distinction is explicit in the FM-focused formulation, which positions FlowTTS-GRPO as a response to a gap left by PPO, DPO, GRPO, and DiffRO-style work on token-generating TTS models (Wang et al., 22 Jun 2026).

Several contemporaneous lines of work define the broader conceptual territory. One line applies GRPO to LLM-based TTS and then explains how the same mechanism can be adapted to semantic-token LLMs followed by flow-matching acoustic refinement, with the LLM acting as the policy and the flow module preserving acoustic detail and naturalness (Liu et al., 23 Sep 2025). A second line, represented by F5R-TTS, reformulates a deterministic flow-matching TTS model into a Gaussian policy so that reinforcement learning can operate directly on stochastic acoustic trajectories (Sun et al., 3 Apr 2025). A third line generalizes GRPO to multilingual low-resource TTS and explicitly derives a flow-based extension using tractable likelihoods under normalizing flows (Hussain et al., 26 Sep 2025). A fourth line extends the same basic logic to text-to-audio generation, where Flow-GRPO is implemented through a reverse-time SDE sampler and rewarded by Large Audio LLMs (LALMs) (Li et al., 12 Mar 2026). A fifth line addresses deterministic few-step flow-map generators through Anchored Stochastic Flow Map Composition (ASFMC), providing a path-preserving stochasticization mechanism for models that are otherwise difficult to optimize by online RL (Li et al., 1 Jul 2026).

This literature suggests that FlowTTS-GRPO is not a single immutable algorithm. Rather, it is a family of GRPO-based post-training methods for flow-conditioned speech generation, differing mainly in how stochasticity is introduced, how policy likelihoods are computed, and which reward components are emphasized.

2. Optimization framework and stochasticization

A standard FM TTS formulation predicts a velocity field on linear interpolants between noise and data. In the FM-focused formulation, the model predicts velocity $v = x_1 - x_0$ from $x_t = (1-t)x_0 + t x_1$, where $x_1 \sim X_1$ is real speech and $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$, and the probability-flow ODE sampler is written as

$dx_t = v_t \, dt .$

The difficulty for RL is that this decoding process is deterministic and therefore exploration-free (Wang et al., 22 Jun 2026).

The standard solution in FlowTTS-GRPO is to convert the ODE trajectory into a reverse-time SDE that preserves marginals while introducing stochasticity. In the FM-based TTS setting, the update is implemented as

$$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$

with

$$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$

where $\epsilon \sim \mathcal{N}(0,\mathbf{I})$ and $\sigma_t = a\sqrt{\frac{1-t}{t}}$ (Wang et al., 22 Jun 2026). The same general construction appears in text-to-audio Flow-GRPO, where the transition density becomes Gaussian and therefore yields tractable per-step log-likelihoods for PPO-style ratios (Li et al., 12 Mar 2026).

GRPO then operates on groups of sampled trajectories. In the FM-based TTS formulation, the decoding process is cast as a Markov decision process with state $s_t \triangleq (c,t,x_t)$, action $x_t = (1-t)x_0 + t x_1$0, deterministic Euler transition, and terminal reward only at $x_t = (1-t)x_0 + t x_1$1 (Wang et al., 22 Jun 2026). For a group of $x_t = (1-t)x_0 + t x_1$2 candidates, the group-relative advantage is

$x_t = (1-t)x_0 + t x_1$3

The resulting objective is a PPO-like clipped surrogate with KL regularization to a reference policy, using the likelihood ratio

$x_t = (1-t)x_0 + t x_1$4

and Gaussian transition likelihoods under the SDE sampler (Wang et al., 22 Jun 2026).

Other FlowTTS-GRPO variants retain the same group-relative principle but alter the policy parameterization. In LLM-based TTS with downstream flow refinement, the policy is the autoregressive speech-token LLM, the advantage is $x_t = (1-t)x_0 + t x_1$5, and KL regularization is computed against a frozen reference initialized from the pre-trained policy (Liu et al., 23 Sep 2025). In low-resource multilingual TTS, a simpler GRPO objective is used without KL or PPO clipping, relying on group-relative centering and a small learning rate for stability (Hussain et al., 26 Sep 2025). In few-step deterministic flow maps, ASFMC introduces anchor-based conditional resampling so that the stochastic policy preserves the original marginal probability path while still providing well-defined likelihood ratios for GRPO updates (Li et al., 1 Jul 2026).

3. Reward design and multi-objective optimization

Reward construction is one of the defining features of FlowTTS-GRPO, and a common misconception is that the method is equivalent to ASR-only reward maximization. The literature instead uses a wide range of multi-objective rewards.

In the FM-based TTS framework, three reward components are used: speaker similarity $x_t = (1-t)x_0 + t x_1$6, ASR-based reward $x_t = (1-t)x_0 + t x_1$7, and perceptual quality reward $x_t = (1-t)x_0 + t x_1$8. Speaker similarity is computed with ERes2Net as cosine similarity between generated and reference embeddings; ASR reward is $x_t = (1-t)x_0 + t x_1$9 for Chinese and $x_1 \sim X_1$0 for English using Paraformer and Faster-Whisper Large v3; perceptual quality uses P.835 DNSMOS OVRL after resampling waveforms to 16 kHz. The final reward is a weighted sum with standard-deviation normalization,

$x_1 \sim X_1$1

and the reported setting is $x_1 \sim X_1$2, $x_1 \sim X_1$3 (Wang et al., 22 Jun 2026). The same work reports that the weighted combination converges faster than a probabilistic reward-assignment scheme, and that omitting classifier-free guidance during training accelerates convergence.

A different reward design appears in GRPO for LLM-based TTS with flow refinement, where the reward is a harmonic mean of intelligibility and ASR confidence:

$x_1 \sim X_1$4

$x_1 \sim X_1$5

The hyperparameters are $x_1 \sim X_1$6, $x_1 \sim X_1$7, $x_1 \sim X_1$8, and $x_1 \sim X_1$9. The reported analysis shows low correlation between $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$0 and $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$1 with Pearson $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$2, and cases with $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$3 still benefit from the discriminative signal of $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$4 (Liu et al., 23 Sep 2025).

Low-resource multilingual GRPO uses a three-way reward from ASR intelligibility, speaker verification similarity, and audio quality estimation, normalized to $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$5 and weighted as $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$6, $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$7, and $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$8 (Hussain et al., 26 Sep 2025). Multi-reward single-codebook TTS adds rule-based length penalty, entropy regularization reward, and an LLM-annotated prosody alignment reward, with $x_0 \sim X_0 = \mathcal{N}(0,\mathbf{I})$9 and $dx_t = v_t \, dt .$0 (Zhong et al., 26 Nov 2025). Text-to-audio Flow-GRPO further replaces CLAP-style rewards with an LALM yes/no reward

$dx_t = v_t \, dt .$1

where $dx_t = v_t \, dt .$2 under a multimodal audio-LLM (Li et al., 12 Mar 2026).

4. Architectural patterns and training practice

FlowTTS-GRPO is applied to several distinct architectural families. In semantic-token-plus-flow systems such as CosyVoice2, the semantic-token LLM is the policy $dx_t = v_t \, dt .$3, a non-autoregressive flow-matching module adds acoustic detail and naturalness, and a recommended integration is to apply GRPO to the autoregressive token generator while keeping the flow module frozen or lightly tuned by supervised losses (Liu et al., 23 Sep 2025). In hybrid CosyVoice 3.0, the pipeline comprises a speech tokenizer, an LM that generates speech tokens, an FM acoustic model conditioned on prompt and generated tokens plus prompt mel and speaker embedding, and a HiFi-GAN vocoder; FM-only FlowTTS-GRPO fine-tunes the FM component while keeping the LM untouched in the FM-RL setting (Wang et al., 22 Jun 2026).

In FM-only systems such as F5-TTS, the FM module maps prompt and generated text plus prompt mel directly to mel spectrograms, and RL is applied to that FM component alone (Wang et al., 22 Jun 2026). F5R-TTS introduces a more structural modification: the final linear layer is changed to predict $dx_t = v_t \, dt .$4 and $dx_t = v_t \, dt .$5 per mel bin and flow step, turning deterministic displacement prediction into a Gaussian output so that the sequence-level likelihood can be used directly in GRPO updates (Sun et al., 3 Apr 2025). In single-codebook TTS LLMs, GRPO is first applied to the autoregressive codec-token backbone, and a flow-matching decoder is then attached post-GRPO to refine acoustic realization, with reported complementary gains in SIM and MOS (Zhong et al., 26 Nov 2025).

Training practice also varies, but several configurations recur. In FM-based TTS, mixed ODE-SDE sampling is restricted to an early-step window, optimization is performed only on stochastic steps, and LoRA is used on the FM component with rank $dx_t = v_t \, dt .$6 and lora_alpha $dx_t = v_t \, dt .$7. For CosyVoice 3.0 FM, the reported learning rate is $dx_t = v_t \, dt .$8 with linear decay to $dx_t = v_t \, dt .$9 at $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$0k steps; for F5-TTS FM it is $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$1 with linear decay to $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$2 at $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$3k steps; the selected noise level is $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$4; CV3 uses group size $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$5 and F5-TTS uses $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$6; groups with zero standard deviation are discarded (Wang et al., 22 Jun 2026). In LLM-policy GRPO with flow refinement, the reported settings are Whisper-large-v3 for rewards, $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$7, $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$8, and learning rate $$x_{t+\Delta t} = x_{t,\mathrm{mean}} + \sigma_t \sqrt{\Delta t}\,\epsilon,$$9 (Liu et al., 23 Sep 2025).

5. Empirical results

The clearest empirical characterization of FlowTTS-GRPO comes from FM-based TTS experiments on F5-TTS and CosyVoice 3.0. For F5-TTS, FlowTTS-GRPO improves both objective quality and intelligibility: on test-zh, CER changes from $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$0 to $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$1, SS2 from $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$2 to $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$3, and P835 from $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$4 to $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$5; on test-en, WER changes from $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$6 to $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$7, SS2 from $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$8 to $$x_{t,\mathrm{mean}} = x_t + \Big[v_\theta(x_t,t) + \frac{\sigma_t^2}{2(1-t)}\big(-x_t + t\,v_\theta(x_t,t)\big)\Big]\Delta t,$$9, and P835 from $\epsilon \sim \mathcal{N}(0,\mathbf{I})$0 to $\epsilon \sim \mathcal{N}(0,\mathbf{I})$1; on test-hard, CER changes from $\epsilon \sim \mathcal{N}(0,\mathbf{I})$2 to $\epsilon \sim \mathcal{N}(0,\mathbf{I})$3. For CosyVoice 3.0 with FM only tuned, speaker similarity and perceptual quality improve while intelligibility is largely LM-governed: on test-zh, CER changes from $\epsilon \sim \mathcal{N}(0,\mathbf{I})$4 to $\epsilon \sim \mathcal{N}(0,\mathbf{I})$5, SS2 from $\epsilon \sim \mathcal{N}(0,\mathbf{I})$6 to $\epsilon \sim \mathcal{N}(0,\mathbf{I})$7, and P835 from $\epsilon \sim \mathcal{N}(0,\mathbf{I})$8 to $\epsilon \sim \mathcal{N}(0,\mathbf{I})$9; on test-en, WER changes from $\sigma_t = a\sqrt{\frac{1-t}{t}}$0 to $\sigma_t = a\sqrt{\frac{1-t}{t}}$1, SS2 from $\sigma_t = a\sqrt{\frac{1-t}{t}}$2 to $\sigma_t = a\sqrt{\frac{1-t}{t}}$3, and P835 from $\sigma_t = a\sqrt{\frac{1-t}{t}}$4 to $\sigma_t = a\sqrt{\frac{1-t}{t}}$5; on test-hard, CER changes from $\sigma_t = a\sqrt{\frac{1-t}{t}}$6 to $\sigma_t = a\sqrt{\frac{1-t}{t}}$7 (Wang et al., 22 Jun 2026).

Closely related flow-refinement results are reported in semantic-token GRPO. On CosyVoice2, GRPO-CER-NLL improves baseline zh CER from $\sigma_t = a\sqrt{\frac{1-t}{t}}$8 to $\sigma_t = a\sqrt{\frac{1-t}{t}}$9, en CER from $s_t \triangleq (c,t,x_t)$0 to $s_t \triangleq (c,t,x_t)$1, ja WER from $s_t \triangleq (c,t,x_t)$2 to $s_t \triangleq (c,t,x_t)$3, and ko CER from $s_t \triangleq (c,t,x_t)$4 to $s_t \triangleq (c,t,x_t)$5, while subjective MOS changes from $s_t \triangleq (c,t,x_t)$6 to $s_t \triangleq (c,t,x_t)$7 for zh, $s_t \triangleq (c,t,x_t)$8 to $s_t \triangleq (c,t,x_t)$9 for en, $x_t = (1-t)x_0 + t x_1$00 to $x_t = (1-t)x_0 + t x_1$01 for ja, and $x_t = (1-t)x_0 + t x_1$02 to $x_t = (1-t)x_0 + t x_1$03 for ko; the improvements are reported as statistically significant with $x_t = (1-t)x_0 + t x_1$04 (Liu et al., 23 Sep 2025). On Llasa-1B, baseline CER changes from zh $x_t = (1-t)x_0 + t x_1$05 and en $x_t = (1-t)x_0 + t x_1$06 to zh $x_t = (1-t)x_0 + t x_1$07 and en $x_t = (1-t)x_0 + t x_1$08 under GRPO-CER-NLL, while SIM changes from zh $x_t = (1-t)x_0 + t x_1$09 and en $x_t = (1-t)x_0 + t x_1$10 to zh $x_t = (1-t)x_0 + t x_1$11 and en $x_t = (1-t)x_0 + t x_1$12 (Liu et al., 23 Sep 2025).

F5R-TTS reports a $x_t = (1-t)x_0 + t x_1$13 relative reduction in WER and a $x_t = (1-t)x_0 + t x_1$14 relative increase in SIM score compared to a conventional flow-matching baseline on zero-shot voice cloning (Sun et al., 3 Apr 2025). Low-resource multilingual GRPO reports that Portuguese with $x_t = (1-t)x_0 + t x_1$15 minutes of paired data changes from CER $x_t = (1-t)x_0 + t x_1$16 in the baseline to $x_t = (1-t)x_0 + t x_1$17 after fine-tuning plus GRPO, and that on English without CFG the base CER $x_t = (1-t)x_0 + t x_1$18 changes to $x_t = (1-t)x_0 + t x_1$19 under GRPO, with SSIM changing from $x_t = (1-t)x_0 + t x_1$20 to $x_t = (1-t)x_0 + t x_1$21 (Hussain et al., 26 Sep 2025). In single-codebook TTS, the GRPO-optimized backbone achieves the best CER on test-zh and the highest MOS among the compared systems, while the additional flow-matching decoder further improves SIM and MOS over the GRPO-optimized autoregressive model (Zhong et al., 26 Nov 2025).

6. Limitations, misconceptions, and open directions

Several limitations recur across the literature. Reward-model bias is a central concern: ASR confidence does not perfectly track naturalness or prosody, and high intelligibility can be achieved with monotone or over-articulated speech if the reward is insufficiently balanced (Liu et al., 23 Sep 2025). Low-resource multilingual GRPO likewise notes dependence on pretrained judges such as Whisper, Titanet, and neural PESQ estimators, together with the possibility of reward hacking and the need for careful normalization (Hussain et al., 26 Sep 2025). FM-based TTS reports that proxy rewards such as DNSMOS and ASR may not perfectly represent human preference, and that multi-objective fusion mitigates but does not eliminate this problem (Wang et al., 22 Jun 2026).

A second misconception is that FlowTTS-GRPO uniformly improves every speech dimension in every architecture. The reported evidence is more specific. In hybrid systems, intelligibility is often LM-governed, so FM-RL primarily improves audio-detail metrics such as speaker similarity and perceptual quality; this is why CosyVoice 3.0 FM-RL improves SS and DNSMOS more consistently than CER or WER (Wang et al., 22 Jun 2026). In acoustic-token LLMs, CER improvements do not necessarily translate into statistically significant MOS gains, whereas semantic-token plus flow systems benefit more in naturalness (Liu et al., 23 Sep 2025).

A third issue concerns stochasticization itself. Excessive sampler noise can induce over-exploration and reward hacking in flow-based audio generation, while too little noise weakens exploration and slows improvement (Li et al., 12 Mar 2026). For deterministic few-step flow maps, standard SDE-based stochasticization does not directly apply; ASFMC addresses this by anchor-based conditional resampling, but the method depends on anchor design and closed-form conditionals under the chosen affine path (Li et al., 1 Jul 2026).

Open directions are stated explicitly across these works. Reported proposals include combining FM-RL with LM-RL to jointly optimize intelligibility and acoustic detail, extending training to multilingual datasets, designing better proxy rewards for prosody and naturalness, and strengthening safeguards against misuse in zero-shot voice cloning (Wang et al., 22 Jun 2026). Related work also suggests that path-preserving stochasticization, LALM-derived rewards, and flow-decoder attachment after GRPO are likely to remain central axes of future research in flow-based speech alignment (Li et al., 12 Mar 2026, Zhong et al., 26 Nov 2025).