VoiceFlow: Rectified Flow Matching in TTS

• VoiceFlow is an acoustic TTS model that leverages rectified flow matching to generate mel-spectrograms via a continuous ODE approach.
• The method retrains on synthetic endpoint pairs to straighten generative trajectories, reducing computational steps while maintaining high perceptual quality.
• Empirical benchmarks show superior performance over diffusion-based models, especially in low-step and multi-speaker scenarios.

VoiceFlow (Rectified Flow Matching in TTS)

VoiceFlow is an acoustic model for efficient text-to-speech synthesis which utilizes rectified flow matching (RFM) to improve both the quality and efficiency of mel-spectrogram generation. In contrast to conventional diffusion models, which require computationally intensive iterative sampling, VoiceFlow recasts generation as solving an ordinary differential equation (ODE) over the spectrogram features, conditional on textual input. The rectified flow matching technique straightens the generative trajectory, resulting in highly effective synthesis with a limited number of sampling steps. Empirical results demonstrate VoiceFlow’s superiority over diffusion-based TTS models across subjective and objective benchmarks.

1. Flow Matching: Formulation and RFM Training

VoiceFlow applies the conditional flow matching framework, in which the mel-spectrogram generation task is formulated as transporting a noise sample $x_0$ to a target spectrogram $x_1$ along a linear path in feature space. The intermediate samples $x_t$ are drawn from the conditional probability:

$$p_t(x \mid x_0, x_1, y) = \mathcal{N}(x \mid t x_1 + (1-t)x_0, \sigma^2 I)$$

where $y$ encodes the conditioning information, such as linguistic or duration features, and $\sigma$ is a small constant ensuring tight coupling at endpoints.

The associated vector field for the flow is

$v_t(x \mid x_0, x_1, y) = x_1 - x_0,$

which describes a constant, straight-line direction. VoiceFlow trains a neural network $u_\theta$ to estimate this vector field by minimizing:

$$\min_\theta \ \mathbb{E}_{t, x_0, x_1, x_t \sim p_t} \left[ \| u_\theta(x_t, y, t) - (x_1 - x_0) \|^2 \right].$$

Rectified flow matching is introduced as a crucial enhancement. After initial training, synthetic endpoints $\hat{x}_1$ are generated by numerically integrating the learned ODE from $x_0$, then retraining $u_\theta$ on these synthetic pairs $(x_0', \hat{x}_1)$, leading to straighter and more direct generative trajectories:

$$\min_\theta \ \mathbb{E}_{t, (x_0', \hat{x}_1), x_t} \left[ \| u_\theta(x_t, y, t) - (\hat{x}_1 - x_0') \|^2 \right].$$

This rectification directly improves both synthesis quality and efficiency, especially when the number of sampling steps is small.

2. Model Architecture

VoiceFlow’s architecture comprises several components:

• Text Encoder & Duration Predictor: Encodes phone sequences into latent features and predicts phone durations, typically via forced alignment. These representations produce frame-level conditioning vectors $y$.
• Duration Adaptor: Aligns latent vectors temporally using predicted durations, enabling correct timing for mel-spectrogram generation.
• Vector Field Estimator: Utilizes a U-Net-style architecture with residual 2D convolutional blocks (scheme derived from GradTTS). Inputs include a sample $x_t$, condition $y$, and time index $t$, the latter being processed through dedicated fully connected layers. The network estimates $u_\theta(x_t, y, t)$ to be used during ODE integration.

The ODE $$\operatorname{d}x/\operatorname{d}t = u_\theta(x, y, t)$$ is solved via discretization (e.g., Euler method) for $K$ steps to obtain the synthesized mel-spectrogram $x_1$.

3. Sampling Efficiency and Performance Benchmarks

VoiceFlow achieves high performance with drastically reduced sampling steps, which is validated with both subjective (Mean Opinion Score, MOS) and objective (MOSNet, MCD) metrics. Subjective evaluations across LJSpeech and LibriTTS datasets illustrate:

Model	Steps	MOS (LJSpeech)	MOS (LibriTTS)
GradTTS	2	↓	↓
VoiceFlow	2	↑	↑
GradTTS	100	Intermediate	Intermediate
VoiceFlow	100	↑	↑

At very low step counts (e.g., 2 steps), VoiceFlow maintains perceptual quality, with MOS degradation observed for GradTTS. Objective measures further confirm superior speed-quality tradeoffs. In multi-speaker settings the benefits are even more pronounced, as the flow matching system adapts efficiently across variable speaker embeddings and acoustic conditions.

4. Ablation of Rectified Flow

Ablation studies isolate the impact of rectification. CMOS scores reveal significant drops when this self-refinement step is omitted (–0.78 on LJSpeech, –1.21 on LibriTTS for 2-step scenarios). Visualizations of ODE paths indicate the rectified model’s trajectories are predominantly straight lines, whereas non-rectified and diffusion paths are more convoluted.

This finding substantiates the efficacy of training on synthetic endpoint pairs in producing a vector field whose flow matches the shortest path, thereby maximizing synthesis efficiency and quality.

5. Comparison to Diffusion-Based TTS

VoiceFlow addresses limitations in diffusion-based models (e.g., GradTTS, DiffVoice), which require many sampling steps due to stochasticity in SDE/ODE-based generation. The ODE formulation in VoiceFlow, combined with straightened flow via RFM, enables competitive or superior synthesis at orders-of-magnitude fewer steps.

Key comparative features:

• No score matching required; direct vector field estimation.
• Linear (straight-line) generative path versus random diffusion trajectories.
• Substantial reduction in computational cost and latency.
• Greater robustness under multi-speaker and variable linguistic durations.

6. Extensions and Future Research Directions

The principles underlying VoiceFlow’s rectified flow matching suggest promising avenues for other TTS and speech-based applications. Noted future directions include:

• Automatic alignment search: using flow matching to align phonetic or linguistic units to acoustic frames.
• Voice conversion: leveraging RFM trajectories for speaker-conditional transformations.
• Improved sampling schemes: further refinement of ODE solvers, conditioning mechanisms, or representation disentanglement.
• Broader flow-based generative models: adapting RFM to other time-series and structured data synthesis domains.

A plausible implication is that the rectified flow approach may continue to support advances in speech synthesis efficiency, quality, and model interpretability.

7. Practical Implications and Limitations

VoiceFlow’s approach yields distinct benefits in real-world deployment scenarios:

• Fast synthesis with low latency, suitable for interactive and on-device TTS.
• Consistent high-quality output with minimal inference steps.
• Adaptability for both single- and multi-speaker environments.

A potential limitation is the reliance on the accuracy of the rectification process; the straightened flow is maximally efficient only when the learned vector field captures the true data manifold. As with all ODE-based neural synthesis, discretization choices (step size, solver method) may influence final sample quality.

In summary, VoiceFlow with rectified flow matching constitutes a significant technical advance in non-autoregressive, ODE-based text-to-speech generation. It leverages direct vector field estimation, flow rectification, and efficient architectural design to surpass conventional diffusion models in both synthesis performance and computational efficiency. This framework establishes new directions for generative modeling of structured audio data.