- The paper presents a one-step MeanFlow framework that converts semantic tokens to latent sequences, enabling rapid audio synthesis.
- It leverages a lightweight VAE and a conditional 1D DiT to reconstruct waveforms, with decoder-only and joint fine-tuning to refine latent mismatches.
- Experimental results show a 17× reduction in real-time factor compared to multi-step approaches while maintaining competitive quality metrics.
Introduction and Motivation
Contemporary LLM-based TTS and multimodal speech systems leverage neural audio codecs to discretize audio for downstream autoregressive or non-autoregressive modeling. The trend towards semantic codec tokenization emphasizes efficiency and controllability but intensifies the burden on the Token2Wav (Token-to-Waveform) decoder, which must synthesize high-fidelity audio from highly compressed representations. While flow-matching-based decoders achieve high perceptual quality, their inference-time cost—due to multi-step ODE integration—creates a stringent quality-latency trade-off, impeding deployment for interactive or edge inference scenarios.
This paper introduces a Token2Wav framework that applies the MeanFlow one-step generative paradigm in a highly compressed latent space. The approach sidesteps the inefficiency and instability associated with direct waveform-space one-step flows, using a combination of a lightweight VAE and a conditional 1D DiT (Diffusion Transformer) as backbone components.
Methodology
The method decomposes Token2Wav into two deterministic stages:
- Semantic Token to Latent Sequence: MeanFlow-based generator predicts the latent sequence in a single forward pass, conditioned on semantic tokens and speaker embeddings.
- Latent to Waveform Synthesis: A lightweight VAE decoder reconstructs the waveform from the generated latents.
This architecture leverages the reduced sequence length and representational dynamic range of the latent domain, which stabilizes training and inference.
MeanFlow One-Step Generation
MeanFlow replaces standard flow matching's learning of instantaneous velocity fields with the estimation of average velocity across the probability path, thereby enabling a one-step update from Gaussian noise to the data manifold. Operating in latent space, the approach enables efficient, fixed-latency synthesis. The model uses a DiT-1D network, which accepts fused conditioning from semantic tokens, speaker embeddings, and temporal information via adaptive normalization.
Refinement for Latent Distribution Mismatch
A challenge of latent-space generative modeling is the possible mismatch between the training latent distribution (from the VAE encoder) and the generator-induced distribution at inference. The paper proposes two refinement strategies without increasing inference-time cost:
- Decoder-Only Fine-Tuning (Decoder-FT): The VAE decoder is fine-tuned with the MeanFlow generator frozen, minimizing waveform-domain losses.
- End-to-End Joint Fine-Tuning (Joint-FT): Both the generator and decoder are updated using waveform-domain objectives, applied through the one-step latent sample path.
This mitigates artifacts arising from latent out-of-distribution error, improving perceptual quality.
Experimental Results
Comprehensive experiments on LibriSpeech test-clean and LibriTTS substantiated the empirical claims.
Quality and Efficiency
The best configuration (140M DiT, latent dimension D=24, joint fine-tuning) achieves:
| Model |
WER (%) ↓ |
SpkSim ↑ |
UTMOS ↑ |
MOS ↑ |
RTF ↓ |
| CosyVoice2 (10-step) |
3.18 |
0.940 |
3.76 |
4.05 ± 0.03 |
0.0775 |
| VAE Upper Bound |
2.14 |
0.966 |
3.67 |
4.10 ± 0.04 |
-- |
| MeanFlow (Joint-FT, D=24) |
3.41 |
0.932 |
3.64 |
3.85 ± 0.03 |
0.0046 |
The proposed method achieves a 17× reduction in RTF compared to the multi-step CosyVoice2 baseline (0.0046 vs. 0.0775), with marginal degradation in MOS and WER. Ablations reveal that increasing latent dimensionality directly enhances all quality metrics, and that decoder or joint fine-tuning is essential to align the latent and waveform distributions, with joint fine-tuning consistently yielding the best results.
Latency Analysis
Runtime breakdown highlights that the replacement of iterative flow-matching with a single DiT evaluation delivers the speedup. The VAE decoder's contribution to total RTF is significant but subdominant compared to the generator's.
Model Capacity
Increasing DiT model size from 140M to 600M does not confer notable gains and in some cases slightly diminishes perceptual quality. This finding suggests that for one-step updates, over-parameterization may hinder global average-velocity learning.
Theoretical and Practical Implications
The shift towards one-step conditional generative decoding in a compressed latent space constitutes a significant architectural simplification for neural audio codecs in TTS pipelines, especially in low-latency and edge settings. Operating in the latent domain mitigates the instability and error propagation endemic to waveform-level large-step generative models, while the MeanFlow paradigm eliminates iterative computation at inference. The demonstrated efficiency gains indicate immediate applicability to interactive or real-time speech systems.
From a theoretical standpoint, these experiments show that the one-step MeanFlow objective, while developed for image generation, can be successfully transferred to the latent audio domain. Unlike classic autoregressive or iterative diffusion/vocoder stacks, the method demonstrates that single-step average-velocity matching is adequate for high-fidelity waveform synthesis—contingent on effective latent representation and careful waveform-domain loss integration.
Future Directions
Potential avenues include:
- Further architecture refinement of the DiT-1D generator for improved velocity estimation stability at high dimensionality
- Incorporation of adversarial and perceptual losses within MeanFlow training rather than post-hoc refinement
- Generalization to multi-speaker, emotional, or cross-lingual TTS and speech editing tasks
- Investigation of hybrid iterative/one-step models that flexibly trade off sample quality and latency, particularly for very low-bitrate semantic codecs
- Exploration of similar one-step generative mechanisms for other modalities in multimodal LLM pipelines
Conclusion
This work establishes latent-space MeanFlow-based one-step generation as a practical and effective architectural pattern for Token2Wav modules in LLM-driven TTS systems. The sublinear latency scaling and competitive synthesis results, combined with robust refinement strategies for latent mismatch, suggest broad viability for deployment scenarios that demand both speed and high-fidelity audio reconstruction. The methodology paves the way for further research into non-iterative generative models across speech and multimodal audio domains.
(2606.18072)