dots.tts Technical Report

Abstract: We present dots.tts, a 2B-parameter continuous autoregressive text-to-speech (TTS) foundation model that models speech in a continuous latent space. Compared with existing continuous autoregressive models, our key innovations are threefold. First, we train an AudioVAE with multiple objectives to build a semantically structured and prediction-friendly continuous speech space. Second, we use full-history conditioning in the flow-matching head to preserve long-range consistency and reduce drift during generation. Third, we apply reward-free self-corrective post-training to the flow-matching head to further improve robustness and acoustic quality. After being trained on a large-scale multilingual corpus, dots.tts achieves the best average performance on Seed-TTS-Eval, with WERs of 0.94%/1.30%/6.60% and SIM scores of 81.0/77.1/79.5 on the zh/en/zh-hard test sets, respectively. Across other benchmarks, dots.tts also consistently demonstrates open-source state-of-the-art performance, exhibiting strong generation stability, voice cloning ability, and emotional expressiveness. For efficient inference, we further apply CFG-aware MeanFlow distillation, enabling low-latency speech generation with first-packet latencies of 85/54 ms in output streaming and dual-streaming modes, respectively. To facilitate reproducible research and practical deployment, we release the training and inference code, together with the pretrained, post-trained, and MeanFlow-distilled checkpoints, under the Apache 2.0 license.

• The paper presents a continuous autoregressive TTS framework using a 2B-parameter model and unified latent space to overcome limitations of discrete systems.
• It leverages a high-fidelity AudioVAE with multi-scale mel, adversarial, and feature-matching losses to align semantic and acoustic components and minimize error accumulation.
• It employs MeanFlow distillation for real-time, low-latency inference with sub-100 ms first-packet latencies, making it ideal for streaming applications.

dots.tts: A Continuous Autoregressive Foundation Model for Text-to-Speech

Model Architecture and Innovations

The dots.tts framework introduces a 2B-parameter, fully continuous autoregressive (AR) TTS system, targeting the limitations of discrete-token AR paradigms. The model architecture is fundamentally decomposed into three subsystems: a high-fidelity semantic AudioVAE, a semantic encoder projecting to a 6.25 Hz audio-semantic domain, and a backbone comprising a pretrained LLM and an autoregressive flow-matching (AR-FM) head. The AudioVAE operates at 48 kHz, encoding speech as $128$-d latents at 25 Hz and utilizing a BigVGAN-style causal decoder.

In contrast to leading discrete AR systems, dots.tts canonically routes all per-step prediction through a single, multitask-trained continuous latent space, mitigating the semantic expressivity limits and acoustic bottlenecks posed by neural codecs and quantization (Chen et al., 2024, Anastassiou et al., 2024). The AudioVAE is pretrained with multi-scale mel, adversarial, and feature-matching losses, then further fine-tuned with multitask (ASR, speaker, emotion) supervision and a WavLM-aligned frame-level loss to simultaneously maintain reconstructive quality and semantic structure, ensuring latent suitability as an LLM target.

The backbone leverages a decoupled LLM (Qwen2.5-1.5B), passively receiving BPE tokens and audio-semantic embeddings, facilitating incremental, streaming-friendly 1T1A inference. The AR-FM head itself is founded on a DiT-style velocity field and is responsible for precise generation and roll-out stability, with full-history conditioning minimizing long-range error accumulation typical of continuous AR models.

Figure 1: Model backbone: text and audio-semantic tokens enter a unified LLM stream; latent patches are autoregressively decoded and fed back to the semantics.

Flow-Matching Training and Reward-Free Self-Correction

The AR-FM head is trained with a block-causal attention-masked DiT stack, directly modeling the velocity between noise and latent patches in the AudioVAE space via rectified flow matching (Shen et al., 2023). A bespoke block-causal mask and RoPE position assignment ensure the model’s context during parallelized training matches inference-time AR roll-outs exactly.

To further address the compounding of small errors in long continual AR generation, the authors integrate reward-free self-corrective alignment, adapting SOAR-style off-trajectory sample exposure but limiting supervision to the AR-FM module. The mechanism produces auxiliary training samples by propagating the current model state with classifier-free guidance and then re-noising, training the model to steer back toward ground truth. This procedure demonstrably resolves ODE mismatch issues, yielding substantial gains in synthesis stability and text faithfulness without additional reward models or external teachers.

Figure 2: Block-causal attention and RoPE positions for flow-matching: autoregressive alignment between training and deployment contexts.

MeanFlow Distillation for Low-Latency Inference

Real-time deployment is a central goal; thus, the authors distill the AR-FM DiT into a MeanFlow student, replacing the original ODE solver with interval-conditioned mean velocity predictions. MeanFlow distillation is performed with fused classifier-free guidance from the teacher, achieving low-step inference (NFE = 2–4) with high fidelity and further reducing latency. The causal design unlocks incremental, interleaved 1T1A inference, supporting “streaming” applications with sub-100 ms first-packet latencies (plain: 85 ms, dual: 54 ms at RTF ~0.23–0.25 on H800).

Empirical Results

Speech Reconstruction and Latent Quality

On LibriSpeech test-other, the latent achieves PESQ 4.09, STOI 0.973, UTMOS 3.75, SIM 0.969, and WER 4.14%, outperforming all reported discrete baselines and matching leading continuous representations, affirming negligible end-to-end degradation from the learned latent.

Zero-Shot TTS Benchmarks

• Seed-TTS-Eval: dots.tts achieves the lowest average WER (2.95, SOAR) and highest SIM (79.2, SOAR), superseding every open-source and commercial baseline under canonical zero-shot conditions. MeanFlow distillation maintains WER with minimal SIM loss (NFE=4).
• MiniMax Multilingual: Attains top SIM (83.9) on the 24-language set, outperforming VoxCPM2 (82.3), and matches the strongest WERs on well-represented languages but trails on extreme low-resource scripts due to lack of phoneme augmentation.
• CV3-Eval: Leads cross-lingual voice cloning, with SOAR reaching SIM 75.0 (en$\to$zh) and 72.8 (zh$\to$en)—substantially ahead of all non-continuous baselines.
• EmergentTTS-Eval: Yields open-source-best performance on Syntactic Complexity (65.7% preference vs all closed/commercial systems) and Emotions (72.7%); overall, dots.tts is at parity with gpt-4o-mini-tts in composite qualitative metrics.

Efficiency

In plain and interleaved streaming modes, the system attains real-time synthesis on contemporary datacenter GPUs with TTFB (first packet) down to $54\,\mathrm{ms}$ (dual-streaming, MF NFE=4).

Theoretical and Practical Implications

The investigation robustly confirms that continuous-latent AR generation—when coupled with multitask latent shaping, explicit semantic–acoustic modularity, and reward-free error-corrective post-training—achieves and frequently surpasses the best discrete-token TTS systems in naturalness, expressiveness, and cross-lingual voice fidelity. The inclusion of MeanFlow-based distillation indicates that continuous-latent AR systems are viable for real-time product deployment, countering prevailing assumptions about computational infeasibility and error accumulation.

The demonstrated separation of semantic and acoustic processes, together with the adoption of a semantically structured, learnable latent, paves the way for nuanced instruction- and style-controllable TTS, as well as further generalization to speech-and-sound synthesis workflows. Moreover, the reward-free self-correction paradigm can likely generalize to other AR generative domains facing similar drift and sample mismatch issues in long sequences.

Future Directions

Future work involves addressing the remaining WER gap on script-divergent or low-resource languages by integrating phoneme-side augmentation and language-balanced post-training. Instruction/explicit style-conditioned tuning leveraging paired descriptive data is likely to augment expressiveness and control. Finally, expanding training to include balanced speech, singing, and environmental audio corpora will generalize continuous AR TTS towards broad audio generation tasks. As with all high-fidelity zero-shot cloning models, robust watermarking and reference-consent protocols become essential for responsible deployment.

Conclusion

dots.tts constitutes a comprehensive demonstration that continuous-latent AR generation, if supported by appropriately structured semantic encoding and error correction, not only rivals but in several aspects surpasses the strongest discrete-token TTS systems in both accuracy and expressiveness. The architecture’s modularity, real-time feasibility, and public release establish it as a reference system for the next generation of text-to-speech research and application (2606.07080).