Papers
Topics
Authors
Recent
Search
2000 character limit reached

FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations

Published 18 Aug 2026 in cs.SD | (2608.17492v1)

Abstract: Recent continuous autoregressive TTS models operate directly on continuous speech representations, preserving rich acoustic details while leveraging the instruction-following capabilities of text LLMs. This paradigm opens new possibilities for voice cloning, instruction-controlled voice design, and speech editing, but remains susceptible to error accumulation during autoregressive generation. Existing solutions often require additional semantic modules, multi-stage tokenizer training pipelines, or complex autoregressive architectures. In this work, we propose FireRedTTS3, a simple yet effective speech generation and editing framework that mitigates error accumulation at the representation level. Specifically, we leverage a frozen Audio Encoder trained on diverse speech understanding tasks as a semantic teacher to regularize the audio feature space. This improves text-speech alignment and stabilizes autoregressive generation while keeping the overall system simple. FireRedTTS3 provides two variants: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for unified voice cloning, instruction-controlled voice design, and speech editing. Experiments show that FireRedTTS3-Base achieves the best average speech intelligibility and speaker similarity among compared systems on Seed-TTS-Eval and MiniMax-MLS-Test, while FireRedTTS3-Instruct outperforms competing systems on InstructTTSEval and Ming-Freeform-Audio-Edit. These results demonstrate that semantically enriched continuous speech representations, combined with a simple architecture, enable stable, controllable, and high-fidelity speech generation and editing. Code and models are available at https://github.com/FireRedTeam/FireRedTTS3.

Citations (12)

Summary

  • The paper introduces RedAE, a continuous speech tokenizer that distills semantic features from a frozen audio-understanding model, reducing autoregressive error accumulation without auxiliary modules.
  • FireRedTTS3-Base achieves strong zero-shot multilingual voice cloning, reporting 3.04% average error and 78.8% speaker similarity on Seed-TTS-Eval and 3.75% error with 84.8% similarity across 24 languages on MiniMax-MLS-Test.
  • FireRedTTS3-Instruct unifies voice cloning, instruction-controlled voice design, and speech editing through textual planning, leading compared systems on InstructTTSEval and improving editing accuracy over Ming-UniAudio-Edit.

FireRedTTS3 is a continuous autoregressive text-to-speech framework that addresses error accumulation in LLM-DiT style generation by enriching the speech representation itself with semantic supervision, rather than adding auxiliary modules or architectural complexity. The system is built around RedAE, a continuous speech tokenizer distilled from a frozen audio understanding model, and ships in two variants: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for unified voice cloning, instruction-controlled voice design, and speech editing. The authors report state-of-the-art average intelligibility and speaker similarity on Seed-TTS-Eval and MiniMax-MLS-Test, and the best results among compared systems on InstructTTSEval and Ming-Freeform-Audio-Edit.

Motivation and positioning

Continuous autoregressive TTS models bypass vector quantization to preserve fine-grained acoustic detail while inheriting the instruction-following capability of pretrained text LLMs. However, because continuous latents live in an unbounded space, small prediction errors compound across autoregressive steps, producing timbre shifts and prosody collapse. Prior remedies fall into two camps: injecting semantics into continuous representations (e.g., Ming-UniAudio's additional semantic module, VibeVoice's separate semantic VAE, dots.tts's ASR objective during VAE training), or regularizing the feature space during modeling (e.g., VoxCPM's FSQ bottleneck). Both add tokenizer-training stages or architectural complexity. FireRedTTS3 instead distills semantics directly into the latent space via a frozen teacher, keeping both tokenizer training and the downstream generator simple.

The RedAE tokenizer

RedAE processes 24 kHz waveforms at 50 Hz (480-sample frames), applies two cascaded Qwen3-style Transformers — one for contextual modeling at 50 Hz, one that downsamples to 25 Hz via attention pooling over two-frame patches — and reconstructs via a Qwen3-style decoder that upsamples back to 50 Hz and predicts an STFT spectrum converted to waveform with iSTFT. Notably, KL regularization is omitted to avoid acoustic over-compression.

The key design choice is semantic distillation: the authors first train FireRedAudio, an audio understanding model covering ASR and speaker verification, then freeze its Audio Encoder as a semantic teacher during RedAE training. An MSE loss between 25 Hz RedAE latents and teacher features grounds the latent space semantically; the teacher is discarded after tokenizer training. Training uses a GAN objective (adversarial, multi-scale Mel reconstruction, feature matching, and semantic losses) for 550k steps on 32 H800 GPUs over 500k hours of mixed audio (50% clean speech, 25% noisy speech, 10% sound effects, 15% music). Encoder and Decoder are jointly optimized in a single stage, which is the paper's central claim of simplicity relative to multi-stage pipelines.

The LLM-DiT generation framework

Both variants share three components. An Aggregator compresses 25 Hz RedAE representations into 6.25 Hz latent patches using attention pooling. A Backbone Transformer, initialized from a pretrained Qwen3 text model, autoregressively models interleaved text tokens and latent patches; its final hidden states condition a DiT module (with AdaLN timestep injection) that performs patch-level denoising. At each step, the DiT receives a noisy 4-frame current patch plus 12 frames of clean historical latents, preserving temporal continuity. Classifier-free guidance is enabled by dropping the Backbone condition (and speaker condition when present) with probability 0.1.

FireRedTTS3-Base initializes its Backbone from Qwen3-1.7B-Base, prepends a CAM++ speaker embedding and a language tag, and trains with flow-matching plus stop-prediction losses. Stage one uses 2.6M hours of Chinese and English speech for 170k steps; stage two extends coverage with 560k hours spanning 24 languages and 21 Chinese dialects.

FireRedTTS3-Instruct initializes from the instruction-tuned Qwen3-1.7B, retains its text head, and adopts ChatML formatting with task-specific system prompts. Instructions are first converted into a structured textual plan — twelve concrete acoustic attributes for voice design, or a target transcription plus edit-region mask for editing — before guiding synthesis. It adds a text loss to the training objective and undergoes a second stage of 330k hours of voice-design and editing data for 40k steps. Unlike Base, it uses no explicit speaker embeddings or language tags.

Voice cloning results

On Seed-TTS-Eval, FireRedTTS3-Base achieves the best average error rate (3.04%) and average speaker similarity (78.8%) among compared systems, ranking second on Test-EN WER (1.64%) and Test-Hard CER (6.50%), and first on speaker similarity for both Test-ZH (80.9%) and Test-EN (77.2%). The authors attribute this to RedAE providing a stable modeling target for text-speech alignment while avoiding quantization-induced acoustic loss.

On MiniMax-MLS-Test across 24 languages, FireRedTTS3-Base attains the lowest average error rate (3.75%) and highest average speaker similarity (84.8%), ranking first or second in 22 of 24 languages on similarity. Two observations strengthen the generality claim: Portuguese and Ukrainian appear in neither the Audio Encoder nor RedAE training data, yet the model achieves competitive error rates (Ukrainian WER of 0.55% is the best reported), indicating the representations generalize to unseen languages. The paper concedes that the uniformly high Cantonese error rates across all systems stem from Whisper-large-v3's limited Cantonese recognition rather than synthesis quality, so those figures should not be taken at face value.

Instruction-following voice design

On InstructTTSEval, FireRedTTS3-Instruct leads all six subtasks, reaching 85.8/82.0/69.7 on Chinese APS/DSD/RP and 80.7/82.3/72.0 on English, ahead of Qwen3-TTS-VD and Ming-Omni-TTS-16B-A3B. The intermediate planning step is credited with disambiguating complex parameter-style instructions and translating abstract style descriptions into explicit acoustic attributes. One methodological caveat: because the official evaluation toolkit relies on Gemini-2.5-pro-preview, which was inaccessible, all systems were re-scored with Gemini-2.5-pro, so absolute numbers may not be comparable to previously published results on this benchmark.

Speech editing results

For acoustic editing on Ming-Freeform-Audio-Edit, FireRedTTS3-Instruct substantially outperforms Ming-UniAudio-Edit: volume alteration reduces RAE from 14.9% to 3.58% (Chinese) with SIM improving from 0.86 to 0.92, speed alteration cuts Chinese WER from 5.88% to 2.27%, and pitch alteration improves SIM from 0.36 to 0.51 (Chinese).

For semantic editing, the model shows consistent performance across deletion, insertion, and substitution in both template-based ("basic") and free-form ("open") settings. In open-setting deletion, it lowers Chinese WER from 22.92% to 16.31% and raises accuracy from 82.92% to 89.32%. Averaged over basic+open settings, it achieves 87.27%/78.91% editing accuracy (Chinese/English) versus 82.91%/71.06% for Ming-UniAudio-Edit, with lower unedited-region WERs (6.49%/20.90%) confirming faithful preservation of untouched content and speaker identity. These results support the claim that RedAE's combination of semantic grounding and acoustic fidelity enables precise edit localization without dedicated editing architectures.

Limitations and open questions

Several caveats bear directly on the reported results. The Cantonese intelligibility figures are confounded by the evaluator (Whisper-large-v3) rather than the synthesizer, and the same evaluator-dependence applies to all WER/CER metrics. The InstructTTSEval comparison required substituting Gemini-2.5-pro for the official judge model, introducing potential scoring inconsistency with prior work. The semantic distillation depends on the quality and language coverage of the FireRedAudio teacher; generalization to unseen languages was demonstrated only for Portuguese and Ukrainian, and the mechanism behind this transfer is not analyzed. The paper does not ablate the contribution of individual loss terms (e.g., λsem\lambda_{sem} sensitivity), the effect of patch size choices (4-frame noisy context, 12-frame history), or inference-time computational cost relative to discrete-token baselines. Whether the frozen-teacher approach scales to richer instruction sets beyond the twelve acoustic attributes used for voice design remains unexamined.

Conclusion

FireRedTTS3 demonstrates that semantic enrichment of continuous speech representations — achieved simply by MSE distillation from a frozen, task-diverse audio encoder during single-stage tokenizer training — is sufficient to stabilize LLM-DiT autoregressive generation against error accumulation. This representation-level intervention yields competitive zero-shot voice cloning across 24 languages, and, combined with intermediate textual planning, enables a single model to unify voice cloning, instruction-controlled voice design, and localized speech editing with strong benchmark results.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We found no open problems mentioned in this paper.