Cross-Lingual F5-TTS for Zero-Shot Voice Cloning

• The paper introduces a transcript-free framework for cross-lingual F5-TTS using flow matching with Diffusion Transformer and ConvNeXt refinement to achieve near-human voice cloning.
• It demonstrates lightweight input adaptation and low-resource multilingual fine-tuning to extend the model to new languages while preserving core voice characteristics.
• It addresses evaluation challenges by proposing methods like ASR-based Best-of-N selection and cross-family rank ensembles to improve synthesis reliability.

Cross-lingual F5-TTS denotes a line of research that extends flow-matching-based zero-shot text-to-speech from monolingual voice cloning to multilingual and language-agnostic synthesis. In its current form, the topic spans the original F5-TTS foundation model, which is a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer and ConvNeXt-refined text representations (Chen et al., 2024), transcript-free cross-lingual voice cloning that removes dependence on prompt transcripts (Liu et al., 18 Sep 2025), lightweight input adaptation for adding new languages while preserving existing capabilities (Chivereanu et al., 13 Dec 2025), and low-resource multilingual fine-tuning that turns an English F5-TTS checkpoint into a near-human polyglot system for Indian languages (Varadhan et al., 27 May 2025). Across these works, the central technical problem is the separation of linguistic content, speaker identity, style, and duration when the prompt language and target language differ.

1. Foundations in flow matching and earlier cross-lingual TTS

F5-TTS is introduced as a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer. Its text input is padded with filler tokens to the same length as input speech, and the denoising is then performed for speech generation. The model replaces the fragile text–speech coupling of earlier alignment-free systems with ConvNeXt V2 blocks that refine text representations before DiT-based acoustic generation, and it adds inference-time Sway Sampling to improve performance and efficiency (Chen et al., 2024). Trained on a public 100K hours multilingual dataset, the model is reported to exhibit highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency (Chen et al., 2024).

The cross-lingual setting addressed by later F5-TTS work has clear antecedents in earlier flow-based systems. A 2022 flow-based cross-lingual TTS method used a non-attentive Tacotron architecture in which the decoder was replaced with a normalizing flow network conditioned on speaker identity, so that both TTS and voice conversion could be performed by the same model (Ellinas et al., 2022). In the cross-lingual pipeline, acoustic features were first produced with a native speaker of the target language and then converted to the target speaker’s voice by the same model, with reported benefits in objective and subjective evaluations and positive results for speakers averaging 7.5 minutes of speech (Ellinas et al., 2022). A 2023 extension factorized the problem into a 4-stage framework: an upstream Voice Conversion model, converted target-locale data, a single-speaker acoustic TTS model, and a locale-independent vocoder, with reported robustness across model architectures, languages, speakers, and amounts of data, and particular benefit in low-resource settings (Piotrowski et al., 2023).

These earlier results suggest that modern cross-lingual F5-TTS inherits two design commitments from the pre-F5 flow literature: first, pronunciation and speaker identity should be disentangled; second, low-resource cross-lingual synthesis is easier when the system can reuse a shared acoustic generator or a shared speech-side teacher across languages (Ellinas et al., 2022).

2. Transcript-free cross-lingual voice cloning

The most direct formulation of Cross-Lingual F5-TTS appears in “Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis” (Liu et al., 18 Sep 2025). Its starting point is the observation that most current flow-matching-based TTS models still rely on reference transcripts corresponding to the audio prompt for synthesis, and that this dependency prevents cross-lingual voice cloning when audio prompt transcripts are unavailable, particularly for unseen languages (Liu et al., 18 Sep 2025). The paper identifies two technical obstacles: identifying word boundaries during training and determining appropriate duration during inference (Liu et al., 18 Sep 2025).

The proposed solution preprocesses audio prompts by forced alignment to obtain word boundaries, enabling direct synthesis from audio prompts while excluding transcripts during training (Liu et al., 18 Sep 2025). During training, the left segment of an utterance serves as the prompt, its transcript is discarded, and the right segment is masked and predicted. The core DiT-based flow-matching model remains the F5-TTS acoustic generator, but the conditioning regime changes: the prompt contributes speaker identity and style acoustically, while the target text supplies only the target-language content (Liu et al., 18 Sep 2025). To address duration modeling, the system trains speaking rate predictors at different linguistic granularities—phonemes per second, syllables per second, and words per second—and uses Gaussian Cross-Entropy for discrete speaking-rate classification (Liu et al., 18 Sep 2025).

The quantitative outcome is that the method matches the performance of F5-TTS while enabling cross-lingual voice cloning (Liu et al., 18 Sep 2025). On cross-lingual test-en, the phoneme-level predictor $M1$ gives WER $= 2.496\%$, SIM $= 0.543$, and UTMOS $= 3.069$; on cross-lingual test-zh, the syllable-level predictor $M2$ gives WER $= 1.801\%$, SIM $= 0.565$, and UTMOS $= 2.503$ (Liu et al., 18 Sep 2025). The same study reports that fine-grained units outperform word-level predictors for duration estimation, and that speaking rate predictors trained only on English and Chinese generalize to prompts in German, French, Hindi, and Korean (Liu et al., 18 Sep 2025).

A concise way to view the method is that it removes prompt transcripts from both the training objective and the inference interface, replacing transcript-derived duration heuristics with acoustic speaking-rate estimation. This makes the prompt language largely irrelevant to the voice-cloning pathway, while keeping the target language explicit on the text side (Liu et al., 18 Sep 2025).

3. Language extension and low-resource adaptation

Cross-lingual F5-TTS also includes work on extending an existing F5-TTS checkpoint to new languages without retraining the full model. The clearest example is “F5-TTS-RO: Extending F5-TTS to Romanian TTS via Lightweight Input Adaptation” (Chivereanu et al., 13 Dec 2025). That system keeps the original weights frozen, appends a sub-network to the model, and trains it as an extension for the textual embedding matrix of the text encoder (Chivereanu et al., 13 Dec 2025). For simplicity, it relies on the ConvNeXt module implemented in F5-TTS to model the co-dependencies between the new character-level embeddings, and the module serves as a “soft” letter-to-sound layer that converts Romanian text into a continuous representation usable by the frozen backbone (Chivereanu et al., 13 Dec 2025). Subjective evaluation with 20 human listeners covers audio similarity between reference and generated speech, pronunciation and naturalness, and Romanian-English code-switching; the reported conclusion is that the approach maintains voice cloning capabilities and enables, to a certain extent, code-switching within the same utterance, although residual English accent characteristics remain (Chivereanu et al., 13 Dec 2025). Objectively, RO-F5TTS reports WER $= 5.27\%$ against $5.77\%$ for MMS-TTS-RON, while its cosine speaker-similarity mean is $= 2.496\%$0, compared with $= 2.496\%$1 for full fine-tuning (Chivereanu et al., 13 Dec 2025).

A different adaptation regime appears in “Phir Hera Fairy,” which evaluates how the English F5-TTS model adapts to 11 Indian languages (Varadhan et al., 27 May 2025). The paper compares training from scratch, fine-tuning English F5 on Indian data, and fine-tuning on both Indian and English data to prevent forgetting (Varadhan et al., 27 May 2025). The reported result is that fine-tuning with only Indian data proves most effective, and the resultant IN-F5 is a near-human polyglot that enables speakers of one language, such as Odia, to fluently speak in another, such as Hindi (Varadhan et al., 27 May 2025). On the main comparison table, overall MUSHRA is $= 2.496\%$2 for EN$= 2.496\%$3IN, $= 2.496\%$4 for EN$= 2.496\%$5EN+IN, $= 2.496\%$6 for training from scratch, and $= 2.496\%$7 for human speech (Varadhan et al., 27 May 2025). In the seen-speaker voice-cloning condition, IN-F5 reports naturalness $= 2.496\%$8 and speaker similarity $= 2.496\%$9, compared with human values $= 0.543$0 and $= 0.543$1 (Varadhan et al., 27 May 2025). The same work also shows that unseen languages such as Bhojpuri and Tulu can be synthesized using a human-in-the-loop approach for zero-resource TTS via synthetic data generation (Varadhan et al., 27 May 2025).

System	Mechanism	Reported outcome
F5-TTS-RO	Frozen backbone plus lightweight input-level adapter	Maintains voice cloning and enables, to a certain extent, code-switching (Chivereanu et al., 13 Dec 2025)
IN-F5	Fine-tune English F5-TTS on Indian data only	Near-human polyglot; speakers of one language can fluently speak in another (Varadhan et al., 27 May 2025)

A plausible implication is that F5-TTS’s main multilingual bottleneck is often the text interface rather than the acoustic generator. That interpretation is consistent with the Romanian adapter, which changes only the input side (Chivereanu et al., 13 Dec 2025), and with the Indian multilingual study, where English pretraining supplies most of the acoustic prior while target-language fine-tuning specializes the text-to-speech mapping (Varadhan et al., 27 May 2025).

4. Prompt robustness, in-the-wild conditions, and hybrid cross-lingual designs

One line of work treats standard F5-TTS as insufficiently robust in difficult cross-lingual prompt conditions. “PFluxTTS: Hybrid Flow-Matching TTS with Robust Cross-Lingual Voice Cloning and Inference-Time Model Fusion” explicitly targets weak cross-lingual voice cloning in flow-matching TTS and introduces a dual-decoder design combining duration-guided and alignment-free models through inference-time vector-field fusion, robust cloning using a sequence of speech-prompt embeddings in a FLUX-based decoder, and a modified PeriodWave vocoder with super-resolution to 48 kHz (Pankov et al., 4 Feb 2026). On cross-lingual in-the-wild data, it reports WER $= 0.543$2 versus $= 0.543$3 for F5-TTS, CER $= 0.543$4 versus $= 0.543$5, and SPK-SIM $= 0.543$6 versus $= 0.543$7 (Pankov et al., 4 Feb 2026). The same paper states that it matches ChatterBox in naturalness with MOS $= 0.543$8, achieves 23% lower WER than ChatterBox ($= 0.543$9 vs. $= 3.069$0), and surpasses ElevenLabs in speaker similarity by $= 3.069$1 SMOS (Pankov et al., 4 Feb 2026). In this comparison, the main limitation of baseline F5-TTS is not naturalness in clean conditions but instability under cross-lingual, long, noisy, or transcript-free prompts.

A complementary robustness direction is “F5-TTS-DPS,” which adds Exponential Moving Average during supervised fine-tuning and dual-scoring prompt selection using a LLM and a large audio LLM (Sun et al., 22 May 2026). The prompt-selection stage uses Qwen2.5-Omni for audio-scoring prompt selection and Qwen3-30B-A3B for semantic alignment verification between target text and reference text (Sun et al., 22 May 2026). The report is not framed as a cross-lingual study, but it is directly relevant to multilingual in-the-wild deployment because it addresses noisy prompts and alignment mismatch in the reference data. On the development set, F5-TTS-DPS achieves UTMOS $= 3.069$2 and speaker similarity $= 3.069$3, and the report states that it achieves the best a-DCF scores across three advanced SASV systems among all submissions (Sun et al., 22 May 2026).

These systems point in two different directions. One direction strengthens prompt conditioning and alignment inside the flow-matching generator itself, as in vector-field fusion and sequence prompt embeddings (Pankov et al., 4 Feb 2026). The other improves prompt quality and training stability around the generator, as in EMA and dual-scoring prompt selection (Sun et al., 22 May 2026). Both are now part of the practical meaning of cross-lingual F5-TTS.

5. Evaluation methodology and the Best-of-$= 3.069$4 confound

A major methodological development is the finding that Best-of-$= 3.069$5 TTS evaluation is confounded by ASR family alignment (Yu et al., 9 Jul 2026). In the studied setup, F5-TTS generates $= 3.069$6 candidates, an ASR verifier scores each candidate with a joint WER+CER criterion against the target text, and the lowest-ranked candidate is selected. The paper shows that on LibriSpeech-PC test-clean with F5-TTS, verifier rankings reverse across Whisper, wav2vec 2.0, and HuBERT evaluators, and same-family verifier-evaluator pairs recover $= 3.069$7–$= 3.069$8 more oracle headroom than cross-family pairs despite near-identical representations with linear CKA $= 3.069$9 (Yu et al., 9 Jul 2026).

Under the official F5-TTS evaluator, the single-shot baseline is WER $M2$0. The best single verifier, distil-v3, drives this to $M2$1, a $M2$2 relative reduction (Yu et al., 9 Jul 2026). However, the broader conclusion is not that a single verifier should be trusted, but that cross-evaluator triangulation should become default reporting practice (Yu et al., 9 Jul 2026). The paper proposes two cross-family rank ensembles—rank-averaging and conjunctive max-rank—and reports that they attain the lowest mean WER across three independent evaluators, $M2$3 at $M2$4, which is a $M2$5 relative improvement over F5-TTS, with no measurable degradation under automatic SIM-o or UTMOS metrics (Yu et al., 9 Jul 2026).

For cross-lingual F5-TTS, this result matters because ASR-based verification is often the easiest way to compare outputs across languages and prompts. The confound means that apparent progress in transcript fidelity can be a property of evaluator choice rather than synthesis quality. A plausible implication is that multilingual F5-TTS studies should report more than one evaluator family whenever ASR-based BoN selection or ASR-based intelligibility metrics are central to the claim.

6. Limitations and emerging research directions

The present literature leaves several limitations explicit. First, transcript dependence was a structural limitation of earlier flow-matching F5-TTS prompting and is only partially removed by the transcript-free framework of Cross-Lingual F5-TTS (Liu et al., 18 Sep 2025). Second, lightweight input adaptation preserves existing capabilities but does not fully remove residual accent; F5-TTS-RO reports that residual English accent characteristics remain (Chivereanu et al., 13 Dec 2025). Third, baseline F5-TTS can be highly brittle in in-the-wild cross-lingual prompt settings, as indicated by the large gap between F5-TTS and PFluxTTS on cross-lingual WER and CER (Pankov et al., 4 Feb 2026). Fourth, evaluation itself is unstable when a single ASR family is used both for verification and judgment (Yu et al., 9 Jul 2026).

The research trajectory nevertheless points toward a coherent design space. One direction is transcript-free prompt handling plus speaking-rate prediction, which aims at language-agnostic voice cloning without prompt transcripts (Liu et al., 18 Sep 2025). A second is language extension by frozen-backbone adapters, which offers a way to add new scripts and character inventories while retaining English and Chinese support (Chivereanu et al., 13 Dec 2025). A third is low-resource multilingual specialization, where English F5-TTS acts as a high-resource initializer for languages with much smaller corpora (Varadhan et al., 27 May 2025). A fourth is robust prompt conditioning and hybrid flow-matching architectures, which target the failure modes of alignment-free prompting in noisy or strongly cross-lingual conditions (Pankov et al., 4 Feb 2026).

Taken together, these results indicate that cross-lingual F5-TTS is no longer just multilingual zero-shot cloning with foreign-language text. It is a composite research program concerned with transcript-free conditioning, duration estimation, low-resource language addition, polyglot and code-mixed fluency, prompt selection in noisy data, and evaluator-robust measurement. The field’s current evidence supports the view that F5-style flow matching remains a strong backbone for cross-lingual speech synthesis, but only when paired with explicit solutions for duration, prompting, language extension, and evaluation (Chen et al., 2024).