Raon-OpenTTS: Open Models for Robust Text-to-Speech

• Raon-OpenTTS is a robust open-source text-to-speech system that integrates large-scale audio corpora with diffusion transformer models for high-quality synthesis.
• It uses a model-based filtering pipeline to curate a high-quality subset from a 615K-hour speech pool, ensuring enhanced intelligibility and noise resilience.
• The project promotes reproducibility by releasing full training code, data manifests, and standardized benchmarks for direct comparison with proprietary systems.

Searching arXiv for the specified paper and closely related TTS works to ground the article with current citations. Raon-OpenTTS is a family of open-weight text-to-speech models, an associated large-scale open speech corpus, and a robustness-oriented evaluation benchmark introduced in “Raon-OpenTTS: Open Models and Data for Robust Text-to-Speech” (Kim et al., 20 May 2026). It is defined by three tightly coupled assets: Raon-OpenTTS-Pool, a 615K-hour open English speech collection; Raon-OpenTTS-Core, a curated 510K-hour subset derived by model-based filtering; and Raon-OpenTTS itself, a series of diffusion transformer (DiT)-based TTS models from 0.3B to 1B parameters trained entirely on open data. The project is positioned around reproducibility, robustness across non-studio conditions, and direct comparison with both open and closed TTS systems under common benchmarks rather than around a single model checkpoint alone (Kim et al., 20 May 2026).

1. Composition of the project

Raon-OpenTTS is organized as a model family, a data pipeline, and an evaluation suite. The flagship model is Raon-OpenTTS-1B, while the surrounding data and benchmark assets are intended to make large-scale TTS training reproducible with open resources rather than proprietary corpora (Kim et al., 20 May 2026).

Component	Scale	Function
Raon-OpenTTS-Pool	615K hours, 240M segments	Large-scale open English speech pool
Raon-OpenTTS-Core	510K hours, 194M segments	Filtered high-quality training subset
Raon-OpenTTS-0.3B	336M parameters	Smaller DiT TTS variant
Raon-OpenTTS-1B	1048M parameters	Flagship DiT TTS variant
Raon-OpenTTS-Eval	clean, noisy, in-the-wild, expressive	Structured robustness benchmark

The project’s central claim is that open-source TTS models trained entirely on open, reproducible data can achieve competitive robustness and quality. In the reported comparisons, Raon-OpenTTS-1B is described as comparable to state-of-the-art closed-data models such as Qwen3-TTS and CosyVoice 3, even though those systems are trained on several million hours of proprietary speech data whereas Raon-OpenTTS is trained on hundreds of thousands of hours of open data (Kim et al., 20 May 2026).

This framing makes the project notable not merely as a checkpoint release but as an attempt to establish an open-data counterpart to closed TTS ecosystems. A plausible implication is that Raon-OpenTTS should be understood as infrastructure for training, benchmarking, and extension, not only as a single inference model.

2. Model architecture and generative formulation

Raon-OpenTTS is a Diffusion Transformer-based TTS system that closely follows the F5-TTS and DiTTo-TTS design, with a transformer backbone operating in latent acoustic space and a diffusion-style generative mechanism that denoises latent representations into speech features (Kim et al., 20 May 2026). Conditioning is on text and on reference audio for zero-shot speaker and style transfer.

The two disclosed configurations differ only in scale. The 0.3B model corresponds to the F5-TTS default configuration, with 22 transformer layers, 16 attention heads, attention dimension 1024, FFN dimension 2048, text embedding dimension 512, and 336M total parameters. The 1B model scales to 28 layers, 22 attention heads, attention dimension 1408, FFN dimension 5632, text embedding dimension 512, and 1048M total parameters (Kim et al., 20 May 2026).

The diffusion formulation is summarized by the standard forward noising process

$$x_t = \sqrt{\alpha_t}\,x_0 + \sqrt{1-\alpha_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$

and the noise-prediction objective

$$\mathcal{L}_{\text{diff}}(\theta) = \mathbb{E}_{x_0,\epsilon,t} \bigl[ \|\epsilon-\epsilon_\theta(x_t,t,y)\|_2^2 \bigr].$$

Here, $x_0$ denotes the target acoustic representation and $y$ the conditioning signal. The project description also states that the DiT uses cross-attention to fuse conditioning from text and reference audio into the denoising process (Kim et al., 20 May 2026).

The architectural emphasis is not solely on naturalness in clean settings. Because the model is trained on Raon-OpenTTS-Core, which includes multi-domain speech rather than only studio-clean data, the design is explicitly oriented toward robust zero-shot cloning under noisy, in-the-wild, and expressive conditions. This distinguishes the system from TTS lines optimized primarily for clean evaluation sets.

3. Data construction, filtering, and reproducibility

Raon-OpenTTS-Pool consists of 615,000 hours and 240 million segments of primarily English speech aggregated from publicly available English speech corpora and web-sourced recordings. The sources listed in the project description span read speech, audiobooks, voice-cloning corpora, lecture speech, meetings, telephony, large ASR corpora, and open web-sourced speech resources. Raon-OpenTTS-Core is the filtered subset produced by a model-based filtering pipeline, leaving 510K hours and 194M segments (Kim et al., 20 May 2026).

The filtering procedure is described as combining ASR-based text-speech alignment and WER filtering, non-intrusive speech quality scoring such as DNSMOS, and speaker or linguistic filters that remove non-English utterances, heavily code-mixed content, problematic segment lengths, overlapping speech, and heavy background music where possible. In conceptual form, the retained training set is

$$\mathcal{D}_{\text{Core}} = \bigl\{ x \in \mathcal{D}_{\text{Pool}} \;\big|\; \text{WER}(x)\le \tau_{\text{WER}}, \; q(x)\ge \tau_{\text{quality}}, \; \text{other predicates}(x) \bigr\}.$$

This data-centric construction is one of the project’s defining features. The description explicitly links the Pool-to-Core transition to the broader claim that quality is more important than raw quantity. In the reported ablations, training on unfiltered Pool data yields worse WER and more instability than training on filtered Core data, while scaling from 0.3B to 1B improves intelligibility and speaker similarity, especially in hard and noisy conditions (Kim et al., 20 May 2026).

The release is designed to preserve reproducibility. The repository at https://github.com/krafton-ai/RAON-OpenTTS is stated to include model checkpoints, full training code, data manifests and download scripts for reconstructing the pool from official sources, the filtering pipeline, and benchmark assets for Raon-OpenTTS-Eval (Kim et al., 20 May 2026). The component datasets retain their original licenses, and the project description notes that downstream use must comply with those licenses.

4. Evaluation methodology and benchmark results

Raon-OpenTTS is evaluated on Seed-TTS-Eval, CV3-Hard-EN, and the project’s own Raon-OpenTTS-Eval benchmark. The latter is structured around four acoustic conditions—clean, noisy, in-the-wild, and expressive—and includes reference audio plus target transcripts for zero-shot cloning evaluation (Kim et al., 20 May 2026).

The objective metrics are Word Error Rate (WER) for intelligibility and speaker similarity (SIM) computed as cosine similarity between reference and synthesized speaker embeddings. The reported definitions are

$\text{WER}=\frac{S+D+I}{N},$

where $S$, $D$, and $I$ are substitution, deletion, and insertion counts and $N$ is the number of reference words, and

$$\mathcal{L}_{\text{diff}}(\theta) = \mathbb{E}_{x_0,\epsilon,t} \bigl[ \|\epsilon-\epsilon_\theta(x_t,t,y)\|_2^2 \bigr].$$0

where $$\mathcal{L}_{\text{diff}}(\theta) = \mathbb{E}_{x_0,\epsilon,t} \bigl[ \|\epsilon-\epsilon_\theta(x_t,t,y)\|_2^2 \bigr].$$1 is the speaker embedding model (Kim et al., 20 May 2026).

Human evaluation is conducted through SMOS and CMOS protocols. SMOS asks raters to compare reference and generated audio on a 1–5 similarity scale. CMOS asks annotators to compare two synthesized samples on a $$\mathcal{L}_{\text{diff}}(\theta) = \mathbb{E}_{x_0,\epsilon,t} \bigl[ \|\epsilon-\epsilon_\theta(x_t,t,y)\|_2^2 \bigr].$$2 to $$\mathcal{L}_{\text{diff}}(\theta) = \mathbb{E}_{x_0,\epsilon,t} \bigl[ \|\epsilon-\epsilon_\theta(x_t,t,y)\|_2^2 \bigr].$$3 relative quality scale. The protocol uses U.S.-based Amazon Mechanical Turk annotators, six ratings per item, instructions to use headphones in a quiet environment, and a quality-control rule that discards responses when an annotator assigns identical scores to all five items on a page (Kim et al., 20 May 2026).

The flagship empirical results are concentrated in three benchmark statements. On Seed-TTS-Eval, Raon-OpenTTS-1B achieves a WER of 1.78% and a SIM of 0.749, ranking second on WER and first on SIM among recent open-weight TTS baselines. On CV3-Hard-EN, it achieves a WER of 6.15% and a SIM of 0.775, ranking first on both metrics. On Raon-OpenTTS-Eval, it attains the best average WER and SIM among all evaluated models and the second-best human preference as measured by CMOS (Kim et al., 20 May 2026).

These numbers define the project’s technical significance. The reported pattern is not simply high quality under clean conditions, but persistent performance under adverse acoustic conditions where many open baselines degrade more sharply.

5. Relation to adjacent research directions

Raon-OpenTTS occupies a specific point in the current open TTS landscape: large-scale open-data training for robust zero-shot cloning. Nearby research directions emphasize control, multilinguality, or low-resource transfer more than large-scale open English robustness.

“UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech” introduces a lightweight adaptation method for multilingual LLM-based TTS that adds LoRA to all Q/K/V/O self-attention projections and uses <PHON_START> and <PHON_END> tags to control segmental pronunciation and pitch accent at the phoneme level for Japanese while maintaining naturalness and speaker similarity in a zero-shot setting (Kato, 13 Aug 2025). Relative to Raon-OpenTTS, this points to a control interface that is far more explicit at the phonological level.

“VoiceSculptor: Your Voice, Designed By You” focuses on instruction-based voice design rather than large-scale open-data training. It integrates instruction-driven voice design, CoT-based attribute tokens, RAG over a 500k instruction database, and downstream prompt-based cloning through CosyVoice2, thereby targeting pitch, speaking rate, age, emotion, and style through natural-language control (Hu et al., 15 Jan 2026). This suggests a complementary axis of development that Raon-OpenTTS does not yet expose explicitly.

Low-resource adaptation is addressed directly in “LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition,” which combines pre-training on rich-resource languages, dual transformation between TTS and ASR, and knowledge distillation, and reports more than 98% intelligibility rate and above 3.5 MOS under extremely low-resource conditions (Xu et al., 2020). “Low-Resource Multilingual and Zero-Shot Multispeaker TTS” instead uses language agnostic meta learning to adapt a multilingual multispeaker system to a new language with 5 minutes of training data while retaining zero-shot speaker transfer (Lux et al., 2022). These works mark a different operational regime from Raon-OpenTTS: adaptation efficiency rather than large-scale open-data scaling.

Multilingual accent disentanglement appears in “Multilingual Multiaccented Multispeaker TTS with RADTTS,” which provides explicit control over accent, language, speaker, and fine-grained $$\mathcal{L}_{\text{diff}}(\theta) = \mathbb{E}_{x_0,\epsilon,t} \bigl[ \|\epsilon-\epsilon_\theta(x_t,t,y)\|_2^2 \bigr].$$4 and energy features without relying on bilingual training data (Badlani et al., 2023). By contrast, Raon-OpenTTS is English-focused and benchmarked primarily as a robust zero-shot TTS system rather than as a multilingual accent-control framework.

Finally, “How Open is Open TTS? A Practical Evaluation of Open Source TTS Tools for Romanian” documents the practical difficulties of adapting FastPitch, VITS, Grad-TTS, and Matcha-TTS to an underserved language, emphasizing installation barriers, preprocessing complexity, and compute constraints (Răgman et al., 25 Mar 2026). In that context, Raon-OpenTTS’s release of data manifests, filtering code, training code, and benchmark assets can be read as an attempt to make openness operational rather than nominal.

6. Limitations, open release, and future directions

The project description identifies several limitations. Raon-OpenTTS-Pool and Raon-OpenTTS-Core are English-focused, with demographic imbalance and domain imbalance. Although the training pool includes expressive speech and the benchmark evaluates expressive and in-the-wild conditions, explicit control over prosody, emotion, duration, and pitch is limited compared with carefully designed auto-regressive systems. The 1B DiT is also computationally heavy, especially with many diffusion steps, and real-time or on-device deployment may require step-count reduction, distillation, smaller variants, better codecs, or faster architectures (Kim et al., 20 May 2026).

The reported failure modes are similarly specific. In extremely noisy or incoherent reference audio, the model may overfit to noise or fail to capture the speaker. Very long or complex texts may degrade prosody and rhythm. Speaker similarity may also drop for accents or styles absent in training (Kim et al., 20 May 2026). These are not unusual failure cases for zero-shot cloning, but in Raon-OpenTTS they are framed as consequences of the current data distribution and inference cost structure rather than of a narrowly tuned clean-speech model.

Future work named in the project description includes multilingual extension, better prosody control through explicit prosody embeddings or duration control, faster inference through model compression and diffusion distillation, and data diversification to reduce bias and improve fairness (Kim et al., 20 May 2026). Taken together, these directions indicate that Raon-OpenTTS is best understood as a strong open baseline for robust TTS, with the surrounding dataset and benchmark designed to make subsequent work on multilinguality, controllability, and efficiency measurable under a common, reproducible framework.