Raon-OpenTTS-Eval: Robust & Phonology Evaluation
- Raon-OpenTTS-Eval is a text-to-speech evaluation framework designed to test robustness and phonological accuracy across diverse speech conditions.
- It uses a structured benchmark with four domains—clean, noisy, in-the-wild, and expressive—combining automatic metrics with human comparative ratings.
- Its multilingual extension audits phonological contrasts via classifier-based methods, revealing systematic issues such as underproduction in ATR phenomena.
Raon-OpenTTS-Eval is a text-to-speech evaluation framework centered on robustness under acoustic variation and, in its later elaboration, on phonological faithfulness. In the "Raon-OpenTTS: Open Models and Data for Robust Text-to-Speech" line of work, it is introduced as a structured benchmark for zero-shot voice cloning across clean, noisy, in-the-wild, and expressive speech, using automatic metrics and human comparative preference to assess intelligibility, speaker similarity, and perceived quality (Kim et al., 20 May 2026). In "Towards a Phonology-Informed Evaluation of Multilingual TTS," the same name is used for a broader evaluation suite blueprint that audits whether synthesized speech preserves language-specific sound contrasts conditioned by grammar, rather than merely sounding natural (Barman et al., 2 Jul 2026). Taken together, these two strands position Raon-OpenTTS-Eval as both a robustness benchmark and a diagnostic methodology for analyzing where multilingual TTS systems fail.
1. Origin and conceptual scope
Raon-OpenTTS-Eval was introduced alongside Raon-OpenTTS to address a specific gap in contemporary TTS assessment: systems that perform well on studio-quality prompts often degrade when the reference speech is noisy, far-field, conversational, or emotionally expressive. The benchmark therefore targets robustness under domain shift in speaker, environment, and style, and is designed to complement existing public evaluations rather than duplicate them (Kim et al., 20 May 2026).
Its scope is defined by three properties. First, it captures multiple acoustic dimensions within one benchmark through four conditions: clean, noisy, in-the-wild, and expressive speech. Second, it aligns evaluation with a realistic zero-shot voice cloning workflow in which a short reference prompt in a given acoustic condition conditions synthesis of new text in the same target style and environment. Third, it combines automatic metrics with human comparative preference, avoiding exclusive dependence on either ASR-derived intelligibility or subjective naturalness (Kim et al., 20 May 2026).
A later extension reframes the benchmark’s purpose for multilingual TTS. The central claim is that naturalness does not guarantee preservation of phonological contrasts that distinguish words from their grammatical forms. Standard metrics such as MOS do not test for this. The phonology-informed formulation therefore treats human speech as a benchmark domain, trains an acoustic-to-phonology classifier on human speech, transfers that classifier to synthesized speech, and measures whether the TTS output realizes the intended categories faithfully (Barman et al., 2 Jul 2026).
This dual lineage gives the framework a layered meaning. In its original English benchmark form, Raon-OpenTTS-Eval measures robustness across acoustic conditions. In its phonology-informed formulation, it measures whether TTS preserves linguistically meaningful contrasts with measurable acoustic cues. This suggests a transition from robustness benchmarking to a more diagnostic evaluation stack.
2. Benchmark composition and evaluation protocol
The original benchmark consists of four condition-specific subsets assembled from widely used open English datasets. These conditions are conceptually defined rather than exhaustively enumerated in the paper, but their intended evaluation roles are explicit (Kim et al., 20 May 2026).
| Condition | Description | Evaluation emphasis |
|---|---|---|
| Clean | Studio-quality read speech with minimal noise and reverberation | Baseline cloning quality |
| Noisy | Speech with additive or environmental noise | Noise robustness |
| In-the-wild | Conversational, far-field, or telephone speech | Device and environment shift |
| Expressive | Emotion-rich, prosodic, or acted speech | Style and prosody transfer |
The benchmark is framed as zero-shot speaker/style/environment cloning. A short reference audio from one of the four conditions is used to condition synthesis for a target text. Automatic scoring uses Word Error Rate and speaker similarity, while human evaluation uses comparative mean opinion score. The core formulas are standard:
and
where is cosine similarity between the speaker embeddings of the reference and synthesized audio (Kim et al., 20 May 2026).
Human evaluation is sampled uniformly across conditions in the published appendix protocol. For each acoustic condition, 30 items are randomly sampled for subjective tests, each rated by 6 annotators from US MTurk; presentation order is randomized, and pages with flat responses are excluded by basic quality control (Kim et al., 20 May 2026). The comparative scale for CMOS is the 7-point set , with positive values indicating that Audio B is preferred over Audio A. The benchmark also uses SMOS as a 1–5 score assessing similarity in speaker, style, and environment, although WER, SIM, and CMOS are the most prominent reported dimensions (Kim et al., 20 May 2026).
The high-level benchmark composition is reported more clearly than its exact inventory statistics. Precise counts for utterances, hours, speakers, accents, and text stratification variables are not explicitly reported in the paper. Likewise, the paper does not explicitly state benchmark audio technical parameters such as sampling rate, bit depth, or numeric SNR ranges (Kim et al., 20 May 2026). These omissions are substantive because they limit exact replication of subset composition even when the evaluation logic is reproducible.
3. Reported performance and interpretation
On Raon-OpenTTS-Eval, Raon-OpenTTS-1B achieves the best average WER and SIM among all evaluated models, and the second-best human preference as measured by CMOS (Kim et al., 20 May 2026). The baselines named in the paper are Qwen3-TTS and CosyVoice 3, both described as strong systems trained on proprietary data (Kim et al., 20 May 2026).
The benchmark’s role in the Raon-OpenTTS paper is best understood in the context of cross-benchmark results. On Seed-TTS-Eval, Raon-OpenTTS-1B achieves a WER of 1.78% and a SIM of 0.749, ranking second on WER and first on SIM among recent open-weight TTS baselines. On CV3-Hard-EN, it achieves a WER of 6.15% and a SIM of 0.775, ranking first on both metrics (Kim et al., 20 May 2026). These figures are not scores from Raon-OpenTTS-Eval itself, but they provide context for the system’s performance profile.
The Raon-OpenTTS-Eval ranking is analytically significant because the system leads on average WER and SIM but not on CMOS. The reported interpretation is that small differences in naturalness, prosody, or timbre quality may cause human listeners to prefer another system despite equivalent or superior intelligibility and speaker similarity (Kim et al., 20 May 2026). This establishes a methodological point that recurs throughout later work: automatic intelligibility and similarity metrics do not collapse into a single perceptual dimension.
The paper also emphasizes robustness trends rather than only leaderboard position. WER typically degrades in noisy and in-the-wild conditions as ASR errors increase, while SIM may decrease if a synthesizer normalizes away environmental cues or fails to capture expressive prosody. The benchmark is therefore intended to test whether a model clones both speaker identity and acoustic context, not merely lexical content (Kim et al., 20 May 2026).
A key limitation of the published results is granularity. Exact numeric results for Raon-OpenTTS-Eval, as opposed to overall rankings, are not provided in the supplied text, and per-condition subset metrics are not included there either (Kim et al., 20 May 2026). The benchmark is thus well specified at the level of purpose, conditions, and protocol, but less complete at the level of published score tables.
4. Phonology-informed evaluation and Assamese ATR auditing
The phonology-informed formulation of Raon-OpenTTS-Eval begins from a different failure mode. Neural TTS can sound natural across languages while systematically failing to preserve phonological contrasts that are conditioned by grammar. The case study in the paper is Assamese advanced tongue root vowel harmony, where [+ATR] suffix vowels trigger harmony on stem vowels and constrain co-occurrence of vowel qualities within a word (Barman et al., 2 Jul 2026).
The evaluation pipeline trains a classifier on human speech to map acoustic features to phonological labels and then transfers that classifier to TTS output. For ATR, the acoustic literature predicts that [+ATR] vowels exhibit lower first formant and narrower F1 bandwidth than [-ATR] vowels. The paper uses this relation to construct an audit of whether synthesized vowels are acoustically realized on the correct side of a human-derived decision boundary (Barman et al., 2 Jul 2026).
The two central token-level metrics are error rate and faithfulness score:
where is the gold phonological label and is the label predicted from acoustics. For directional bias, the paper defines an underproduction measure such as
with the complementary overgeneration probability defined analogously (Barman et al., 2 Jul 2026).
The Assamese results establish three claims that motivate the design. First, domain transfer works: a simple logistic regression trained on human vowels transfers to Assamese MMS TTS with minimal loss. Reported scores are H→H accuracy of approximately 81.7% with macro-F1 of approximately 0.81, and H→TTS accuracy of approximately 83% with macro-F1 of approximately 0.81 (Barman et al., 2 Jul 2026). Second, the faithfulness audit reveals a directional bias. Human out-of-fold mismatch is approximately 0.185 with nearly symmetric overgeneration and underproduction, whereas TTS mismatch is approximately 0.164 but is dominated by underproduction of [+ATR], approximately 0.142 versus approximately 0.021 overgeneration, yielding an approximately 7:1 asymmetry absent in human speech (Barman et al., 2 Jul 2026). Third, the misrealization is concentrated: roughly one in three tokens of the [+ATR] mid vowels /e/ and /o/ are classified as [-ATR] by the human-trained model (Barman et al., 2 Jul 2026).
At the word level, the divergence between intended phonology and produced acoustics becomes sharper. The paper defines a harmony function over per-vowel labels and evaluates whether word-level harmony classification based on predicted labels matches that based on gold labels. On transfer to TTS, acoustic plus predicted ATR sequence features yield macro-F1 of approximately 0.62, whereas acoustic plus gold ATR sequence features yield approximately 0.49 (Barman et al., 2 Jul 2026). The paper’s interpretation is direct: intended phonology is not a reliable proxy for produced acoustics. In practice, predicted labels are the better input for harmony classification of synthesized speech (Barman et al., 2 Jul 2026).
This part of Raon-OpenTTS-Eval therefore changes the unit of evaluation. Instead of asking only whether the speech is intelligible or similar to a prompt, it asks whether acoustics instantiate the contrastive structure that the language requires.
5. Architecture, metrics, and multilingual extensibility
The phonology-informed documentation specifies Raon-OpenTTS-Eval as a modular suite with five core modules: data ingestion, feature extraction, classifier training and inference, metric computation, and reporting (Barman et al., 2 Jul 2026). Data ingestion loads human and TTS audio, metadata, phonological specifications, and segmentation, with support for standardized formats such as TextGrid, CSV, and JSON. Feature extraction includes formants, F1 bandwidth, duration, F0, and MFCCs, together with normalization and outlier filtering. Classifier training is performed on human data with speaker-disjoint cross-validation, and inference is then run on TTS audio. Metric computation covers token-level faithfulness, directional bias, confusion-derived metrics, word-level harmony accuracy, transfer losses, and optional distributional divergences. Reporting includes language-pack-aware dashboards, confusion matrices, error heatmaps by vowel identity, per-word harmony decisions, and exportable JSON summaries (Barman et al., 2 Jul 2026).
The acoustic and annotation requirements are correspondingly explicit. The suite assumes parallel utterances from human speakers and synthesized outputs over the same stimuli, phoneme-level segmentation with vowel boundaries, underlying phonological specifications, and language-specific word-level labels. For formants in human speech, it uses speaker-level Lobanov z-score normalization; for single-voice TTS, it applies global z-scoring within the corpus. Outlier handling excludes tokens outside plausible acoustic bounds, with example bounds reported for F1, F2, F3, and B1 (Barman et al., 2 Jul 2026).
The generalization mechanism is the "language pack." Each language pack defines the contrast, its acoustic correlates, feature extraction configuration, normalization rules, and token- and word-level metrics (Barman et al., 2 Jul 2026).
| Contrast type | Primary correlates | Example metrics |
|---|---|---|
| Tone | F0 trajectory, range, slope, turning points | F0 trajectory RMSE, contour accuracy |
| Vowel length | Segment duration | Duration error, length confusion |
| Nasality | Nasal formant, antiformants, bandwidth changes | Nasal/oral accuracy, antiformant detection |
| Aspiration | VOT, aspiration noise spectrum, H1–H2 | VOT RMSE, class-wise F1 |
| Ejectives/Glottalization | Burst amplitude, high-frequency energy, negative VOT | Classification accuracy, confusion analysis |
| Other vowel harmony | F2 shifts, rounding effects on F2/F3 | Token fidelity, word-level harmony accuracy |
The paper explicitly states that Raon-OpenTTS-Eval should complement MOS, STOI/PESQ, and ASR-based intelligibility rather than replace them (Barman et al., 2 Jul 2026). A balanced profile therefore includes naturalness, intelligibility, phonological faithfulness scores, directional biases, and word-level harmony accuracy. The stated interpretive principle is that reports should avoid collapsing all dimensions into a single scalar and should instead present a dashboard of scores with targeted error analysis (Barman et al., 2 Jul 2026).
6. Relation to adjacent evaluation paradigms
Raon-OpenTTS-Eval occupies a particular niche within the broader TTS evaluation literature. Its original benchmark form is closest to robustness-oriented cloning evaluations, but its later formulation extends into phonological auditing and multilingual diagnostic analysis.
ClonEval provides an adjacent open benchmarking layer for cloning-specific evaluation. It standardizes single-reference voice cloning, computes speaker similarity via pretrained WavLM embeddings and cosine similarity, and avoids human rating by design. The ClonEval paper explicitly presents the framework as a component that can plug into broader open TTS evaluation workflows such as Raon-OpenTTS-Eval, particularly on the axis of speaker identity transfer and emotion transfer (Christop et al., 29 Apr 2025). Relative to Raon-OpenTTS-Eval, ClonEval is narrower in metric scope and intentionally omits MOS- or ASR-based intelligibility in its core protocol.
PashtoTTS-Bench develops a different critique of single-metric evaluation, this time for low-resource non-Latin-script languages. Its INSV framework separates Intelligibility, Naturalness, Script fidelity, and Verification, arguing that a single ASR round-trip WER can fail when a system produces no audio, speaks the wrong language, preserves target script only in the ASR transcript, or is unnatural to native listeners (Rahman, 26 May 2026). This is methodologically aligned with the Raon-OpenTTS-Eval claim that intelligibility or naturalness alone is insufficient, although the failure modes emphasized are different.
The Rapid Prosody Transcription paradigm addresses another blind spot: localization of prosodic errors within an utterance. Instead of assigning a global MOS, listeners mark transcript words where the intonation sounds incorrect, yielding a word-level probabilistic error map. The reported strong negative correlation between PMOS and error rates, with pooled Pearson , shows that localized diagnostic evaluation can track system rankings while revealing boundary-specific and context-specific prosodic failures that MOS alone obscures (Gutierrez et al., 2021). This is conceptually similar to Raon-OpenTTS-Eval’s move from global scores toward targeted diagnostics.
Large-scale pairwise multilingual preference evaluation provides a complementary human-centered perspective. The Indic multilingual framework based on over 120K pairwise comparisons uses a locked overall-preference step followed by six perceptual dimensions and Bradley–Terry modeling to derive statistically grounded multilingual leaderboards. Its analysis identifies expressiveness and intelligibility as the strongest drivers of overall preference once robustness to hallucinations and noise is adequate (Anand et al., 23 Apr 2026). That result is relevant to Raon-OpenTTS-Eval because it clarifies which perceptual dimensions dominate human judgments even when automatic metrics are strong.
A plausible implication of these adjacent frameworks is that Raon-OpenTTS-Eval is best understood not as a single fixed benchmark, but as a hub architecture into which cloning, prosody, multilingual human preference, and language-specific phonological auditing can be integrated.
7. Limitations, controversies, and future directions
The original Raon-OpenTTS-Eval benchmark is English-focused, and multilingual robustness is not covered in the initial release (Kim et al., 20 May 2026). This is not a minor gap: the phonology-informed extension is motivated precisely by multilingual settings in which language-specific contrasts may be lost while surface naturalness remains high (Barman et al., 2 Jul 2026). The benchmark’s current identity is therefore split between a published English robustness benchmark and a multilingual diagnostic blueprint.
Several reporting limitations are explicit. Exact numeric results for Raon-OpenTTS-Eval are not given in the supplied text beyond overall rankings; per-condition subset metrics and detailed composition statistics are not included there; and the exact ASR and speaker encoder versions are not specified (Kim et al., 20 May 2026). On the phonology-informed side, the approach requires controlled human benchmark corpora, phoneme-level segmentation, language-specific phonological specifications, and well-motivated acoustic correlates, which makes deployment substantially more demanding than applying WER or SIM alone (Barman et al., 2 Jul 2026).
The framework also inherits known methodological sensitivities from related evaluation paradigms. ASR-based intelligibility and embedding-based similarity are backend-dependent, as emphasized in both the Raon-OpenTTS and PashtoTTS-Bench discussions (Kim et al., 20 May 2026, Rahman, 26 May 2026). Human preference judgments require careful design to control rater variance, as shown by pairwise multilingual evaluation work (Anand et al., 23 Apr 2026). Prosodic diagnostics benefit from localization rather than global MOS, but inter-annotator agreement can be low even when relative system differences are stable (Gutierrez et al., 2021).
The future directions named across the source materials converge on expansion rather than replacement. For the original benchmark, future work includes multilingual conditions, device diversity expansion, more challenging text containing numerals, rare words, and homographs, and explicit punctuation and casing normalization tests (Kim et al., 20 May 2026). For the phonology-informed suite, future work consists of shipping language packs, adding contrasts such as tone, nasality, aspiration, and other harmony systems, and reporting faithfulness dashboards alongside conventional TTS metrics rather than reducing evaluation to a single score (Barman et al., 2 Jul 2026).
In that sense, Raon-OpenTTS-Eval names an evolving research program. Its initial form benchmarks robustness under realistic acoustic conditions; its later formulation argues that robust multilingual TTS evaluation must also determine whether a system preserves the phonological structure of the language it claims to synthesize.
