PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech

Abstract: Standard text-to-speech (TTS) evaluation measures intelligibility (WER, CER) and overall naturalness (MOS, UTMOS) but does not quantify accent. A synthesiser may score well on all four yet sound non-native on features that are phonemic in the target language. For Indic languages, these features include retroflex articulation, aspiration, vowel length, and the Tamil retroflex approximant (letter zha). We present PSP, the Phoneme Substitution Profile, an interpretable, per-phonological-dimension accent benchmark for Indic TTS. PSP decomposes accent into six complementary dimensions: retroflex collapse rate (RR), aspiration fidelity (AF), vowel-length fidelity (LF), Tamil-zha fidelity (ZF), Frechet Audio Distance (FAD), and prosodic signature divergence (PSD). The first four are measured via forced alignment plus native-speaker-centroid acoustic probes over Wav2Vec2-XLS-R layer-9 embeddings; the latter two are corpus-level distributional distances. In this v1 we benchmark four commercial and open-source systems (ElevenLabs v3, Cartesia Sonic-3, Sarvam Bulbul, Indic Parler-TTS) on Hindi, Telugu, and Tamil pilot sets, with a fifth system (Praxy Voice) included on all three languages, plus an R5->R6 case study on Telugu. Three findings: (i) retroflex collapse grows monotonically with phonological difficulty Hindi < Telugu < Tamil (~1%, ~40%, ~68%); (ii) PSP ordering diverges from WER ordering -- commercial WER-leaders do not uniformly lead on retroflex or prosodic fidelity; (iii) no single system is Pareto-optimal across all six dimensions. We release native reference centroids (500 clips per language), 1000-clip embeddings for FAD, 500-clip prosodic feature matrices for PSD, 300-utterance golden sets per language, scoring code under MIT, and centroids under CC-BY. Formal MOS-correlation is deferred to v2; v1 reports five internal-consistency signals plus a native-audio sanity check.

• The paper introduces a novel PSP metric suite that decomposes accent evaluation into six phonological dimensions to diagnose TTS accent errors.
• The approach leverages forced alignment and embedding-based features to assess contrast fidelity, revealing significant performance differences across Hindi, Telugu, and Tamil.
• The experimental results demonstrate that traditional metrics like WER miss accent errors, highlighting the need for detailed, actionable diagnostic measures in Indic TTS.

An Authoritative Review of "PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech" (2604.25476)

Motivation and Limitations of Standard Metrics

Standard evaluation protocols in TTS—most notably WER, CER, MOS, UTMOS—emphasize intelligibility and general naturalness but fail to quantify accent nativity. In Indic languages, native phonological distinctions (retroflex vs. dental contrasts, aspiration, vowel length, and the Tamil retroflex approximant) are frequently collapsed in TTS outputs without negatively impacting WER or MOS, leading to systems that are intelligible yet perceptibly accented. This deficit motivates the need for interpretable, per-phonological-dimension benchmarks that surface these accent artifacts, particularly in the context of rich phonological inventories found in Hindi, Telugu, and Tamil.

PSP Metric Suite: Structure and Rationale

The Phoneme Substitution Profile (PSP) decomposes Indic accent evaluation across six theoretically and perceptually salient dimensions:

• Retroflex Collapse Rate (RR): Quantifies the rate at which retroflexes are replaced by their dental cognates.
• Aspiration Fidelity (AF): Measures fidelity of aspiration contrasts, crucial for Hindi and present in Telugu.
• Vowel Length Fidelity (LF): Assesses preservation of phonemic vowel length distinctions.
• Tamil-zha Fidelity (ZF): Captures the proper realization of the Tamil retroflex approximant /\b{r}/, a notorious error source for non-native systems.
• Fréchet Audio Distance (FAD): A distributional metric in the XLS-R embedding space, capturing overall spectral and prosodic congruence with native corpora.
• Prosodic Signature Divergence (PSD): Quantifies divergence in a 5-dimensional prosodic space (pitch range, log-$F_0$, speech rate, nPVI, log-duration).

The first four dimensions are indexed by explicit forced alignment and vector-probe acoustic similarity (XLS-R layer-9) to native-speaker centroids, while FAD and PSD leverage large-scale embedding or feature space statistics. This design enables decoupling of phoneme-level and global sequence/prosody discrepancies. The system is robust to ASR errors and does not depend on high-quality ASR for the target language—a critical practical consideration for low-resource Indic settings.

Implementation and Resource Release

The PSP benchmark is implemented as an open-source, GPU-accelerated toolkit, with native-speaker centroids (500 clips/language), reference distributions for FAD (1000 utterances) and PSD (500 utterances), and 300-utterance held-out golden evaluation sets for Hindi, Telugu, and Tamil. All artifacts are released under permissive licenses to promote reproducibility.

Experimental Results and Key Findings

Per-Dimension Accent Profiling

PSP was applied to four state-of-the-art commercial/open-source TTS systems (ElevenLabs v3, Cartesia Sonic-3, Sarvam Bulbul, Indic Parler-TTS) and an in-progress system (Praxy Voice) across three Indic languages.

Principal findings are:

• Monotonic Gradient in Retroflex Collapse: Hindi systems achieve near-native retroflex fidelity ($\sim$1% collapse), Telugu systems show marked degradation ($\sim$40%), and Tamil systems further degrade ($\sim$68%). This correlates precisely with phonological complexity and known TTS difficulty for these languages.
• Decoupling of WER and PSP Rankings: Systems leading in WER do not consistently lead across all phonological dimensions. In several cases, WER is uninformative for accent (e.g., Cartesia's strong WER in Hindi, but poor retroflex fidelity in Telugu and Tamil).
• No Pareto Optimality: No system simultaneously optimizes all PSP dimensions, e.g., one system may lead FAD but lose in RR or PSD, supporting the need for decomposed, multi-dimensional accent profiling.
• Prosodic Errors Not Captured by WER: Systems such as ElevenLabs achieve competitive WER while exhibiting anomalously flat prosodies (narrow pitch range, altered nPVI), which are effectively surfaced by PSD but missed by intelligibility-centered metrics.
• Voice-Prompt Recovery as a Remedy: Conditioning Praxy R6 with native speaker reference prompts at inference substantially reduced collapse rates and PSD, even without retraining the TTS acoustic generator weights. This demonstrates that certain accent errors stem from conditioning and can be mitigated in deployment through user-supplied reference prompts.

Calibration and Metric Validity

Internal consistency signals validate the metric suite:

• Gradient in RR tracks community expectations for TTS quality by language.
• Indic-specialized systems outperform general/commercial ones on Indic-specific dimensions.
• A native-audio sanity check confirms that FAD and PSD have a well-defined native noise floor on held-out speech.
• Language-dependent noise floors exist for per-phoneme metrics owing to forced aligner limitations in Telugu/Tamil. Absolute interpretation of these metrics is robust only for Hindi, while Telugu/Tamil results should be used for cross-system ranking, not absolute scaling.
• Large training data increases and LoRA adaptation move FAD but not phoneme-level RR, exposing which model components/paths must be targeted for further phonological improvement.

Limitations and Future Roadmap

Critical limitations include dependency on public CTC aligners (which constrain per-phoneme probe granularity in Telugu/Tamil), small pilot sample sizes in v1, and prototype-level centroid construction. The v2 roadmap includes formal MOS studies (50 utterances $\times$ 5 raters per language) and centroids derived from strictly disjoint reference/test speaker pools.

PSP is designed for granular diagnostic analysis of TTS system errors, not leaderboard ranking. It is positioned to complement, not supersede, ASR-based (WER/CER) or MOS-style benchmarks, and is tailored specifically for the accent dimension, orthogonal to intelligibility.

Implications for Indic TTS Research and Deployment

The introduction of dimensionally decomposed, interpretable accent metrics will directly advance TTS development for Indic languages by making previously opaque failure modes actionable. For system developers, the ability to localize errors in retroflex realization, vowel length, or prosody ensures efficient model iteration (e.g., selecting retraining vs. prompt-conditioning interventions). For downstream applications, improved native-likeness is essential for user trust in tasks such as voice assistants and digital learning platforms targeting millions of speakers across the Indic linguistic area.

Integration of PSP-class metrics into Indic TTS leaderboards and development pipelines will likely catalyze further research in forced alignment, accented speech modeling, and cross-lingual representation learning.

Conclusion

"PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech" (2604.25476) systematically addresses the failure of existing metrics to capture accent nativity in Indic TTS. Through carefully designed, open-source per-dimension probes and embedding-based corpus-level metrics, it establishes a rigorous, actionable framework for analyzing and closing accent gaps across multiple Indic languages. By demonstrating the orthogonality of accent to intelligibility and enabling error localization, PSP sets a methodological precedent for future phonological evaluation benchmarks, both within and potentially beyond the Indic context.