TTS-PRISM: A Perceptual Reasoning and Interpretable Speech Model for Fine-Grained Diagnosis

Abstract: While generative text-to-speech (TTS) models approach human-level quality, monolithic metrics fail to diagnose fine-grained acoustic artifacts or explain perceptual collapse. To address this, we propose TTS-PRISM, a multi-dimensional diagnostic framework for Mandarin. First, we establish a 12-dimensional schema spanning stability to advanced expressiveness. Second, we design a targeted synthesis pipeline with adversarial perturbations and expert anchors to build a high-quality diagnostic dataset. Third, schema-driven instruction tuning embeds explicit scoring criteria and reasoning into an efficient end-to-end model. Experiments on a 1,600-sample Gold Test Set show TTS-PRISM outperforms generalist models in human alignment. Profiling six TTS paradigms establishes intuitive diagnostic flags that reveal fine-grained capability differences. TTS-PRISM is open-source, with code and checkpoints at https://github.com/xiaomi-research/tts-prism.

• The paper introduces a 12-dimensional diagnostic schema that quantifies both acoustic stability and expressive qualities for Mandarin TTS systems.
• It employs targeted data synthesis and schema-driven instruction tuning to enhance evaluation accuracy and align performance with expert human annotations.
• Benchmarking across multiple TTS paradigms reveals system-specific diagnostic flags, enabling actionable insights for addressing stability and expressiveness issues.

TTS-PRISM: A Multi-Dimensional Diagnostic Framework for Mandarin Text-to-Speech

Motivation and Framework Overview

Current generative TTS systems demonstrate near-human performance but lack transparent, fine-grained evaluative methods. Conventional metrics such as Mean Opinion Score (MOS) obscure localized artifacts and perceptual failures due to their monolithic nature, impeding actionable diagnosis and model development. To address these deficiencies, TTS-PRISM introduces a multi-dimensional evaluation schema, targeted data synthesis strategies, and schema-driven instruction tuning for Mandarin TTS diagnosis. The proposed approach quantifies both acoustic stability and expressiveness, facilitating interpretable profiling and granular capability discrimination across systems.

Figure 1: The schema comprises 12 well-defined dimensions spanning acoustic stability and expressiveness.

Multi-Dimensional Diagnostic Schema

TTS-PRISM formalizes a 12-dimensional schema structured in two layers:

• Basic Capability Layer (Scores 1–5): Encompasses eight sub-dimensions covering audio clarity, pronunciation accuracy (including Mandarin-specific tone sandhi and polyphone disambiguation), prosody accuracy (intonation, pauses, speech rate), and consistency (speaker, style, emotion). These criteria leverage explicit acoustic thresholds, mitigating subjective ambiguity during evaluation.
• Advanced Expressiveness Layer (Scores 0–2 Bonus): Quantifies expressive dimensions—stress, lengthening, paralinguistics (non-verbal cues), and emotion expression—by defining positive anchors through high-fidelity recordings and state-of-the-art TTS systems.

Explicit scoring guidelines for each dimension enable objective mapping of subjectivity into actionable standards for system improvement and research benchmarking.

Targeted Data Synthesis and Instruction Tuning

To construct a diagnostic dataset with balanced diversity and discriminative power, TTS-PRISM leverages adversarial perturbations, expert anchors, and linguistically diverse sources. Perturbations target prosodic, pronunciation, and audio artifacts, bolstered by expert-annotated subsets to resolve Mandarin-specific challenges.

Figure 2: Targeted synthesis strategy sharpens decision boundaries; schema-driven instruction tuning enables efficient single-pass, 12-dimensional diagnosis.

Gemini-2.5-Pro performs dimension-wise decomposition for annotation, ensuring rationales for stress and lengthening are refined to avoid hallucinations. The final dataset comprises over 200k samples, capturing positive and negative distributions across paradigm, text domain, and quality scale.

Figure 3: Distribution of diverse TTS sources and text domains.

Schema-driven instruction tuning is employed on MiMo-Audio as the backbone, enforcing interpretable reasoning via strictly conditioned rationales prior to scoring. The interleaved target sequence ensures logical consistency, minimizing hallucination and detached rationales frequently observed in unconstrained CoT approaches.

Experimental Results and Comparative Profiling

TTS-PRISM demonstrates superior alignment with expert human annotation in a stratified 1,600-sample Mandarin Gold Test Set. Notable findings include:

• Robust Performance: High LCC and SRCC in both Basic Capability and Advanced Expressiveness layers, with persistent accuracy on out-of-distribution samples.
• Rationale Quality: Achieves Rationale Support Consistency (RSC) of 0.98, integrating logical coherence and perceptual accuracy, in contrast to baselines that exhibit coherent reasoning but poor acoustic alignment.
• Ablation Analysis: Removal of targeted negatives drastically degrades performance, underscoring the importance of hard negative construction. Instruction tuning and explicit rationale generation are critical for extracting latent diagnostic capabilities from ASR-pretrained backbones.
• System Profiling: Multi-dimensional scores for six leading TTS paradigms reveal pronounced differentiation in both stability and expressiveness. IndexTTS 2 demonstrates peak emotion expression and lengthening; CosyVoice 3 excels in paralinguistics and stress; Qwen3-TTS achieves best-in-class pronunciation accuracy but limited expressiveness. Diagnostic flags extracted from score profiles reflect underlying architectural tendencies—e.g., “Stable but Flat” vs. “Highly Expressive”—enabling actionable insights for targeted enhancements.

Practical and Theoretical Implications

TTS-PRISM advances the diagnostic paradigm for generative TTS evaluation by replacing scalar aggregation with interpretable, dimension-wise profiling. Practically, this enables developers to localize bottlenecks, optimize configurations for specific use cases (e.g., broadcast, conversational, emotive applications), and fine-tune models for desired expressiveness. Theoretically, explicit mapping of perceptual criteria onto acoustic artifacts facilitates research into architecture-specific inductive biases and the development of more human-aligned synthesis algorithms.

The revealed limitations in pronunciation accuracy highlight the inherent tolerance of ASR-pretrained audio models to intelligibility defects—a bias that instruction tuning alone cannot override. Reinforcement learning approaches, incorporating human-in-the-loop and preference feedback, are posited as future directions to further calibrate fine-grained diagnostic capabilities.

Conclusion

TTS-PRISM provides an interpretable, schema-driven framework for comprehensive Mandarin speech diagnosis. The methodology outperforms generalist models in human alignment and enables nuanced profiling of advanced TTS architectures. Persisting pronunciation limitations suggest deeper architectural biases in current ASR-pretrained models; resolving these requires new forms of perceptual preference alignment and targeted reinforcement learning. The framework, criteria, code, and checkpoints are open-sourced for adoption and further study within the research community.