---
title: 'InstructTTSEval-Zh: Mandarin TTS Benchmark'
url: https://www.emergentmind.com/topics/instructttseval-zh
type: topic
---

# InstructTTSEval-Zh: Mandarin TTS Benchmark

InstructTTSEval-Zh is the Chinese subset of the InstructTTSEval benchmark suite, designed to measure the ability of modern text-to-speech (TTS) systems to follow complex, free-form natural-language style instructions, with a particular focus on nuanced paralinguistic control and instruction adherence in Mandarin Chinese speech synthesis [2506.16381]. It is characterized by a structured set of style-controlled tasks, robust automatic evaluation using large-language models (LLMs) as judges, and coverage of both low-level acoustic parameters and high-level social or contextual instructions.

## 1. Task Taxonomy and Benchmark Design

InstructTTSEval-Zh comprises 1,000 Chinese instruction–reference-audio pairs drawn from expressive media sources, partitioned into three hierarchical tasks that target progressively more abstract dimensions of paralinguistic control:

1. **Acoustic-Parameter Specification (APS):**  
   APS tests the model’s precise control over twelve predefined acoustic features, spanning physiological (e.g., gender, pitch), linguistic (clarity, fluency, speed), social (accent, age, volume), and psychological/pragmatic (emotion, tone, personality) tiers. Each APS instruction is a free-form caption, generated by Gemini (gemini-2.5-pro), that matches all these attributes in the paired reference audio, enabling direct assessment of feature-level synthesis fidelity.

2. **Descriptive-Style Directive (DSD):**  
   DSD evaluates a system’s generalization from structured prompts to under-specified, unstructured ones. Instructions are derived by rewriting APS prompts with GPT-4o, randomly omitting 20–50% of the feature mentions to create varying degrees of underspecification. This requires models to infer omitted attributes holistically and map qualitative stylistic descriptions to appropriate prosodic patterns.

3. **Role-Play (RP):**  
   RP probes contextual and social reasoning, with prompts requiring the model to perform according to character roles or scene settings (e.g., “angry officer,” “anxious interviewee”). Instructions are generated via GPT-4o chain-of-thought prompts that abstract from explicit feature values to a scenario, requiring the TTS system to infer suitable paralinguistic realization.

All three tasks share the same 1,000 base audio segments, ensuring controlled ablation of explicitness and complexity across instruction types [2506.16381].

## 2. Data Collection and Annotation Pipeline

**Source Material:**  
Half the data is drawn from the NCSSD (presumably a Chinese expressive speech dataset), and the remainder from Chinese movies, TV dramas, and variety shows. Audio is segmented and diarized to isolate single-speaker clips of ≤30 seconds using pyannote.audio. Transcriptions are produced with whisper-large-v3, followed by punctuation and quality filtering (DNSMOS ≥2.8, minimum segment duration and text length, WhisperD to ensure single-speaker validity).

**Expressiveness Filtering:**  
Segments are further filtered via the DVA toolkit, with only those exhibiting both Dominance >0.8 and Arousal >0.8 retained. Final statistics for the ZH subset are:

| Subset       | # Segments | Duration (h) |
| ------------ | ---------- | ------------ |
| NCSSD        | 500        | 0.93         |
| Media        | 500        | 1.61         |
| **Total**    | 1,000      | 2.54         |

**Instruction Construction:**  
- **APS:** Gemini produces sentences naming all 12 features, including any temporal modulation.
- **DSD:** GPT-4o rewrites these captions into style directives, randomly omitting features to generate three dropout-weight variations.
- **RP:** GPT-4o uses scenario inference (chain-of-thought prompting) to abstract specific features into role- or scene-based prompts, explicitly omitting labels.

All instruction–audio pairs are thus backed by expressively annotated and contextually diverse reference material, with manual and LLM-based quality control ensuring robust coverage [2506.16381].

## 3. Evaluation Protocol and Scoring

**Automatic LLM-as-Judge:**  
For each instruction-synthesis pair, Gemini is used as an automatic judge to assess alignment: it produces a binary label (“true” if primary style attributes align, “false” otherwise), guided by a detailed rubric. This LLM-as-judge methodology enables scalable, replicable, and cost-efficient evaluation, and was empirically shown to match human rater agreement levels across all task types (APS: 88%, DSD: 80%, RP: 76%; overall: 81.3%) [2506.16381].

**Scoring Metric:**  
For each task, style-adherence is the macro-average of Gemini’s binary judgments over 1,000 items:
$$
S_{\mathrm{task}} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}(\text{Align}_i)
$$
where $\text{Align}_i=1$ if Gemini judges sample $i$ “true,” and 0 otherwise.

## 4. Benchmark Results and Model Comparisons

Performance on InstructTTSEval-Zh is summarized below, covering both commercial (closed-source) and open-source TTS systems alongside the “oracle” score of the original human audio:

| Model              | APS (%) | DSD (%) | RP (%) | Avg (%) |
|--------------------|---------|---------|--------|---------|
| reference_audio    | 90.9    | 86.7    | 69.8   | 82.5    |
| gemini-flash       | 88.2    | 90.9    | 77.3   | 85.4    |
| gemini-pro         | 89.0    | 90.1    | 75.5   | 84.8    |
| gpt-4o-mini-tts    | 54.9    | 52.3    | 46.0   | 51.1    |
| VoxInstruct        | 47.5    | 52.3    | 42.6   | 47.5    |

**Observations:**
- Closed-source Gemini models approach the oracle upper bound on APS/DSD but lag on RP, reflecting persistent difficulties in abstract social role inference.
- GPT-4o-mini-tts shows sharp degradation for fine-grained acoustic control (APS) and contextual complexity (RP).
- VoxInstruct (open-source) matches gpt-4o-mini-tts in DSD/RP but shows limited APS adherence, indicating difficulties with explicit low-level feature control.
- Commercial TTS models often display non-native prosody in Mandarin Chinese, plausibly due to predominantly English training data [2506.16381].

## 5. Analysis of Strengths, Weaknesses, and Task Properties

**Strengths:**
- Gemini-flash/pro robustly follow explicit APS-style feature lists (APS >88%).
- DSD generalization is strong for commercial systems (~90%), indicating capacity to interpolate between seen style configurations.

**Weaknesses:**
- Role-play remains challenging: even reference audio achieves only 69.8%, with the best TTS models achieving 77.3%.
- Extreme paralinguistic phenomena (sighs, screams), dynamic emotion transitions, and singing are not adequately reproduced.
- Open-source models display orthogonal strengths, able to imitate certain timbral qualities (e.g., child-like) yet failing in emotional nuance and scenario inference.

These findings highlight both the granularity of InstructTTSEval-Zh and the limits of current TTS models in modeling abstract, context-dependent expressive speech in Mandarin.

## 6. Methodological Insights and Recommendations

The InstructTTSEval-Zh analysis prompts several recommendations for advancing instruction-following TTS in Chinese [2506.16381]:

- **Unified Timbre–Emotion Modeling:**  
  Existing systems often display trade-offs between flexible timbre control and emotional expressiveness. Joint modeling of speaker identity (timbre) and affective dynamics is needed for comprehensive style coverage.

- **Dynamic Event Synthesis:**  
  Paralinguistic events (e.g., laughter, screams) should be promoted to first-class outputs, demanding explicit annotation and generation benchmarks.

- **Enhanced Role-Inference:**  
  Current bottlenecks in RP tasks reflect a gap in abstract social reasoning, which may be addressed by integrating semantic scenario understanding into the prosody generation pipeline.

- **Chinese-Native Prosody Adaptation:**  
  Commercial TTS systems benefit from fine-tuning on Mandarin corpora to restore native tonal contours and authentic rhythmic patterns, counteracting biases from English-centric pretraining.

- **Cost-Efficient Automatic Evaluation:**  
  While Gemini enables reliable evaluation, its resource demands suggest a need for open-source multi-modal judges with strong correlation to human ratings for accelerated research cycles.

- **Style Coverage Expansion:**  
  Addressing data imbalance by enriching underrepresented emotions and roles (e.g., comedic irony, calm urgency) would yield a more uniform and diagnostic benchmark.

## 7. Significance and Future Directions

InstructTTSEval-Zh provides a rigorous, automatic, multi-task framework for evaluating instruction-driven Mandarin TTS systems along both explicit acoustic and abstract expressive axes. The separation of APS, DSD, and RP tasks allows disentanglement of low-level and high-level style control, revealing distinct strengths and failure modes across system classes [2506.16381].

A plausible implication is that as TTS research migrates toward open-ended style and contextual control, benchmarks such as InstructTTSEval-Zh—paired with instruction fidelity metrics and robust LLM-as-judge scoring—will drive the methodological evolution required for genuinely expressive and human-like synthetic speech in Chinese.

Source: https://www.emergentmind.com/topics/instructttseval-zh