Toward Natural Emotional Text-To-Speech System with Fine-Grained Non-Verbal Expression Control

Abstract: While current emotional Text-to-Speech (TTS) models have successfully controlled verbal prosody, they often ignore non-verbal vocalizations (NVs), which are essential for authentic human emotion. Although some non-verbal datasets have recently emerged, they often lack high-quality, fine-grained annotations, which restricts a model's ability to precisely control NV generation. To address this limitation, we propose a novel approach for fine-grained non-verbal expression synthesis. We curate and reprocess female NV utterances from the EARS corpus, develop a new annotation scheme using tags to encode NV types, frequencies, and durations, and build an emotional TTS benchmark to demonstrate its effectiveness. Our evaluation shows that while our NV approach leads to minor trade-offs in perceived naturalness, it significantly improves expressiveness (eMOS 4.20) and emotional recognition accuracy (78.8%). Emotion-specific analysis further reveals that NV cues are highly effective for high-arousal emotions like happy (82.5%) and fear (82.7%), and almost perfectly convey sadness (98.3%).

• The paper introduces a fine-grained non-verbal vocalization (NV) dataset to enhance emotional text-to-speech synthesis.
• It employs Grad-TTS with a trainable emotion encoder, ensuring precise emotional expression through NV control.
• Results show 78.8% accuracy in emotion recognition using NVs, significantly improving over verbal-only methods.

Fine-Grained Control of Non-Verbal Vocalizations in Emotional Text-to-Speech

Motivation and Background

The synthesis of emotional speech for conversational AI requires more than prosodic manipulation of verbal utterances; it demands nuanced handling of non-verbal vocalizations (NVs) such as laughter, cries, and cheers. NVs play a central role in affective human communication, yet most emotional TTS systems either lack NVs altogether or rely on coarse annotation schemes that restrict the model’s expressiveness and granularity of control. Existing NV datasets—including NVTTS and AMI Meeting Corpus—suffer from limitations such as poor acoustic quality, restricted NV categories, coarse-grained tagging, and cultural domain specificity. These constraints preclude modeling the frequency, duration, and style of NVs required for authentic emotional synthesis.

This work addresses these deficiencies through the construction of a fine-grained NV expression dataset and the design of an emotional TTS system capable of precise NV control. The goal is to enable TTS models to produce speech with emotionally congruent NVs whose type, frequency, and duration can be rigorously specified.

Data Construction and Annotation Scheme

Utilizing the EARS corpus as a foundation, the authors curated a dataset of female NV utterances spanning six NV categories: laughter-open, laughter-closed, cheering, yelling, crying, and screaming. Each audio file was segmented using pydub with noise thresholds and silence duration criteria, resulting in 739 utterances of 2–6 seconds each.

Crucially, NVs were annotated via a structured scheme:

• Discrete Vocalizations: Frequency controlled via repeated syllables (e.g., <(Laughter-open) ha ha ha>).
• Continuous Vocalizations: Duration specified by elongating phonemes (e.g., <(crying) wuuuuu>), with each additional letter incrementing duration.
• Style-Level Tags: Encode NV type (e.g., cheering, crying), enabling categorical structure. This enables the model to learn NVs on multiple granularity axes—style, frequency, and temporal duration—unlike prior datasets which only tag NV presence or simple types.

Model Architecture and Processing Pipeline

Grad-TTS was selected as the backbone for synthesis, enhanced with a trainable emotion encoder using continuous arousal and valence attributes derived from Russell’s circumplex model. The pipeline incorporates a custom NV processor comprising:

• Style Parser: Identifies categorical NV style.
• Discrete Unit Parser: Counts discrete NV occurrences.
• Duration Parser: Quantifies temporal NV length for continuous sounds.

NV tokens are created by structurally encoding these facets, ensuring that the emotional TTS model parses and interprets non-verbal cues with high fidelity. The hierarchical annotation is processed as a preprocessing step, so both verbal and NV segments are jointly fed into the model.

Acoustic preprocessing utilized 80-dimensional mel-spectrograms, with high-fidelity audio generation via Hifi-GAN vocoder, enabling robust waveform synthesis suitable for subjective evaluation.

Experimental Setup and Metrics

A 9-hour corpus compiled from EXPRESSO, SEMAINE, and ESD (with arousal/valence predicted via pre-trained SER model) served as training data. The NV module was trained and compared on both the curated fine-grained dataset and the NVTTS coarse-grained corpus. Subjective evaluation involved rating naturalness (nMOS) and expressiveness (eMOS) using ambiguous utterances, alongside a four-way emotion recognition task (happy, sad, fear, anger). Additionally, preference ranking was performed for various NV combinations associated with happy and sad emotions to assess listener expectations.

Results and Analysis

Overall Performance

• Expressiveness: The fine-grained NV approach yielded eMOS 4.20, outperforming verbal-only (eMOS 3.81) and coarse-grained NV systems.
• Emotional Recognition: Average recognition accuracy increased dramatically, with 78.8% for fine-grained NV, 13.3% higher than verbal-only. For sadness, accuracy was 98.3%; for happy and fear, 82.5% and 82.7%, respectively.
• Naturalness: Minor decrease in nMOS (fine-grained NV 3.43) relative to verbal-only (3.54), indicating a trade-off offset by gains in expressiveness and recognition.

Emotion-Specific Findings

• Sad: Near-perfect accuracy in sadness recognition with multi-element non-verbal cues, validating the benefit of compositional NV expression.
• Happy: Cheering NVs (e.g., <(cheering) Wo ho>) were strongly preferred over laughter, challenging default assumptions that laughter is the optimal NV for conveying joy.
• Fear: Screaming NVs provided substantial accuracy improvement, highlighting the importance of salient NV design for high-arousal emotions.
• Anger: Yelling NVs did not yield substantial accuracy improvements due to lack of uniquely identifiable markers, suggesting further NV category development is needed.

Preference Testing

Participants significantly favored multi-component NV expressions over single NV cues (e.g., prolonged crying plus sob), reinforcing that nuanced NV construction enhances perceived authenticity. Cheering was preferred for happy emotions, indicating that NV selection should be informed by empirical listener preference rather than standard practice.

Implications and Future Directions

The fine-grained annotation and modeling scheme demonstrates that detailed NV control is indispensable for expressive and intelligible emotional speech synthesis. These results challenge existing approaches reliant on coarse-grained NV tagging and reveal that explicit handling of frequency and duration is critical for listener recognition and expressiveness.

On a theoretical level, this work advocates for hierarchical NV representation and structured tokenization within TTS pipelines. Practically, it enables conversational AI to synthesize emotionally rich responses that are perceived as natural and authentic, with explicit parameterization for affective NV elements.

Future research should investigate:

• Expansion to multi-speaker and multi-lingual corpora with corresponding fine-grained NV annotations.
• Automatic NV detection and annotation in large-scale datasets.
• Integration of TTS systems with multi-modal emotion expression, such as facial gestures.
• Further development of NV categories associated with underrepresented emotions (e.g., anger) and cultural adaptation.

Conclusion

This paper establishes that fine-grained NV annotation and control results in significant improvements to expressiveness and emotional recognition accuracy in TTS synthesis, with minor trade-offs in naturalness. The empirical findings substantiate the claim that NVs, when structured by type, frequency, and duration, are pivotal in conveying affect. Preference evaluation further demonstrates that NV design should be empirically guided. The implications extend to both practical deployment in conversational AI and the theoretical design of emotionally congruent TTS systems.