Emotional Speech Synthesis

• Emotional Speech Synthesis (ESS) is a subfield of neural speech synthesis that generates synthetic speech with rich, controllable emotional expressivity using advanced emotion encoding techniques.
• Recent advances enable fine-grained, multi-scale control through both categorical and continuous emotion representations, enhancing the naturalness and versatility of synthetic speech.
• Open challenges include dataset quality, annotation robustness, and cross-lingual adaptation, with emerging solutions leveraging human–LLM pipelines and comprehensive, multi-dimensional corpora.

Emotional Speech Synthesis (ESS) is the subfield of neural speech synthesis that aims to generate synthetic speech with rich, controllable emotional expressivity. ESS extends conventional text-to-speech (TTS) architectures by incorporating mechanisms for emotion representation, conditioning, and evaluation, addressing both categorical and continuous emotion dimensions. Recent research advances have driven ESS from simple global emotion labels to systems with fine-grained, multi-scale control, continuous intensity morphing, and open-domain caption-guided generation.

1. Emotional Speech Datasets and Annotation Paradigms

A foundational challenge in ESS is the acquisition of high-quality corpora with annotated emotion labels or descriptors. Early emotional speech datasets were small, acted by professionals, and labeled categorically (e.g., IEMOCAP, RAVDESS, EmoV-DB) (Adigwe et al., 2018). Modern corpora such as AffectSpeech (253,799 utterances from multiple sources) provide multi-granular annotations along six axes: categorical polarity, open-vocabulary captions, binned intensity, prosodic tags, segmental prominence, and emotion-related semantics (Qi et al., 5 Apr 2026). The EMOVIE dataset specifically introduces fine-grained polarity annotations on Mandarin movie dialogue (9.7k utterances, 5-level scalar polarity from –1 to +1) (Cui et al., 2021).

Annotation methodologies now include human–LLM collaborative pipelines: algorithmic pre-labeling (acoustic/prosodic features; valence–arousal–dominance inference), LLM description (multi-model prompting), and human-in-the-loop adjudication. To minimize stylistic bias and promote generalization, single reference captions are expanded into multiple functional styles (narrative, bullet-point, technical, etc.), as in AffectSpeech (Qi et al., 5 Apr 2026). However, inter-annotator agreement and speaker label completeness (e.g., in EMOVIE) remain open issues for dataset robustness.

2. Representations and Conditioning of Emotion

2.1 Categorical and Continuous Emotion Encoding

ESS systems historically encoded emotion as categorical labels (e.g., "angry," "neutral"), mapped to either one-hot or low-dimensional learned embeddings (Lee et al., 2017, Adigwe et al., 2018). The emergence of continuous representations—scalar polarity, valence-arousal, or latent vectors aligning with affect dimensions—enables fine intensity control and smooth interpolation between emotions (Cui et al., 2021, Oh et al., 2022). For example, EMOVIE’s polarity is a real scalar, whereas systems like MsEmoTTS incorporate categorical, utterance-level, and syllable-level (local) emotion vectors simultaneously (Lei et al., 2022).

2.2 Embedding and Control Mechanisms

Advanced systems employ style tokens (GST), relative attribute vectors (pairwise ordinal rankings), or multi-scale embeddings incorporating utterance-level and fine-grained, frame- or phoneme-level supervisors (Um et al., 2019, Lei et al., 2020, Tang et al., 2024). Prompt-based, instruction-conditioned models such as VoxInstruct (fine-tuned on AffectSpeech) cross-attend on free-form emotional captions, supporting open-domain input (Qi et al., 5 Apr 2026).

Conditioning is achieved via concatenation or sum-injection of emotion embeddings into encoder or decoder layers, with variants using Conditional LayerNorm, Conditional Cross-Attention, or broadcast addition at each time step. Systems now support:

• Direct specification (manual selection of global/local emotion or intensity vectors)
• Reference-based transfer (extracting fine-grained emotion descriptors from audio)
• Automatic prediction (emotion class/strength from text using LMs: GPT-3, BERT, RoBERTa) (Yoon et al., 2022, De et al., 2024, Lei et al., 2022)

3. Model Architectures and Training Strategies

3.1 Backbone Models

Most ESS architectures extend standard neural TTS models:

• Sequence-to-sequence (Tacotron, Tacotron2): Encoder-decoder with attention; emotion embeddings injected into decoder inputs (Lee et al., 2017, Tits, 2019, Cho et al., 2021).
• Feed-forward (FastSpeech2): Parallel non-autoregressive generation; enables high-speed inference and per-token conditioning; now common for multi-scale ESS (Diatlova et al., 2023, Oh et al., 2022, De et al., 2024).
• Diffusion-based models: Conditional DDPMs (e.g., Grad-TTS) with multi-scale control at the denoising stage allow frame-wise and utterance-level emotion guidance (Tang et al., 2024).

3.2 Emotion Modeling Modules

State-of-the-art ESS systems integrate multiple emotion-conditioning modules:

Module	Level	Control Signal	Example Systems
GM	Global (category)	Categorical/soft-embedding	MsEmoTTS, ED-TTS, EMOVIE
UM	Utterance	Conv-pool over reference / BERT output	MsEmoTTS, ED-TTS
LM	Syllable/phoneme	Scalar attribute/ranker-predicted	MsEmoTTS, (Lei et al., 2020, Oh et al., 2022)
SED	Frame-level diar.	WavLM-encoded soft labels	ED-TTS
Prompt	Open-domain	Instructional LLM-generated captions	AffectSpeech/VoxInstruct, EmoPro

3.3 Training Objectives and Loss Functions

Typical compounded objectives: mel-spectrogram or WORLD parameter reconstruction (L1/L2), duration/pitch/energy MSE, emotion classification (cross-entropy), adversarial losses (GANs or discriminators), and, in some models, style or token alignment penalties.

Some ESS models (e.g., GANtron) utilize full adversarial training (Wasserstein-GAN) for mel-spectrogram realism with explicit label/conditioned tokens (Hortal et al., 2021). Diffusion models optimize noise prediction loss alongside emotion supervision, while cross-domain diarization losses combine frame-level cross-entropy with domain adaptation (MLMMD) (Tang et al., 2024).

4. Fine-Grained, Multi-Scale, and Mixed Emotion Control

A significant progression from early categorical systems is the capability for localized and continuous emotional control:

• Phoneme/syllable-level control: By predicting or specifying per-unit emotion strengths, models can produce speech with local peaks, ramps, or even alternating affective patterns within an utterance (Lei et al., 2020, Oh et al., 2022, Lei et al., 2022).
• Continuous interpolation: Inter- or intra-class emotion attribute interpolation allows for perception-preserving morphing; semi-supervised learning with pseudo-labels and adversarial objectives yields uniform grid geometry and robust expressivity across the full intensity spectrum (Oh et al., 2022, Um et al., 2019, Zhou et al., 2022).
• Mixed emotion synthesis: Pairwise or vectorized relative difference modeling in the style/attribute space enables synthesis of unprecedented blends (e.g., surprise+anger→"outrage"), with subjective and emotion recognizer-backed validation (Zhou et al., 2022).
• Instructional caption-driven control: Open-text descriptions with granularity over intensity, prosody, and segmental prominence serve as conditioning for fully flexible, instruction-based ESS (Qi et al., 5 Apr 2026). Prompt selection strategies optimize the emotional transferability of prompts in zero-shot, large-model pipelines (Wang et al., 2024).

5. Evaluation Protocols and Metrics

ESS systems are evaluated by a combination of objective and subjective protocols:

• Objective: Mel-cepstral distortion (MCD), F0 RMSE, Word Error Rate (WER), Emotion Reclassification Accuracy (ERA), Emotion Similarity/Diversity (cosine or pairwise measures via Emotion2Vec), and diarization error rates where fine-grained annotation exists (De et al., 2024, Tang et al., 2024, Qi et al., 5 Apr 2026).
• Subjective: MOS (naturalness, expressivity), CMOS, A/B preference, best-worst scaling, and emotional saliency/recognizability via forced-choice or rating scale (Cui et al., 2021, Lei et al., 2022, Wang et al., 2024, Qi et al., 5 Apr 2026).

Human–LLM collaborative evaluation schemes are emerging to ensure reliability and consistency in large-scale subjective ratings (Qi et al., 5 Apr 2026). The alignment between synthesized and ground-truth emotion can be validated through emotion recognition models (e.g., SED or SER networks in a closed loop).

6. Multilingual, Multi-Speaker, and Application-Specific Developments

ESS has been demonstrated across multiple languages (Mandarin, Korean, English, French) and with multi-speaker architectures:

• Speaker-independence/adaptation: Embedding fusion, curriculum learning with speaker–emotion balancing, and disentangled architectures allow multi-speaker and multilingual ESS (Cho et al., 2021, Yoon et al., 2022).
• Application domains: Accessibility—improving screen-reader expressivity with automated emotion detection from raw text (De et al., 2024); personalized companions and robots—full pipeline extraction and alignment of caregiver affect to synthetic speech (Homma et al., 2021).
• Open-source datasets and tools: The release of datasets such as EMOVIE, AffectSpeech, and EmoV-DB accelerates benchmarking and reproducibility.

Quality on less-common combinations, control of intra-utterance emotion trajectories, and realistic transfer across demographic or linguistic boundaries remain areas of ongoing development (Cui et al., 2021, De et al., 2024).

7. Open Challenges and Future Directions

Persistent limitations arise from label scarcity, lack of fine-grained ground-truth, uneven data quality, and overfitting to template annotation styles. Proposed future directions include:

• Expansion to multi-dimensional emotion and fine-grained time/segmental labels (Cui et al., 2021, Qi et al., 5 Apr 2026).
• End-to-end, jointly optimized architectures integrating text, acoustic, and emotion-prediction modules for in-the-wild, real-time synthesis (Oh et al., 2022, De et al., 2024).
• Personalization, cross-lingual adaptation, and integration of multimodal emotion cues (e.g., visual, contextual) (Homma et al., 2021, Qi et al., 5 Apr 2026).
• More nuanced, prompt-ensemble or interpolation strategies for instruction-based, zero-shot, or mixed emotion control (Wang et al., 2024, Qi et al., 5 Apr 2026).
• Systematic human and classifier-in-the-loop evaluation frameworks to correlate perceptual accuracy, naturalness, and emotional diversity.

The canonical trajectory is toward models capable of generating, transferring, and understanding human-affective speech at arbitrary granularity under open-domain control, while ensuring naturalness, intelligibility, and application-specific suitability (Qi et al., 5 Apr 2026, Tang et al., 2024, Lei et al., 2022).