ExpressiveSpeech: Advances in Synthesis

• ExpressiveSpeech is a domain in speech research that synthesizes speech with affective, prosodic, and stylistic cues beyond lexical content using multi-scale control methods.
• Recent studies utilize diverse control mechanisms including latent variables, contextual text conditioning, and prosodic sketches to enhance expressive speech generation.
• Curated bilingual datasets and multimodal evaluation frameworks are driving improvements in assessing expressive appropriateness and long-form synthesis quality.

Searching arXiv for recent and foundational papers on expressive speech synthesis, retrieval, datasets, and evaluation. In contemporary speech research, expressive speech refers to speech that conveys affect, intent, and interpersonal cues beyond lexical content, spanning prosodic, stylistic, and paralinguistic variation; recent work also uses the name “ExpressiveSpeech” for a curated bilingual dataset selected with the DeEAR framework for speech-to-speech fine-tuning (Tits et al., 2019, Lin et al., 23 Oct 2025). The area now covers utterance-level “expressive primitives,” discourse-level and long-term behaviors, natural-language and sketch-based control, articulatory conditioning, expressive speech retrieval, and context-rich evaluation of whether a performance is appropriate for its narrative or interactional setting (Triantafyllopoulos et al., 2024, Wang et al., 10 May 2026).

1. Conceptual scope and theoretical framing

Expressive speech synthesis has been defined as the generation of speech that not only conveys propositional content but is also “coloured with inflections” spanning affective, prosodic, stylistic, and paralinguistic variations. The covered phenomena include categorical emotions, dimensional affect, stance, mood, personality, politeness, sincerity, pitch, timing, intensity, rhythm, pronunciation, vocal bursts, backchannels, and speaking style (Triantafyllopoulos et al., 2024). A complementary formulation from the controllable expressive speech literature treats expressive speech as speech that conveys affect, intent, and interpersonal cues beyond lexical content, with controllability ranging from explicit low-level controls over pitch, energy, and duration to discrete tags, latent embeddings, and prosody transfer (Tits et al., 2019).

A persistent misconception in the area is to equate expressiveness with emotion labels alone. The literature instead treats emotion as one component within a broader expressive manifold. This is explicit in work that separates utterance-contained “expressive primitives” from longer-term, discourse-level behavior, and in work that models articulatory voice production dimensions such as Glottalization, Tenseness, and Resonance rather than only high-level emotions (Triantafyllopoulos et al., 2024, Li et al., 2024). This suggests that “expressive speech” is best understood as a multi-scale control problem: local acoustic realization, utterance-level prosodic shape, and discourse-conditioned behavior all matter.

A second theoretical distinction concerns temporal scope. Stage I systems target single utterances, often with latent variables, style tokens, or prompt control. Stage II systems seek contextual, personalized, and longer-term behavior across turns or narrative spans, where the relevant target is not merely strong emotion but coherent expressive evolution (Triantafyllopoulos et al., 2024). Much of the recent literature can be read as a transition from Stage I toward Stage II.

2. Control mechanisms and generative architectures

The control problem has been approached through several architectural families. Early neural approaches used latent variables to model utterance-level “expressions” without labels. “Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder” conditions VoiceLoop on a global latent variable $z$ inferred from acoustics, with $p(z)=\mathcal{N}(0,I)$ and KL annealing to mitigate latent-variable collapse; the latent is held constant across frames to provide utterance-level control, and interpolation in $z$ yields smooth changes in F0 trajectories and speaking style (Akuzawa et al., 2018). “Using VAEs and Normalizing Flows for One-shot Text-To-Speech Synthesis of Expressive Speech” extends this line by adding a Householder Flow to a 64-dimensional style posterior, reducing KL divergence by 22% and enabling one-shot style transfer from a roughly one-second expressive reference utterance (Aggarwal et al., 2019).

A second family conditions synthesis on text that describes context or style. “Contextual Expressive Text-to-Speech” introduces CTTS, in which free-form textual context is concatenated with phoneme representations and encoded jointly, replacing fixed emotion labels with scene- or discourse-derived text (Tu et al., 2022). “Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis” moves further toward discourse conditioning through a hierarchical context encoder over phrases and neighboring sentences, distilled from a speech-based teacher and integrated into a FastSpeech 2 backbone; the model uses two past and two future sentences in experiments and improves both objective metrics and MOS relative to sentence-only baselines (Lei et al., 2022). The abstract of “EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting” places the same agenda in an LLM-centered setting, claiming fine-grained freestyle natural-language emotion control, a phoneme boost variant that outputs phoneme tokens and audio tokens in parallel, and a 40-hour English dataset with fine-grained emotion labels and natural-language descriptions (Yang et al., 17 Apr 2025).

A third family exposes low-level or mid-level controls directly. “DrawSpeech: Expressive Speech Synthesis Using Prosodic Sketches as Control Conditions” allows users to draw pitch and energy sketches, uses a sketch-to-contour predictor to recover detailed phoneme-level contours, and conditions a latent diffusion model on both sketches and reconstructed contours; its reported MOS is 4.49 with Sketch Correlation 4.30, markedly above adapted FastSpeech 2 and NaturalSpeech 2 baselines (Chen et al., 8 Jan 2025). “GTR-Voice: Articulatory Phonetics Informed Controllable Expressive Speech Synthesis” defines a three-dimensional articulatory framework—Glottalization, Tenseness, Resonance—with 125 GTR combinations recorded by a professional voice actor, and shows that both FastPitch and StyleTTS can be conditioned on these discrete articulatory labels (Li et al., 2024). For human-robot interaction, “EmojiVoice: Towards long-term controllable expressivity in robot speech” uses phrase-level emoji prompting as a style ID injected into Matcha-TTS, emphasizing real-time, offline control on robot-class hardware rather than maximal generative scale (Tuttösí et al., 18 Jun 2025).

These lines differ in interface and inductive bias. Latent-variable models prioritize unsupervised factorization; context-text systems prioritize semantic and discourse grounding; sketch- and articulatory-conditioned systems prioritize fine-grained interpretability. A plausible implication is that no single control mechanism dominates across all use cases: narrative TTS, audiobook rendering, role-playing, and robot interaction impose different constraints on latency, precision, and user effort.

3. Corpora, annotation regimes, and data curation

Recent progress has depended on datasets that expose richer labels than conventional neutral TTS corpora. The field now spans acted emotional corpora, spontaneous expressive speech, articulatory recordings, natural-language style descriptions, and evaluation-oriented datasets.

Resource	Scale	Annotation focus
Expresso	47 hours	26 spontaneous expressive styles, read speech, improvised dialogues (Nguyen et al., 2023)
SpeechCraft	2,381.54 hours, 2,249,579 clips	bilingual natural-language style descriptions (Jin et al., 2024)
Rasa	10 hours neutral and 1–3 hours expressive speech for each of the 6 Ekman emotions across 3 languages	low-resource expressive TTS (Varadhan et al., 2024)
GTR-Voice	2,500 clips, ~3.6 hours	125 articulatory GTR combinations (Li et al., 2024)
CEAEval-D	16.1 hours annotated subset	15 dimensions of expressive appropriateness (Wang et al., 10 May 2026)
ExpressiveSpeech	~14,000 utterances, 51 hours	DeEAR-selected bilingual expressive S2S data (Lin et al., 23 Oct 2025)

Expresso is central for textless expressive speech resynthesis because it combines 37% expressive reading and 72% improvised dialog, both studio-recorded, across 26 styles including angry, calm, projected, sarcastic, sympathetic, whispered, and non_verbal (Nguyen et al., 2023). SpeechCraft addresses a different bottleneck: it replaces template-style labels with individualized natural-language descriptions produced by a multi-expert annotation pipeline plus LLM rewriting, covering English and Chinese at very large scale (Jin et al., 2024). Rasa, by contrast, is explicitly designed for low-resource conditions and shows that syllabically balanced neutral data and relatively small expressive subsets can yield usable systems in Assamese, Bengali, and Tamil (Varadhan et al., 2024).

Several datasets are specialized. GTR-Voice isolates articulatory phonetics through 5 Glottalization levels, 5 Tenseness levels, and 7 Resonance categories, recorded over 20 Mandarin utterances and validated both by listener agreement and SVM classifiers (Li et al., 2024). CEAEval-D is not an overview corpus but a context-rich evaluation dataset for Mandarin conversational speech, pairing target utterances with narrative context and 15 annotations including overall expressive appropriateness, TTS difficulty, emotion, rhythm, intonation, recording conditions, paralinguistic vocalizations, and sound events (Wang et al., 10 May 2026). The DeEAR-selected ExpressiveSpeech dataset is explicitly curation-driven: approximately 14,000 utterances totaling 51 hours are retained from five public corpora by thresholding an objective expressiveness score, with average expressiveness 80.2 on a 0–100 scale (Lin et al., 23 Oct 2025).

This diversification of corpora changes what can be trained. Fine-grained natural-language prompts, articulatory control, context-sensitive evaluation, and low-resource recipes all depend on data regimes that older emotional TTS datasets did not provide. It also means that “expressive speech” no longer denotes a single annotation tradition.

4. Context, long-form generation, and adjacent retrieval tasks

Long-form and contextual expressivity have become a distinct research direction. In Mandarin lecture synthesis, hierarchical context modeling over phrases and adjacent sentences improves F0 RMSE from 54.395 to 51.871, Energy RMSE from 3.733 to 3.138, Duration MSE from 0.1267 to 0.1175, MCD from 4.614 to 4.535, and MOS from 3.679 to 4.067 relative to FastSpeech 2, with ABX preferences favoring the proposed system by 39.6% over FastSpeech 2 (Lei et al., 2022). In kids’ story synthesis, emotion-coherent augmentation forms two-sentence training examples by merging emotionally congruent utterances and sampling pauses from a Normal distribution fit to real inter-sentence silences with mean 509 ms and standard deviation 223 ms; the full system improves TP-GST style alignment to 0.075 L1 and yields synthesized pause distributions closer to ground truth (Chung, 10 Feb 2026).

Contextual modeling is also being absorbed into spoken LLMs. “VITA-QinYu: Expressive Spoken LLM for Role-Playing and Singing” extends interleaved text-audio generation with eight parallel audio codebooks, trains on 15.8K hours of natural conversation, role-playing, and singing data, and reports superior role-playing and singing performance while maintaining state-of-the-art conversational accuracy on C3 and URO (Xu et al., 7 May 2026). This broadens expressive speech from TTS toward end-to-end spoken interaction.

Adjacent tasks now include expressive speech retrieval. “Expressive Speech Retrieval using Natural Language Descriptions of Speaking Style” formulates retrieval based on how something was said rather than what was said, learns a joint speech-text latent space with a CLIP/CLAP-style bidirectional InfoNCE loss, an adversarial modality discriminator, and an auxiliary style classifier, and evaluates on 22 speaking styles with Recall@k. On Expresso, RoBERTa plus emotion2vec reaches $R@1=0.8159$ and $R@5=0.9826$, while prompt augmentation is shown to be critical for generalization to arbitrary phrasing (Kang et al., 15 Aug 2025).

These works collectively shift expressive speech from isolated sentence rendering to document-level narration, dialogue, role-playing, singing, and corpus search. A plausible implication is that expressive speech is converging with multimodal language modeling: the same system is increasingly expected to reason about context, generate speech, and interpret or retrieve styles from language.

5. Evaluation, benchmarking, and the problem of expressive appropriateness

Evaluation has become one of the field’s central controversies. Classical metrics such as MCD, F0 RMSE, voiced/unvoiced error, CER/WER, MOS, and ABX remain common, and Expresso’s resynthesis benchmark combines bitrate, ABX discrimination, phone-normalized mutual information, ASR-based WER, F0 Frame Error, and style-classification accuracy to analyze trade-offs between fidelity, speaker invariance, and style preservation (Nguyen et al., 2023). Yet several recent works argue that these metrics do not fully capture what listeners mean by expressive quality.

One challenge is that emotional intensity is not equivalent to expressive appropriateness. “Evaluating the Expressive Appropriateness of Speech in Rich Contexts” formalizes expressive appropriateness as the alignment between the expressive realization of a spoken utterance and the latent communicative intent implied by discourse-level narrative context, introduces CEAEval-D, and trains CEAEval-M with knowledge distillation, a planner-based text module, adaptive audio attention bias, and reinforcement learning. With CoT, CEAEval-M reaches LCC 0.72 and ACC 70.80% at context size 15, outperforming a range of speech-capable baselines (Wang et al., 10 May 2026). This directly addresses cases in which a strongly emotional utterance is still contextually wrong.

Another challenge is turning human preference into a scalable objective score. “Decoding the Ear: A Framework for Objectifying Expressiveness from Human Preference Through Efficient Alignment” decomposes expressiveness into Emotion, Prosody, and Spontaneity, learns a non-linear fusion from fewer than 500 annotated clips, and reports PCC 0.91 and SRCC 0.86 against human judgments for overall expressiveness, with system-level SRCC 0.96 for ranking seven speech-to-speech models (Lin et al., 23 Oct 2025). The same framework is then used to curate the ExpressiveSpeech dataset and to show that fine-tuning raises an S2S model’s overall expressiveness from 2.0 to 23.4 on a 100-point scale (Lin et al., 23 Oct 2025).

Evaluation itself is becoming multimodal and LLM-assisted. The abstract of EmoVoice states that the work investigates the reliability of existing emotion evaluation metrics and their alignment with human perceptual preferences, and explores using GPT-4o-audio and Gemini to assess emotional speech (Yang et al., 17 Apr 2025). This suggests a methodological transition: benchmarks are moving from isolated acoustic proxies toward richer, context-aware, model-assisted judgments, although the stability and bias properties of such judges remain an open question.

6. Applications, limitations, and open directions

Expressive speech systems now serve several distinct application classes. Retrieval models support voice casting, dubbing, TTS dataset curation, analytics, and assistive technologies by enabling free-form natural-language queries over style rather than content (Kang et al., 15 Aug 2025). EmojiVoice targets social robots, where offline real-time synthesis and long-term variation matter; in its storytelling case study, varied emoji prompting improves perceived expressiveness and suitability relative to a baseline, whereas in an assistant use case a consistently pleasant voice is preferred (Tuttösí et al., 18 Jun 2025). SpeakEasy addresses expressive content creation as an interaction-design problem, using high-level context and sentence-level iteration to help users produce performances closer to their own standards than with conventional TTS interfaces (Brade et al., 7 Apr 2025). Audiobook, narrative, and conversational-agent scenarios motivate evaluation frameworks such as CEAEval that judge whether an utterance fits its context rather than merely whether it sounds emotional (Wang et al., 10 May 2026).

Low-resource deployment remains a major constraint. Rasa shows that just 1 hour of neutral and 30 minutes of expressive data can yield a Fair system as indicated by MUSHRA scores, and that increasing neutral data to 10 hours, with minimal expressive data, significantly enhances expressiveness (Varadhan et al., 2024). At the same time, several hard problems recur across the literature: disentangling speaker identity from style, scaling long-context conditioning, controlling difficult emotions such as fear and surprise, handling cross-lingual transfer, and preserving intelligibility under strong expressive modulation (Varadhan et al., 2024, Triantafyllopoulos et al., 2024).

The societal issues are equally prominent. Reviews of expressive speech synthesis highlight misuse and deepfakes, persuasion and manipulation, attention-economy escalation, psychological impacts, fairness and representation issues, and the broader risks associated with foundation models; proposed mitigations include legal constraints, built-in alignment objectives, machine-based auditing, spoof detection, and RLHF-style guardrails (Triantafyllopoulos et al., 2024). Because expressive systems increasingly model personality, mood, and social stance, the ethical problem is not limited to speaker imitation. It also concerns the controlled generation of persuasive, emotionally calibrated behavior.

The current trajectory points toward integrated systems that combine rich data curation, context-sensitive planners, human-aligned evaluation, and multiple control channels—text, reference audio, articulatory labels, sketches, or emojis—within a single expressive interface. The literature suggests that future progress will depend less on a single synthesis backbone than on how well these components are aligned to discourse, user intent, and human judgment.