Conversation Context-Aware End-to-End TTS

• Conversation context-aware end-to-end TTS models integrate preceding dialogue turns, using multi-modal and hierarchical architectures to modulate prosody and style for natural, spontaneous speech.
• These approaches fuse textual, semantic, and acoustic features through recurrent, transformer, and graph-based encoders, enabling nuanced prosody adaptation and improved dialogue coherence.
• Recent advances incorporate latent style predictors and diffusion-based decoders that enhance expressiveness, achieve high MOS scores, and demonstrate significant improvements over traditional TTS systems.

Conversation context-aware end-to-end text-to-speech (TTS) refers to neural models that synthesize speech not solely from input text, but with explicit integration of preceding conversational turns—enabling modulation of prosody, style, and turn-level coherence at inference time. These systems employ end-to-end architectures that condition generation on features spanning lexical, semantic, prosodic, and acoustic histories, supporting contextually appropriate, expressive, and spontaneous-sounding speech in both dialogue and narrative tasks.

1. Architectural Foundations and Modeling Paradigms

Early end-to-end conversational TTS systems extended sequence-to-sequence backbones such as Tacotron2 with auxiliary encoders and context fusion modules, incorporating both current utterance features (phonemes, syntactic embeddings, semantics) and representations of conversational history. This led to richer intra- and inter-utterance prosodic variation and the emergence of spontaneous behaviors (e.g., fillers, repetitions) (Guo et al., 2020). Subsequent work advanced architectural depth and fusion mechanisms, exemplified by multi-scale, multi-modal systems like M²-CTTS (Xue et al., 2023), which aggregate both textual (sentence- and token-level embeddings, speaker cues) and acoustic (wav2vec2.0 features, fine-grained prosody) histories at coarse and fine timescales. Context modules typically precede a conditional decoder (FastSpeech2-type or waveform generator), with context vectors modulating acoustic output via adaptive normalization and gating.

Recent models emphasize context fusion not only for spoken dialogue, but also for long-form expressive synthesis, as in audiobook-style TTS. Here, context encoders often employ transformer stacks over concatenated or windowed sequences of sentences and their style embeddings, achieving cross-sentence coherence in prosody and discourse-level style (Guo et al., 9 Jun 2024, Dai et al., 19 Sep 2025). The architectural trend is toward hierarchical or graph-based models, leveraging global and local scale context, and, increasingly, highly-parameterized diffusion modules for speech latent generation.

2. Context Modeling Mechanisms

Approaches to modeling conversational history fall into several technical categories:

• Recurrent and GRU-based history summarizers: Early systems used GRUs over sentence-level BERT or other text embeddings, speaker tags, and simple statistics (turn index, position). Supplied as context vectors to the main attention-based decoder, these captured recent dialogue flow for utterance-level prosody adaptation (Guo et al., 2020).
• Graph-based context encoders: DialogueGCN is employed to construct a directed conversational graph, with nodes as utterances (features: text plus style tokens) and edge types parameterizing inter-speaker influence and intra-speaker inertia. Attention mechanisms summarize node features into context-aware style vectors, balancing self-consistency with adaptation to interlocutors (Li et al., 2021).
• Multi-scale, multi-modal fusion: M²-CTTS splits context extraction into coarse-grained history modules (e.g., Sentence-BERT for semantic context, wav2vec2.0 for acoustic), summarized via self-attention, and fine-grained modules (token-level cross-attention over linguistic/acoustic representations), with subsequent fusion at decoder layers using style-adaptive normalization and gating (Xue et al., 2023).
• Transformer-based context encoders: For modeling long-form narrative context or multi-sentence dialogue, transformer stacks operate over concatenated phoneme, token, and style representations, supporting propagation of discourse-level prosodic cues and accommodating variable-length context windows (Guo et al., 9 Jun 2024).
• Latent style predictors: Gaussian mixture VAEs and LSTM-based predictors independently infer latent speaking style vectors based on BERT-derived context encodings and acoustic history, which are then fed into VITS-style synthesizers to align TTS output with conversational flow (Mitsui et al., 2022).

3. Hierarchical and Tokenizer-Free Modeling Advances

Recent breakthroughs in context-aware TTS stem from hierarchical semantic-acoustic architectures that circumvent the limitations of pre-trained speech tokenizers. The VoxCPM model introduces a semi-discrete quantization bottleneck: a text-semantic LLM (TSLM) generates a semantic-prosodic plan, processed via differentiable finite scalar quantization (FSQ) and further refined by a residual acoustic LLM (RALM) that injects frame-level acoustic detail. These representations drive a local diffusion-based decoder under a unified diffusion objective, enabling stable yet expressive speech generation. The architectural stack, inherently end-to-end and tokenizer-free, excels at fusing both text-driven and acoustic context, supporting context-dependent prosody across dialogue and narrative genres (Zhou et al., 29 Sep 2025).

A distinguishing feature in VoxCPM is that both TSLM and RALM can condition on real or synthetic conversational history. TSLM, as a pre-trained LLM, incorporates past and current utterances, ingesting turn markers to structure context. Empirically, context-aware quantized representations cluster by discourse type and produce MOS scores on par with or exceeding human expressivity benchmarks.

4. Prosody, Style Control, and Expressive Synthesis

Conversation context-aware end-to-end TTS models enable the explicit shaping of both low-level prosodic attributes (pitch, energy, duration) and higher-order style phenomena (emotional tone, speaking act, dialogue turn structure):

• Context-sensitive prosody: Models integrate context via cross-attention (BERT, wav2vec2.0, etc.), self-attention, and graph propagation, resulting in utterance prosody that adapts to question-answer flow, emphasis, and spontaneous phenomena such as fillers and repetitions (Guo et al., 2020, Xue et al., 2023).
• Style embedding and transfer: Systems leverage GST tokens (style learned unsupervised via Tacotron-GST) or contrastively learned HuBERT/T5/T5-derived style spaces, allowing the inference and transfer of speaking styles from text, past speech, or reference signals. Graph- or transformer-based context fusion enhances inter-speaker adaptation and intra-speaker consistency (Li et al., 2021, Guo et al., 9 Jun 2024).
• Instruction- and emotion-driven synthesis: Context-aware instruct-TTS models synthesize speech conditioned on fine-grained, LLM-generated emotion/scene instructions and timbre descriptions, producing audiobook narration and dialogue with contextually appropriate, emotionally expressive articulation (Dai et al., 19 Sep 2025).

5. Training Objectives, Data, and Evaluation

Training protocols for conversational context-aware TTS integrate signal reconstruction, adversarial, variational, and prosody/style supervision losses. Notable formulations include:

• Adversarial and diffusion objectives: Models such as VoxCPM and Deep Dubbing train conditional flow-matching components, with loss functions defined over denoising flows parameterized by context-aware embeddings (Zhou et al., 29 Sep 2025, Dai et al., 19 Sep 2025).
• Contrastive and alignment losses: Cross-modal InfoNCE-type and cosine alignment terms ensure text-derived style codes remain faithful to speech-style groundings, supporting text-driven expressiveness in absence of explicit style labels (Guo et al., 9 Jun 2024).
• Reconstruction and auxiliary losses: Standard $\ell_1$, mel-spectral, and KL-divergence losses for acoustic prediction are augmented with style-weight prediction, pitch/energy/duration fitting, and, where applicable, dialogue-level coherence objectives (Li et al., 2021, Xue et al., 2023, Mitsui et al., 2022).

Empirical evaluation relies on objective measures (MCD, CER/WER), subjective MOS/naturalness/expressiveness tests, ABX/preference rates, and analyses of prosody/variance. Statistically significant gains in MOS and contextual expressiveness are reported across models with robust context integration.

6. Applications, Corpora, and Limitations

Applications encompass voice agents, multi-speaker audiobook narration, open-domain spoken dialogue, and expressive TTS for reading-style or conversational content. Multiple bespoke and public corpora underpin advances: English Conversation Corpus (24h, annotated turns, no style labels) (Li et al., 2021), DailyTalk (20h, balanced gender, multi-turn) (Xue et al., 2023), large-scale synthetic and transcribed audiobook datasets for narrative TTS (Guo et al., 9 Jun 2024, Dai et al., 19 Sep 2025), and Japanese spontaneous dialogue (18k utts, real fillers, variable disfluency) (Mitsui et al., 2022).

Key limitations include constraints on training data volume (especially genuinely spontaneous dialogue), the challenge of style controllability beyond unsupervised representations, and the handling of very long-term conversational dependencies. Some models employ fixed context windows, and context granularity remains an open research direction. Fine control over disfluency and explicit dialog act space incorporation are also active research questions.

7. Comparative Experimental Results

A summary table of representative evaluation results in context-aware end-to-end TTS models:

Model/System	MOS/Naturalness	Expressiveness/EMOS	Context/Dialogue-level MOS	Notable Subjective/Objective Results
VoxCPM (Zhou et al., 29 Sep 2025)	EN: 4.11, ZH: 4.10	High pitch/energy variance	Higher multi-turn appropriateness	WER: 1.85%, CER: 0.93%, SIM: 73–77%
GST-FastSpeech2 + DialogueGCN (Li et al., 2021)	3.584 ± 0.100	—	Preference 55.11%	Stat. sig. improvement over RNN context, ABX pref. +29%
M²-CTTS (Xue et al., 2023)	3.74 ± 0.07	—	CMOS +0.34 vs. coarse-only	Fine-grained acoustic context most effective
TACA-VITS (Guo et al., 9 Jun 2024)	3.90 ± 0.10	3.93 ± 0.11	—	EMOS gain +0.3–0.4; listeners favor paragraph-level continuity
CA-Instruct-TTS (Dai et al., 19 Sep 2025)	3.33	4.15 (emotion)	—	MOS-Emotion gain +0.48 vs. no-instruction baseline
GMVAE-VITS + predictor (Mitsui et al., 2022)	—	—	3.53 ± 0.11	Dialogue-level MOS +0.19 vs. VITS; significant improvement
Tacotron2+context (M₃) (Guo et al., 2020)	—	—	CMOS +0.39	Context encoder aligns intonation, handles disfluency

All MOS and subjective results are as reported, with statistically significant differences where stated. These demonstrate that conversation context-aware end-to-end TTS architectures consistently yield more natural, expressive, and context-driven synthetic speech than models lacking explicit context integration.