Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity

Abstract: Personalized text-to-speech (TTS) aims to clone the target speaker in the synthesized speech, imitating both the voice and speaking style. Current LLM-based TTS methods ignore the style-specific prosodic patterns in generated speech, resulting in deficient style learning and thus limiting speaker similarity in synthesized speech. To this end, we investigate the prosody learning conditioned on the synthesized speech, and propose to predict the prosody of the current syllable based on previously predicted speech. Experimental results obtained on three datasets demonstrated the efficacy of the proposed dynamic prosody prediction method in enhancing the prosody learning capability, thereby improving the speaker similarity of the generated speech. Audio samples are available at https://muzw.github.io/dynapros/.

• The paper introduces a dynamic, syllable-level prosody prediction mechanism that enhances speaker similarity in LLM-based TTS.
• The model employs autoregressive conditioning on text, reference speech, and previously generated outputs for precise prosody control.
• Experimental results show significant improvements in MOS, CER, and emotion transfer over static prosody methods across multiple benchmarks.

Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity

Introduction

This work addresses a fundamental limitation of contemporary LLM-based TTS systems: the insufficient explicit modeling of style-specific prosody, which is critical for achieving high speaker similarity in synthesized speech. Existing LLM-based TTS approaches, including recent frameworks such as CosyVoice and RALL-E, either leverage fixed-length embeddings for speaker/style or statically pre-compute prosody prior to synthesis. These paradigms fail to capture dynamic prosodic variations that are essential for realistic speaker cloning. This paper introduces a dynamic, syllable-level prosody prediction mechanism, conditioning the prosody of each generated syllable on the target text, reference speech, and previously generated target speech, and demonstrates strong improvements in prosody transfer and speaker similarity across multiple benchmarks.

Figure 1: The CosyVoice LLM baseline architecture, illustrating its input structure for text and speech tokens and the role of reference speaker embedding.

Background and Problem Formulation

LLM-based TTS architectures such as CosyVoice utilize quantized speech tokens and text embeddings, employing a reference utterance to derive fixed speaker embeddings to condition the decoder-only Transformer. While effective for many generative tasks, CosyVoice and similar approaches lack explicit modeling or controllability over prosody/style, leading to style imitation deficiencies. Other frameworks introduce external prosody embeddings (e.g., BASE TTS), but these typically neglect the stylistic cues in the target text. Approaches such as Vevo and RALL-E (utilizing chain-of-thought, CoT, prompting) predict phoneme-level prosody for the entire utterance prior to decoding, thus ignoring autoregressive dependencies and context sensitivity in prosodic realization.

The Proposed Dynamic Prosody Prediction Architecture

The proposed method reframes the prosody modeling step, leveraging the autoregressive capabilities of modern LLMs to perform dynamic, context-sensitive prediction of syllable-level prosody tokens. The key architectural innovation is the conditioning of each current syllable’s prosody prediction on (i) the target text, (ii) reference speaker embedding, (iii) reference prosody, and crucially, (iv) all previously generated target speech and prosody tokens.

Rather than statically predicting prosody for the full utterance before synthesis, this approach alternates between prosody token and speech token generation at each syllable. Each prosody token is derived as a quantized representation of four per-syllable features: duration, mean energy, mean pitch, and pitch range (via k-means clustering, with 512 clusters), and the prediction distribution for the next syllable’s prosody explicitly conditions on past outputs. Speech tokens are then generated autoregressively, conditioned on both text and all prosody/speech history to date.

Figure 2: Overall depiction of the LLM architecture integrating dynamic prosody prediction, showing alternating prosody/speech token generation and dependency on previously predicted outputs (see top: overall architecture, bottom: sequence schedule during inference).

The training objective is the weighted sum of cross-entropy losses for prosody and speech tokens, making the whole process end-to-end differentiable. At inference, both prosody and speech tokens for the target utterance are sampled with top-$p$/top-$k$ strategies and synthesized via a flow matching module.

Experimental Setup

The model is trained on a large-scale Mandarin dataset (approx. 50k hours) including WenetSpeech and the Mandarin portion of Emilia, with syllable boundaries provided by Montreal Forced Aligner. Test evaluations span three datasets: ESD (emotion-rich, 10 speakers, 5 emotions), an unlabeled internal set (diverse styles), and AISHELL-3 (prosodically neutral, 214 speakers).

Comparative baselines include:

• CosyVoice (50k): Original CosyVoice without explicit prosody modeling.
• CoT: CosyVoice with chain-of-thought, statically pre-computed, syllable-level prosody tokens.
• Proposed: CosyVoice with dynamic prosody prediction.

Evaluations include subjective MOS and preference tests (naturalness, speaker similarity), as well as objective measures such as CER (intelligibility), emotion similarity/accuracy (cosine similarity and recognition accuracy via emotion2vec+), and prosodic feature correlation/RMSE (pitch and energy).

Results

Subjective Evaluation: The proposed model achieves comparable or higher MOS than both baseline methods across all datasets, and consistently higher preference scores for speaker similarity, especially on emotion-rich and stylistically diverse corpora.

Objective Evaluation: The proposed model yields the lowest CER across all test sets, reflecting improved intelligibility. In emotion transfer, it surpasses both CosyVoice and CoT in emotion embedding similarity and recognition accuracy. In terms of prosodic correlation metrics, the model consistently achieves the highest correlations and lowest RMSE (for both pitch and energy), underscoring its ability to accurately model and reconstruct prosodic cues essential for style transfer and speaker similarity.

Further, head-to-head preference evaluations against other large-scale, open-source TTS models (Vevo1.5, F5-TTS, CosyVoice multilingual variant) indicate that the dynamic prosody prediction method outperforms all challengers on prosodically rich datasets even when trained on smaller data, strongly suggesting better data efficiency and improved generalization in style learning.

Theoretical and Practical Implications

This research convincingly demonstrates that LLM-based TTS systems benefit substantially from autoregressive, context-sensitive dynamic prosody modeling. By providing fine-grained, syllable-level control, the method directly mitigates the limitations of fixed or statically pre-computed prosody representations, resulting in improved style fidelity and voice imitation. Importantly, the increased data efficiency and the ability to better exploit limited training data address a critical barrier in scaling high-fidelity, style-diverse TTS models.

On a practical level, the framework is architecture-agnostic and can be retrofitted onto existing LLM-based TTS pipelines. This enables enhanced speaker and style similarity for personalized TTS in prosodically rich and emotionally expressive applications, even in low-resource training conditions.

Prospects for future research include extending dynamic prosody modeling to multi-lingual and code-switching scenarios, incorporating more refined stylistic controls (e.g., global-to-local attention over historical context), or integrating with style/control interfaces for user-driven expressive speech generation.

Conclusion

The dynamic prosody prediction framework delivers demonstrable improvements in speaker similarity for LLM-based TTS by integrating incremental, autoregressive prosody modeling conditioned on prior speech and context. Extensive empirical evidence confirms both subjective and objective gains over static and implicit alternatives, showing clear superiority in expressive, prosodically rich voice cloning for personalized speech generation. The findings motivate further exploration of context-sensitive prosody representations in scalable TTS architectures.