Dynamic Prosody Prediction

Updated 4 July 2026

Dynamic prosody prediction is the modeling of time-varying features like pitch, energy, and duration from text and contextual signals to enhance speech naturalness.
Recent methodologies integrate explicit control interfaces, latent-variable models, and discrete token approaches to address the one-to-many mapping in TTS.
Evaluation techniques now focus on perceptual quality and style transfer efficacy, balancing diversity, naturalness, and computation efficiency.

Searching arXiv for recent and foundational papers on dynamic prosody prediction in TTS and related prosody modeling. Dynamic prosody prediction is the modeling of time-varying prosodic structure—typically pitch, energy, duration, rhythm, pauses, and higher-level prominence—as a conditional process driven by text and, in some systems, by speaker identity, acoustic prompts, visual context, or previously generated speech. Across the recent literature, it is treated as a response to the one-to-many mapping from text to speech: the same text admits multiple valid prosodic realizations, and models that collapse this variation tend to produce over-smoothed or stylistically deficient speech (Li et al., 2023, Mou et al., 13 Jun 2026, Liu et al., 2024). The topic also extends beyond synthesis into evaluation: conditional prediction of prosody tokens has been used to score whether synthesized speech exhibits an expected prosodic structure for its text, without reference audio (Ulgen et al., 24 Sep 2025).

1. Conceptual scope and historical development

Early formulations treated dynamic prosody prediction as the generation of context-sensitive contour structure rather than as a monolithic style label. The Variational Prosody Model built on the Superposition of Functional Contours paradigm by representing prosody as overlapping function-specific prototypes whose shapes vary with linguistic and paralinguistic context, and by learning a latent space for this variation (Gerazov et al., 2018). This established a line of work in which prosody is decomposed into interpretable, temporally organized components rather than inferred solely from end-to-end acoustic regression.

Subsequent neural TTS systems moved toward explicit control of durations and pitch trajectories. Prosody-TTS modeled phoneme durations and frame-level $f_0$ as auxiliary tasks inside an end-to-end waveform generator, replacing attention with duration-based alignment and exposing rhythm and intonation as editable variables at inference (Pamisetty et al., 2021). In parallel, "Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selection" selected sentence-level acoustic embeddings by semantic and syntactic similarity, using linguistic structure to choose contextually appropriate prosody for isolated sentences and long-form reading (Tyagi et al., 2019).

A second strand introduced stochastic latent-variable modeling at finer temporal scales. CHiVE predicted frame-level $F0(t)$ , energy via $c0(t)$ , and phone durations with a hierarchical conditional variational architecture whose latent variable is a sentence-level prosody embedding (Wan et al., 2019). CAMP then shifted the bottleneck to the word level, learning one latent prosody vector per word or pause and predicting that sequence from semantic and syntactic context, motivated by the claim that prosody varies at a slower rate than segmental content (Hodari et al., 2020).

More recent work has diversified both the conditioning signals and the learning objectives. CLAPSpeech learned prosody from text context through contrastive language-audio pre-training (Ye et al., 2023). Diffusion-based models treated phoneme-wise prosody as a generative distribution rather than a deterministic target (Li et al., 2023, Oh et al., 2023, Liu et al., 2024). LLM-based TTS introduced syllable-level autoregressive prosody prediction conditioned on previously predicted speech tokens to improve style learning and speaker similarity (Mou et al., 13 Jun 2026). Evaluation research reframed the topic once more: TTScore-pro predicts phoneme-level prosody tokens from text and scores synthesized speech by the conditional likelihood of those tokens, turning dynamic prosody prediction into an objective evaluation mechanism (Ulgen et al., 24 Sep 2025).

2. Prosodic representations and temporal granularity

A central design choice is the granularity at which prosody is represented. Several systems use explicit continuous features. Prosody-TTS predicts phoneme-level durations and frame-level $f_0(t)$ , with band aperiodicity and mel spectrogram prediction downstream (Pamisetty et al., 2021). VisualSpeech uses the FastSpeech2 variance adaptor to predict per-phoneme duration, pitch, and energy, then expands them to the frame level by a length regulator (Que et al., 31 Jan 2025). Audiobook prosody prediction operates at the phrase level, modeling one value each for pitch, volume, and speaking rate per aligned segment (Pethe et al., 2023).

Other systems move to latent or discrete representations. CHiVE uses a sentence-level latent embedding to drive explicit phoneme and frame-level prosodic outputs (Wan et al., 2019). CAMP learns one latent vector per word or pause, explicitly imposing a word-rate bottleneck (Hodari et al., 2020). CLAPSpeech constructs phoneme-level and word-level embeddings in a joint text-audio space, with prosody defined as the pitch and duration variance of the same token under different contexts (Ye et al., 2023).

Discrete tokenization has become especially important in recent work. TTScore-pro uses FACodec prosody tokens originating from NaturalSpeech 3. These tokens are originally frame-level at 12.5 ms steps with a codebook size of 1024, but TTScore refactors them to phoneme level by pooling continuous prosody representations over phoneme spans and then applying residual vector quantization (Ulgen et al., 24 Sep 2025). In the LLM-based TTS method, each syllable is assigned a discrete prosody token obtained by k-means quantization of the vector $g_i = [d_i, e_i, h_i, r_i]$ , where the four features are duration, mean energy, mean pitch, and pitch range (Mou et al., 13 Jun 2026). ProMode predicts frame-level $F0$ , energy, Mel-10, and voiced/unvoiced structure from masked acoustics and aligned text, using a fixed-length latent prosody embedding inside a conditional decoder with temporally dependent modulation (Eren et al., 12 Aug 2025).

These representational choices imply different emphases. FACodec prosody tokens in TTScore-pro primarily encode $F0$ contour information, whereas the LLM-based syllable tokens explicitly include duration, mean energy, mean pitch, and pitch range (Ulgen et al., 24 Sep 2025, Mou et al., 13 Jun 2026). This suggests that "dynamic prosody prediction" is not a single representation class but a family of temporal abstractions ranging from explicit low-level trajectories to discrete symbolic surrogates.

3. Conditioning signals and temporal context

Text is the universal conditioning source, but the literature differs sharply on how much additional context is required. The redundancy study on audiobook speech found that a word’s prosodic features require an extended past context of 3–8 words to be reliably predicted, while redundancy with future words is restricted to 1–2 words (Regev et al., 14 Mar 2025). This provides a quantitative context horizon for streaming or low-latency systems.

Several models enrich text with semantic and syntactic context. CLAPSpeech combines phoneme and BPE streams, aligns them through word-level pooling and Word2Ph expansion, and learns token-specific prosody representations by contrasting same-token instances in different textual contexts (Ye et al., 2023). CAMP uses BERTBASE contextual embeddings and additional syntactic features such as POS tags, word class, compound-noun flags, and punctuation, although its ablations show that BERT alone suffices and that adding the syntactic encoders yields no further improvement (Hodari et al., 2020). Linguistics-driven acoustic embedding selection for newscaster speech likewise uses BERT sentence embeddings and syntactic distance vectors as retrieval criteria for sentence-level acoustic embeddings (Tyagi et al., 2019).

Other systems condition prosody on modalities beyond text. VisualSpeech cross-attends phoneme-level text queries to frame-level video features from Omnivore or ResNet50, producing a phone-aligned fused sequence that drives pitch, energy, and duration prediction (Que et al., 31 Jan 2025). ProMode conditions on both masked acoustics and time-aligned textual content, so that acoustic context contributes speaker idiosyncrasy and temporal continuity while text contributes phonological and structural cues (Eren et al., 12 Aug 2025). The LLM-based method goes further by conditioning the prosody of the current syllable on both previous prosody tokens $q_{1:i-1}$ and previously predicted speech tokens $S_{1:i-1}$ , formalized as

$y_i = p(C^p \mid v, X, q_{1:i-1}, S_{1:i-1}),$

with speech generation then conditioned on the newly predicted $F0(t)$ 0 (Mou et al., 13 Jun 2026).

This conditioning diversity reflects different notions of what must remain invariant and what must adapt. Speaker embeddings anchor timbre, while dynamic conditioning on previous speech or acoustic prompts attempts to capture how a speaker’s style unfolds over time rather than treating style as a fixed global code (Mou et al., 13 Jun 2026, Eren et al., 12 Aug 2025).

4. Modeling paradigms

Paradigm	Core representation	Representative papers
Superpositional and variational contour models	Function-specific contour prototypes and latent spaces	(Gerazov et al., 2018)
Hierarchical latent-variable TTS	Sentence- or word-level latents driving explicit prosody	(Wan et al., 2019, Hodari et al., 2020)
Explicit deterministic predictors	Duration, pitch, and energy heads in non-AR TTS	(Pamisetty et al., 2021, Que et al., 31 Jan 2025, Lam et al., 2024)
Autoregressive and token-based models	Syllable- or phoneme-level discrete prosody tokens	(Mou et al., 13 Jun 2026, Ulgen et al., 24 Sep 2025)
Flow- and diffusion-based generators	Sampleable distributions over phoneme-wise or latent prosody	(Klapsas et al., 2022, Li et al., 2023, Oh et al., 2023, Liu et al., 2024)

Deterministic explicit predictors remain important because they offer direct control interfaces. Prosody-TTS exposes duration and $F0(t)$ 1 editing (Pamisetty et al., 2021), VisualSpeech augments FastSpeech2’s variance adaptor with visual context (Que et al., 31 Jan 2025), and PRESENT modifies the inference-time outputs of duration, pitch, and energy predictors in a pretrained JETS model to enable zero-shot text-to-prosody control, including subphoneme-level pitch editing for Mandarin tones and question intonation (Lam et al., 2024).

Variational and flow-based systems address the one-to-many problem by making prosody stochastic. CHiVE uses a sentence-level latent in a hierarchical CVAE (Wan et al., 2019), while "Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesis" compares autoregressive priors, Glow-based priors, and a Dynamical VAE over phoneme-level latents extracted by an FVAE (Klapsas et al., 2022). The reported trade-off is explicit: normalizing flows increase variability and expressiveness but slightly reduce perceived quality, whereas DVAE improves quality at the cost of decreased expressiveness and variability (Klapsas et al., 2022).

Diffusion models generalize this stochastic view. "Diverse and Expressive Speech Prosody Prediction with Denoising Diffusion Probabilistic Model" predicts phoneme-wise pitch, energy, and duration vectors with a DDPM denoiser conditioned on text encoder features, using $F0(t)$ 2 diffusion steps and a non-causal WaveNet denoiser (Li et al., 2023). DiffProsody instead generates a latent prosody sequence with a DDGAN-style model and prosody conditional adversarial training, reducing sampling to $F0(t)$ 3 and reporting that prosody generation is 16 times faster than the conventional diffusion model (Oh et al., 2023). DiffStyleTTS combines utterance-level implicit style from GST with phoneme-level explicit prosody and uses improved classifier-free guidance to control the intensity of style guidance during diffusion sampling (Liu et al., 2024).

Autoregressive token models emphasize local temporal dependency. In the LLM-based TTS system, the full factorization is

$F0(t)$ 4

so prosody and speech tokens are generated in alternating syllable-level steps (Mou et al., 13 Jun 2026). TTScore-pro uses a formally similar autoregressive factorization for evaluation rather than synthesis,

$F0(t)$ 5

with teacher forcing used to compute average log-likelihood as a prosody score (Ulgen et al., 24 Sep 2025).

5. Evaluation methodologies and empirical findings

Evaluation remains a major difficulty because traditional reference-based prosody metrics correlate weakly or inconsistently with perception. TTScore directly addresses this issue. On SOMOS, traditional log $F0(t)$ 6-RMSE and $F0(t)$ 7-correlation yield near-zero or inconsistent correlations, whereas TTScore-pro shows moderate positive correlations with MOS; on VoiceMOS, TTScore-pro reaches utterance-level LCC/SRCC around $F0(t)$ 8 on all systems and system-level around $F0(t)$ 9 (Ulgen et al., 24 Sep 2025). The same work reports that scores on real speech from LibriSpeech-dev and VCTK are higher than on synthesized speech, and that TTScore-pro assigns lower scores when $c0(t)$ 0 is inverted around the mean or time-flipped (Ulgen et al., 24 Sep 2025).

Generation papers use a broader set of objective and subjective metrics. The LLM-based dynamic prosody model reports MOS values of $c0(t)$ 1 on ESD, $c0(t)$ 2 on an internal set, and $c0(t)$ 3 on AISHELL-3, alongside lower CER than CosyVoice(50k) and CoT and preference gains in speaker-similarity tests (Mou et al., 13 Jun 2026). DiffProsody reports MOS $c0(t)$ 4, CER $c0(t)$ 5, WER $c0(t)$ 6, and improved prosody metrics relative to FastSpeech 2 and ProsoSpeech on VCTK (Oh et al., 2023). DiffStyleTTS reports MOS $c0(t)$ 7, JS divergence of $c0(t)$ 8 for $c0(t)$ 9, $f_0(t)$ 0 for energy, and $f_0(t)$ 1 for duration, with RTF $f_0(t)$ 2 (Liu et al., 2024).

Contextual and multimodal predictors are also evaluated through downstream acoustics. VisualSpeech reduces prosody MSE relative to FastSpeech2, with pitch MSE improving from $f_0(t)$ 3 to $f_0(t)$ 4 and duration MSE from $f_0(t)$ 5 to $f_0(t)$ 6 or $f_0(t)$ 7 depending on the visual backbone (Que et al., 31 Jan 2025). ProMode improves frame-level and phoneme-level $f_0(t)$ 8 and energy metrics over StyleTTS2*, Wav2Vec2-SER*, and Emotion2Vec*, and when integrated into a diffusion TTS system improves WER, UTMOS, AutoPCP, and listener preference (Eren et al., 12 Aug 2025). Audiobook prosody prediction predicts pitch, volume, and speaking rate from phrase embeddings, with enhanced TTS showing higher correlation to human pitch in 22 out of 24 books and to human volume in 23 out of 24 books (Pethe et al., 2023).

Taken together, these results indicate a shift from scalar error analysis toward distributional, perceptual, and task-aligned evaluation. A plausible implication is that dynamic prosody prediction is increasingly judged by whether it supports naturalness, style transfer, intelligibility, and speaker similarity rather than by framewise proximity to a single reference contour.

6. Applications, controllability, and limitations

Dynamic prosody prediction supports several application classes. In synthesis, it enables controllable rhythm and intonation, as in Prosody-TTS and PRESENT (Pamisetty et al., 2021, Lam et al., 2024). In style transfer, it supports reference-driven or token-driven conditioning, as in CHiVE, CLAPSpeech, DiffStyleTTS, and the LLM-based syllable predictor (Wan et al., 2019, Ye et al., 2023, Liu et al., 2024, Mou et al., 13 Jun 2026). In evaluation, TTScore-pro functions as a reference-free criterion for prosodic appropriateness and has been proposed as a prospective auxiliary training signal or selection criterion, although the paper itself does not implement that extension (Ulgen et al., 24 Sep 2025). In zero-shot and cross-lingual settings, PRESENT uses explicit duration–pitch–energy editing to improve CER for German, Hungarian, Spanish, and Mandarin with an English-only JETS model (Lam et al., 2024).

The control interfaces vary. Some are latent and sampled, as in CHiVE, CAMP, VPM, and diffusion models (Wan et al., 2019, Hodari et al., 2020, Gerazov et al., 2018, Liu et al., 2024). Some are explicit continuous controls over duration or pitch scaling (Pamisetty et al., 2021, Lam et al., 2024). Some are discrete token sequences predicted autoregressively (Mou et al., 13 Jun 2026). DiffStyleTTS adds a guidance scale $f_0(t)$ 9 and correction scale $g_i = [d_i, e_i, h_i, r_i]$ 0 that control guidance intensity and mitigate overexposure artifacts at high guidance strengths (Liu et al., 2024).

Common limitations recur across the literature. TTScore-pro is limited by FACodec’s $g_i = [d_i, e_i, h_i, r_i]$ 1-centric prosody pathway: energy, duration, and rhythm are not directly modeled (Ulgen et al., 24 Sep 2025). Dynamic LLM-based TTS depends on reliable syllable boundaries from Montreal Forced Aligner and on the fidelity of the speech tokenizer (Mou et al., 13 Jun 2026). Diffusion-based methods often improve diversity at the cost of latency, as explicitly noted by the DDPM prosody predictor with real-time factor $g_i = [d_i, e_i, h_i, r_i]$ 2 for 500 diffusion steps (Li et al., 2023). CAMP does not model discourse beyond single sentences, and CLAPSpeech states that inter-sentence prosody for long-form TTS remains open (Hodari et al., 2020, Ye et al., 2023). Several studies are language- or domain-specific: audiobook prosody is derived from read narrative English (Pethe et al., 2023), the LLM-based dynamic syllable predictor is specific to Mandarin (Mou et al., 13 Jun 2026), and DiffStyleTTS is evaluated on a private Mandarin Chinese multi-speaker corpus (Liu et al., 2024).

Across these works, dynamic prosody prediction has become a unifying problem formulation rather than a single algorithmic recipe. It encompasses explicit variance-adaptor control, latent-variable modeling, token autoregression, multimodal conditioning, and reference-free evaluation. The shared technical objective is to model how prosodic structure evolves over time in a way that is context-sensitive, sampleable or controllable, and compatible with downstream judgments of naturalness, intelligibility, and style.