---
title: 'ProsodyLM: Prosody-Aware Speech Language Model'
url: https://www.emergentmind.com/topics/prosodylm
type: topic
---

# ProsodyLM: Prosody-Aware Speech Language Model

ProsodyLM is a speech language modeling approach that makes prosody an explicit part of the token stream rather than leaving it entangled inside generic acoustic tokens. Introduced as a speech LM with a speech encoder, an autoregressive language model, and a speech decoder, it represents each utterance as ordinary text followed by word-level prosody tokens, with the stated goal of enabling a text-initialized LLM to learn the interdependency between linguistic content and prosodic realization through pre-training alone. The reported emergent capabilities include prosody-sensitive speech generation, understanding of emotion and emphasis, and prosody consistency across longer contexts [2507.20091].

## 1. Conceptual basis and scope

ProsodyLM is motivated by the claim that a useful speech LM should learn three dependency types: **Content \(\rightarrow\) Prosody**, **Prosody \(\rightarrow\) Content**, and **Prosody \(\rightarrow\) Prosody**. In this formulation, prosody includes intonation, pitch movement, speaking rate, energy, pauses, emphasis, and expressive cues such as emotion or contrastive focus. The central criticism of mainstream speech-LM pipelines is that converting speech into generic discrete acoustic tokens is sub-optimal for prosody learning: such tokenizers often prioritize semantic or phonetic content, and the resulting tokens are argued to be too abstract for text-based LLMs to learn prosodic regularities efficiently. The paper further states that even with approximately **29.9k hours of LibriLight audiobooks**, LLMs trained on conventional speech tokens do not exhibit obvious emerging prosody processing capabilities from pre-training alone [2507.20091].

A closely related information-theoretic result appears in work on the redundancy between prosody and text. Using English audiobooks, that study finds a high degree of redundancy between words and prosodic features including intensity, duration, pauses, and pitch contours, while also showing that prosodic features cannot be fully predicted from text [2311.17233]. This is consistent with the ProsodyLM premise that prosody should neither be collapsed into text nor treated as fully independent of it.

The broader significance of ProsodyLM lies in a representational claim: the LM should not infer prosody only indirectly from low-level speech symbols, but should instead receive prosody in an aligned, interpretable form. In that sense, ProsodyLM is as much a tokenization proposal as it is a model architecture.

## 2. Tokenization and representational design

Each utterance is divided into sentences, and each sentence is serialized into a **[Text]** section and a **[Prosody]** section separated by **`<SEP1>`** and **`<SEP2>`**. The [Prosody] section contains an optional **[Global]** token and then alternates between pause tokens and word-level prosody descriptions in the pattern:

```text
[Global] <SIL> [Dur] word [5-dim Prosody] <SIL> [Dur] word [5-dim Prosody] ...
```

The word-level prosody vector has five dimensions [2507.20091]:

| Feature | Definition |
|---|---|
| Duration | Average duration of the phones in the word; later clarified as log-transformed |
| Log-F0 range | Difference between the 95% and 5% percentiles of the frame-level log-F0 contour within the word |
| Log-F0 median | Median of the frame-level log-F0 contour of the word |
| Log-F0 slope | Slope of the best-fit line to the word’s log-F0 contour |
| Log-energy | Log-norm of the mel-spectrogram for the word |

Pause duration is represented separately by **`<SIL> [Dur]`** between consecutive words. Each prosody dimension is clipped above and below, normalized into \([0,1]\), and quantized into **512 equally spaced bins**. The appendix specifies clipping percentiles for the five dimensions: Duration \(0.1\%/99.9\%\), Log-F0 Range \(0\%/99.9\%\), Log-F0 Median \(0.1\%/99.9\%\), Log-F0 Slope \(0.5\%/99.5\%\), and Energy \(0.1\%/100\%\) [2507.20091].

A notable design choice is that the word identity is repeated in the [Prosody] section even though it already appears in [Text]. Two reasons are given. First, LLMs are described as poor at counting and alignment; without repetition, the model would have to infer which prosody vector belongs to which word. Second, the [Text] section may contain written forms that differ from the spoken realization because of text normalization, such as “1996” versus “nineteen ninety six” [2507.20091].

The optional **[Global]** token summarizes sentence-level prosodic extremity. It is computed by aggregating frequencies of prosody tokens in the pre-training set; lower aggregate score corresponds to rarer, more “extreme” prosody. Empirically, the paper reports that including [Global] helps generation and control in some settings, especially energy, whereas omitting it is generally better for understanding tasks and dialogue continuation [2507.20091].

## 3. Model architecture, training objective, and decoder interface

ProsodyLM uses a speech encoder, an autoregressive LM, and a speech decoder, but the novelty is concentrated in the tokenization and the decoder conditioning path. The LM is initialized from **Llama3.1-8B-Instruct** and continuously pre-trained with **LoRA rank 64, \(\alpha = 16\)** under standard next-token prediction:

$$
p(x_1, x_2, \dots, x_T) = \prod_{t=1}^{T} p(x_t \mid x_{<t}).
$$

The training sequence concatenates instruction text, a speaker-F0 token, and alternating [Text]/[Prosody] sections. A speaker average log-F0 token is included because male and female speakers have different pitch ranges; to reduce overfitting to a single prompt, the pre-training uses **65 paraphrases** of “Spin a narrative” [2507.20091].

The preprocessing pipeline is explicitly word-aligned. Audio is segmented into sentences, transcribed with **Whisper**, aligned using **StyleTTS2’s text aligner**, converted into phone boundaries in spectrogram frames, and then aggregated into word-level prosody features. Pause durations are inserted as **`<SIL> [Dur]`** tokens. The main pre-training corpus is **29.9k hours of LibriLight audiobooks**. Speakers with **less than 1 hour** total speech are removed, and utterances where **more than 20%** of prosodic tokens are extreme or invalid are filtered out. Group-A models are trained for **3 epochs** with **batch size 64**, **initial learning rate \(1\times10^{-4}\)**, **warmup ratio 0.1**, and a **cosine LR scheduler**; the reported compute for ProsodyLM and MIMI-token is **< 2000 H100 GPU hours** [2507.20091].

The decoder adaptation is implemented on top of **StyleTTS2**. Instead of conditioning StyleTTS2’s duration and prosody predictors on a generic style embedding, ProsodyLM conditions them on LM-generated word-level prosody:

$$
\hat{\bm d}_{phone} = F_d(\bm h_{phone}, \bm d_{word}), \quad [\hat{\bm f}_{frame}, \hat{\bm e}_{frame}] = F_p(\bm h_{frame}, [\bm f_{word}, \bm e_{word}]),
$$

where \(\bm d_{word}\) is word-level duration, \(\bm f_{word}\) contains word-level F0 information, and \(\bm e_{word}\) is word-level energy. Word-level prosody tokens are embedded, repeated to phone or frame resolution, and added to input phone embeddings. Only the duration and prosody predictors are retrained; the rest of StyleTTS2 remains fixed [2507.20091].

The system supports at least two generation modes. In TTS mode, the [Text] section is provided and the LM generates the [Prosody] section before speech decoding. In speech continuation mode, tokenized reference speech is given as context and the LM continues with future [Text]/[Prosody] segments [2507.20091].

## 4. Reported emergent prosody capabilities

ProsodyLM organizes evaluation around the three dependency types that motivate the model: **Content \(\rightarrow\) Prosody**, **Prosody \(\rightarrow\) Content**, and **Prosody \(\rightarrow\) Prosody**. Across these settings, the paper’s central claim is that a prosody-friendly representation enables capabilities that do not emerge under matched pre-training with standard speech tokens [2507.20091].

For **Content \(\rightarrow\) Prosody**, direct style specification is tested using prompts such as “in a high voice,” “slowly,” or “quietly.” ProsodyLM reports **F0 pair 18.51**, **Duration pair 1.48**, and **Energy pair 0.171**, versus **3.25 / 0.28 / 0.125** for StyleTTS2, **1.86 / 0.42 / 0.126** for MIMI-token, and **5.53 / 0.54 / 0.037** for GLM4V-token. In indirect style inference, where style must be inferred from character descriptions rather than explicit adjectives, ProsodyLM reports **21.88 / 1.17 / 0.061**. In clarification passages, the model is described as capturing both **on-focus stress** and **post-focus compression**, and an appendix result shows that ProsodyLM **slows down by about 5%** in the final corrective utterance, whereas most baselines speed up [2507.20091].

Human evaluation also supports the claim that the model’s pre-trained prosody capabilities transfer to synthesis quality. In the **consistent-voice** condition, ProsodyLM obtains **4.07** for prosody MOS and **3.96** for naturalness, compared with **3.51 / 3.80** for StyleTTS2 and **2.82 / 2.34** for MIMI-token. In the **inconsistent-voice** condition, it reports **4.00** for prosody and **3.90** for naturalness, with prosody quality roughly comparable to stronger commercial systems in that setting [2507.20091].

For **Prosody \(\rightarrow\) Content**, ProsodyLM is probed on emphasis detection and emotion recognition. On **EmphAssess**, it reports **0.0656**, whereas MIMI-token and GLM4V-token report **0.0044** and **0.0073**. On the **JL corpus**, the average increase in log output probability for the correct emotion word is reported as **0.014** for happy, **0.203** for sad, **0.203** for excited, **0.021** for angry, and **0.063** for neutral; several baselines are near zero or negative on multiple emotions [2507.20091].

For **Prosody \(\rightarrow\) Prosody**, dialogue continuation tests whether style contrast in an initial turn is preserved in a later generated turn. ProsodyLM reports **F0 pair 47.9**, **Duration pair 1.09**, and **Energy pair 0.038**, while MIMI-token reports **0.73 / -0.61 / -0.147** and GLM4V-token **1.56 / 0.37 / -0.052**. The paper treats this as evidence that ProsodyLM transfers reference style contrast into subsequent dialogue more effectively than matched-tokenization baselines [2507.20091].

The paper also measures whether these gains come at the cost of ordinary language modeling. On held-out audiobook text, perplexity is **13.87** for ProsodyLM, compared with **24.53** for base Llama3.1-8B and **11.70** for a text-only audiobook-fine-tuned Llama. On Fluent Speech Commands intent classification, ProsodyLM reports **89.80**, compared with **91.10** for base Llama3.1-8B and **89.43** for the text-only fine-tuned model [2507.20091]. This suggests that explicit prosody modeling only modestly perturbs text capability.

## 5. Relation to adjacent prosody-aware modeling paradigms

ProsodyLM sits within a broader shift away from treating prosody as an implicit by-product of either text prediction or generic speech tokenization. One immediate contrast is the use of raw next-word predictability as a prosodic control signal. Work on LLM surprisal for TTS found only moderate correlation with acoustic prominence, showed that larger models and longer contexts often underpredict prominent words, and reported only minimal improvement over a FastPitch baseline when surprisal was used as conditioning [2306.09814]. That result places a lower ceiling on purely text-internal predictability as a proxy for salience and helps explain why ProsodyLM chooses explicit word-level prosody attributes rather than raw surprisal.

A second contrast concerns tokenization. A benchmark of discrete speech tokens shows that prosodic sensitivity depends strongly on SSL backbone, layer, cluster size, k-means training data, and smoothing; HuBERT and ContentVec are strongest for local relative prosodic contours, emotion2vec is stronger for utterance-level style, and intermediate layers often provide the most stable compromise between phonetic and prosodic information [2508.11224]. A related tokenizer study argues for a middle ground between phonetic and acoustic units: the **Phonological Tokenizer** fine-tunes phonetic tokens with ASR and resynthesis so that tokens retain linguistic and prosodic information while discarding much of speaker identity, improving emotion recognition and speech-LM continuation quality [2601.19781]. ProsodyLM is more explicit than either approach: instead of hoping prosody survives in clustered SSL space, it serializes prosody as separate aligned tokens.

A third adjacent direction converts speech into text-like semantics plus a parallel prosodic channel at the input side. **TextPro-SLM** uses WhisperPro to output synchronized text tokens and prosody embeddings, then feeds an LLM mostly text plus compact prosodic side information. Its stated goal is to make spoken input resemble that of a prosody-aware text LLM, and it reports the lowest modality gap among leading SLMs at both 3B and 7B scales while also performing strongly on paralinguistic tasks [2605.05927]. Relative to TextPro-SLM, ProsodyLM is closer to a discrete symbolic prosody language: it uses explicit word-level quantized attributes rather than continuous hidden-state embeddings.

In generative TTS, explicit prosody planning is increasingly handled with structured latent variables. **ProsodyFM** models phrase break location, break duration, and terminal intonation shape with a Phrase Break Encoder, a Duration Predictor, and a Terminal Intonation Encoder over a small token bank, improving phrasing, intonation, intelligibility, and OOD generalization [2412.11795]. **Dynamic Prosody Prediction in LLM-based TTS** instead predicts a **syllable-level discrete prosody token** online from text, speaker embedding, previous prosody tokens, and previously generated speech tokens, arguing that static whole-utterance prosody prediction is insufficient for speaker similarity [2606.15267]. These systems differ from ProsodyLM in granularity and downstream task, but they reinforce the same design trend: prosody becomes more tractable when represented as an aligned, manipulable structure rather than as an opaque global style vector.

## 6. Limitations and directions opened by the ProsodyLM formulation

The ProsodyLM paper explicitly notes two primary limitations. First, the proposed prosody token has **limited expressive power**, especially for **voice quality** changes. Second, decoder quality may be compromised because most of the speech decoder remains inherited from pre-trained StyleTTS2 while only some predictor modules are retrained [2507.20091]. The paper also reports an internal trade-off around the **[Global]** token: with [Global], direct style and indirect style results are **18.5 / 1.48 / 0.171** and **21.88 / 1.17 / 0.061**, whereas without [Global] they become **23.9 / 1.87 / 0.032** and **23.36 / 1.42 / 0.042**; dialogue continuation is also stronger without [Global], improving from **34.5 / 0.46 / -0.004** to **47.9 / 1.09 / 0.038** [2507.20091]. The representation is therefore not uniformly optimal across control and understanding.

A broader limitation is representational granularity. ProsodyLM uses word-level pause and five-dimensional prosody tokens, which is substantially richer than generic speech tokens but still coarser than phone-level local priors or phrase-structured prosody plans. Phone-level GMM-MDN modeling shows that local prosody distributions are inherently multimodal and that a single Gaussian prior under-represents plausible realizations [2102.00851]. This suggests that a future ProsodyLM may need more explicitly sequential local latent structure than a single word-level attribute bundle.

Another limitation concerns factorization. Multilingual TTS with decoupled pronunciation and prosody modeling reports gains when pronunciation and prosody are given separate encoders, separate decoders, and different language-conditioned sharing patterns, rather than being forced into one shared text representation [2209.06789]. ProsodyLM’s explicit prosody stream moves in this direction, but it still conditions a largely fixed decoder rather than fully separating phonological content, prosody, and speaker/style pathways.

A further extension is multimodal prosody learning. ProMode learns a fixed-length latent prosodic embedding from partially masked acoustics and aligned text, predicts explicit F0 and energy in masked regions, and improves downstream TTS prosody preference relative to several style encoders [2508.09389]. That result suggests a complementary route to ProsodyLM: retain symbolic word-level prosody tokens for LM readability, but also train richer latent prosody encoders that can capture variation not easily expressed by duration, log-F0 range, log-F0 median, log-F0 slope, and log-energy alone.

Taken together, these directions imply that ProsodyLM is best viewed not as a final account of prosody in speech LMs, but as a decisive representational proposal. Its main contribution is to show that explicit, aligned, interpretable prosody tokens can make pre-training itself yield nontrivial prosody-aware behavior in a speech LM [2507.20091]. A plausible implication is that subsequent systems will increasingly combine this explicit symbolic layer with finer local latents, stronger decoder adaptation, phrase-structured planning, and richer input-side prosody representations rather than reverting to generic acoustic token streams.

Source: https://www.emergentmind.com/topics/prosodylm