ProsodyLM: Prosody-Aware Speech Language Model

• ProsodyLM is a speech language model that explicitly integrates aligned word-level prosody tokens, bridging linguistic content with prosodic realization.
• It employs a specialized tokenization scheme that segments utterances into textual and prosodic components, enabling precise control over intonation, pitch, and timing.
• Emergent benefits include improved prosody-sensitive speech generation, enhanced emotion detection, and more consistent dialogue continuation.

ProsodyLM is a speech language modeling approach that makes prosody an explicit part of the token stream rather than leaving it entangled inside generic acoustic tokens. Introduced as a speech LM with a speech encoder, an autoregressive LLM, and a speech decoder, it represents each utterance as ordinary text followed by word-level prosody tokens, with the stated goal of enabling a text-initialized LLM to learn the interdependency between linguistic content and prosodic realization through pre-training alone. The reported emergent capabilities include prosody-sensitive speech generation, understanding of emotion and emphasis, and prosody consistency across longer contexts (Qian et al., 27 Jul 2025).

1. Conceptual basis and scope

ProsodyLM is motivated by the claim that a useful speech LM should learn three dependency types: Content $\rightarrow$ Prosody, Prosody $\rightarrow$ Content, and Prosody $\rightarrow$ Prosody. In this formulation, prosody includes intonation, pitch movement, speaking rate, energy, pauses, emphasis, and expressive cues such as emotion or contrastive focus. The central criticism of mainstream speech-LM pipelines is that converting speech into generic discrete acoustic tokens is sub-optimal for prosody learning: such tokenizers often prioritize semantic or phonetic content, and the resulting tokens are argued to be too abstract for text-based LLMs to learn prosodic regularities efficiently. The paper further states that even with approximately 29.9k hours of LibriLight audiobooks, LLMs trained on conventional speech tokens do not exhibit obvious emerging prosody processing capabilities from pre-training alone (Qian et al., 27 Jul 2025).

A closely related information-theoretic result appears in work on the redundancy between prosody and text. Using English audiobooks, that study finds a high degree of redundancy between words and prosodic features including intensity, duration, pauses, and pitch contours, while also showing that prosodic features cannot be fully predicted from text (Wolf et al., 2023). This is consistent with the ProsodyLM premise that prosody should neither be collapsed into text nor treated as fully independent of it.

The broader significance of ProsodyLM lies in a representational claim: the LM should not infer prosody only indirectly from low-level speech symbols, but should instead receive prosody in an aligned, interpretable form. In that sense, ProsodyLM is as much a tokenization proposal as it is a model architecture.

2. Tokenization and representational design

Each utterance is divided into sentences, and each sentence is serialized into a [Text] section and a [Prosody] section separated by <SEP1> and <SEP2>. The [Prosody] section contains an optional [Global] token and then alternates between pause tokens and word-level prosody descriptions in the pattern:

$\rightarrow$2

The word-level prosody vector has five dimensions (Qian et al., 27 Jul 2025):

Feature	Definition
Duration	Average duration of the phones in the word; later clarified as log-transformed
Log-F0 range	Difference between the 95% and 5% percentiles of the frame-level log-F0 contour within the word
Log-F0 median	Median of the frame-level log-F0 contour of the word
Log-F0 slope	Slope of the best-fit line to the word’s log-F0 contour
Log-energy	Log-norm of the mel-spectrogram for the word

Pause duration is represented separately by <SIL> [Dur] between consecutive words. Each prosody dimension is clipped above and below, normalized into $[0,1]$, and quantized into 512 equally spaced bins. The appendix specifies clipping percentiles for the five dimensions: Duration $0.1\%/99.9\%$, Log-F0 Range $0\%/99.9\%$, Log-F0 Median $0.1\%/99.9\%$, Log-F0 Slope $0.5\%/99.5\%$, and Energy $0.1\%/100\%$ (Qian et al., 27 Jul 2025).

A notable design choice is that the word identity is repeated in the [Prosody] section even though it already appears in [Text]. Two reasons are given. First, LLMs are described as poor at counting and alignment; without repetition, the model would have to infer which prosody vector belongs to which word. Second, the [Text] section may contain written forms that differ from the spoken realization because of text normalization, such as “1996” versus “nineteen ninety six” (Qian et al., 27 Jul 2025).

The optional [Global] token summarizes sentence-level prosodic extremity. It is computed by aggregating frequencies of prosody tokens in the pre-training set; lower aggregate score corresponds to rarer, more “extreme” prosody. Empirically, the paper reports that including [Global] helps generation and control in some settings, especially energy, whereas omitting it is generally better for understanding tasks and dialogue continuation (Qian et al., 27 Jul 2025).

3. Model architecture, training objective, and decoder interface

ProsodyLM uses a speech encoder, an autoregressive LM, and a speech decoder, but the novelty is concentrated in the tokenization and the decoder conditioning path. The LM is initialized from Llama3.1-8B-Instruct and continuously pre-trained with LoRA rank 64, $\alpha = 16$ under standard next-token prediction:

$\rightarrow$0

The training sequence concatenates instruction text, a speaker-F0 token, and alternating [Text]/[Prosody] sections. A speaker average log-F0 token is included because male and female speakers have different pitch ranges; to reduce overfitting to a single prompt, the pre-training uses 65 paraphrases of “Spin a narrative” (Qian et al., 27 Jul 2025).

The preprocessing pipeline is explicitly word-aligned. Audio is segmented into sentences, transcribed with Whisper, aligned using StyleTTS2’s text aligner, converted into phone boundaries in spectrogram frames, and then aggregated into word-level prosody features. Pause durations are inserted as <SIL> [Dur] tokens. The main pre-training corpus is 29.9k hours of LibriLight audiobooks. Speakers with less than 1 hour total speech are removed, and utterances where more than 20% of prosodic tokens are extreme or invalid are filtered out. Group-A models are trained for 3 epochs with batch size 64, initial learning rate $\rightarrow$1, warmup ratio 0.1, and a cosine LR scheduler; the reported compute for ProsodyLM and MIMI-token is < 2000 H100 GPU hours (Qian et al., 27 Jul 2025).

The decoder adaptation is implemented on top of StyleTTS2. Instead of conditioning StyleTTS2’s duration and prosody predictors on a generic style embedding, ProsodyLM conditions them on LM-generated word-level prosody:

$\rightarrow$2

where $\rightarrow$3 is word-level duration, $\rightarrow$4 contains word-level F0 information, and $\rightarrow$5 is word-level energy. Word-level prosody tokens are embedded, repeated to phone or frame resolution, and added to input phone embeddings. Only the duration and prosody predictors are retrained; the rest of StyleTTS2 remains fixed (Qian et al., 27 Jul 2025).

The system supports at least two generation modes. In TTS mode, the [Text] section is provided and the LM generates the [Prosody] section before speech decoding. In speech continuation mode, tokenized reference speech is given as context and the LM continues with future [Text]/[Prosody] segments (Qian et al., 27 Jul 2025).

4. Reported emergent prosody capabilities

ProsodyLM organizes evaluation around the three dependency types that motivate the model: Content $\rightarrow$6 Prosody, Prosody $\rightarrow$7 Content, and Prosody $\rightarrow$8 Prosody. Across these settings, the paper’s central claim is that a prosody-friendly representation enables capabilities that do not emerge under matched pre-training with standard speech tokens (Qian et al., 27 Jul 2025).

For Content $\rightarrow$9 Prosody, direct style specification is tested using prompts such as “in a high voice,” “slowly,” or “quietly.” ProsodyLM reports F0 pair 18.51, Duration pair 1.48, and Energy pair 0.171, versus 3.25 / 0.28 / 0.125 for StyleTTS2, 1.86 / 0.42 / 0.126 for MIMI-token, and 5.53 / 0.54 / 0.037 for GLM4V-token. In indirect style inference, where style must be inferred from character descriptions rather than explicit adjectives, ProsodyLM reports 21.88 / 1.17 / 0.061. In clarification passages, the model is described as capturing both on-focus stress and post-focus compression, and an appendix result shows that ProsodyLM slows down by about 5% in the final corrective utterance, whereas most baselines speed up (Qian et al., 27 Jul 2025).

Human evaluation also supports the claim that the model’s pre-trained prosody capabilities transfer to synthesis quality. In the consistent-voice condition, ProsodyLM obtains 4.07 for prosody MOS and 3.96 for naturalness, compared with 3.51 / 3.80 for StyleTTS2 and 2.82 / 2.34 for MIMI-token. In the inconsistent-voice condition, it reports 4.00 for prosody and 3.90 for naturalness, with prosody quality roughly comparable to stronger commercial systems in that setting (Qian et al., 27 Jul 2025).

For Prosody $\rightarrow$0 Content, ProsodyLM is probed on emphasis detection and emotion recognition. On EmphAssess, it reports 0.0656, whereas MIMI-token and GLM4V-token report 0.0044 and 0.0073. On the JL corpus, the average increase in log output probability for the correct emotion word is reported as 0.014 for happy, 0.203 for sad, 0.203 for excited, 0.021 for angry, and 0.063 for neutral; several baselines are near zero or negative on multiple emotions (Qian et al., 27 Jul 2025).

For Prosody $\rightarrow$1 Prosody, dialogue continuation tests whether style contrast in an initial turn is preserved in a later generated turn. ProsodyLM reports F0 pair 47.9, Duration pair 1.09, and Energy pair 0.038, while MIMI-token reports 0.73 / -0.61 / -0.147 and GLM4V-token 1.56 / 0.37 / -0.052. The paper treats this as evidence that ProsodyLM transfers reference style contrast into subsequent dialogue more effectively than matched-tokenization baselines (Qian et al., 27 Jul 2025).

The paper also measures whether these gains come at the cost of ordinary language modeling. On held-out audiobook text, perplexity is 13.87 for ProsodyLM, compared with 24.53 for base Llama3.1-8B and 11.70 for a text-only audiobook-fine-tuned Llama. On Fluent Speech Commands intent classification, ProsodyLM reports 89.80, compared with 91.10 for base Llama3.1-8B and 89.43 for the text-only fine-tuned model (Qian et al., 27 Jul 2025). This suggests that explicit prosody modeling only modestly perturbs text capability.

5. Relation to adjacent prosody-aware modeling paradigms

ProsodyLM sits within a broader shift away from treating prosody as an implicit by-product of either text prediction or generic speech tokenization. One immediate contrast is the use of raw next-word predictability as a prosodic control signal. Work on LLM surprisal for TTS found only moderate correlation with acoustic prominence, showed that larger models and longer contexts often underpredict prominent words, and reported only minimal improvement over a FastPitch baseline when surprisal was used as conditioning (Kakouros et al., 2023). That result places a lower ceiling on purely text-internal predictability as a proxy for salience and helps explain why ProsodyLM chooses explicit word-level prosody attributes rather than raw surprisal.

A second contrast concerns tokenization. A benchmark of discrete speech tokens shows that prosodic sensitivity depends strongly on SSL backbone, layer, cluster size, k-means training data, and smoothing; HuBERT and ContentVec are strongest for local relative prosodic contours, emotion2vec is stronger for utterance-level style, and intermediate layers often provide the most stable compromise between phonetic and prosodic information (Onda et al., 15 Aug 2025). A related tokenizer study argues for a middle ground between phonetic and acoustic units: the Phonological Tokenizer fine-tunes phonetic tokens with ASR and resynthesis so that tokens retain linguistic and prosodic information while discarding much of speaker identity, improving emotion recognition and speech-LM continuation quality (Onda et al., 27 Jan 2026). ProsodyLM is more explicit than either approach: instead of hoping prosody survives in clustered SSL space, it serializes prosody as separate aligned tokens.

A third adjacent direction converts speech into text-like semantics plus a parallel prosodic channel at the input side. TextPro-SLM uses WhisperPro to output synchronized text tokens and prosody embeddings, then feeds an LLM mostly text plus compact prosodic side information. Its stated goal is to make spoken input resemble that of a prosody-aware text LLM, and it reports the lowest modality gap among leading SLMs at both 3B and 7B scales while also performing strongly on paralinguistic tasks (Cui et al., 7 May 2026). Relative to TextPro-SLM, ProsodyLM is closer to a discrete symbolic prosody language: it uses explicit word-level quantized attributes rather than continuous hidden-state embeddings.

In generative TTS, explicit prosody planning is increasingly handled with structured latent variables. ProsodyFM models phrase break location, break duration, and terminal intonation shape with a Phrase Break Encoder, a Duration Predictor, and a Terminal Intonation Encoder over a small token bank, improving phrasing, intonation, intelligibility, and OOD generalization (He et al., 2024). Dynamic Prosody Prediction in LLM-based TTS instead predicts a syllable-level discrete prosody token online from text, speaker embedding, previous prosody tokens, and previously generated speech tokens, arguing that static whole-utterance prosody prediction is insufficient for speaker similarity (Mou et al., 13 Jun 2026). These systems differ from ProsodyLM in granularity and downstream task, but they reinforce the same design trend: prosody becomes more tractable when represented as an aligned, manipulable structure rather than as an opaque global style vector.

6. Limitations and directions opened by the ProsodyLM formulation

The ProsodyLM paper explicitly notes two primary limitations. First, the proposed prosody token has limited expressive power, especially for voice quality changes. Second, decoder quality may be compromised because most of the speech decoder remains inherited from pre-trained StyleTTS2 while only some predictor modules are retrained (Qian et al., 27 Jul 2025). The paper also reports an internal trade-off around the [Global] token: with [Global], direct style and indirect style results are 18.5 / 1.48 / 0.171 and 21.88 / 1.17 / 0.061, whereas without [Global] they become 23.9 / 1.87 / 0.032 and 23.36 / 1.42 / 0.042; dialogue continuation is also stronger without [Global], improving from 34.5 / 0.46 / -0.004 to 47.9 / 1.09 / 0.038 (Qian et al., 27 Jul 2025). The representation is therefore not uniformly optimal across control and understanding.

A broader limitation is representational granularity. ProsodyLM uses word-level pause and five-dimensional prosody tokens, which is substantially richer than generic speech tokens but still coarser than phone-level local priors or phrase-structured prosody plans. Phone-level GMM-MDN modeling shows that local prosody distributions are inherently multimodal and that a single Gaussian prior under-represents plausible realizations (Du et al., 2021). This suggests that a future ProsodyLM may need more explicitly sequential local latent structure than a single word-level attribute bundle.

Another limitation concerns factorization. Multilingual TTS with decoupled pronunciation and prosody modeling reports gains when pronunciation and prosody are given separate encoders, separate decoders, and different language-conditioned sharing patterns, rather than being forced into one shared text representation (Peng et al., 2022). ProsodyLM’s explicit prosody stream moves in this direction, but it still conditions a largely fixed decoder rather than fully separating phonological content, prosody, and speaker/style pathways.

A further extension is multimodal prosody learning. ProMode learns a fixed-length latent prosodic embedding from partially masked acoustics and aligned text, predicts explicit F0 and energy in masked regions, and improves downstream TTS prosody preference relative to several style encoders (Eren et al., 12 Aug 2025). That result suggests a complementary route to ProsodyLM: retain symbolic word-level prosody tokens for LM readability, but also train richer latent prosody encoders that can capture variation not easily expressed by duration, log-F0 range, log-F0 median, log-F0 slope, and log-energy alone.

Taken together, these directions imply that ProsodyLM is best viewed not as a final account of prosody in speech LMs, but as a decisive representational proposal. Its main contribution is to show that explicit, aligned, interpretable prosody tokens can make pre-training itself yield nontrivial prosody-aware behavior in a speech LM (Qian et al., 27 Jul 2025). A plausible implication is that subsequent systems will increasingly combine this explicit symbolic layer with finer local latents, stronger decoder adaptation, phrase-structured planning, and richer input-side prosody representations rather than reverting to generic acoustic token streams.