Emphasis Meets Emotion TTS

• The paper introduces EME-TTS, a framework that jointly models emotional expression and lexical emphasis using an emotional FastSpeech-2 backbone.
• It leverages weakly supervised emphasis pseudo-labels, variance-based prosodic features, and an Emphasis Perception Enhancement block to maintain clear emphasis under varying emotions.
• Evaluation shows improved emphasis recognition, emotion accuracy, and naturalness compared to baseline TTS models.

Searching arXiv for the primary paper and closely related works mentioned in the provided data. Emphasis Meets Emotion TTS (EME-TTS) is a text-to-speech framework for jointly modeling emotional expression and lexical emphasis in synthesized speech. It is introduced to address two specific research questions: how to effectively utilize emphasis to enhance the expressiveness of emotional speech, and how to maintain the perceptual clarity and stability of target emphasis across different emotions. The system is built on an emotional FastSpeech-2 backbone and combines weakly supervised emphasis pseudo-labels, variance-based emphasis features, and an Emphasis Perception Enhancement block to preserve locally salient emphasis under emotion conditioning. Reported results indicate gains in emphasis recognition, emotion recognition, expressiveness preference, and naturalness relative to the compared systems (Li et al., 16 Jul 2025).

1. Problem formulation and scope

Recent work in emotional TTS synthesis and emphasis-controllable speech synthesis has advanced significantly, but their interaction remains underexplored. EME-TTS is explicitly framed around that interaction. Its central premise is not merely that emotion and emphasis are both useful controls, but that they can interfere with or reinforce one another during synthesis. The framework therefore treats emphasis as a structured component of emotional prosody rather than as an isolated post-processing signal.

Within this formulation, emphasis has two roles. First, it is used to enhance the expressiveness of emotional speech. Second, it must remain perceptually stable when the same textual emphasis target is rendered under different emotions. These goals are operationalized through phoneme-level emphasis position pseudo-labels, variance-based local prosodic features, and attention-level enhancement of emphasized positions.

A common simplification in controllable TTS is to assume that global emotional style is sufficient to recover local salience. EME-TTS instead encodes local emphasis positions directly and couples them with emotion-dependent conditioning. This design choice reflects the paper’s emphasis on perceptual clarity and stability rather than on emotion transfer alone.

2. Architectural organization

EME-TTS is built on top of an emotional FastSpeech-2 backbone, specifically EmoSpeech, and consists of three major sub-networks: a phoneme encoder, a variance adapter with integrated emphasis modeling, and a frame decoder. The data flow is the standard non-autoregressive TTS pipeline described in the work: text is converted to a phoneme sequence, passed through the phoneme encoder, processed by the variance adapter, upsampled, decoded into a mel-spectrogram, and then rendered by a vocoder.

The model uses two conditioning signals. The first is an emotion embedding, denoted $c_{\text{emotion}}$, obtained from a one-hot emotion representation followed by a learned embedding. The second is a set of emphasis position pseudo-labels represented as binary start/end markers at the phoneme level. Emotion modeling resides in Conditional Cross-Attention layers and conditional layer normalization in the encoder and decoder, and the global prosodic predictors for pitch, duration, and energy in the variance adapter are also conditioned on emotion.

Emphasis control is realized in two places. In the variance adapter, emphasis pseudo-labels are injected to produce variance-based emphasis features. In the encoder and decoder stacks, every Transformer block is replaced by an Emphasis Perception Enhancement block, which explicitly boosts attention weights at emphasized positions. The resulting architecture is therefore hybrid in a precise sense: emotion enters as a global conditioning signal distributed through the stack, whereas emphasis is introduced both as a local prosodic perturbation and as an attention-level salience mechanism.

This organization is significant because it separates two different functions of prosody. Global emotional style is propagated through conditional modules that modulate the full sequence representation, while local emphasis is handled through phoneme-aligned masks and local deviations in pitch and duration. The architecture thereby encodes the interaction of style and salience without collapsing them into a single latent factor.

3. Weak supervision and pseudo-label generation

The framework avoids expensive human annotation of emphasis by using EmphaClass, a fine-tuned SSL speech model, to perform frame-level emphasis classification on raw emotional speech from the ESD corpus (Li et al., 16 Jul 2025). The pseudo-label generation procedure is defined in four steps. Each utterance audio sample from ESD is first processed by EmphaClass to obtain a frame-wise emphasis probability $p_t \in [0,1]$. Consecutive frames are then collapsed into word-level decisions using the known phoneme-to-frame alignment from the TTS aligner: if the mean of $p_t$ over frames aligned to a word $w_j$ exceeds a threshold $\theta$, that word is marked as emphasized. For each phoneme $i$, the binary label $e_i \in \{0,1\}$ and the word’s start and end indices are recorded. The resulting labels are described as emphasis pseudo-labels.

These pseudo-labels are used as high-quality proxies for human emphasis annotations. The weakly supervised setting is therefore not annotation-free; rather, supervision is shifted from manual labeling to automatic emphasis inference from aligned emotional speech. A plausible implication is that the quality of emphasis supervision depends materially on the classifier and on the phoneme-to-frame alignment, even though the paper’s core contribution lies in how these labels are used inside TTS.

The use of pseudo-labels also clarifies what “emphasis” means operationally in the framework. It is not an abstract latent property but a word-localized binary decision mapped down to phoneme spans. That choice aligns the control interface with the needs of a FastSpeech-style variance adapter and with the masking required by the attention modules.

4. Variance-based emphasis modeling

EME-TTS explicitly models local prosodic deviations in emphasized regions through two variance predictors in the variance adapter. Let $F_0(i)$ and $Dur(i)$ denote the ground-truth pitch and phoneme duration for phoneme $i$. Sentence-level means are defined as

$p_t \in [0,1]$0

For emphasized regions, the corresponding means are

$p_t \in [0,1]$1

The variance features are then defined as the differences between emphasized-region means and sentence means:

$p_t \in [0,1]$2

$p_t \in [0,1]$3

for all phonemes $p_t \in [0,1]$4 in an emphasized word; phonemes outside that word receive zero. These $p_t \in [0,1]$5-features are described as capturing the “extra” pitch and duration boost that human speakers deploy on emphasized words.

This formulation is notable for two reasons. First, the features are anchored to sentence-level baselines rather than learned as unconstrained residuals. Second, they are region-specific rather than token-generic: only phonemes inside emphasized words receive nonzero emphasis variance values. The ablation results attribute a large part of the emotional recognition gain, especially for negative emotions, to these variance-based emphasis features, indicating that the relevant acoustic cues are not exhausted by global emotion embeddings alone.

A common misconception would be to treat emphasis control here as a binary mask with no acoustic semantics. In EME-TTS, the mask is only part of the mechanism. The variance predictors encode explicit local prosodic deviations in pitch and duration, which the paper associates with the acoustic realization of emphasis.

5. Emphasis Perception Enhancement block

Every Transformer block in the phoneme encoder and frame decoder is replaced by an Emphasis Perception Enhancement (EPE) block. The block takes as input the hidden sequence $p_t \in [0,1]$6, the emotion embedding $p_t \in [0,1]$7, and an emphasis mask $p_t \in [0,1]$8 with ones at emphasized phoneme positions.

The internal computation proceeds through multi-head self-attention, conditional cross-attention, an emphasis adapter, conditional layer normalization, and a two-layer 1D convolution block with ReLU nonlinearity. In the conditional cross-attention stage,

$p_t \in [0,1]$9

$p_t$0

$p_t$1

The emphasis adapter modifies the attention weights through

$p_t$2

$p_t$3

$p_t$4

Here, $p_t$5 is a learned scalar and is set to $p_t$6 in the reported experiments. The stated purpose is to boost the attention weights at emphasized positions so that emphasized words remain salient even under strong emotional conditioning.

Conditional layer normalization is applied as in AdaSpeech4:

$p_t$7

where $p_t$8 and $p_t$9 are linear projections of $w_j$0. Residual connections and feed-forward projections produce the output $w_j$1.

The paper characterizes the end-to-end effect of the block as a representation that carries both global emotional style and locally enhanced emphasis perception (Li et al., 16 Jul 2025). Conceptually, the EPE block is the mechanism that prevents emotion conditioning from washing out local emphasis cues. The ablations further support this division of labor: variance-based features contribute heavily to emotional recognition, whereas the EPE block primarily improves emphasis discrimination without harming global prosody or speaker consistency.

6. Training objective and inference-time control

The total training loss $w_j$2 is a weighted sum of spectrogram reconstruction loss, prosody predictor losses in the variance adapter, and, optionally, a small adversarial or style-consistency loss on the emotion embedding if EmoSpeech’s adversarial speaker classifier is used. The spectrogram reconstruction term is

$w_j$3

The prosody losses are all mean squared error terms:

$w_j$4

$w_j$5

$w_j$6

$w_j$7

together with the standard energy predictor loss

$w_j$8

An example combined objective is

$w_j$9

The reported setting with all $\theta$0 values equal to $\theta$1 gave stable training, and slight up-weighting such as $\theta$2 is said to further improve emphasis clarity.

At inference time, human-defined emphasis pseudo-labels are unavailable. The framework therefore prompts GPT-4 with the sentence and a desired emotion chosen from neutral, angry, happy, sad, or surprise, asking which single word should be emphasized to best convey that emotion and requesting the 1-based word index. The returned index $\theta$3 is converted into a mask $\theta$4 with ones on all phonemes belonging to word $\theta$5. That mask is then used both in the variance adapter and in the emphasis adapter within each EPE block.

This inference procedure is a distinctive feature of EME-TTS. It converts emphasis prediction into a text-driven, emotion-aware control problem without additional annotation. A plausible implication is that passage-level context becomes important when selecting emphasis positions, a point consistent with the preference results reported for short-passage evaluation.

7. Evaluation, ablations, and interpretive significance

The experiments use the English subset of ESD, comprising 10 speakers, 5 emotions, and 350 utterances, with train, validation, and test splits following EmoSpeech. The compared baselines are CosyVoice2-0.5B-Instruct and EmoSpeech, and the vocoder is iSTFTNet. Evaluation uses the Emphasis Accuracy Test, Emotion Accuracy, Emotional Expressiveness Preference Test, Mean Opinion Score for naturalness, and NISQA.

The reported quantitative results are summarized below.

Evaluation aspect	Reported result
Emphasis recognition	EME-TTS w/ EPE: 78% correct; w/o EPE: 73%
Objective emotion accuracy	EME-TTS: 0.73; CosyVoice2: 0.68; EmoSpeech: 0.72
Subjective emotion accuracy	EME-TTS: 0.67; CosyVoice2: 0.48; EmoSpeech: 0.58
Naturalness	EME-TTS MOS: 4.22 ±0.28; NISQA: 3.76; EmoSpeech MOS: 4.14; NISQA: 3.71

In addition, EME-TTS is chosen most often as “most expressive” in the Emotional Expressiveness Preference Test, especially in passages where context cues can be exploited by the LLM emphasis predictor (Li et al., 16 Jul 2025). Emphasis recognition is described as nearly uniform across emotions except for a slight dip on surprise. The subjective emotion-recognition results indicate that angry and sad are markedly better recognized once emphasis is introduced.

The ablation findings are especially informative about the internal logic of the model. Disabling the variance-based emphasis features reduces emphasis accuracy by approximately five points. Replacing pseudo-labels with randomly assigned emphasis yields no improvement over plain EmoSpeech. Disabling the EPE block while retaining variance features recovers some expressiveness but weakens emphasis clarity, with listeners misidentifying emphasis 27% of the time versus 22% with EPE. The paper further states that the largest single gain in emotional recognition, especially for negative emotions, comes from variance-based emphasis, whereas the EPE block primarily improves emphasis discrimination without harming global prosody or speaker consistency.

Taken together, these results support a specific interpretation of the emotion–emphasis relation. The framework does not treat emphasis as an ornamental add-on to emotional speech synthesis. Instead, it models emphasis as a local prosodic and attentional variable whose interaction with emotion affects both expressiveness and recognizability. This suggests that controllable emotional TTS may benefit from explicitly separating global style conditioning from local salience control, while still allowing them to interact at the representation level.