---
title: 'Emo-FiLM: Word-Level Emotional Control in TTS'
url: https://www.emergentmind.com/topics/emo-film
type: topic
---

# Emo-FiLM: Word-Level Emotional Control in TTS

Emo-FiLM is a framework for fine-grained emotional speech synthesis in LLM-based text-to-speech, introduced to address the limitation of sentence-level emotional control in existing emotional TTS systems. Rather than conditioning an utterance with a single global label, reference utterance, or natural-language prompt, Emo-FiLM models emotion as a sequence of local states at roughly the word level, and injects these states into a pretrained LLM-TTS backbone through Feature-wise Linear Modulation (FiLM). The method aligns frame-level features from emotion2vec to words with Montreal Forced Aligner (MFA), predicts for each word both a discrete emotion category and a continuous intensity, and uses those signals to generate FiLM scale and shift parameters that directly modulate text hidden states before speech decoding [2509.20378].

## 1. Problem formulation and conceptual scope

Emo-FiLM targets **fine-grained emotional speech synthesis**, defined as speech synthesis in which emotion can vary within a sentence rather than only across utterances. The motivating observation is that many natural utterances are not emotionally homogeneous: a sentence may begin uncertain, turn disappointed midway, and end relieved or excited. Sentence-level conditioning is therefore inadequate because its control signal is global and static, whereas the desired control signal is a sequence of local emotional states aligned to specific words [2509.20378].

The framework is built around two modules. The first is **Fine-grained Emotion Annotation**, which derives word-level emotion annotations from speech. The second is **Emotion-modulated Generation**, which inserts an **E-FiLM** layer into a pretrained **LLM-TTS** system. The stated motivation for FiLM is that appending emotion tokens or adding emotion embeddings does not explicitly reshape the internal linguistic representation in a feature-wise manner. FiLM instead performs an affine transformation of hidden features, making emotional control more direct and localized at the word level [2509.20378].

This design places Emo-FiLM within a broader transition from global style control toward temporally resolved expressive control. A plausible implication is that the method treats emotional rendering as part of linguistic planning rather than as a late-stage prosodic correction, because the modulation acts directly on text embeddings or hidden states before speech-token decoding [2509.20378].

## 2. Word-level emotion annotation from speech

The annotation module begins with **frame-level emotion features** extracted from speech using **emotion2vec**. These frame-level features are aligned to transcript words using **Montreal Forced Aligner (MFA)**, which yields word boundaries. For each word, the method collects the emotion2vec frames falling within the corresponding interval and applies **masked average pooling** to obtain a fixed-dimensional pooled representation [2509.20378].

After frame-to-word aggregation, the method trains a lightweight Transformer over frame sequences to produce enhanced word-level emotional representations. The model includes multi-head self-attention, feed-forward layers, and residual connections, and the pooled representation is sent to two parallel heads: a **classification head** for a discrete emotion category and a **regression head** for a continuous emotion intensity in $[0,1]$. The training loss for this annotation model is a weighted sum of classification and regression losses, specifically cross-entropy for the category prediction and mean squared error for the intensity prediction [2509.20378].

An important implementation detail is that the paper states the authors use **global emotion labels from IEMOCAP and ESD to pseudo-label words** and train the word-level emotion prediction model. The supervision is therefore not described as fully manual word-level annotation throughout the main corpora. This suggests that the method relies on a weakly supervised conversion from utterance-level affect to word-level emotional trajectories, with emotion2vec and forced alignment supplying the local structure [2509.20378].

The annotation pipeline is thus explicitly alignment-dependent. It requires MFA during annotation or data construction, does not rely on an ASR-based alignment step in the described training setup, and does not introduce a separate duration model as part of the Emo-FiLM mechanism. This dependence on forced alignment is one of the system’s explicit practical constraints [2509.20378].

## 3. FiLM-based emotion-modulated generation

Emo-FiLM is built on a **frozen pretrained CosyVoice2** LLM-TTS framework. According to the paper, only the **E-FiLM module** and the **decoding head** are optimized during training. The generation module contains an **Emotion Encoder**, a **FiLM Layer**, and an **Emotion Supervision Mechanism**. The emotion encoder maps discrete emotion categories and continuous intensity labels into dense emotional feature vectors, which are fused with text embeddings to form word-aligned conditioning features [2509.20378].

The central modulation is the FiLM transformation
$$
\tilde{\mathbf{h}}_{\text{text}} = \gamma \odot \mathbf{h}_{\text{text}} + \beta,
$$
where $\mathbf{h}_{\text{text}}$ is the text hidden representation, $\gamma$ is the dimension-wise scaling factor, $\beta$ is the dimension-wise shifting factor, and $\odot$ denotes element-wise multiplication. Conceptually, for each word, the model uses its emotion category and intensity to generate $\gamma$ and $\beta$, then modulates the corresponding text representation. The stated rationale is that this permits local emotional variation across both words and feature dimensions, rather than imposing a single sentence-level style vector [2509.20378].

The training setup is multi-task. The speech generation objective is a label-smoothed cross-entropy over speech tokens,
$$
\mathcal{L}_{\text{TTS}} = -\frac{1}{M} \sum_{i=1}^{B} \sum_{t=1}^{L_i} \sum_{k=1}^{V} q(y_{i,t},k)\log p_{i,t}(k),
$$
and an auxiliary step-wise emotion classification loss is applied on hidden states,
$$
\mathcal{L}_{\text{emo}} = -\frac{1}{N} \sum_{i=1}^{B} \sum_{t=1}^{T_i} \log p_{i,t}(y_{i,t}).
$$
The joint objective is
$$
\mathcal{L} = \mathcal{L}_{\text{TTS}} + \lambda \mathcal{L}_{\text{emo}},
$$
with $\lambda$ balancing speech generation and emotion supervision. The implementation details reported in the paper include **Adam**, **batch size 4**, and **5 training epochs**. The paper also states that Emo-FiLM integrates **Flow Matching with HiFi-GAN** to achieve word-level emotion-controllable speech synthesis, although this integration is not described in detail within the reported method section [2509.20378].

The inference description states that the input text is jointly encoded with word-level emotional features, modulated by the FiLM layer, and then passed to the LLM-TTS decoder. However, the paper does not fully specify the user interface for supplying those word-level controls. A plausible implication is that controllability is architecturally present but operationally under-specified at inference time: the existence of word-level control is clear, but the exact mechanism by which a practitioner provides the per-word labels and intensities is not fully detailed [2509.20378].

## 4. Datasets and evaluation protocol

To support fine-grained evaluation, the paper introduces the **Fine-grained Emotion Dynamics Dataset (FEDD)**. FEDD contains **1,000 utterances** with emotional shifts across **5 speakers** and **5 emotions**. It includes **500 mild transitions** generated by **natural language instructions** and **500 strong transitions** created by **concatenating segments with different emotions from the same speaker**. The paper describes FEDD as carrying detailed annotations of emotional transitions, but does not fully specify the annotation schema, inter-annotator agreement, train/dev/test split proportions, speaker identities, or full language metadata [2509.20378].

For the global emotion synthesis benchmark, the paper constructs a test set from the **English portion of ESD** consisting of **10 speakers**, **5 emotions**—**Angry, Happy, Sad, Surprise, Neutral**—and **30 utterances per category**, for a total of **1,500 samples**. FEDD serves as the fine-grained dynamics benchmark, while ESD serves as the global emotion benchmark [2509.20378].

The baseline suite covers three representative emotional TTS paradigms: **EmoSpeech**, a label-based FastSpeech2-style system; **GenerSpeech**, a reference-audio-based emotional or style transfer system; and **CosyVoice2**, a natural-language-instruction or prompt-based LLM-TTS system. EmoSpeech and CosyVoice2 use test data with global emotion labels, whereas GenerSpeech is given reference audio with the same emotion and speaker as ground truth [2509.20378].

The evaluation protocol includes both objective and subjective metrics. Objective metrics are **Emo SIM**, **DTW**, and **WER** computed with **Whisper-Large-v3**. Emo SIM measures emotion similarity between synthesized and real speech, but the paper notes that it averages frame-level emotion vectors and may therefore obscure dynamic emotional variation; **DTW** is introduced specifically to better reflect emotional trajectories and transitions over time. Subjective metrics are **EMOS**, the mean emotion similarity opinion score, and **NMOS**, the mean opinion score for naturalness [2509.20378].

## 5. Empirical results and ablation evidence

On the global **ESD** benchmark, Emo-FiLM with **Global Label** achieves **98.78** Emo SIM, **23.98** DTW, **3.12** WER, **4.13** EMOS, and **4.23** NMOS. On **FEDD**, using **Fine-grained Label**, it achieves **99.32** Emo SIM, **49.62** DTW, **7.32** WER, **4.19** EMOS, and **4.23** NMOS. The paper emphasizes a **12.7% relative improvement in DTW over CosyVoice2** on ESD and a **9.1% DTW gain** on FEDD, and interprets these results as evidence that Emo-FiLM improves both global emotional fidelity and the temporal alignment of emotional dynamics [2509.20378].

The ablation study isolates several factors. Training with only global-level labels increases FEDD DTW from **49.6** to **52.7**. Removing word-level data tuning causes the largest degradation, raising FEDD DTW to **134.0**. Removing the auxiliary emotion loss increases FEDD DTW from **49.6** to **73.9**. Replacing the FiLM layer with simple addition worsens DTW to **70.5** according to the paper’s discussion. These results are the paper’s primary evidence that fine-grained supervision, explicit emotion preservation in hidden states, and affine modulation are all functionally important [2509.20378].

The paper also reports per-emotion classification accuracies on ESD, with Emo-FiLM obtaining **65.6%** on Happy, **71.2%** on Surprise, **70.4%** on Angry, and **78.5%** on Neutral. In a case study on the sentence “I thought the project was going to fail, but at the last minute it worked!”, the authors state that CosyVoice2 and EmoSpeech produce relatively flat $F_0$ trajectories, while Emo-FiLM tracks the ground-truth pitch evolution more closely around emotional boundaries [2509.20378].

Within the paper’s own interpretation, the key empirical claim is not merely higher average emotional similarity, but better modeling of **emotion dynamics across time**. This suggests that the benefit of FiLM is not only stronger conditioning strength but also better temporal placement of emotional changes, because the modulation is word-aligned rather than utterance-global [2509.20378].

## 6. Relation to adjacent emotion-conditioning paradigms and stated limitations

Emo-FiLM is explicit about using canonical **FiLM** for emotion control, and this distinguishes it from several neighboring emotion-conditioned architectures. **EmoDubber** addresses emotion-controllable audiovisual movie dubbing, but its emotion mechanism is not feature-wise affine modulation; instead, it steers a flow-matching generative trajectory in mel space with positive and negative classifier guidance, while speaker adaptation uses style-affine conditioning [2412.08988]. **VidEmo** is an emotion-centric video foundation model organized around affective-tree reasoning and GRPO-based post-training, and does not introduce FiLM or feature-wise affine modulation [2511.02712]. **EmoVid** improves emotional controllability in text-to-video and image-to-video generation through prompt-level emotion conditioning plus LoRA fine-tuning of Wan2.1, rather than FiLM-style internal modulation [2511.11002]. **EmoFace** uses separate speech and emotion encoders to predict MetaHuman rig controls for emotional 3D face animation, but does not present FiLM or a formal disentanglement framework [2407.12501]. Earlier affective modeling in narratives and cross-modal alignment, such as emotion-flow encoding for movie-tag prediction and EMID’s emotion-aware music–image matching, likewise uses temporal emotion representations or reranking rather than feature-wise linear modulation [1808.04943]; [2308.07622].

This comparison clarifies the specific role of Emo-FiLM within the broader literature. In adjacent systems, emotion is often injected as global prompting, reference transfer, classifier guidance, structured reasoning, or auxiliary affective alignment. Emo-FiLM instead places emotion control inside the linguistic hidden-state pipeline itself, via per-word affine modulation of the representations from which speech tokens are generated. A plausible implication is that the method is more tightly coupled to local lexical realization than approaches that steer generation only at the prompt level or only in the decoder trajectory [2509.20378].

The paper also reports several limitations. Some equations are typeset incorrectly in the manuscript. The inference-time control interface is not fully described. The exact FiLM placement inside the backbone and the dimensional details of the emotion encoder are sparse. FEDD is described at a high level rather than with a fully specified annotation protocol. The method depends on MFA and is therefore not alignment-free. The authors further claim generality for the approach, in the sense that FiLM is a lightweight conditioning mechanism that could in principle be inserted into other neural TTS backbones, but they do not experimentally demonstrate integration across multiple backbones beyond the frozen CosyVoice2 setting [2509.20378].

In that sense, Emo-FiLM is best understood as a word-level emotional control mechanism for LLM-based TTS whose central technical claim is narrow but consequential: local emotion trajectories should directly modulate text hidden states, not merely accompany them as global side information. The available evidence supports that claim within one pretrained LLM-TTS backbone and under both global and dynamic-emotion evaluation regimes [2509.20378].

Source: https://www.emergentmind.com/topics/emo-film