Emo-FiLM: Word-Level Emotional Control in TTS

• The paper presents a novel FiLM-based approach that modulates text embeddings with word-level emotional states for fine-grained TTS.
• It employs forced alignment with emotion2vec and a lightweight Transformer to derive both discrete emotion categories and continuous intensity measures per word.
• Empirical results show improved DTW scores and naturalness, validating the benefits of localized, temporally resolved emotional control over global labels.

Emo-FiLM is a framework for fine-grained emotional speech synthesis in LLM-based text-to-speech, introduced to address the limitation of sentence-level emotional control in existing emotional TTS systems. Rather than conditioning an utterance with a single global label, reference utterance, or natural-language prompt, Emo-FiLM models emotion as a sequence of local states at roughly the word level, and injects these states into a pretrained LLM-TTS backbone through Feature-wise Linear Modulation (FiLM). The method aligns frame-level features from emotion2vec to words with Montreal Forced Aligner (MFA), predicts for each word both a discrete emotion category and a continuous intensity, and uses those signals to generate FiLM scale and shift parameters that directly modulate text hidden states before speech decoding (Wang et al., 20 Sep 2025).

1. Problem formulation and conceptual scope

Emo-FiLM targets fine-grained emotional speech synthesis, defined as speech synthesis in which emotion can vary within a sentence rather than only across utterances. The motivating observation is that many natural utterances are not emotionally homogeneous: a sentence may begin uncertain, turn disappointed midway, and end relieved or excited. Sentence-level conditioning is therefore inadequate because its control signal is global and static, whereas the desired control signal is a sequence of local emotional states aligned to specific words (Wang et al., 20 Sep 2025).

The framework is built around two modules. The first is Fine-grained Emotion Annotation, which derives word-level emotion annotations from speech. The second is Emotion-modulated Generation, which inserts an E-FiLM layer into a pretrained LLM-TTS system. The stated motivation for FiLM is that appending emotion tokens or adding emotion embeddings does not explicitly reshape the internal linguistic representation in a feature-wise manner. FiLM instead performs an affine transformation of hidden features, making emotional control more direct and localized at the word level (Wang et al., 20 Sep 2025).

This design places Emo-FiLM within a broader transition from global style control toward temporally resolved expressive control. A plausible implication is that the method treats emotional rendering as part of linguistic planning rather than as a late-stage prosodic correction, because the modulation acts directly on text embeddings or hidden states before speech-token decoding (Wang et al., 20 Sep 2025).

2. Word-level emotion annotation from speech

The annotation module begins with frame-level emotion features extracted from speech using emotion2vec. These frame-level features are aligned to transcript words using Montreal Forced Aligner (MFA), which yields word boundaries. For each word, the method collects the emotion2vec frames falling within the corresponding interval and applies masked average pooling to obtain a fixed-dimensional pooled representation (Wang et al., 20 Sep 2025).

After frame-to-word aggregation, the method trains a lightweight Transformer over frame sequences to produce enhanced word-level emotional representations. The model includes multi-head self-attention, feed-forward layers, and residual connections, and the pooled representation is sent to two parallel heads: a classification head for a discrete emotion category and a regression head for a continuous emotion intensity in $[0,1]$. The training loss for this annotation model is a weighted sum of classification and regression losses, specifically cross-entropy for the category prediction and mean squared error for the intensity prediction (Wang et al., 20 Sep 2025).

An important implementation detail is that the paper states the authors use global emotion labels from IEMOCAP and ESD to pseudo-label words and train the word-level emotion prediction model. The supervision is therefore not described as fully manual word-level annotation throughout the main corpora. This suggests that the method relies on a weakly supervised conversion from utterance-level affect to word-level emotional trajectories, with emotion2vec and forced alignment supplying the local structure (Wang et al., 20 Sep 2025).

The annotation pipeline is thus explicitly alignment-dependent. It requires MFA during annotation or data construction, does not rely on an ASR-based alignment step in the described training setup, and does not introduce a separate duration model as part of the Emo-FiLM mechanism. This dependence on forced alignment is one of the system’s explicit practical constraints (Wang et al., 20 Sep 2025).

3. FiLM-based emotion-modulated generation

Emo-FiLM is built on a frozen pretrained CosyVoice2 LLM-TTS framework. According to the paper, only the E-FiLM module and the decoding head are optimized during training. The generation module contains an Emotion Encoder, a FiLM Layer, and an Emotion Supervision Mechanism. The emotion encoder maps discrete emotion categories and continuous intensity labels into dense emotional feature vectors, which are fused with text embeddings to form word-aligned conditioning features (Wang et al., 20 Sep 2025).

The central modulation is the FiLM transformation

$$\tilde{\mathbf{h}}_{\text{text}} = \gamma \odot \mathbf{h}_{\text{text}} + \beta,$$

where $\mathbf{h}_{\text{text}}$ is the text hidden representation, $\gamma$ is the dimension-wise scaling factor, $\beta$ is the dimension-wise shifting factor, and $\odot$ denotes element-wise multiplication. Conceptually, for each word, the model uses its emotion category and intensity to generate $\gamma$ and $\beta$, then modulates the corresponding text representation. The stated rationale is that this permits local emotional variation across both words and feature dimensions, rather than imposing a single sentence-level style vector (Wang et al., 20 Sep 2025).

The training setup is multi-task. The speech generation objective is a label-smoothed cross-entropy over speech tokens,

$$\mathcal{L}_{\text{TTS}} = -\frac{1}{M} \sum_{i=1}^{B} \sum_{t=1}^{L_i} \sum_{k=1}^{V} q(y_{i,t},k)\log p_{i,t}(k),$$

and an auxiliary step-wise emotion classification loss is applied on hidden states,

$$\mathcal{L}_{\text{emo}} = -\frac{1}{N} \sum_{i=1}^{B} \sum_{t=1}^{T_i} \log p_{i,t}(y_{i,t}).$$

The joint objective is

$$\tilde{\mathbf{h}}_{\text{text}} = \gamma \odot \mathbf{h}_{\text{text}} + \beta,$$0

with $$\tilde{\mathbf{h}}_{\text{text}} = \gamma \odot \mathbf{h}_{\text{text}} + \beta,$$1 balancing speech generation and emotion supervision. The implementation details reported in the paper include Adam, batch size 4, and 5 training epochs. The paper also states that Emo-FiLM integrates Flow Matching with HiFi-GAN to achieve word-level emotion-controllable speech synthesis, although this integration is not described in detail within the reported method section (Wang et al., 20 Sep 2025).

The inference description states that the input text is jointly encoded with word-level emotional features, modulated by the FiLM layer, and then passed to the LLM-TTS decoder. However, the paper does not fully specify the user interface for supplying those word-level controls. A plausible implication is that controllability is architecturally present but operationally under-specified at inference time: the existence of word-level control is clear, but the exact mechanism by which a practitioner provides the per-word labels and intensities is not fully detailed (Wang et al., 20 Sep 2025).

4. Datasets and evaluation protocol

To support fine-grained evaluation, the paper introduces the Fine-grained Emotion Dynamics Dataset (FEDD). FEDD contains 1,000 utterances with emotional shifts across 5 speakers and 5 emotions. It includes 500 mild transitions generated by natural language instructions and 500 strong transitions created by concatenating segments with different emotions from the same speaker. The paper describes FEDD as carrying detailed annotations of emotional transitions, but does not fully specify the annotation schema, inter-annotator agreement, train/dev/test split proportions, speaker identities, or full language metadata (Wang et al., 20 Sep 2025).

For the global emotion synthesis benchmark, the paper constructs a test set from the English portion of ESD consisting of 10 speakers, 5 emotions—Angry, Happy, Sad, Surprise, Neutral—and 30 utterances per category, for a total of 1,500 samples. FEDD serves as the fine-grained dynamics benchmark, while ESD serves as the global emotion benchmark (Wang et al., 20 Sep 2025).

The baseline suite covers three representative emotional TTS paradigms: EmoSpeech, a label-based FastSpeech2-style system; GenerSpeech, a reference-audio-based emotional or style transfer system; and CosyVoice2, a natural-language-instruction or prompt-based LLM-TTS system. EmoSpeech and CosyVoice2 use test data with global emotion labels, whereas GenerSpeech is given reference audio with the same emotion and speaker as ground truth (Wang et al., 20 Sep 2025).

The evaluation protocol includes both objective and subjective metrics. Objective metrics are Emo SIM, DTW, and WER computed with Whisper-Large-v3. Emo SIM measures emotion similarity between synthesized and real speech, but the paper notes that it averages frame-level emotion vectors and may therefore obscure dynamic emotional variation; DTW is introduced specifically to better reflect emotional trajectories and transitions over time. Subjective metrics are EMOS, the mean emotion similarity opinion score, and NMOS, the mean opinion score for naturalness (Wang et al., 20 Sep 2025).

5. Empirical results and ablation evidence

On the global ESD benchmark, Emo-FiLM with Global Label achieves 98.78 Emo SIM, 23.98 DTW, 3.12 WER, 4.13 EMOS, and 4.23 NMOS. On FEDD, using Fine-grained Label, it achieves 99.32 Emo SIM, 49.62 DTW, 7.32 WER, 4.19 EMOS, and 4.23 NMOS. The paper emphasizes a 12.7% relative improvement in DTW over CosyVoice2 on ESD and a 9.1% DTW gain on FEDD, and interprets these results as evidence that Emo-FiLM improves both global emotional fidelity and the temporal alignment of emotional dynamics (Wang et al., 20 Sep 2025).

The ablation study isolates several factors. Training with only global-level labels increases FEDD DTW from 49.6 to 52.7. Removing word-level data tuning causes the largest degradation, raising FEDD DTW to 134.0. Removing the auxiliary emotion loss increases FEDD DTW from 49.6 to 73.9. Replacing the FiLM layer with simple addition worsens DTW to 70.5 according to the paper’s discussion. These results are the paper’s primary evidence that fine-grained supervision, explicit emotion preservation in hidden states, and affine modulation are all functionally important (Wang et al., 20 Sep 2025).

The paper also reports per-emotion classification accuracies on ESD, with Emo-FiLM obtaining 65.6% on Happy, 71.2% on Surprise, 70.4% on Angry, and 78.5% on Neutral. In a case study on the sentence “I thought the project was going to fail, but at the last minute it worked!”, the authors state that CosyVoice2 and EmoSpeech produce relatively flat $$\tilde{\mathbf{h}}_{\text{text}} = \gamma \odot \mathbf{h}_{\text{text}} + \beta,$$2 trajectories, while Emo-FiLM tracks the ground-truth pitch evolution more closely around emotional boundaries (Wang et al., 20 Sep 2025).

Within the paper’s own interpretation, the key empirical claim is not merely higher average emotional similarity, but better modeling of emotion dynamics across time. This suggests that the benefit of FiLM is not only stronger conditioning strength but also better temporal placement of emotional changes, because the modulation is word-aligned rather than utterance-global (Wang et al., 20 Sep 2025).

6. Relation to adjacent emotion-conditioning paradigms and stated limitations

Emo-FiLM is explicit about using canonical FiLM for emotion control, and this distinguishes it from several neighboring emotion-conditioned architectures. EmoDubber addresses emotion-controllable audiovisual movie dubbing, but its emotion mechanism is not feature-wise affine modulation; instead, it steers a flow-matching generative trajectory in mel space with positive and negative classifier guidance, while speaker adaptation uses style-affine conditioning (Cong et al., 2024). VidEmo is an emotion-centric video foundation model organized around affective-tree reasoning and GRPO-based post-training, and does not introduce FiLM or feature-wise affine modulation (Zhang et al., 4 Nov 2025). EmoVid improves emotional controllability in text-to-video and image-to-video generation through prompt-level emotion conditioning plus LoRA fine-tuning of Wan2.1, rather than FiLM-style internal modulation (Qiu et al., 14 Nov 2025). EmoFace uses separate speech and emotion encoders to predict MetaHuman rig controls for emotional 3D face animation, but does not present FiLM or a formal disentanglement framework (Liu et al., 2024). Earlier affective modeling in narratives and cross-modal alignment, such as emotion-flow encoding for movie-tag prediction and EMID’s emotion-aware music–image matching, likewise uses temporal emotion representations or reranking rather than feature-wise linear modulation (Kar et al., 2018); (Zou et al., 2023).

This comparison clarifies the specific role of Emo-FiLM within the broader literature. In adjacent systems, emotion is often injected as global prompting, reference transfer, classifier guidance, structured reasoning, or auxiliary affective alignment. Emo-FiLM instead places emotion control inside the linguistic hidden-state pipeline itself, via per-word affine modulation of the representations from which speech tokens are generated. A plausible implication is that the method is more tightly coupled to local lexical realization than approaches that steer generation only at the prompt level or only in the decoder trajectory (Wang et al., 20 Sep 2025).

The paper also reports several limitations. Some equations are typeset incorrectly in the manuscript. The inference-time control interface is not fully described. The exact FiLM placement inside the backbone and the dimensional details of the emotion encoder are sparse. FEDD is described at a high level rather than with a fully specified annotation protocol. The method depends on MFA and is therefore not alignment-free. The authors further claim generality for the approach, in the sense that FiLM is a lightweight conditioning mechanism that could in principle be inserted into other neural TTS backbones, but they do not experimentally demonstrate integration across multiple backbones beyond the frozen CosyVoice2 setting (Wang et al., 20 Sep 2025).

In that sense, Emo-FiLM is best understood as a word-level emotional control mechanism for LLM-based TTS whose central technical claim is narrow but consequential: local emotion trajectories should directly modulate text hidden states, not merely accompany them as global side information. The available evidence supports that claim within one pretrained LLM-TTS backbone and under both global and dynamic-emotion evaluation regimes (Wang et al., 20 Sep 2025).