FEDD: Fine-Grained Emotion Dynamics Dataset

• FEDD is a speech dataset that provides fine-grained emotional annotations at the word level, combining discrete emotion categories with continuous intensity values.
• It comprises 1,000 utterances from five professional voice actors, balanced across ten emotion transition pairs with both mild and strong shifts.
• The dataset’s dual-label approach, using MFA alignment and emotion2vec, supports dynamic emotion modeling and refined control in TTS systems.

Searching arXiv for the specified paper and supporting work on emotion2vec. The Fine-grained Emotion Dynamics Dataset (FEDD) is a speech dataset introduced to support fine-grained emotional speech synthesis, particularly settings in which emotion changes within a sentence rather than remaining globally uniform. FEDD was presented in connection with the Emo-FiLM framework for LLM-based TTS, where it serves as an evaluation resource with detailed annotations of emotional transitions at word level rather than at sentence level (Wang et al., 20 Sep 2025). Its defining characteristic is the combination of discrete emotion categories and continuous intensity values for individual words, enabling analysis of both the loci and the magnitude of emotion shifts within an utterance. The dataset is positioned against existing emotional text-to-speech paradigms that rely on sentence-level control through predefined labels, reference audio, or natural language prompts, which the source work identifies as insufficient for capturing dynamic within-sentence shifts (Wang et al., 20 Sep 2025).

1. Definition and research setting

FEDD is a dataset of English emotional speech containing 1,000 utterances from five professional voice actors. It was constructed specifically to support the study of fine-grained emotional dynamics in speech synthesis, with annotations designed to represent emotion transitions at word-level temporal granularity (Wang et al., 20 Sep 2025). In this setting, each word receives both a discrete category label and a continuous intensity value, permitting a representation of emotional expression that is temporally localized and amplitude-sensitive.

The source work situates FEDD within emotional text-to-speech research, where sentence-level emotional control is common but limited. FEDD addresses this limitation by operationalizing emotion as a trajectory over words rather than as a single utterance-wide condition. This suggests a shift from global style conditioning toward token- or span-level affective control, which is particularly relevant for models that directly modulate text representations during synthesis (Wang et al., 20 Sep 2025).

2. Corpus composition and transition design

FEDD contains 1,000 utterances in English, all recorded by five professional voice actors. Audio segments were drawn from scripted recordings in studio conditions at 16 kHz and 24-bit using the same five speakers. The corpus is divided into two equal subsets with distinct transition regimes (Wang et al., 20 Sep 2025).

Five hundred utterances are “mild-transition” examples elicited in a single take by directing speakers via natural-language emotion instructions such as “start surprised, finish joyful.” Five hundred further utterances are “strong-transition” examples created by concatenating two single-emotion segments from the same speaker after trimming silences. Global per-utterance emotion labels were bootstrapped via an off-the-shelf emotion2vec-plus-large model for reference (Wang et al., 20 Sep 2025).

The transition space is explicitly balanced. Each of the five emotions appears roughly 200 times as the start emotion and 200 times as the end emotion. Ten possible ordered transitions are instantiated, and each occurs in 100 utterances, divided into 50 mild and 50 strong examples. The emotion categories used are Angry, Happy, Sad, Surprise, and Neutral (Wang et al., 20 Sep 2025).

Property	Value
Total utterances	1,000
Speakers	5 professional voice actors
Language	English
Emotion categories	Angry, Happy, Sad, Surprise, Neutral
Transition types	Mild-transition, strong-transition
Ordered transitions	10, each with 100 utterances

This design makes FEDD unusual among emotional speech resources because it is organized around transitions rather than only static states. The source text further characterizes the dataset as balancing “natural” in-take mild transitions and “engineered” concatenated strong transitions across all pairwise emotion combinations (Wang et al., 20 Sep 2025).

3. Annotation targets and temporal granularity

FEDD uses word-level annotation. For each word, the dataset defines a discrete category label and a continuous intensity label. The discrete label is denoted

$$y^{class} \in \{\text{Angry, Happy, Sad, Surprise, Neutral}\},$$

and the continuous intensity is denoted

$y^{dim} \in [0,1].$

The temporal unit is the word, not the sentence, phrase, or fixed-duration frame sequence (Wang et al., 20 Sep 2025).

The alignment and annotation procedure begins with Montreal Forced Aligner (MFA), which aligns transcript to audio and produces frame-to-word boundaries. A pre-trained emotion2vec model then extracts frame-level feature vectors $x_{1:T}$ for each word span. These frame-level vectors are aggregated by masked average pooling into a pooled representation $f \in \mathbb{R}^d$ (Wang et al., 20 Sep 2025). The source specifies two output heads on a lightweight Transformer: one predicts the discrete category via softmax, and the other predicts intensity via sigmoid. Word-level shifts are detected whenever the predicted category changes between adjacent words. To distinguish mild from strong shifts, the intensity difference $|y_i^{dim} - y_{i-1}^{dim}|$ is compared against a threshold $\tau = 0.3$, described as empirically chosen (Wang et al., 20 Sep 2025).

The source also reports human verification of the alignment stage. Annotators reviewed 10% of the data to confirm boundary placements, with alignment accuracy reported as greater than 95% (Wang et al., 20 Sep 2025). This review protocol is distinct from full manual relabeling: the provided description emphasizes confirmation of boundary placements rather than wholesale replacement of the automated annotation procedure.

4. Construction pipeline and quality control

The FEDD construction pipeline is explicitly specified. Audio is first preprocessed by trimming leading and trailing silences, resampling to 16 kHz, and normalizing loudness. Frame-level features $x_{1:T}$ are then extracted using emotion2vec, described in the source as a self-supervised speech-emotion encoder. MFA aligns frames to words to obtain per-word frame index ranges (Wang et al., 20 Sep 2025, Ma et al., 2023).

For each word $w_i$, the pooled feature is computed as

$$f_i = \frac{1}{|\Omega_i|} \sum_{t \in \Omega_i} x_t,$$

where $\Omega_i$ is the set of frames aligned to that word. The category and intensity labels are then computed as

$y^{dim} \in [0,1].$0

and

$y^{dim} \in [0,1].$1

Quality control is performed at several points. Words shorter than 0.05 s or with fewer than 5 frames are dropped. Intensity trajectories are median-filter smoothed with window size 3 words. Manual spot checks are carried out on 100 utterances to verify emotion boundaries and labels (Wang et al., 20 Sep 2025).

These steps define a reproducible labeling pipeline that combines forced alignment, self-supervised affective feature extraction, pooled word representations, and dual-head prediction. The source explicitly attributes the robustness and reproducibility of the resulting labels to the combination of an emotion2vec backbone, MFA alignment, and dual-head prediction (Wang et al., 20 Sep 2025).

5. Statistical structure and annotated example

The dataset summary reports an average utterance duration of 3.2 s with $y^{dim} \in [0,1].$2, and a mean word duration of 0.35 s with $y^{dim} \in [0,1].$3. Average segment duration per emotion is reported as 2.0 s for Angry, 2.1 s for Happy, 1.9 s for Sad, 1.8 s for Surprise, and 2.2 s for Neutral. Mild and strong transitions differ in mean total duration, reported as 2.8 s versus 3.4 s respectively (Wang et al., 20 Sep 2025).

An excerpted distribution table in the source lists transition counts and average durations for specific examples, including Angry $y^{dim} \in [0,1].$4 Happy with 100 utterances and average duration 3.3 s, and Sad $y^{dim} \in [0,1].$5 Surprise with 100 utterances and average duration 3.1 s (Wang et al., 20 Sep 2025). The broader balancing statement in the source indicates that this structure generalizes across all ten ordered transitions.

The paper also provides a worked annotation example for utterance #742: “I thought the project was going to fail, but at the last minute it worked!” In that example, the early words are labeled Sad with gradually increasing intensity, “fail” remains Sad at intensity 0.78, and a shift to Surprise is marked beginning at “but.” Later, “last” is labeled Happy, followed by “minute,” “it,” and “worked” with Happy intensities reaching 0.92 at the final word (Wang et al., 20 Sep 2025). This example illustrates that the annotation scheme can represent both abrupt categorical transitions and smoother intensity evolution over adjacent words.

A plausible implication is that FEDD is intended to capture not merely a single turning point but a piecewise emotional trajectory, since the example includes a Sad $y^{dim} \in [0,1].$6 Surprise shift followed by a subsequent Happy region within the same utterance. The source does not separately formalize multi-shift typologies beyond adjacent-word category change detection, but the example demonstrates that the word-level labeling scheme can express them (Wang et al., 20 Sep 2025).

6. Mathematical role in synthesis and evaluation

Within the source paper, FEDD is not only a corpus but also part of a broader formulation of dynamic emotional speech synthesis. The word-level emotion prediction loss is defined as

$y^{dim} \in [0,1].$7

This loss couples categorical prediction with scalar intensity regression, reflecting the dual-label design of the dataset (Wang et al., 20 Sep 2025).

For synthesis, the paper defines an emotion modulation mechanism termed the E-FiLM layer:

$y^{dim} \in [0,1].$8

where

$y^{dim} \in [0,1].$9

In this formulation, word-level emotion labels are embedded and transformed into affine modulation parameters that act directly on text embeddings. FEDD provides the annotation substrate for such conditioning, since it supplies the $x_{1:T}$0 pairs at the same granularity as the textual units being modulated (Wang et al., 20 Sep 2025).

For evaluation of emotional dynamics, the source uses a DTW-based metric over frame-level emotion2vec trajectories:

$x_{1:T}$1

where $x_{1:T}$2 and $x_{1:T}$3 are the target and synthesized emotion2vec trajectories. This metric links FEDD’s fine-grained annotations to trajectory-matching evaluation rather than only global emotion classification, consistent with the dataset’s emphasis on intra-utterance temporal structure (Wang et al., 20 Sep 2025).

7. Distinctiveness, scope, and interpretive cautions

The source identifies several properties as making FEDD unique. Unlike sentence-level benchmarks, it provides word-level emotion category and intensity annotations, thereby capturing the locations and strengths of emotion shifts. It balances mild in-take transitions and strong concatenated transitions across all pairwise emotion combinations. The paper further states that this combination of a self-supervised emotion2vec backbone, MFA alignment, and dual-head prediction makes FEDD the first large public resource explicitly designed to support fine-grained dynamic emotion modeling in TTS (Wang et al., 20 Sep 2025).

At the same time, several interpretive cautions follow directly from the construction protocol. The labels are produced through a prediction pipeline rather than exhaustive manual annotation, with human review applied to boundary confirmation on 10% of the data and spot checks on 100 utterances (Wang et al., 20 Sep 2025). Strong-transition examples are formed by concatenating single-emotion segments, whereas mild-transition examples are elicited in a single take. This means the corpus intentionally combines natural and engineered transition types. A plausible implication is that FEDD supports controlled comparison between different forms of emotional change, but it also means transition realism may vary by subset.

Another potential misconception is that FEDD is a generic emotion corpus. Its actual design goal is narrower and more technical: it is explicitly built for fine-grained dynamic emotion modeling in TTS, especially for architectures that require word-level conditioning and evaluation of temporal emotion trajectories (Wang et al., 20 Sep 2025). In that sense, its importance lies less in broad coverage of spontaneous affect and more in the precise operationalization of within-sentence emotional modulation.