Hierarchical Emotion Distribution in TTS
- Hierarchical ED is a multi-level representation that defines continuous emotion intensities at the utterance, word, and phoneme scales for precise TTS control.
- The methodology uses explicit segmentation with ranking-based and learned intensity extractors to derive soft, interpretable emotion labels from aligned speech segments.
- Empirical findings demonstrate that integrating Hierarchical ED enhances prosodic control, naturalness, and expressive variability compared to global emotion embeddings.
Searching arXiv for the cited and related papers to ground the article in current literature. arXiv search query: "Hierarchical Emotion Distribution speech synthesis emotional TTS" Hierarchical Emotion Distribution (ED) is a multi-level emotion representation developed chiefly for controllable emotional text-to-speech (TTS), in which emotion is modeled not as a single utterance-level label or a single global style vector, but as a set of continuous, segment-linked emotion distributions or intensity vectors defined over phonemes, words, and utterances. In the TTS literature, ED functions simultaneously as a training target, a conditioning signal, and an inference-time control interface: during training it is extracted from aligned reference speech as a set of “soft labels,” and during inference it can be predicted from text or manually edited to alter emotional rendering at multiple temporal scales (Inoue et al., 2024, Inoue et al., 2024, Inoue et al., 2024, Inoue et al., 7 Jul 2025).
1. Conceptual foundations
The central motivation for Hierarchical ED is the mismatch between the multi-scale structure of speech emotion and the single-scale control variables used in many emotional TTS systems. The underlying claim across the main ED papers is that speech emotion is intrinsically hierarchical: utterance-level emotion corresponds to global prosody such as pitch range, tempo, and rhythm; word-level emotion captures lexical emphasis and emotional salience; phoneme-level emotion reflects local prosodic realization through pitch, energy, and duration. A single global embedding or a hard utterance-level emotion label cannot represent this within-utterance variation, and therefore cannot support localized editing (Inoue et al., 2024, Inoue et al., 2024, Inoue et al., 2024).
Within this research line, the word “distribution” is used in a technically specific but non-uniform sense. In the flow-matching formulation, the classifier outputs are treated as segment-wise emotion distributions, and the authors use “emotion distribution” and “emotion intensity” almost interchangeably; for each segment, the model outputs a probability distribution over emotions or emotion-presence scores (Inoue et al., 2024). In the earlier FastSpeech2-based formulations and their multi-step successor, ED is implemented as a vector of normalized per-emotion intensity values in , derived from ranking functions rather than explicitly constrained to sum to $1$ (Inoue et al., 2024, Inoue et al., 2024, Inoue et al., 7 Jul 2025). Accordingly, Hierarchical ED is best understood as a continuous multi-emotion intensity representation organized by linguistic granularity rather than as a simplex-valued posterior in the strict probabilistic sense.
A second foundational distinction is architectural. ED is not introduced in these papers as a latent variable learned end-to-end by a VAE or diffusion prior. Instead, it is constructed from aligned speech segments by explicit extractors and then injected into TTS as an external or semi-external supervisory/control signal. This procedural design is essential to the interpretability claims of the framework: users manipulate explicit per-segment emotion values rather than opaque latent style codes (Inoue et al., 2024, Inoue et al., 2024).
2. Representation structure and extraction pipelines
The canonical hierarchy comprises three levels: one ED vector for the whole utterance, one ED vector per word, and one ED vector per phoneme. For TTS conditioning, higher-level vectors are aligned to the phoneme sequence by replication: the utterance-level vector is copied across all phoneme positions, and each word-level vector is expanded over the phonemes belonging to that word. The resulting representation is therefore phoneme-synchronous while still preserving global and intermediate emotional context (Inoue et al., 2024, Inoue et al., 2024, Inoue et al., 7 Jul 2025).
In the initial FastSpeech2-based ED systems, extraction follows a ranking-based pipeline. Speech is segmented with Montreal Forced Aligner (MFA) into phoneme-, word-, and utterance-level units; an 88-dimensional acoustic feature set is extracted from each segment using OpenSMILE; and a pre-trained ranking function is applied at each segmental level. Support vector machines (SVMs) are used to rank emotion intensity, and the resulting scores are normalized to , where larger values represent stronger emotion intensity. The output for a segment is an ED vector in which each value represents the intensity of a specific emotion encoded in that segment (Inoue et al., 2024, Inoue et al., 2024, Inoue et al., 7 Jul 2025).
The later flow-matching formulation replaces ranking-based extractors with learned emotion intensity extractors. Its hierarchical ED extractor comprises speech segmentation and three identical emotion intensity extractors, one for each level. Each extractor contains an acoustic feature extraction module, a normalization layer, a trained feature extractor, and a classifier module. The acoustic feature extractor is frozen; the paper explores OpenSMILE, WavLM, and HuBERT, and the best-performing setup uses WavLM for utterance-level ED and OpenSMILE for word- and phoneme-level ED. The learned feature extractor consists of two fully connected layers separated by a ReLU activation layer, and the classifier is either a multiclass Speech Emotion Recognizer (SER) or an Emotion Presence Recognizer (EPR) built from one binary classifier per emotion (Inoue et al., 2024).
That flow-matching paper also introduces explicit smoothing of classifier outputs to avoid overconfident near-binary probabilities when those outputs are intended to represent graded emotion intensity. Standard in softmax is replaced by , with swept from $1.1$ to $3.0$ in $0.1$ increments and selected by minimizing the KL divergence between the empirical emotion-intensity distribution on training samples and the uniform distribution. The stated goal is to smooth the distribution of emotion intensities (Inoue et al., 2024).
Training supervision in the learned extractor remains weakly localized. The extractor is trained before TTS; the acoustic feature extractor is frozen; cross-entropy is used with loss weights adjusted based on the segment count; and utterance emotion labels are propagated to all segments in that utterance. To reduce contamination by speaker or gender information, the extractor includes an adversarial classifier with a Gradient Reversal Layer (GRL), scaled by $0.5$, and the selected extractor maximizes validation emotion accuracy while pushing speaker or gender prediction toward random chance (Inoue et al., 2024). This design makes explicit that hierarchical ED is induced from segmentation and shared extractor structure rather than from direct fine-grained human annotation.
3. Integration into speech synthesis and control mechanisms
In the FastSpeech2 line of work, ED is integrated through the variance adaptor. The text encoder produces linguistic embeddings, ED vectors are transformed into ED embeddings by fully connected layers with Tanh, and the variance adaptor, using both text and ED embeddings, predicts pitch, duration, and energy. The utterance-level distribution is duplicated across all phonemes and the word-level distribution is replicated across the corresponding phonemes, so the variance adaptor learns a link between linguistic structure and hierarchical emotional intensity (Inoue et al., 2024, Inoue et al., 2024).
The 2024 “prediction and control” variant adds a text-side ED predictor. A BERT-based linguistic encoder is pretrained on BookCorpus and then integrated into the TTS framework; during training, hierarchical ED is extracted from ground-truth audio and guides the predictor to establish a connection between emotional and linguistic prosody. At run time, no reference audio is required: the model predicts phoneme-, word-, and utterance-level ED from text and permits manual modification of those values before synthesis (Inoue et al., 2024).
The flow-matching system relocates ED into a different synthesis backbone. It uses MatchaTTS with optimal-transport conditional flow matching (OT-CFM). The text encoder produces phoneme-level linguistic embeddings; Resemblyzer provides a speaker embedding; and the hierarchical ED extractor provides aligned per-phoneme emotional conditioning. A fully connected layer projects the concatenated speaker-plus-ED conditioning to match the text encoder output dimension, and this conditioning is expanded by phoneme durations to compute the predicted average mel-spectrogram $1$0, which then conditions the OT-CFM decoder. During inference, ED can be extracted from a reference utterance for emotion transfer or edited manually before synthesis (Inoue et al., 2024).
The 2025 multi-step extension makes the dependency structure explicit. Instead of predicting all levels independently, it predicts utterance-level ED first, then word-level ED conditioned on the utterance level, then phoneme-level ED conditioned on both higher levels. Two integration modes are proposed: an external ED prediction module, trainable with the text encoder frozen and attachable to multiple TTS systems, and an internal variance-adaptor integration in which the encoder is jointly trained with an ED-related MSE loss. The explicit prediction order is utterance $1$1 word $1$2 phoneme, and the stated purpose is to let global emotional context refine local emotional variation (Inoue et al., 7 Jul 2025).
Across the literature, inference-time control is quantitative and direct. Users edit selected ED entries numerically, typically within $1$3. Reported evaluation protocols vary emotion intensity from $1$4 to $1$5 in increments of $1$6, or compare $1$7, $1$8, and $1$9 conditions in perceptual studies. Control can be applied globally at the utterance level, locally at the word level, or very locally at the phoneme level; some experiments also combine word- and phoneme-level edits (Inoue et al., 2024, Inoue et al., 2024, Inoue et al., 7 Jul 2025).
4. Empirical performance and controllability
The earliest ED study, which introduced the hierarchical ED extractor into FastSpeech2, reported modest but consistent improvements over MsEmoTTS in quality and expressiveness. On MOS, Hierarchical ED reached 0 versus 1 for MsEmoTTS; on the A/B preference test for emotional expressiveness, Hierarchical ED obtained 2 versus 3 for MsEmoTTS and 4 for Neutral. Objective metrics also improved: MCD 5 versus 6, pitch distortion 7 versus 8, energy distortion 9 versus 0, and frame disturbance 1 versus 2 (Inoue et al., 2024).
The subsequent text-predicted ED framework showed that hierarchical ED could be learned from text while preserving synthesis quality and improving controllability. Relative to FastSpeech2, the proposed system reported MOS 3 versus 4 and MCD 5 versus 6, while the ablation without the ED predictor degraded to MOS 7 and MCD 8. In best-worst scaling for expressiveness similarity to the reference, the proposed method was chosen as best in 9 of cases and worst in 0. Its controllability analysis used the prosodic change ratio 1 and found the strongest effects at word and phoneme level, with weaker utterance-level control (Inoue et al., 2024).
The flow-matching formulation provides the most detailed quantitative evidence for learned ED extraction. The proposed system with EPR achieved Positive 2, Negative 3, and Score 4 on the SER-based controllability metric, clearly exceeding the prior SVM-based hierarchical baseline at Positive 5, Negative 6, Score 7. In MUSHRA, Proposed w/ EPR obtained emotional similarity 8 and naturalness 9, slightly above the SVM-HED baseline at 0 and 1. The acoustic-feature ablation further showed that the combined setup—WavLM for utterance-level ED and OpenSMILE for word and phoneme levels—was best on WER 2, MCD 3, pitch distortion 4, energy distortion 5, WeSpeaker SECS 6, and SER controllability score 7 (Inoue et al., 2024).
Perceptual controllability results in that same study were strongly monotonic. In the utterance-level BWS test, the 8-intensity sample of the proposed EPR model was chosen as most expressive at rates including 9 for Angry, $1.1$0 for Happy, $1.1$1 for Sad, and $1.1$2 for Surprise, averaging $1.1$3. At the word level, the average “most expressive” preference for the $1.1$4-intensity condition reached $1.1$5. Acoustic analyses showed expected prosodic trends: happiness increasing mean pitch, anger increasing energy mean and standard deviation, sadness lengthening duration and lowering pitch, and surprise increasing pitch with slight energy increases (Inoue et al., 2024).
The multi-step extension argues that hierarchical dependency modeling improves synthesis even when raw ED prediction error changes little. With ground-truth ED, GT + VA(Multi-Step) improved MUSHRA from $1.1$6 to $1.1$7 and WER from $1.1$8 to $1.1$9 over GT + VA(single-step). With predicted ED, External multi-step improved over External single-step from MUSHRA $3.0$0 to $3.0$1 and WER $3.0$2 to $3.0$3, while emotional similarity MUSHRA rose from $3.0$4 to $3.0$5. In word-level controllability against MsEmoTTS, the high-intensity $3.0$6 condition was chosen as “Most expressive” at $3.0$7 for Angry, $3.0$8 for Happy, $3.0$9 for Sad, and $0.1$0 for Surprise (Inoue et al., 7 Jul 2025).
5. Extensions and adjacent formulations beyond TTS
Although Hierarchical ED is most concretely developed in emotional TTS, several adjacent lines of work broaden the conceptual landscape. One neighboring strand derives hierarchical structure from learned emotion representations rather than from segmental prosody. “Enhancing Cognitive Models of Emotions with Representation Learning” learns 32-way text emotion representations, then induces an emotion graph from layer-wise confusion dynamics and an emotion wheel from averaged emotion embeddings, showing that hierarchy can emerge from flat supervision rather than being predefined (Guo et al., 2021). “Emergence of Hierarchical Emotion Organization in LLMs” similarly treats next-token probabilities over 135 emotion words as per-instance emotion distributions, aggregates them into a matching matrix $0.1$1, and induces directed parent-child relations from asymmetric conditional dependence; the resulting hierarchies grow deeper with model size and align significantly with human emotion-wheel structure (Zhao et al., 12 Jul 2025).
A second neighboring strand provides symbolic rather than probabilistic hierarchy. “TONE: A 3-Tiered ONtology for Emotion analysis” builds an OWL ontology from Parrott’s hierarchy, with 6 primary emotions, secondary and tertiary descendants, curated vocabularies, and typed links such as isComposedOf, isOppositeOf, and plus-LeadsTo. This is not an ED model in the TTS sense, but it supplies a hierarchical label space and explicit child-to-parent aggregation structure that can support hierarchical scoring or roll-up (Gupta et al., 2024).
A third strand combines hierarchy with distributional regularization in recognition tasks without implementing TTS-style ED. HADUA uses hierarchical attention for intra-modal and inter-modal fusion of EEG and eye-movement signals, while its distributional component regularizes soft pseudo-labels and batch-level class frequencies during domain adaptation; the hierarchy is in multimodal representation learning, not in the emotion label space itself (Tang et al., 29 Jan 2026). “Temporal Label Hierachical Network for Compound Emotion Recognition” organizes compound emotions through a temporal pyramid and a coarse-to-fine valence-arousal-informed label structure, but trains with cross-entropy on hard categories rather than with explicit hierarchical distributions (Li et al., 2024). “StyleEDL” predicts image emotion distributions with hierarchical stylistic representations across CNN layers and graph-based emotion dependency modeling, yet its hierarchy is in visual style representation rather than in a coarse-to-fine emotion ontology (Jing et al., 2023).
Taken together, these works show that “hierarchical emotion” can denote at least three distinct objects: hierarchical segmental control in speech synthesis, hierarchical latent or induced structure among emotion labels, and hierarchical feature fusion or weakly structured label organization in emotion recognition. Hierarchical ED, in the narrow and now standard TTS sense, is the first of these.
6. Limitations, ambiguities, and open directions
The principal limitation of current Hierarchical ED systems is that their hierarchy is representational and operational rather than formally probabilistic. The main TTS papers do not define a recursive graphical model over utterance-, word-, and phoneme-level emotion variables, and they do not introduce explicit cross-level consistency losses enforcing agreement between levels. In the flow-matching paper, the hierarchy is induced by segmentation and aligned concatenation rather than by a mathematical coupling between segment scales (Inoue et al., 2024). This makes the framework highly practical but theoretically loose.
A second limitation is supervision quality. In the ranking-based systems, fine-grained ED labels are derived from handcrafted acoustic features and level-specific ranking functions rather than from direct human annotation at word or phoneme level (Inoue et al., 2024, Inoue et al., 2024). In the learned-extractor system, local segments inherit utterance emotion labels during extractor training, so phoneme- and word-level EDs remain weakly supervised even when the extractor architecture is more sophisticated (Inoue et al., 2024). A plausible implication is that some local ED patterns encode extractor heuristics rather than directly observed localized emotion.
A third limitation is infrastructural dependence. Existing ED pipelines rely heavily on forced alignment, segment extraction, OpenSMILE or SSL feature frontends, and separately trained ranking or recognition modules. This enables interpretability, but it also makes the pipeline less end-to-end and sensitive to segmentation quality. The same concern appears in the multi-step model, which additionally reports error accumulation at lower levels when higher-level predicted EDs are used as conditioning, especially at the phoneme level (Inoue et al., 7 Jul 2025).
The term “distribution” itself remains somewhat ambiguous across the literature. In some formulations ED is a smoothed classifier probability vector; in others it is a normalized vector of relative-attribute scores; and across several papers it is explicitly described as continuous emotion intensity labels rather than as a strict simplex. This terminological looseness has not prevented practical gains, but it complicates comparison with label-distribution learning, calibration, or probabilistic affect modeling (Inoue et al., 2024, Inoue et al., 2024, Inoue et al., 7 Jul 2025).
Current evaluation is also uneven across levels. Utterance- and word-level control are well supported by both objective and subjective experiments, whereas phoneme-level control is often illustrated qualitatively or evaluated less extensively. The papers themselves note broader data limitations: reliance on acted emotional corpora such as ESD, limited language coverage, and the need for more naturalistic emotional speech, more advanced emotion diarization or SER, more varied voice qualities, and stronger cross-lingual robustness (Inoue et al., 2024, Inoue et al., 7 Jul 2025).
In its present form, Hierarchical ED is therefore best characterized as a practical, interpretable, and increasingly effective framework for multi-level emotion control in speech synthesis: a phoneme-aligned collection of continuous emotion descriptors defined at utterance, word, and phoneme scales, extracted or predicted with explicit segmental structure, and used to make emotional rendering quantitatively editable. Its most mature instantiations have already shown consistent gains in emotional similarity, controllability, and localized prosodic manipulation, while its outstanding theoretical challenge is to turn this operational hierarchy into a formally coupled hierarchical emotion model (Inoue et al., 2024, Inoue et al., 2024, Inoue et al., 7 Jul 2025).
