A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models

Abstract: While prior work has explored emotion control in hybrid text-to-speech systems, the geometric properties of these modules, and their implications for steerability, remain poorly understood. We present the first comparative study of speech LLM (SLM) and conditional flow-matching (CFM) modules as activation steering sites for mixed emotion speech synthesis. We first characterize emotion representations using linear probing and local intrinsic dimensionality (LID), and then evaluate single-site and joint steering for mixed-emotion synthesis. Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement. Joint steering increases emotion intensity but degrades proportional control and speech quality on in-distribution data. These findings provide practical guidance for multi-site activation steering in hybrid TTS systems and highlight the importance of representation geometry in controllable speech generation.

• The paper demonstrates that SLM’s low-dimensional, speaker-invariant subspace enables robust, proportional emotion control in TTS.
• It employs linear probing and local intrinsic dimensionality analysis to compare representation quality and emotion–speaker entanglement between SLM and CFM.
• Experiments reveal that while CFM achieves higher emotion intensity, it suffers from significant cross-speaker degradation compared to SLM.

A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models

Introduction

This work provides a systematic comparative analysis of two distinct modules in hybrid text-to-speech (TTS) systems—Speech LLMs (SLM) and Conditional Flow-Matching (CFM) decoders—as sites for activation steering in composable emotional speech synthesis. Focusing on the underlying geometry of internal representations, the study investigates how these properties translate to controllability, composability, and preservation of target attributes, addressing the practical and theoretical limits of emotion steering in current hybrid TTS architectures.

Geometric Analysis of SLM and CFM Representations

A core contribution is the comprehensive representation analysis using linear probing and local intrinsic dimensionality (LID). Linear probing quantifies the linear discriminability of emotion classes at every layer, both within and across speakers. In the SLM, emotion information is concentrated in a low-dimensional, speaker-invariant subspace peaking between layers 10–17. Both within-speaker (0.80) and cross-speaker (0.71) classification accuracies converge with a minimal gap (0.08), indicating robust generalization. In contrast, CFM exhibits high within-speaker accuracy (0.89) but substantial degradation across speakers (0.62, gap of 0.32), with discriminability spread broadly, reflecting emotion–speaker entanglement.

LID trajectory analysis reveals that SLM representations, particularly in mid-to-late layers, exhibit the geometric separation requisite for composable control, with pooled LID values indicative of moderately compressed subspaces ($\sim$28). The positive $\Delta$LID (+0.84) in SLM signals that layering multiple emotion categories increases the manifold dimensionality—implying distinct composable directions. In contrast, the CFM features a more complex but less distinct topology, as evidenced by lower-dimensional pooled LID ($\sim$13) and negative $\Delta$LID (–1.48), indicating that emotions reside on an acoustically entangled manifold.

Figure 1: Per-layer emotion discriminability (linear-probe accuracy) in SLM, showing robust separation of emotional states, especially for cross-speaker (red) vs. within-speaker (blue) cases.

These geometric contrasts directly inform the ease or difficulty of extracting activation directions that can be linearly composed to realize arbitrary emotion mixtures.

Methodology for Activation Steering

Using the geometric findings, the study constructs steering vectors by mean differencing between emotional and neutral activations: for SLM, these are extracted at the last-token attention outputs; for CFM, vectors are computed from L2-normalized residual streams from the top-k emotion frames, guided by per-frame emotion predictions. For emotion mixing, steering vectors are linearly combined proportionally to the desired affective ratios.

At inference, steering consists of adding a multiple of the resultant steering vector (scaled by $\alpha$) to the target activation and renormalizing. This framework is applied either at the optimal SLM layers (14, 17), every 5th CFM layer (for each denoising step), or both, to assess single-site and joint-steering regimes.

Experimental Results

The evaluation spans large emotion-labeled multi-speaker datasets (ESD, CREMA-D, RAVDESS, IEMOCAP), with test protocols separating seen/unseen speakers and in/out-of-distribution samples. Metrics assess not only raw emotion induction (E-SIM, TEP), but proportionality of the mix ($\rho$, H-Rt) and preservation of speaker identity (S-SIM) and word accuracy (WER).

Key quantitative outcomes include:

• SLM steering achieves the best proportional emotion control ($\rho = 0.209$ in-distribution, $0.215$ out-of-distribution) with minimal speaker degradation (S-SIM 0.870), leveraging the distinct emotional subspaces revealed by geometric analysis.
• CFM steering maximizes total emotion intensity (TEP = 0.160 in-distribution, 0.272 out-of-distribution) but at the cost of proportionality and significant speaker identity loss (S-SIM drops to 0.807 in-distribution), consistent with the observed speaker–emotion entanglement.
• Joint SLM+CFM steering amplifies intensity but degrades proportionality, confirming that uncoordinated interventions do not simply add: rather, interference accumulates and speaker confusion rises.

Practical and Theoretical Implications

The findings underline critical practical guidance for hybrid TTS systems: SLM is the optimal intervention point for proportional and composable emotion control in speech, suitable for scenarios requiring nuanced affect synthesis with high speaker fidelity. CFM steering is preferable only where maximized intensity is paramount; however, it introduces speaker artifacts and less predictable mixing. Joint steering, without explicit coordination or orthogonalization, does not yield additive benefits and can compromise both proportionality and intelligibility.

Theoretically, the study clarifies the link between the geometric topology of representation spaces and downstream controllability—specifically, how positive $\Delta$LID in the SLM enables reliable mixing via composable directions, while negative $\Delta$LID in CFM signals geometric redundancy detrimental to controllability.

Looking forward, the work suggests that improved multi-site steering requires orthogonalization of emotion and speaker directions in CFM, dynamic per-site strength tuning, and possibly adaptive or frame-level steering conditioned on SLM output. Broader architectural validation and per-layer/per-step analysis could generalize these observations.

Conclusion

This analysis establishes that composable emotion steering in text-to-speech benefits fundamentally from the geometric properties of internal representations. The SLM component in hybrid architectures, with its well-separated, speaker-invariant emotion subspaces, enables robust and proportional affective control. In comparison, the CFM introduces increased intensity at the expense of speaker disentanglement and mixing fidelity. These results have significant consequences for both the design and interpretability of future controllable speech generation systems.