Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech

Abstract: Integrating LLMs into text-to-speech (TTS) systems has improved speech expressiveness, yet interpretable emotional control remains challenging. Existing approaches primarily rely on external conditioning or global activation steering, offering limited insight into the internal representations underlying emotional control. In this work, we analyze emotion-related variation in the semantic hidden states of LLM-based TTS models using sparse autoencoders (SAEs) to identify sparse latent features. Our analysis shows that emotional variation is distributed across multiple sparse latent features, while intervening on a small subset enables interpretable emotion control. Building on this observation, we introduce a feature-level intervention framework for bidirectional emotion induction and suppression without modifying backbone parameters. We further show that distinct latent features are associated with specific acoustic attributes (e.g., pitch), suggesting that emotional expression arises from coordinated latent contributions rather than a single global shift. Empirically, steering these sparse latent features achieves comparable or superior emotion induction and suppression performance relative to global steering and existing TTS baselines.

• The paper introduces sparse autoencoders that extract latent emotion features from TTS semantic backbones for controlled, bidirectional emotion modulation.
• It employs a k-sparse approach with Top-k activation and auxiliary loss to enhance feature selectivity and prevent dead latent collapse.
• Quantitative and perceptual evaluations demonstrate superior emotion similarity, naturalness, and speaker consistency compared to state-of-the-art baselines.

Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech

Introduction

The paper addresses the challenge of interpretable and flexible emotional control in LLM integrated Text-to-Speech (TTS) systems. Conventional TTS emotion control methods primarily utilize label-based and reference-based conditioning, which fail to provide internal interpretability and fine-grained modularity. Prior activation-steering approaches, which modify hidden states at inference, offer training-free control but rely on dense global directions, limiting expressive interpretability. This work introduces sparse autoencoders (SAEs) applied to the semantic backbone of LLM-based TTS models, extracting sparse latent features that encode emotional variation and enabling bidirectional, interpretable emotion manipulation without modifying backbone parameters.

Sparse Autoencoder Methodology in Semantic TTS Backbones

The SAE is trained on semantic backbone activations during the autoregressive semantic-token generation phase of IndexTTS2 (a GPT-based TTS model). A $k$-sparse autoencoder, with an overcomplete latent space (4096 latent features, $k=32$ active per token), transforms dense residual stream activations into sparse latent vectors. Feature competition (Top-$k$) and auxiliary loss on inactive features mitigate dead-latent collapse, enhancing capacity utilization. Emotion-related latent features are identified via controlled activation-frequency analysis under matched text and speaker conditions, yielding robust selectivity scores for emotion–neutral pairs.

Bidirectional emotion control is formulated as steering along a set of selectivity-ranked sparse latent features. Activations of chosen features are incremented or decremented with a continuous coefficient $\alpha_e$; the SAE decoder reconstructs modified residuals, which are fed to the downstream acoustic generation modules. This mechanism enables both emotion induction and emotion suppression, supporting intensity gradation, without requiring retraining or architectural changes.

Analysis of SAE Latent Features and Emotion Selectivity

The selectivity-score distribution for emotion–neutral pairs is heavy-tailed and centered near zero, with only a sparse subset of SAE features exhibiting high selectivity for specific emotions (Figure 1).

Figure 1: Distribution of activation-frequency differences between anger and neutral conditions across all SAE latent features, demonstrating the heavy-tailed selectivity score distribution.

Quantitative steering experiments validate the interpretability of these features. Intervening on a single selectivity-ranked latent feature causes localized mid- to high-frequency amplification and increased short-time energy, while preserving overall time-frequency structure (Figure 2).

Figure 2: Acoustic effects of steering one emotion-related latent feature, producing localized spectral amplification and energy modulation.

Systematic modulation of spectral centroid correlates smoothly with steering scale; negative scales reduce centroid ("brightness") and positive scales increase it (Figure 3).

Figure 3: Mean spectral-centroid deviation relative to baseline under steering; negative scales reduce brightness, positive scales increase it, validating interpretable spectral control.

Activation density analysis shows stable Top-$k$ sparse utilization, with no dead latents (Figure 4).

Figure 4: Feature activation density distribution, confirming robust Top-$k$ sparsity and active latent utilization.

Emotion Intensity Control and Prototype Alignment

Steering scale $\alpha_e$ provides a monotonic control knob for emotion intensity, as measured by emotion-embedding cosine similarity against emotion prototypes. Target-emotion similarity increases continuously with $\alpha_e$, indicating precision control (Figure 5).

Figure 5: Single-scalar control of target-emotion intensity, with mean emotion-prototype similarity increasing smoothly with steering coefficient.

Prototype-calibrated experiments show that steering selectivity-ranked latent features shifts generated speech toward real emotion references in embedding space, not just via acoustic confounds (Figure 6).

Figure 6: Prototype-calibrated emotion-level alignment, with steering moving generation toward real happiness reference.

Across emotion categories, latent-feature control is sparse but multifactorial: happiness is concentrated in a single dominant feature, while anger/sadness require multiplexed latent-feature combinations for optimal alignment (Figure 7).

Figure 7: Scale-dependent alignment to emotion prototypes across emotion categories and latent-feature budgets; multidimensional emotion control is demonstrated.

Quantitative Acoustic Effects of SAE Steering

Latent-feature intervention yields statistically significant, interpretable acoustic changes. For matched text-speaker pairs, steering increases mean $F_0$ by $+23.11$ Hz ($k=32$0) and RMS energy ($k=32$1), with no significant change in length (Figure 8).

Figure 8: Paired F0 comparison between neutral and steered generations evidencing pitch modulation by SAE feature steering.

Bidirectional intervention (positive/negative scaling of emotion-latent features) yields stronger prosodic effects than unidirectional steering (Figure 9).

Figure 9: Distribution of mean $k=32$2 comparing positive-only and bidirectional latent steering; bidirectional controls pitch more effectively.

Empirical Evaluation and Baseline Comparison

SAE-Emotion steering achieves superior or competitive emotion-induction and suppression relative to global steering and state-of-the-art TTS baselines (VALL-E-X, Spark-TTS, EmoVoice, CosyVoice). For all target emotions, it attains highest or second-highest emotion similarity, competitive WER, and robust speaker similarity preservation.

Human evaluation corroborates improved perceptual emotion accuracy (EMOS) and naturalness (NMOS) for SAE-Emotion steering compared to dense global and random SAE interventions.

Strong numerical results:

• Highest EMOS (3.22) and NMOS (3.49) in human listening tests
• Highest or second-best emotion similarity (Emo-SIM) across anger, happiness, sadness
• SAE steering maintains low WER and strong speaker similarity, outperforming dense steering in linguistic integrity under strong interventions

Practical and Theoretical Implications

Practically, the framework provides an efficient, post-hoc, interpretable control interface for LLM-based TTS systems. It supports training-free modular control, emotion suppression, and intensity gradation while preserving backbone architectural invariance and speaker identity. Theoretically, the results evidence that emotional representation in TTS semantic backbones is sparse and multifactorial, challenge the assumption of global mean-shift emotion modeling, and validate the utility of SAE-driven feature-level manipulation for expressive speech synthesis.

This approach may encourage future research into monosemanticity in TTS, scalable latent-feature discovery, and hybrid representation-level and linguistic prompt-driven emotional generation. The findings suggest future investigation into cross-modal SAE architectures (cf. VLMs), systematic charting of the full emotional latent space, and integration with multimodal affective computing pipelines.

Conclusion

The paper demonstrates that interpretable, bidirectional emotion control in LLM-based TTS models can be achieved via sparse autoencoders operating over the semantic backbone. Emotional variation is distributed across multiple sparse latent features, and targeted, selectivity-based feature interventions modulate emotional expression with precision and expressivity, outperforming dense global steering and non-interpretable baselines. Empirical and perceptual evaluations show superior emotional fidelity, naturalness, and speaker identity preservation. The framework advances practical controllability and theoretical understanding in expressive speech synthesis, setting a research agenda for interpretable, modular emotion modeling and control in generative speech models.