HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech

Abstract: Recently, LLM-based Text-to-Speech (TTS) models have achieved remarkable naturalness. However, the standard Supervised Fine-Tuning paradigm often converges to statistically averaged prosody, limiting emotional expressiveness. While preference-driven optimization offers a promising alternative, existing approaches suffer from two structural mismatches: information conflict, where content and emotion in a shared latent space produce conflicting gradients, leading to reward hacking and semantic degradation; and scale gap, where sparse sentence-level rewards struggle to guide dense frame-level generation. To overcome these challenges, we propose HPRO, a hierarchical progressive reward optimization framework. Within HPRO, we introduce the HD-Emo codec as a novel differentiable reward model to resolve the information conflict. It extracts speech into distinct content and style preference tokens, structurally isolating emotional optimization from semantic content. Building upon this structured preference space, HPRO bridges the scale gap by progressively aligning frame-, word- and sentence-level objectives. Experiments demonstrate that HPRO significantly enhances emotional expressiveness, while effectively preserving linguistic intelligibility. The code and audio samples are publicly available at https://xxh333.github.io/hpro-demo/.

• The paper presents a hierarchical reward optimization strategy that disentangles semantic and emotional objectives using a dual quantization HD-Emo codec.
• It employs a progressive three-stage training mechanism—frame-level, word-level, and sentence-level—to enhance linguistic clarity and capture fine-grained emotional cues (WER 4.02%, wVAD-CCC 0.339).
• Ablation studies confirm that isolating content and emotion objectives prevents semantic degradation and emotional flattening, setting a new benchmark for controllable TTS.

Hierarchical Progressive Reward Optimization for Emotional Text-to-Speech

Introduction

The paper "HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech" (2606.28249) addresses core limitations in LLM-driven emotional text-to-speech (TTS) synthesis, specifically the inability of traditional supervised fine-tuning and monolithic reward paradigms to generate speech with both high semantic intelligibility and fine-grained, authentic emotional expressiveness. The authors attribute these limitations to two structural mismatches: an information conflict caused by entangled semantic and emotional objectives in a unified latent space, and a scale gap between sparse sentence-level rewards and dense frame-level token generation. HPRO (Hierarchical Progressive Reward Optimization) is proposed as a comprehensive hierarchical optimization strategy that exploits structured preference extraction to independently optimize semantics and emotion at multiple granularity levels, thus systematically resolving these foundational mismatches.

Figure 1: Illustration of the motivation for HPRO. (a) DiffRO Framework uses single-scale reward optimization over a unified speech space. (b) HPRO Framework decomposes rewards hierarchically in a structured preference space.

Methodology

HD-Emo Codec: Differentiable, Structured Reward Modeling

Central to the HPRO framework is the HD-Emo codec—a differentiable reward model that disentangles speech signals into discrete content and style preference tokens in separate subspaces. This compression is achieved via dual finite scalar quantization (FSQ) bottlenecks. Content tokens $T_c$ are directly supervised under an ASR objective built on a Whisper-initialized decoder, with a stop-gradient operation to prevent prosodic "leakage" into content, ensuring strictly linguistic supervision. In contrast, style tokens $T_s$ are supervised hierarchically: globally by soft emotion distributions from emotion2vec, and locally by word-level Valence–Arousal–Dominance (wVAD) trajectories, temporally aligned using Montreal Forced Aligner (MFA). Style modulation is enforced via a dynamic feature modulation (Emo-FiLM), which combines tokens for faithful speech reconstruction.

Figure 2: HD-Emo codec design: monotonic speech tokens are processed into content and style preference tokens to structurally isolate semantic and emotional pathways, with hierarchical supervision aligning both streams for final reconstruction.

Hierarchical Progressive Reward Optimization (HPRO)

HPRO leverages the HD-Emo codec to enable granular, structured supervision throughout LLM-based TTS training. It structures rewards over three granularities:

• Frame-level rewards: $L_1$ regression aligns generated and ground-truth discrete latent tokens for both content and style.
• Word-level rewards: CCC losses align predicted and target wVAD values, and CE losses penalize semantic errors at word boundaries.
• Sentence-level rewards: CE losses enforce global emotion class consistency.

During training, HPRO employs a three-stage progressive strategy:

1. Frame-level warm-up grounds acoustic features;
2. Word-level refinement further tunes prosody and ensures local semantic preservation;
3. Sentence-level alignment unifies the global emotional profile. Gumbel-Softmax sampling with annealed temperature controls token discreteness across stages.

   Figure 3: Overview of HPRO: The LLM outputs differentiable speech tokens via Gumbel-Softmax, post-processed by HD-Emo, with hierarchical frame-, word-, and sentence-level rewards applied progressively.

Experimental Results

Quantitative Evaluation

HPRO is benchmarked against state-of-the-art zero-shot TTS baselines (CosyVoice2/3, IndexTTS2, HD-PPT, and a DiffRO-style simulated baseline) on LibriSpeech, LSSED, and EmoVoice-DB. Performance is measured in terms of naturalness (MOS-N), emotional consistency (MOS-E), word error rate (WER), wVAD-CCC, EMO-SIM, and DNSMOS. HPRO achieved:

• Lowest WER (4.02%)—demonstrating strong preservation of linguistic intelligibility.
• Highest wVAD-CCC (0.339) and EMO-SIM (0.672)—indicating superior capture of fine-grained and global emotional intents.
• Best MOS-N (4.171) and competitive MOS-E, with the latter reflecting a stable balance between emotional salience and speech clarity.

In direct contrast, IndexTTS2, while maximally expressive (highest MOS-E), suffers from higher semantic loss, suggesting that targeted, coarse-to-fine global conditioning alone does not suffice for nuanced control.

Ablation Analysis

Ablation experiments confirm the necessity and cumulative benefit of hierarchical constraints:

• Frame-level supervision: Dramatically reduces WER and anchors acoustic foundations.
• Word-level constraints: Yield the best WER and wVAD-CCC, refining prosodic alignment.
• Sentence-level objective: Maximizes global emotional expressiveness, though with a slight precision tradeoff in local prosody—underscoring the inherent tension between local detail and global uniformity.

Isolating (removing) content or emotion objectives results in either severe semantic degradation (WER up to 13.6%) or significant emotional flattening, which confirms the risk of information conflict in untangled reward spaces.

Critically, simulating the DiffRO approach—optimizing only monolithic sentence-level reward—results in noticeably poorer trade-offs between emotion and content, reinforcing HPRO's hierarchical strategy as essential.

Implications and Future Directions

HPRO demonstrates that hierarchical and structured preference extraction, coupled with progressive training, effectively resolves the fundamental trade-offs in emotional TTS. By separating and isolating semantic and affective optimization within LLM-based TTS architectures, HPRO not only prevents reward hacking and reward domination but also enables more precise, interpretable control over the expressive dimensions of synthetic speech. Its multi-scale, modular reward design is broadly applicable, with future extensions suggested toward style control beyond emotion—potentially facilitating granular style transfer, spoken dialogue modelling, and general controllable speech generation in LLM-based pipelines.

Conclusion

HPRO introduces a principled, hierarchical optimization paradigm to emotional TTS that directly overcomes the inherent information conflict and scale gap of contemporary preference-driven approaches. By disentangling content and style through the HD-Emo codec and aligning generation across frame, word, and sentence scales, HPRO achieves superior expressive fidelity and semantic preservation. The framework sets a new standard for controllable, emotionally-aligned speech synthesis and opens further research avenues in fine-grained prosody and style-aware TTS.