Emotion-Aware Stepwise Preference Optimization

• EASPO is a framework that adapts diffusion-based TTS models using stepwise reinforcement learning to provide fine-grained, temporal emotional control.
• It employs the EASPM module to compute dense, timestep-conditioned emotional scores through a modified CLEP model for dynamic candidate evaluation.
• Empirical results show improved expressiveness, naturalness, and prosody alignment over baselines, making it valuable for conversational AI and assistive tools.

Emotion-Aware Stepwise Preference Optimization (EASPO) is a framework for aligning machine-generated content—most notably in diffusion-based text-to-speech (TTS) models—with temporally fine-grained emotional preferences through reinforcement learning over intermediate generative steps (Shi et al., 29 Sep 2025). The approach addresses the limitations of conventional preference optimization methods that rely on coarse, utterance-level labels, enabling dynamic emotional shaping and more expressive, natural synthetic outputs. EASPO has implications in conversational AI, accessibility, and multimodal content generation, as well as offering a basis for broader adaptation in sequential generative modeling.

1. Foundational Concepts and Motivation

EASPO is motivated by the observation that emotional feedback in generative systems is most commonly applied either as endpoint utterance-level signals or through discrete labels provided by proxy classifiers (Shi et al., 29 Sep 2025). These approaches fail to capture the evolving emotional trajectory found in natural speech and human interaction, particularly in tasks such as text-to-speech synthesis, empathetic dialogue modeling, multimodal captioning, or music recommendation.

By introducing stepwise preference supervision at each generative timestep—rather than only at completion—EASPO enables models to adaptively control affect, prosody, and expressively nuanced output throughout generation. This approach is grounded in reinforcement learning formulations, treating each generative step as a Markov Decision Process (MDP) state-action pair, with emotion-aware rewards provided by an auxiliary scoring model.

2. EASPM: Emotion-Aware Stepwise Preference Model

At the core of EASPO is the Emotion-Aware Stepwise Preference Model (EASPM), which provides dense, timestep-conditioned emotional scoring for intermediate candidate outputs (Shi et al., 29 Sep 2025). EASPM is based on a modified CLEP (CLAP-based contrastive language–audio encoder), with separate text and audio branches producing L₂-normalized embeddings.

For each candidate mel-spectrogram $x_{t-1}^i$ at denoising step $t$, EASPM computes:

$$s_i = \langle \frac{f_\mathrm{CLEP-A}(x_{t-1}^i, t)}{\|f_\mathrm{CLEP-A}(x_{t-1}^i, t)\|_2},\ \frac{f_\mathrm{CLEP-T}(c)}{\|f_\mathrm{CLEP-T}(c)\|_2} \rangle$$

where $c$ is the conditioning text. The highest and lowest scoring candidates constitute preference pairs (win/lose), with the pairwise probability modeled via a logistic function, $\hat{p}_w = \sigma[\tau(s_w - s_l)]$.

Time-aware normalization in EASPM ensures sensitivity to denoising step context, mitigating temporal mismatch between noisy and clean states. This structure allows the model to reinforce local emotional adjustments while preserving instability in early, noise-dominated steps.

3. Stepwise Optimization via Reinforcement Learning

EASPO’s optimization process is formalized as RL over generative trajectories in the diffusion TTS model (Shi et al., 29 Sep 2025):

• State: At timestep $t$, the state $s_t$ includes the latent $x_t$ and text $c$.
• Action: $a_t$ selects a candidate $x_{t-1}$ for the next step via policy $\pi_t = p_\theta(x_{t-1}|x_t, c)$.
• Reward: Dense stepwise rewards $\tilde{R}_t^j = s(x_{t-1}^j, c, t)$ are provided for candidates, forming a difference $\Delta \tilde{R}_t$ between win/lose.
• Alignment Objective: The log-likelihood ratio between the model’s policy and a frozen reference policy is computed:

$$\rho_t^j(\theta) = \log \pi_\theta(x_{t-1}^j | s_t) - \log \pi_\mathrm{ref}(x_{t-1}^j | s_t)$$

$$\Delta \rho_t = \rho_t^w(\theta) - \rho_t^l(\theta)$$

The alignment loss is then:

$$\mathcal{L}_t(\theta) = [\beta_t \Delta \rho_t - \Delta \tilde{R}_t]^2$$

$\beta_t = (\lambda^{T-t-1})/\eta$ weights time steps, with $\lambda \in (0,1], \eta > 0$.

Random timestep sampling (excluding initial noisy steps) and averaging over batch samples ensures robust stepwise alignment across a wide range of emotional targets.

4. Empirical Validation and Comparative Performance

EASPO has been empirically evaluated on large-scale emotional speech datasets. EASPM was fine-tuned on the MSP-Podcast corpus (55k utterances, 1,200 speakers, English), while the RL optimization utilized ESD (multiple emotions, multiple speakers).

Metrics include:

• Emo_SIM: Cosine similarity of emotional embeddings.
• Prosody_SIM: Alignment of prosodic features.
• WER: Word error rate for intelligibility.
• UTMOS: Subjective mean opinion score for naturalness and emotional consistency.

Experimental findings report EASPO outperforming seven baseline methods (PromptTTS, EmoDiff, CosyVoice2, etc.) with stronger metrics on both expressiveness and naturalness (Shi et al., 29 Sep 2025). Subjective and objective evaluations indicate robust gains in emotion and prosody alignment.

5. Relation to Broader Preference Optimization Frameworks

EASPO generalizes principles established in prior preference optimization literature. Direct Preference Optimization (DPO) has previously targeted pairwise comparisons of endpoint outputs, as in empathetic response modeling and emotional TTS (Gao et al., 16 Sep 2024, Sotolar et al., 27 Jun 2024, Sotolar et al., 27 Jun 2024). Recent advancements have explored fine-grained optimization using both positive (“preferred”) and negative (“dis-preferred”) feedback, with extensions into unpaired and continuous feedback via expectation–maximization (EM) mechanisms (Abdolmaleki et al., 5 Oct 2024, Zhang, 3 Jul 2025).

The stepwise methodology in EASPO extends these ideas by applying RL-based alignment over intermediate states, enabling temporal control and adaptation to evolving emotional signals. Time-conditioned scoring and dense feedback are central to enabling this level of granularity.

6. Applications and Implementation Implications

EASPO is applicable to domains where fine-grained emotional control is required:

• Conversational Agents: Enhanced expressivity, more engaging dialogue (Sotolar et al., 27 Jun 2024, Kim et al., 23 Dec 2024).
• Accessibility Tools: Emotionally nuanced synthetic speech for assistive reading.
• Content Creation: Tailored emotional audio for storytelling, media, and entertainment.

Implementation challenges include balancing candidate pool size against fidelity, preventing artifacts or weak supervision, and refining candidate selection for local emotional contrast. Another open issue is the domain transfer of temporal preference supervision to modalities beyond speech (video, music).

7. Challenges, Limitations, and Prospective Directions

Dense supervision at intermediate steps increases computational complexity. The efficacy of candidate sampling and time-aware normalization require further empirical investigation, particularly in contexts where localized emotional variations are nuanced or subtle. Ensuring that strong emotional contrast does not compromise linguistic or semantic consistency remains a topic for further research.

Future work may involve:

• Enhanced candidate sampling or weighting strategies.
• Reduced domain mismatches between noisy and clean representations.
• Extending the paradigm to non-speech generative models with temporally evolving emotional control.
• Advanced time conditioning architectures for modeling emotion/prosody evolution over extended generative sequences.

Summary Table: Core Components of EASPO

Component	Function	Reference
EASPM	Timestep-aware emotion scoring for intermediate candidates	(Shi et al., 29 Sep 2025)
Stepwise RL	Alignment of policy with dense emotional rewards at each step	(Shi et al., 29 Sep 2025)
Candidate Pairing	Automatic win/lose construction via EASPM scores	(Shi et al., 29 Sep 2025)
Temporal Weighting	Loss weighting by denoising step for reward alignment	(Shi et al., 29 Sep 2025)

EASPO thus establishes a rigorous, fine-grained framework for emotional shaping in diffusion-based TTS and related sequential generative models, leveraging temporally conditioned preference optimization for advanced multimodal applications.