Emotion Similarity-Weighted Rewards

• Emotion similarity-weighted rewards provide graded reinforcement by assigning partial credit based on the affective proximity between predicted and ground-truth emotions.
• They leverage formal similarity metrics, including Plutchik-based and cosine similarity measures, integrated into RL and affective computing frameworks to overcome binary reward limitations.
• Empirical results demonstrate performance gains and enhanced generalization across tasks like speech emotion recognition, EEG analysis, and RLHF, validating their practical impact.

Emotion similarity-weighted rewards constitute a class of reinforcement signals in deep RL and related optimization settings where feedback is graded not solely on categorical accuracy, but rather by the degree of affective proximity between predicted and ground-truth emotions. This concept formalizes the intuition that the cost or reward for a prediction should scale with the psychological or semantic closeness of emotions, providing “partial credit” for near-misses, and has become central to numerous advances in affective computing, speech generation, EEG analysis, dialogue agents, and RLHF with emotion-driven supervision.

1. Formal Definitions and Core Mechanisms

At the heart of emotion similarity-weighted rewards is a well-defined function $S_{i,j}$ measuring the similarity between two emotion states $y_i$ and $y_j$. In speech emotion recognition (SER) and open-vocabulary emotion recognition, $S_{i,j}$ often leverages domain-structured priors such as Plutchik’s wheel of emotions. For $C$ discrete emotions, $S \in \mathbb{R}^{C\times C}$ encodes pairwise proximities:

• Plutchik-based similarity:

$$S_{i,j} = \dfrac{1}{2}(\cos(\mathrm{Pl}(y_i, y_j)) + 1)$$ for non-neutral emotions, where $\mathrm{Pl}(y_i, y_j) \in [0, \pi]$ is the central angle between two emotions on the wheel; “neutral” is fixed at $S_{i,j}=.5$ to all others (Li et al., 19 Sep 2025, Lian, 2 Aug 2025).

• Cosine similarity in embedding space: In speech generation, let $e_i, e_j \in \mathbb{R}^D$ be emotion embeddings from a fixed encoder. The mean-centered similarity,

$y_i$0

with $y_i$1 the dataset mean, is used to compare referent and generated samples (Tsai et al., 29 Apr 2026).

• Prototype-based similarity: For continuous signals (e.g., EEG), per-sample reward combines prototype proximity and separability:

$y_i$2

with $y_i$3 the nearest prototype, and $y_i$4 weighted by cluster variances (Zhou et al., 2024).

Reward assignment is then a function of similarity, e.g., $y_i$5 for near-misses, $y_i$6 for exact matches, and $y_i$7 for substantially dissimilar predictions, with hyperparameters (e.g., threshold $y_i$8) controlling granularity (Li et al., 19 Sep 2025).

2. Rationale and Theoretical Advantages

Emotion similarity-weighted rewards address a fundamental limitation of traditional RL objectives in affective tasks: binary reward signals ({0, 1}) create sparse gradients, penalize near-correct outputs equally with gross errors, and fail to encode psychosemantic relationships among affective states. By incorporating $y_i$9, these rewards provide:

• Dense signal propagation: “Partial credit” for predictions close in affective space smooths the reward landscape, enabling more sample-efficient and stable convergence in tasks with ambiguous or overlapping emotional boundaries (Li et al., 19 Sep 2025, Lian, 2 Aug 2025).
• Alignment with human perception: Similarity-weighted rewards operationalize psychological theories (e.g., Plutchik’s circumplex), leading to RL policies that better reflect graded human consensus and affective transitions.
• Enhanced generalization: Shaping the RL objective with emotion similarity allows models to transfer knowledge across fine-grained/distal classes, facilitating generalization in open-vocabulary settings and cross-dataset evaluation (Li et al., 19 Sep 2025, Lian, 2 Aug 2025).

3. Domain-Specific Implementations and Algorithms

Speech Emotion Recognition and Generation

• In EMO-RL, reward for a model-predicted emotion $y_j$0 relative to ground-truth $y_j$1 is $y_j$2 if identical, else $y_j$3 if $y_j$4 (thresholded at $y_j$5), else $y_j$6. This is combined additively with format compliance rewards and optimized with GRPO (Li et al., 19 Sep 2025).
• Open-vocabulary emotion recognition (AffectGPT-R1) computes reward based on Plutchik-embedded angular proximity between predicted and reference emotion labels, $y_j$7, and combines it with format adherence using $y_j$8-weighted sum, trained via GRPO (Lian, 2 Aug 2025).

Continuous Signal (EEG) Analysis

• Emotion-Agent applies prototype-based per-segment rewards, maximizing both representativeness (prototype proximity) and distinctiveness (inter-prototype separation), using PPO to select emotionally relevant EEG segments (Zhou et al., 2024).

RLHF and Preference Optimization

• ARF-RLHF interprets the output of a high-accuracy emotion/satisfaction analyzer ($y_j$9) as a scalar reward via $S_{i,j}$0 (e.g., $S_{i,j}$1), with further adaptation via experience replay and trace-biased learning to propagate fine-grained, dynamically tracked emotion supervision to the policy (Zhang, 3 Jul 2025).

4. Empirical Performance and Impact

Quantitative evidence indicates substantial performance gains from similarity-weighted rewards:

Study [arXiv id]	Task	Baseline (UA/WA/F1)	+ Similarity-Weighted Reward	Absolute Improvement
EMO-RL (Li et al., 19 Sep 2025)	MELD (SER, UA/WA/F1)	31.60 / 55.41 / 33.22	36.23 / 63.85 / 38.57	+4.6 / +8.4 / +5.3
	IEMOCAP (UA/WA)	81.74 / —	87.42 / 87.28	+5.68 / —
AffectGPT-R1 (Lian, 2 Aug 2025)	OV-MERD+ (EW)	62.52 %	66.35 %	+3.8 pts
Emotion-Agent (Zhou et al., 2024)	SEED (accuracy, KNN)	42.78 %	62.31 %	+19.53 pts
ARF-RLHF (Zhang, 3 Jul 2025)	RLHF (various tasks)	PPO: baseline	PPO + ARF	+3.3 %

These results robustly demonstrate superiority over token-level, binary, or purely categorical objectives in affective domains. In cross-dataset OV-MER, similarity-weighted RL improved generalization by up to +23 points in WA (Li et al., 19 Sep 2025).

5. Architectural and Methodological Variants

Key approaches differ in how similarity is estimated and integrated:

• Semantic/psychological similarity: Plutchik’s wheel-derived proximity is prevalent when working with labeled, discrete emotions (Li et al., 19 Sep 2025, Lian, 2 Aug 2025).
• Latent embedding similarity: Raw or mean-centered cosine similarity in encoder latent spaces is common in zero-shot or expressive speech generation, though this is controversial (see §6) (Tsai et al., 29 Apr 2026).
• Prototype-based: Clustering and reward assignment in continuous signals leverage K-Means/GMM prototypes to define distributional centers and reward discriminativity (Zhou et al., 2024).
• Human-in-the-loop continuous scoring: RLHF systems compute reward as a continuous function of satisfaction or affect intensity derived from user feedback through an emotion analyzer and preference adaptation mechanism (Zhang, 3 Jul 2025).

Optimization algorithms include GRPO, PPO, and trace-biased variants, with group-relative normalization and exploration to stabilize training in the presence of ambiguous or sparse affective feedback.

6. Limitations, Failure Modes, and Open Controversies

Despite clear successes, emotion similarity-weighted rewards are subject to notable caveats:

• Representation collapse and false resonance: In “The False Resonance” (Tsai et al., 29 Apr 2026), off-the-shelf emotion embedding similarity (EMO-SIM) rewards fail to align with human affective perception, especially under speaker or linguistic distractors—leading to reward signals that enforce acoustic mimicry (voice, prosody, text) rather than genuine emotion transfer. This raises the risk of RL agents optimizing for non-affective features.
• Insufficient disentanglement: The geometric structure of current SER encoders is often dominated by speaker or linguistic cues, rendering cosine similarity unreliable as a zero-shot reward metric unless embeddings are further disentangled or adversarially regularized.
• Validation requirements: Practitioners are strongly counseled to validate any similarity-based reward under adversarial sampling (distractor controls) and to supplement with auxiliary metrics or human validation (Tsai et al., 29 Apr 2026).
• Supervision source reliability: RLHF scenarios depend on the accuracy and adaptivity of the emotion/satisfaction analyzer, which, despite ~70–76% accuracy, may propagate annotator bias or misalignment if not dynamically recalibrated (Zhang, 3 Jul 2025).

A plausible implication is that naïve application of similarity-weighted rewards in zero-shot generation or unsupervised RL can entrench unwanted confounds unless explicit steps—representation calibration, metric benchmarking, and composite reward design—are taken.

7. Recommended Practices and Future Directions

Best practices to realize the benefit of emotion similarity-weighted rewards while mitigating pitfalls include:

• Embedding calibrations such as mean-centering and, where possible, orthogonalization of speaker/text factors (Tsai et al., 29 Apr 2026).
• Disentangled or adversarially trained embedding spaces for improved affective discriminability.
• Composite reward engineering: combining similarity-based metrics with signal-level prosodic, distributional, or shallow classifier-based criteria (Tsai et al., 29 Apr 2026).
• Periodic human-in-the-loop validation (MOS/preference) to verify alignment between optimization trajectory and genuine emotional expressiveness.
• Careful empirical evaluation under adversarially sampled distractors to reveal failure modes before deployment as reward signals.
• In RLHF and preference optimization, continual adaptation and trace-level preference tracking to ensure dynamic response to user tastes, as realized in ARF-RLHF (Zhang, 3 Jul 2025).

Ongoing research seeks to resolve remaining issues around robust, disentangled emotion representation and to extend these techniques to new domains including open-vocabulary, multimodal, and unsupervised affective computing.