Emotion Similarity-Weighted Reward (ESWR)

• ESWR is a reinforcement learning reward shaping method that assigns partial credit to predictions based on emotional similarity using Plutchik’s wheel.
• It replaces traditional binary rewards with a thresholded, similarity-weighted mechanism that improves signal density and stability in emotion recognition tasks.
• Empirical evaluations on datasets like MELD and IEMOCAP demonstrate that integrating ESWR enhances policy performance and cross-domain generalization.

Searching arXiv for the cited papers to ground the article in current preprints. Emotion Similarity-Weighted Reward (ESWR) is a reinforcement-learning reward-shaping mechanism introduced for generalized speech emotion recognition in the EMO-RL framework. Its central purpose is to replace brittle binary correctness signals with a psychologically structured reward that assigns partial credit to emotionally similar predictions, using Plutchik’s wheel of emotions as the similarity geometry. In the literature considered here, ESWR is explicit in EMO-RL, while adjacent mechanisms appear in open-vocabulary multimodal emotion recognition, empathetic dialogue, and cross-domain retrieval as related but non-identical forms of emotion-aware optimization (Li et al., 19 Sep 2025, Lian, 2 Aug 2025, Wang et al., 3 Jul 2025, Doh et al., 2023).

1. Emergence and problem setting

ESWR was proposed in response to a specific failure mode of applying generic RL objectives to Speech Emotion Recognition (SER): ordinary classification-style rewards are sparse and overly punitive when emotion categories have ambiguous boundaries. In the EMO-RL formulation, the conventional binary classification reward gives reward $1$ only for an exact emotion match and $0$ otherwise. The paper argues that this is poorly matched to affective structure, because some prediction errors are psychologically closer to the target than others. Its motivating example is that predicting “sad” instead of “neutral” is less severe than predicting “happy” instead of “angry” (Li et al., 19 Sep 2025).

The paper further states that direct GRPO with binary rewards is too sparse for SER. On MELD, about $60\%$ of accuracy rewards are $0$, which makes policy updates unstable. This sparsity is especially problematic under ambiguous emotional boundaries and noisy annotations. ESWR is designed to densify the reward while preserving discriminative pressure, thereby turning reinforcement learning from a strict right-or-wrong supervision signal into an affect-aware signal grounded in inter-emotion similarity (Li et al., 19 Sep 2025).

Within EMO-RL, ESWR is one of two central design components, the other being Explicit Structured Reasoning (ESR). The broader framework targets Large Audio-LLMs (LALMs), where the paper identifies two obstacles to direct RL application in SER: convergence instability caused by ambiguous emotional boundaries, and limited reasoning ability when using relatively small models such as $7$B-parameter architectures (Li et al., 19 Sep 2025).

2. Formal definition

The mathematical core of ESWR is an emotion-state-transition matrix $S \in \mathbb{R}^{C \times C}$, where $C$ is the number of emotion classes. The matrix encodes pairwise similarity between emotions using Plutchik’s wheel. If either emotion is “neutral,” similarity is fixed to $\frac{1}{2}$. Otherwise, similarity is computed from the angular distance on the wheel and normalized into $[0,1]$ through a cosine transform (Li et al., 19 Sep 2025).

A concise rendering of the similarity definition is:

$$S_{i,j} = \begin{cases} \frac{1}{2}, & \text{if either emotion is `neutral'} \ \frac{1}{2}\left(\cos\left(\mathrm{Pl}(y_i,y_j)\right)+1\right), & \text{otherwise} \end{cases}$$

where $0$0 and $0$1 are emotion types and $0$2 denotes their angular distance on Plutchik’s wheel. The cosine term maps angular distance into a similarity score in $0$3, and the special handling of “neutral” treats it as moderately similar or ambiguous rather than strongly aligned or opposed (Li et al., 19 Sep 2025).

ESWR then converts similarity into a piecewise reward:

$0$4

where $0$5 is the predicted emotion, $0$6 is the ground-truth emotion, $0$7 is the partial-matching coefficient, and $0$8 is the contradiction threshold. The paper sets $0$9, and $60\%$0 is dynamically adjusted from $60\%$1 to $60\%$2 during training (Li et al., 19 Sep 2025).

This definition establishes three reward regimes. Exact matches receive reward $60\%$3. Predictions that are sufficiently similar but not identical receive fractional reward $60\%$4. Predictions that are emotionally contradictory or sufficiently dissimilar receive $60\%$5. ESWR is therefore not a globally continuous similarity reward; it is a thresholded partial-credit mechanism that preserves a hard exclusion region for low-similarity predictions (Li et al., 19 Sep 2025).

3. Integration into GRPO-based optimization

EMO-RL is built on Group Relative Policy Optimization (GRPO) with emotion constraints. For each input state $60\%$6, the policy samples a group of actions,

$60\%$7

where each $60\%$8 is a full generated response containing reasoning and a final answer. Each sampled response receives a total reward defined as the sum of a format reward and an accuracy reward:

$60\%$9

Under ESWR-based training, the accuracy term becomes the emotion similarity-weighted reward, so $0$0 (Li et al., 19 Sep 2025).

The group reward is normalized into relative advantages:

$0$1

and these normalized advantages drive the GRPO update. The paper also states that the policy is constrained by KL divergence to a reference model $0$2, providing regularization against unstable drift (Li et al., 19 Sep 2025).

In the full EMO-RL formulation, ESWR is coupled with a format reward that enforces valid reasoning/answer structure, and in the ESR setting the model must both follow the structured reasoning template and receive emotion-aware reward for the predicted label. This pairing is significant because it places ESWR within a composite reward design rather than treating emotion similarity as the sole optimization target. A common misconception is therefore to read ESWR as a standalone reward; in the implemented RL system it operates as the accuracy component of a broader reward function (Li et al., 19 Sep 2025).

4. Empirical characterization

The main quantitative evidence for ESWR comes from controlled comparisons on MELD and IEMOCAP. On MELD, the paper reports the following progression: GRPO + IR achieves UA $0$3, WA $0$4, F1 $0$5; adding ESWR gives UA $0$6, WA $0$7, F1 $0$8; combining ESWR with ESR yields UA $0$9, WA $7$0, F1 $7$1. On IEMOCAP, the corresponding values are $7$2, then $7$3, then $7$4. These comparisons show that ESWR improves over GRPO even before the full reasoning enhancement is applied (Li et al., 19 Sep 2025).

The reward-mechanism ablation on MELD isolates ESWR more directly. The paper reports: GRPO+IR with UA $7$5, WA $7$6, F1 $7$7; GRPO+EUR with $7$8; GRPO+ESR with $7$9; GRPO+BCR with $S \in \mathbb{R}^{C \times C}$0; and GRPO+ESWR with $S \in \mathbb{R}^{C \times C}$1. The specific comparison between GRPO+BCR and GRPO+ESWR is the clearest evidence that similarity-weighted reward contributes beyond ordinary binary correctness (Li et al., 19 Sep 2025).

The paper also reports cross-dataset generalization improvements. When training on MELD and testing on RAVDESS, SAVEE, and IEMOCAP, GRPO+BCR improves modestly over SFT+IR, GRPO+ESWR improves more, and combining ESWR with ESR gives the best results overall. This suggests that ESWR is not merely increasing in-domain fit, but is helping the model internalize relations among emotion labels that transfer across datasets (Li et al., 19 Sep 2025).

A plausible implication is that ESWR functions partly as reward densification and partly as inductive bias over emotional geometry. The empirical pattern is consistent with the paper’s claim that psychologically grounded partial credit reduces instability and promotes generalization, especially when exact category boundaries are ambiguous (Li et al., 19 Sep 2025).

5. Adjacent formulations and terminological scope

The explicit term “Emotion Similarity-Weighted Reward” appears in EMO-RL, but several nearby works use related mechanisms that illuminate what ESWR is and is not.

Work	Signal	Relation to ESWR
EMO-RL (Li et al., 19 Sep 2025)	Thresholded similarity reward from Plutchik-based matrix $S \in \mathbb{R}^{C \times C}$2	Canonical ESWR formulation
AffectGPT-R1 (Lian, 2 Aug 2025)	EW-based accuracy reward plus format reward via GRPO	Close equivalent in spirit
RLVER (Wang et al., 3 Jul 2025)	Deterministic simulator emotion score $S \in \mathbb{R}^{C \times C}$3, terminal $S \in \mathbb{R}^{C \times C}$4	Emotion-based reward, not similarity-weighted
EmoSim (Doh et al., 2023)	Similarity regularization term $S \in \mathbb{R}^{C \times C}$5	Regularization analogue, not RL reward

AffectGPT-R1 is the closest analogue. It does not introduce a separately named ESWR module, but it replaces token-level supervision with an EW-based accuracy reward taken directly from the official open-vocabulary emotion recognition protocol, combines it with a format reward, and optimizes the result with GRPO. The paper’s rationale is that token-level loss is misaligned with EW-based evaluation, and that EW-based metrics cannot be directly optimized via gradient backpropagation. Its reward is therefore evaluation-as-reward:

$S \in \mathbb{R}^{C \times C}$6

with total reward

$S \in \mathbb{R}^{C \times C}$7

Because the EW metric gives partial credit according to closeness on the emotion wheel, the mechanism can reasonably be interpreted as emotion-similarity-weighted even though no explicit wheel-distance formula is exposed in the paper. AffectGPT-R1 reports a main comparison of $S \in \mathbb{R}^{C \times C}$8 for AffectGPT versus $S \in \mathbb{R}^{C \times C}$9 for AffectGPT-R1, and states that RL improves sampling quality and performance by more than $C$0 (Lian, 2 Aug 2025).

RLVER occupies a different position. It does not define ESWR and does not compare predicted and target emotions via a similarity kernel. Instead, it trains against a self-consistent affective user simulator that produces a deterministic emotion score $C$1, with terminal reward normalized as $C$2. Conceptually, this is an emotion-state reward or verifiable affective reward rather than a similarity-weighted reward (Wang et al., 3 Jul 2025).

The 2023 retrieval paper on textless speech-to-music retrieval is further removed from RL, but its Emotion Similarity Regularization (EmoSim) is a relevant precursor in functional terms. There, the model adds a similarity-matching loss,

$C$3

and optimizes

$C$4

with $C$5. EmoSim is not a reward in the RL sense, but it similarly preserves graded emotional neighborhood structure rather than treating labels as isolated categories (Doh et al., 2023).

6. Reproducibility, limitations, and interpretive boundaries

Several implementation details are essential for reproducing ESWR as reported in EMO-RL. The backbone model is Qwen2-Audio-7B-Instruct. RL training uses 8 NVIDIA RTX A6000 GPUs, per-device batch size $C$6, gradient accumulation $C$7, $C$8 optimization steps, learning rate $C$9, softmax temperature $\frac{1}{2}$0, and $\frac{1}{2}$1 sampled responses per sample at each RL step. The contradiction threshold is $\frac{1}{2}$2, and $\frac{1}{2}$3 decreases dynamically from $\frac{1}{2}$4 to $\frac{1}{2}$5 during training (Li et al., 19 Sep 2025).

The paper also leaves several reproduction-relevant points underspecified. It does not fully spell out the schedule for $\frac{1}{2}$6, so exact replication may require guesswork. The exact emotion categories and the full emotion-state-transition matrix are not printed explicitly, which means reconstruction must proceed from the dataset labels and Plutchik’s wheel. Neutral is treated specially with fixed similarity $\frac{1}{2}$7. The paper also notes higher inference and training complexity from using LALMs and RL (Li et al., 19 Sep 2025).

These details delimit the scope of the method. ESWR is tailored to speech emotion labels and to a manually specified similarity geometry. It is therefore not a task-agnostic reward recipe. A plausible implication is that transfer to other affective tasks would require redefining the similarity structure, the contradiction threshold, and possibly the role of neutrality. The same boundary appears in AffectGPT-R1, where the closest ESWR-like mechanism delegates similarity structure to the official EW metric without exposing its internal geometry (Lian, 2 Aug 2025).

A further interpretive caution concerns nomenclature. In the literature surveyed here, “emotion similarity-weighted reward” is not yet a universal label for all emotion-aware reward shaping. The explicit ESWR name belongs to EMO-RL. AffectGPT-R1 uses an effectively equivalent EW-based partial-credit reward, RLVER uses deterministic simulator emotion scores without similarity matching, and EmoSim provides a loss-based rather than reward-based analogue. The common thread is alignment of optimization with graded affective structure, but the mechanisms differ in whether they operate over categorical similarity, simulator state, or embedding-space regularization (Li et al., 19 Sep 2025, Lian, 2 Aug 2025, Wang et al., 3 Jul 2025, Doh et al., 2023).