GRPO-PTR: RL for Explainable Speech Emotion

• The paper introduces GRPO-PTR, a method that integrates multi-dimensional reasoning supervision, progressive reward scheduling, and dynamic trust weighting to improve emotion prediction and interpretability.
• It employs a composite reward function combining format, outcome, and learned reasoning scores to fine-tune SpeechLLMs for accurate and interpretable emotion recognition.
• The approach refines speech emotion models by ensuring structured reasoning traces and increased output reliability via group-relative advantage and delayed reward integration.

Group-Relative-Policy-Optimization with Progressive-Trust-aware-Reasoning-Reward (GRPO-PTR) is a reinforcement learning (RL)-based fine-tuning strategy introduced for the development of explainable speech emotion reasoning systems. Specifically applied in the EmotionThinker framework, GRPO-PTR advances beyond prior approaches by integrating multi-dimensional supervision of the intermediate reasoning process, a progressive reward schedule, and a dynamic trustworthiness-weighted mechanism to align reasoning reward with outcome correctness. The method addresses the need for both accurate and interpretable emotion predictions grounded in prosodic and acoustic cues, moving speech-based LLMs (SpeechLLMs) toward deeper multimodal reasoning (Wang et al., 22 Jan 2026).

1. Conceptual Principles

GRPO-PTR is designed to optimize SpeechLLMs not only for final decision accuracy (e.g., emotion label classification) but also for generating structured, high-quality, and interpretable reasoning traces. The RL objectives are multi-pronged:

• Preserve output structural correctness via a format reward.
• Guarantee accuracy of the final answer via an outcome reward.
• Supervise the compositional reasoning steps via a learned, multi-dimensional reward assessing reasoning quality.
• Modulate the reasoning reward dynamically according to a trustworthiness weight reflecting the alignment between reasoning quality and answer correctness within a sampled group.
• Introduce the reasoning reward progressively, withholding it until the model demonstrates baseline competence on format and outcome, thereby avoiding early-stage optimization instability (Wang et al., 22 Jan 2026).

2. Comparison with Standard GRPO

The standard Group-Relative Policy Optimization (GRPO) framework employs only rule-based outcome and format rewards. Specifically, it uses:

• A format reward $R_f$ to enforce the required output schema (e.g., > …</think><answer>…</answer>). > > - An outcome reward $R_o$ that is binary, set to 1 if the predicted label matches the gold label and 0 otherwise. > > GRPO-PTR introduces three principal innovations over this baseline: > > - A small, trained reward model provides fine-grained, multi-dimensional scores along axes such as factual alignment, interpretative quality, caption completeness, and fluency/structure. > > - A dynamically computed trustworthiness weight $T$ down-weights the reasoning reward in cases where it fails to preferentially reward correct over incorrect answers within the group of samples. > > - A progressive schedule delays the inclusion of the reasoning reward until the model reliably meets baseline accuracy and formatting constraints, thus preventing destabilization in early training (Wang et al., 22 Jan 2026). > > ## 3. Formal Definitions and Mathematical Structure > > Given input $x$ (audio and transcript) and ground-truth label $y^*$, the policy $\pi_\theta(o\,|\,x)$ emits outputs $o$ structured as `<think>…<answer>y</answer>$. At each RL step, a group of$K$candidates${o_i}_{i=1}^K</sup>issampled.</li></ul><h3class=′paper−heading′id=′rule−based−rewards′>3.1Rule−basedRewards</h3><ul><li><strong>Formatreward</strong>:</li></ul><p></sup> is sampled.</li> </ul> <h3 class='paper-heading' id='rule-based-rewards'>3.1 Rule-based Rewards</h3> <ul> <li><strong>Format reward</strong>:</li> </ul> <p>R_f(o) = \begin{cases} 1, & o \text{ follows the required XML schema} \ 0, & \text{otherwise} \end{cases}</p><ul><li><strong>Outcomereward</strong>:</li></ul><p></p> <ul> <li><strong>Outcome reward</strong>:</li> </ul> <p>R_o$0 3.2 Learned Multi-dimensional Reasoning Reward A reward model $R_o$1 assigns four ratings $R_o$2 to each reasoning trace, which are normalized and aggregated: $R_o$3 3.3 Trustworthiness Weight For each candidate group: Compute group means $R_o$4 (correct outputs) and $R_o$5 (incorrect outputs). Define: $R_o$6 This ensures that reasoning reward is only trusted (i.e., upweighted) when it aligns with, or at least does not misalign with, outcome correctness. 3.4 Composite Reward and Policy Objective Total reward for output $R_o$7: $R_o$8 The group-relative advantage is $R_o$9, and the surrogate objective is the PPO-style: $T$0 where $T$1, $T$2, and $T$3 (Wang et al., 22 Jan 2026). 4. Progressive Reward Scheduling and Algorithm The GRPO-PTR process consists of two phases: Stage 1: Format+Outcome Warm-up $T$4 (reasoning reward weight) is set to 0. Only $T$5 and $T$6 shape the reward until rolling-average emotion accuracy exceeds a threshold ($T$7). Stage 2: Full GRPO-PTR $T$8 is set to its full value (typically 0.5), and the complete composite reward is applied. Pseudocode for both phases is detailed in (Wang et al., 22 Jan 2026), emphasizing sampling of $T$9 candidates, reward aggregation, policy update via group-relative advantage, and delayed introduction of the reasoning reward. 5. Multi-dimensional Reward Model The multi-dimensional reward model (base architecture: Qwen2.5-Omni-3B) is fine-tuned on 101.4k triples of (prompt, reasoning, four-dimensional label). Synthetic data generated by GPT-4o is used to obtain reasoning traces at varying quality levels. The four evaluation criteria are: Factual Alignment Interpretative Quality Caption Completeness Fluency & Structural Clarity Each criterion is scored from 1–5, normalized to $x$0 before aggregation. Learned weights $x$1 combine these into a scalar used in the composite reward (Wang et al., 22 Jan 2026). 6. Core Hyperparameters and Implementation Considerations Typical hyperparameter choices and practical recommendations are as follows: Parameter Default Value Notes Number of candidates $x$2 8 Balances sample diversity and compute Reward Weights $x$3, $x$4, $x$5 As in full training phase KL Penalty Coefficient $x$6 For KL regularization in PPO objective PPO Clipping $x$7 Standard stability measure Learning Rate $x$8 For policy update Warm-up Threshold $x$9 Rolling emotion accuracy before reasoning reward enabled Key implementation notes: Delaying reasoning reward avoids random fluctuations in initial policy, which otherwise degrade advantage estimates required for stable RL. The trustworthiness mechanism ($y^*$0) acts as a safeguard, preventing propagation of spurious signals where the learned reward model lacks alignment with true outcome correctness (Wang et al., 22 Jan 2026). 7. Context and Implications GRPO-PTR was introduced in the context of EmotionThinker to reformulate speech emotion recognition as a deep reasoning task, rather than a pure classification problem. This approach improves both emotion accuracy and explanation quality, measured by standard benchmarks within multimodal reasoning. A plausible implication is the extensibility of GRPO-PTR principles to other structure-conditioned, explainable AI tasks that require simultaneous optimization of outcome correctness and high-fidelity intermediate reasoning. The mechanisms for progressive reward introduction and trust-weighted reasoning scoring provide a generalizable strategy for stabilizing RL-based fine-tuning in low-signal or reward-misaligned settings (Wang et al., 22 Jan 2026).