GRPO-PTR: RL for Explainable Speech Emotion

• The paper introduces GRPO-PTR, a method that integrates multi-dimensional reasoning supervision, progressive reward scheduling, and dynamic trust weighting to improve emotion prediction and interpretability.
• It employs a composite reward function combining format, outcome, and learned reasoning scores to fine-tune SpeechLLMs for accurate and interpretable emotion recognition.
• The approach refines speech emotion models by ensuring structured reasoning traces and increased output reliability via group-relative advantage and delayed reward integration.

Group-Relative-Policy-Optimization with Progressive-Trust-aware-Reasoning-Reward (GRPO-PTR) is a reinforcement learning (RL)-based fine-tuning strategy introduced for the development of explainable speech emotion reasoning systems. Specifically applied in the EmotionThinker framework, GRPO-PTR advances beyond prior approaches by integrating multi-dimensional supervision of the intermediate reasoning process, a progressive reward schedule, and a dynamic trustworthiness-weighted mechanism to align reasoning reward with outcome correctness. The method addresses the need for both accurate and interpretable emotion predictions grounded in prosodic and acoustic cues, moving speech-based LLMs (SpeechLLMs) toward deeper multimodal reasoning (Wang et al., 22 Jan 2026).

1. Conceptual Principles

GRPO-PTR is designed to optimize SpeechLLMs not only for final decision accuracy (e.g., emotion label classification) but also for generating structured, high-quality, and interpretable reasoning traces. The RL objectives are multi-pronged:

• Preserve output structural correctness via a format reward.
• Guarantee accuracy of the final answer via an outcome reward.
• Supervise the compositional reasoning steps via a learned, multi-dimensional reward assessing reasoning quality.
• Modulate the reasoning reward dynamically according to a trustworthiness weight reflecting the alignment between reasoning quality and answer correctness within a sampled group.
• Introduce the reasoning reward progressively, withholding it until the model demonstrates baseline competence on format and outcome, thereby avoiding early-stage optimization instability (Wang et al., 22 Jan 2026).

2. Comparison with Standard GRPO

The standard Group-Relative Policy Optimization (GRPO) framework employs only rule-based outcome and format rewards. Specifically, it uses:

• A format reward $R_f$ to enforce the required output schema (e.g., > …</think><answer>…</answer>). > > - An outcome reward $R_o$ that is binary, set to 1 if the predicted label matches the gold label and 0 otherwise. > > GRPO-PTR introduces three principal innovations over this baseline: > > - A small, trained reward model provides fine-grained, multi-dimensional scores along axes such as factual alignment, interpretative quality, caption completeness, and fluency/structure. > > - A dynamically computed trustworthiness weight $T$ down-weights the reasoning reward in cases where it fails to preferentially reward correct over incorrect answers within the group of samples. > > - A progressive schedule delays the inclusion of the reasoning reward until the model reliably meets baseline accuracy and formatting constraints, thus preventing destabilization in early training (Wang et al., 22 Jan 2026). > > ## 3. Formal Definitions and Mathematical Structure > > Given input $x$ (audio and transcript) and ground-truth label $y^*$, the policy $\pi_\theta(o\,|\,x)$ emits outputs $o$ structured as `<think>…<answer>y</answer>$. At each RL step, a group of$K$candidates${o_i}_{i=1}^K</sup>issampled.</li></ul><h3class=′paper−heading′id=′rule−based−rewards′>3.1Rule−basedRewards</h3><ul><li><strong>Formatreward</strong>:</li></ul><p></sup> is sampled.</li> </ul> <h3 class='paper-heading' id='rule-based-rewards'>3.1 Rule-based Rewards</h3> <ul> <li><strong>Format reward</strong>:</li> </ul> <p>R_f(o) = \begin{cases} 1, & o \text{ follows the required XML schema} \ 0, & \text{otherwise} \end{cases}</p><ul><li><strong>Outcomereward</strong>:</li></ul><p></p> <ul> <li><strong>Outcome reward</strong>:</li> </ul> <p>R_o(o, y^*) = \mathbb{I}[\text{predicted\_label}(o) = y^*]</p><h3class=′paper−heading′id=′learned−multi−dimensional−reasoning−reward′>3.2LearnedMulti−dimensionalReasoningReward</h3><p>Arewardmodel</p> <h3 class='paper-heading' id='learned-multi-dimensional-reasoning-reward'>3.2 Learned Multi-dimensional Reasoning Reward</h3> <p>A reward model r_\phiassignsfourratings assigns four ratings (\hat r_1, \hat r_2, \hat r_3, \hat r_4) \in [1,5]^4toeachreasoningtrace,whicharenormalizedandaggregated:</p><p> to each reasoning trace, which are normalized and aggregated:</p> <p>R_t(o) = \sum_{j=1}^4 w_j \tilde r_j(o),\quad \tilde r_j = \frac{\hat r_j - 1}{4},\quad \sum_j w_j = 1</p><h3class=′paper−heading′id=′trustworthiness−weight′>3.3TrustworthinessWeight</h3><p>Foreachcandidategroup:</p><ul><li>Computegroupmeans</p> <h3 class='paper-heading' id='trustworthiness-weight'>3.3 Trustworthiness Weight</h3> <p>For each candidate group:</p> <ul> <li>Compute group means \overline R_t^{\rm corr}(correctoutputs)and (correct outputs) and \overline R_t^{\rm wrong}(incorrectoutputs).</li><li>Define:</li></ul><p> (incorrect outputs).</li> <li>Define:</li> </ul> <p>T = \begin{cases} 1, & \overline R_t^{\rm corr} \ge \overline R_t^{\rm wrong} \ \exp(\overline R_t^{\rm corr} - \overline R_t^{\rm wrong}), & \text{otherwise} \end{cases}</p><p>Thisensuresthatreasoningrewardisonlytrusted(i.e.,upweighted)whenitalignswith,oratleastdoesnotmisalignwith,outcomecorrectness.</p><h3class=′paper−heading′id=′composite−reward−and−policy−objective′>3.4CompositeRewardandPolicyObjective</h3><p>Totalrewardforoutput</p> <p>This ensures that reasoning reward is only trusted (i.e., upweighted) when it aligns with, or at least does not misalign with, outcome correctness.</p> <h3 class='paper-heading' id='composite-reward-and-policy-objective'>3.4 Composite Reward and Policy Objective</h3> <p>Total reward for output o_i:</p><p>:</p> <p>R_i = \alpha_f R_f(o_i) + \alpha_o R_o(o_i, y^*) + \alpha_t T R_t(o_i)</p><p>Thegroup−relativeadvantageis</p> <p>The group-relative advantage is \hat R_i = R_i - \frac{1}{K}\sum_{j=1}^K R_j,andthesurrogateobjectiveisthePPO−style:</p><p>, and the surrogate objective is the PPO-style:</p> <p>L(\theta) = -\frac{1}{K}\sum_{i=1}^K \min\left(\rho_i(\theta) \hat R_i,\, \mathrm{clip}(\rho_i(\theta), 1-\epsilon, 1+\epsilon)\hat R_i\right) + \beta\,{\rm KL}[\pi_\theta \| \pi_{\theta_{\rm old}}]</p><p>where</p> <p>where \rho_i(\theta) = \frac{\pi_\theta(o_i|x)}{\pi_{\theta_{\rm old}}(o_i|x)},, \epsilon=0.2,and, and \beta\approx0.04(<ahref="/papers/2601.15668"title=""rel="nofollow"data−turbo="false"class="assistant−link"x−datax−tooltip.raw="">Wangetal.,22Jan2026</a>).</p><h2class=′paper−heading′id=′progressive−reward−scheduling−and−algorithm′>4.ProgressiveRewardSchedulingandAlgorithm</h2><p>TheGRPO−PTRprocessconsistsoftwophases:</p><ul><li><strong>Stage1:Format+OutcomeWarm−up</strong> (<a href="/papers/2601.15668" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Wang et al., 22 Jan 2026</a>).</p> <h2 class='paper-heading' id='progressive-reward-scheduling-and-algorithm'>4. Progressive Reward Scheduling and Algorithm</h2> <p>The GRPO-PTR process consists of two phases:</p> <ul> <li><strong>Stage 1: Format+Outcome Warm-up</strong> \alpha_t(reasoningrewardweight)issetto0.Only (reasoning reward weight) is set to 0. Only R_fand and R_oshapetherewarduntilrolling−averageemotionaccuracyexceedsathreshold( shape the reward until rolling-average emotion accuracy exceeds a threshold (\tau \approx 50\%).</li><li><strong>Stage2:FullGRPO−PTR</strong>).</li> <li><strong>Stage 2: Full GRPO-PTR</strong> \alpha_tissettoitsfullvalue(typically0.5),andthecompletecompositerewardisapplied.</li></ul><p>Pseudocodeforbothphasesisdetailedin(<ahref="/papers/2601.15668"title=""rel="nofollow"data−turbo="false"class="assistant−link"x−datax−tooltip.raw="">Wangetal.,22Jan2026</a>),emphasizingsamplingof is set to its full value (typically 0.5), and the complete composite reward is applied.</li> </ul> <p>Pseudocode for both phases is detailed in (<a href="/papers/2601.15668" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Wang et al., 22 Jan 2026</a>), emphasizing sampling of K$ candidates, reward aggregation, policy update via group-relative advantage, and delayed introduction of the reasoning reward.</p> <h2 class='paper-heading' id='multi-dimensional-reward-model'>5. Multi-dimensional Reward Model</h2> <p>The multi-dimensional reward model (base architecture: Qwen2.5-Omni-3B) is fine-tuned on 101.4k triples of (prompt, reasoning, four-dimensional label). Synthetic data generated by <a href="https://www.emergentmind.com/topics/llm-judge-gpt-4o" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">GPT-4o</a> is used to obtain reasoning traces at varying quality levels. The four evaluation criteria are:</p> <ol> <li>Factual Alignment</li> <li>Interpretative Quality</li> <li>Caption Completeness</li> <li>Fluency &amp; Structural Clarity</li> </ol> <p>Each criterion is scored from 1–5, normalized to $[0,1]beforeaggregation.Learnedweights before aggregation. Learned weights w_jcombinetheseintoascalarusedinthecompositereward(<ahref="/papers/2601.15668"title=""rel="nofollow"data−turbo="false"class="assistant−link"x−datax−tooltip.raw="">Wangetal.,22Jan2026</a>).</p><h2class=′paper−heading′id=′core−hyperparameters−and−implementation−considerations′>6.CoreHyperparametersandImplementationConsiderations</h2><p>Typicalhyperparameterchoicesandpracticalrecommendationsareasfollows:</p><divclass=′overflow−x−automax−w−fullmy−4′><tableclass=′tableborder−collapsew−full′style=′table−layout:fixed′><thead><tr><th>Parameter</th><th>DefaultValue</th><th>Notes</th></tr></thead><tbody><tr><td>Numberofcandidates combine these into a scalar used in the composite reward (<a href="/papers/2601.15668" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Wang et al., 22 Jan 2026</a>).</p> <h2 class='paper-heading' id='core-hyperparameters-and-implementation-considerations'>6. Core Hyperparameters and Implementation Considerations</h2> <p>Typical hyperparameter choices and practical recommendations are as follows:</p> <div class='overflow-x-auto max-w-full my-4'><table class='table border-collapse w-full' style='table-layout: fixed'><thead><tr> <th>Parameter</th> <th>Default Value</th> <th>Notes</th> </tr> </thead><tbody><tr> <td>Number of candidates K</td><td>8</td><td>Balancessample<ahref="https://www.emergentmind.com/topics/diversity−beta−recall"title=""rel="nofollow"data−turbo="false"class="assistant−link"x−datax−tooltip.raw="">diversity</a>andcompute</td></tr><tr><td>RewardWeights</td><td></td> <td>8</td> <td>Balances sample <a href="https://www.emergentmind.com/topics/diversity-beta-recall" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">diversity</a> and compute</td> </tr> <tr> <td>Reward Weights</td> <td>\alpha_f=0.3,, \alpha_o=1.0,, \alpha_t=0.5</td><td>Asinfulltrainingphase</td></tr><tr><td>KLPenaltyCoefficient</td><td></td> <td>As in full training phase</td> </tr> <tr> <td>KL Penalty Coefficient</td> <td>\beta=0.04</td><td>ForKLregularizationin<ahref="https://www.emergentmind.com/topics/transformer−based−proximal−policy−optimization−ppo"title=""rel="nofollow"data−turbo="false"class="assistant−link"x−datax−tooltip.raw="">PPO</a>objective</td></tr><tr><td>PPOClipping</td><td></td> <td>For KL regularization in <a href="https://www.emergentmind.com/topics/transformer-based-proximal-policy-optimization-ppo" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">PPO</a> objective</td> </tr> <tr> <td>PPO Clipping</td> <td>\epsilon=0.2</td><td>Standardstabilitymeasure</td></tr><tr><td>LearningRate</td><td></td> <td>Standard stability measure</td> </tr> <tr> <td>Learning Rate</td> <td>1\times 10^{-6}</td><td>Forpolicyupdate</td></tr><tr><td>Warm−upThreshold</td><td></td> <td>For policy update</td> </tr> <tr> <td>Warm-up Threshold</td> <td>\tau \approx 50\%</td><td>Rollingemotionaccuracybeforereasoningrewardenabled</td></tr></tbody></table></div><p>Keyimplementationnotes:</p><ul><li>Delayingreasoningrewardavoidsrandomfluctuationsininitialpolicy,whichotherwisedegradeadvantageestimatesrequiredforstableRL.</li><li>Thetrustworthinessmechanism(</td> <td>Rolling emotion accuracy before reasoning reward enabled</td> </tr> </tbody></table></div> <p>Key implementation notes:</p> <ul> <li>Delaying reasoning reward avoids random fluctuations in initial policy, which otherwise degrade advantage estimates required for stable RL.</li> <li>The trustworthiness mechanism (T$) acts as a safeguard, preventing propagation of spurious signals where the learned reward model lacks alignment with true outcome correctness (Wang et al., 22 Jan 2026).

7. Context and Implications

GRPO-PTR was introduced in the context of EmotionThinker to reformulate speech emotion recognition as a deep reasoning task, rather than a pure classification problem. This approach improves both emotion accuracy and explanation quality, measured by standard benchmarks within multimodal reasoning. A plausible implication is the extensibility of GRPO-PTR principles to other structure-conditioned, explainable AI tasks that require simultaneous optimization of outcome correctness and high-fidelity intermediate reasoning. The mechanisms for progressive reward introduction and trust-weighted reasoning scoring provide a generalizable strategy for stabilizing RL-based fine-tuning in low-signal or reward-misaligned settings (Wang et al., 22 Jan 2026).