--- title: Multi-Aspect Reward Optimization url: https://www.emergentmind.com/topics/multi-aspect-reward-optimization type: topic --- # Multi-Aspect Reward Optimization Multi-Aspect Reward Optimization, in the current literature on post-training large language models, denotes optimization schemes in which policy updates are driven by reward signals encoding more than one criterion or more than one structural view of quality. In GRPO-centered work, these criteria include explicit alignment dimensions such as politeness, meaningfulness, actionability, and safety; correctness-plus-format rewards for reasoning; prefix-level continue/stop guidance derived from outcome-only supervision; entropy-weighted token or sequence shaping; semantic-uncertainty modulation; pairwise or groupwise preferences; diversity and coverage rewards; and separate utilities for candidate generation and ranking [2503.21819, 2602.17025, 2508.04349, 2505.12346, 2511.12596, 2605.12995]. The unifying mechanism is typically a group-based policy optimizer—most often GRPO—that samples multiple outputs for the same input, computes rewards or reward-derived advantages within that group, and propagates the resulting signal to tokens, sequences, prefixes, phases, or group-level structures. ## 1. Formal setting and optimization substrate The dominant substrate for multi-aspect reward optimization in this literature is Group Relative Policy Optimization. In a standard formulation, for each prompt \(s\), one samples \(G\) completions \(a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)\), evaluates each with a reward \(r(s,a_j)\), and defines \[ \mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}. \] The corresponding policy-gradient form is \[ \nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr]. \] This critic-free construction replaces a learned value network with group-wise normalization, and thereby makes the reward design itself the primary carrier of alignment structure [2503.21819]. A separate theoretical line studies GRPO through a U-statistic lens. There, the gradient estimator with a group leave-one-out mean baseline is shown to be a second-order U-statistic, with mean-squared error \[ \mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr), \] and a universal scaling law for optimal group size \[ G^* =\sqrt{\frac{c_3}{c_1}}. \] That analysis further states that GRPO is asymptotically equivalent to an oracle policy gradient algorithm with access to a value function and achieves asymptotically optimal performance within a broad class of policy gradient algorithms [2603.01162]. In the context of multi-aspect rewards, this establishes that richer reward composition does not require abandoning critic-free optimization; instead, the reward itself becomes the locus of modeling complexity. ## 2. Explicit multi-label reward models and scalarization The clearest explicit instance of multi-aspect reward optimization is the GRPO framework with a multi-label reward regression model for safe and aligned language generation. In that setup, the reward model is a multi-label regressor \[ r_\phi(s,a)\in[0,1]^K \] with \(K=4\) aspects: politeness, meaningfulness, actionability, and safety. It is trained on human-normalized scores by minimizing \[ \mathcal{L}_{\text{RM}}(\phi)\;=\;\frac1N\sum_{j=1}^N\sum_{k=1}^K\bigl(r_{\phi,k}(s^{(j)},a^{(j)})-y^{(j)}_k\bigr)^2. \] The aspect scores are then linearly aggregated into a scalar reward, \[ R(s,a)\;=\;\sum_{k=1}^K w_k\,r_{\phi,k}(s,a),\qquad w_k=1\;\forall k. \] The paper reports that a RoBERTa-base encoder with sigmoid outputs achieved \(R^2\approx0.85\) on validation, and that ablations showed uneven weighting or a single-scalar reward model leads to overemphasis on safety at the expense of meaningfulness [2503.21819]. This formulation is notable for separating *reward representation* from *policy optimization*. The reward model is multi-aspect, but the policy still receives a single scalar \(R(s,a)\) through GRPO. That scalarization preserves compatibility with standard group-relative updates while retaining interpretability at the aspect level. A plausible implication is that multi-aspect reward optimization, in this explicit sense, is not a replacement for scalar RL objectives but a method for constructing a scalar objective from a structured alignment model. The same paper emphasizes that GRPO naturally accommodates such scalarized multi-aspect rewards while avoiding a separate critic and the attendant PPO-style value-function complexity. In this construction, multi-objective handling is explicit rather than implicit: the aspect decomposition is part of the reward model, and the optimizer is responsible only for relative ranking within prompt-conditioned groups [2503.21819]. ## 3. Reward decomposition across tokens, sequences, prefixes, groups, and phases Multi-aspect reward optimization is not confined to multiple semantic labels. A broader line of work decomposes reward by *where* it is attached in the generated object. | Locus of reward | Construction | Representative method | |---|---|---| | Token level | Entropy-weighted token reward \(\tilde r_{i,t}\) or conflict-token masking \(\lambda_{i,t}\) | "GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy" [2508.04349]; "GTPO: Trajectory-Based Policy Optimization in Large Language Models" [2508.03772] | | Sequence level | Entropy-shaped sequence reward \(\hat r_i\) | "GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy" [2508.04349] | | Prefix / partial trajectory | \(R^{\rm WS}_i=\lambda R^{\rm pref}_i + R^{\rm final}_i\) from consecutive-prefix comparisons | "WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning" [2602.17025] | | Group level | Reward vector \(\tilde R(\mathbf o)\) defined over the whole sampled group | "Group-Aware Reinforcement Learning for Output Diversity in Large Language Models" [2511.12596] | | Two-phase sequence | Separate \(R_{\rm slate}\) and \(R_{\rm rank}\) with separate group-relative advantages | "F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking" [2605.12995] | At the token and sequence levels, dynamic entropy weighting provides a canonical example. GTPO assigns each token in successful sequences a shaped reward \[ \tilde r_{i,t} = r_i \;+\; \alpha\; \frac{H_{i,t}}{\sum_{k=1}^n H_{k,t}}\;d_t, \] while GRPO-S uses a sequence-level entropy score \[ H_i=\frac1{|o_i|}\sum_{t=1}^{|o_i|} H_{i,t},\qquad \hat r_i = r_i\;+\;\beta\;\frac{H_i}{\sum_{k=1}^n H_k}\;n. \] The stated motivation is that token-level uncertainty often peaks at critical decision points in a reasoning chain, so high-entropy tokens in correct answers deserve a larger share of the reward [2508.04349]. A different token-sensitive mechanism appears in the other GTPO paper, which focuses on conflict tokens and entropy collapse. There, tokens appearing in the same position across completions with opposite rewards are masked so that negative updates are skipped and positive ones amplified, through \[ \lambda_{i,t} = \begin{cases} 1, & \mathcal M_{i,t}=0,\\ 0, & \mathcal M_{i,t}=1\ \land\ A_i<0,\\ 2, & \mathcal M_{i,t}=1\ \land\ A_i>0. \end{cases} \] The final objective also includes an entropy-based completion filter and a trajectory-level entropy penalty \(-\gamma\,\langle H\rangle_i\), replacing KL-based stabilization with direct entropy control [2508.03772]. Prefix-level reward decomposition is developed in WS-GRPO. A Bradley–Terry preference model \(P_\omega(q,\tau_A,\tau_B)\) is trained from outcome-only correctness labels, and then converted into prefix-level signals by comparing consecutive prefixes: \[ r^{\rm step}_{i,t} = P_\omega\bigl(q,\;s_{i,1:t-1},\;s_{i,1:t}\bigr),\qquad R^{\rm pref}_i=\sum_{t=2}^{|\tau_i|}r^{\rm step}_{i,t}, \] which are mixed with the sparse final reward as \[ R^{\rm WS}_i=\lambda\,R^{\rm pref}_i + R^{\rm final}_i. \] This construction turns a single terminal reward into correctness-aware continue/stop guidance over partial trajectories [2602.17025]. Group-level reward decomposition appears in GAPO, where the reward for rollout \(i\) can depend on the entire sampled group \(\mathbf o=(o_1,\dots,o_G)\). For a known valid set \(\mathcal V=\{v_1,\dots,v_L\}\), the frequency-aware reward is \[ \tilde R_i(\mathbf o) = \begin{cases} 1-\bigl(f_{o_i}(\mathbf o)-\tfrac1L\bigr), & o_i\in\mathcal V,\\ -1, & \text{otherwise}. \end{cases} \] This allows the optimizer to target diversity and uniform coverage, objectives that cannot be expressed by independent per-rollout rewards alone [2511.12596]. Phase-specific decomposition is central to F-GRPO. The policy is factorized as \[ \pi_\theta(\tau,\sigma\mid x)=\pi^{\rm slate}_\theta(\tau\mid x)\cdot\pi^{\rm rank}_\theta(\sigma\mid x,\tau), \] with separate rewards for coverage and ranking quality, \[ R_{\rm slate}(x,\tau),\qquad R_{\rm rank}(x,\tau,\sigma;k), \] and separate group-relative advantages \[ \hat A_{\rm slate}^{(i)} = R_{\rm slate}^{(i)}-\bar R_{\rm slate},\qquad \hat A_{\rm rank}^{(i)} = R_{\rm rank}^{(i)}-\bar R_{\rm rank}. \] The total loss is then \(L=L_{\rm slate}+\lambda L_{\rm rank}+\beta_{\rm KL}D_{\rm KL}\), avoiding what the paper calls cross-phase gradient contamination [2605.12995]. A related two-layer decomposition appears in MGRPO, where standard GRPO is followed by a second GRPO pass trained to identify and correct errors in the initial response. The second layer retains only successful corrections or confirmations, thereby supplying implicit process-level supervision without an explicit dense reward model [2506.04746]. This suggests that “aspect” can refer not only to reward semantics but also to reward placement in a staged reasoning-and-revision process. ## 4. Preference, uncertainty, causal interaction, and distillation as auxiliary reward aspects Another major branch of the literature expands the reward signal through auxiliary signals that are not simple semantic labels. A direct bridge to preference optimization is established by the observation that GRPO can be reframed as contrastive learning. One formulation shows that both GRPO and DPO have gradients of contrastive form, and that GRPO with group size \(2\) is effectively the same form as DPO up to a choice of weight. The resulting 2-GRPO objective is \[ \mathcal{J}_{2\text{-GRPO}}(\theta) =\mathbb{E}_{q}\,\mathbb{E}_{o^+,o^-\sim\pi_\theta} \frac12\Bigl[\log\pi_\theta(o^+\mid q)-\log\pi_\theta(o^-\mid q)\Bigr]. \] The same analysis states that the equations extend verbatim to any scalar reward \(\in\mathbb R\) or learned preference model \(r_\phi(q,o)\), and explicitly notes that pairwise or groupwise human preference data can be used to define positives and negatives [2510.00977]. Uncertainty-aware reward modulation is developed in SEED-GRPO, which uses semantic entropy over clusters of semantically equivalent outputs as a prompt-level uncertainty signal. After defining \(SE(q)\) over meaning clusters, the GRPO advantage is rescaled as \[ \hat H_i = A_i \cdot f\!\left(\alpha\cdot \frac{SE(q)}{SE_{\max}}\right). \] Low semantic entropy implies semantically consistent rollouts and therefore larger updates; high semantic entropy implies conflicting meanings and therefore smaller updates. The paper explicitly frames this as dynamic adjustment of policy update magnitudes based on question uncertainty [2505.12346]. AMIR-GRPO introduces an implicit DPO-style contrastive regularizer mined directly from within-group reward rankings. For reward-differentiated pairs \[ S(q)=\{(i,j)\mid r_i>r_j \text{ and } r_i-r_j>\delta\}, \] it defines \[ J_{\rm pref}(\theta)=\mathbb E_q\Bigl[\sum_{(i,j)\in S(q)}\log\sigma(Z_{i,j}(\theta))\Bigr], \] and jointly optimizes \[ J_{\rm AMIR}(\theta)=J_{\rm GRPO}(\theta)+\alpha_{\rm reg}\cdot J_{\rm pref}(\theta). \] The stated effect is to amplify suppression of low-reward trajectories, attenuate response-level length bias, and transform each rollout group into a denser set of supervision constraints [2601.03661]. GCPO incorporates semantic interaction through a structural causal model. It argues that candidate responses are not independent once conditioned on a final integrated output, and uses a causally informed weight \[ \Upsilon_i = \alpha \cdot \cos(z_i,\bar Z_i-\bar Z_i' + \bar z), \qquad B_i=A_i\cdot \Upsilon_i, \] together with a second KL penalty to a causally projected reference distribution \(\pi'_{\rm ref}\). The full objective therefore contains the usual group-relative term, a standard KL to \(\pi_{\rm ref}\), and a second KL to \(\pi'_{\rm ref}\) [2508.05428]. This suggests that multi-aspect reward optimization can be extended to semantic complementarity and contradiction among responses, rather than only to properties of individual responses. CoDistill-GRPO introduces yet another auxiliary aspect: on-policy knowledge distillation. The small model’s effective reward is \[ \widetilde r_i = r_i + \alpha\,\mathrm{KD}_i,\qquad \mathrm{KD}_i =\frac1N\sum_{t=1}^N\log\frac{\pi_\theta(o_{i,t}\mid q,o_{i,