Multi-Aspect Reward Optimization

• Multi-Aspect Reward Optimization is a framework that integrates multiple reward signals—such as safety, meaningfulness, and politeness—via group-relative policy gradients.
• It decomposes rewards across tokens, sequences, prefixes, groups, and phases, enabling precise control over aspects like correctness, diversity, and efficiency.
• Recent empirical studies demonstrate that this approach improves alignment metrics and output diversity while addressing challenges like scalarization bias and instability.

Multi-Aspect Reward Optimization, in the current literature on post-training LLMs, denotes optimization schemes in which policy updates are driven by reward signals encoding more than one criterion or more than one structural view of quality. In GRPO-centered work, these criteria include explicit alignment dimensions such as politeness, meaningfulness, actionability, and safety; correctness-plus-format rewards for reasoning; prefix-level continue/stop guidance derived from outcome-only supervision; entropy-weighted token or sequence shaping; semantic-uncertainty modulation; pairwise or groupwise preferences; diversity and coverage rewards; and separate utilities for candidate generation and ranking (Li et al., 26 Mar 2025, Mundada et al., 19 Feb 2026, Tan et al., 6 Aug 2025, Chen et al., 18 May 2025, Anschel et al., 16 Nov 2025, Surana et al., 13 May 2026). The unifying mechanism is typically a group-based policy optimizer—most often GRPO—that samples multiple outputs for the same input, computes rewards or reward-derived advantages within that group, and propagates the resulting signal to tokens, sequences, prefixes, phases, or group-level structures.

1. Formal setting and optimization substrate

The dominant substrate for multi-aspect reward optimization in this literature is Group Relative Policy Optimization. In a standard formulation, for each prompt $s$, one samples $G$ completions $a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$, evaluates each with a reward $r(s,a_j)$, and defines

$$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$

The corresponding policy-gradient form is

$$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$

This critic-free construction replaces a learned value network with group-wise normalization, and thereby makes the reward design itself the primary carrier of alignment structure (Li et al., 26 Mar 2025).

A separate theoretical line studies GRPO through a U-statistic lens. There, the gradient estimator with a group leave-one-out mean baseline is shown to be a second-order U-statistic, with mean-squared error

$$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$

and a universal scaling law for optimal group size

$G^* =\sqrt{\frac{c_3}{c_1}}.$

That analysis further states that GRPO is asymptotically equivalent to an oracle policy gradient algorithm with access to a value function and achieves asymptotically optimal performance within a broad class of policy gradient algorithms (Zhou et al., 1 Mar 2026). In the context of multi-aspect rewards, this establishes that richer reward composition does not require abandoning critic-free optimization; instead, the reward itself becomes the locus of modeling complexity.

2. Explicit multi-label reward models and scalarization

The clearest explicit instance of multi-aspect reward optimization is the GRPO framework with a multi-label reward regression model for safe and aligned language generation. In that setup, the reward model is a multi-label regressor

$r_\phi(s,a)\in[0,1]^K$

with $K=4$ aspects: politeness, meaningfulness, actionability, and safety. It is trained on human-normalized scores by minimizing

$G$0

The aspect scores are then linearly aggregated into a scalar reward,

$G$1

The paper reports that a RoBERTa-base encoder with sigmoid outputs achieved $G$2 on validation, and that ablations showed uneven weighting or a single-scalar reward model leads to overemphasis on safety at the expense of meaningfulness (Li et al., 26 Mar 2025).

This formulation is notable for separating reward representation from policy optimization. The reward model is multi-aspect, but the policy still receives a single scalar $G$3 through GRPO. That scalarization preserves compatibility with standard group-relative updates while retaining interpretability at the aspect level. A plausible implication is that multi-aspect reward optimization, in this explicit sense, is not a replacement for scalar RL objectives but a method for constructing a scalar objective from a structured alignment model.

The same paper emphasizes that GRPO naturally accommodates such scalarized multi-aspect rewards while avoiding a separate critic and the attendant PPO-style value-function complexity. In this construction, multi-objective handling is explicit rather than implicit: the aspect decomposition is part of the reward model, and the optimizer is responsible only for relative ranking within prompt-conditioned groups (Li et al., 26 Mar 2025).

3. Reward decomposition across tokens, sequences, prefixes, groups, and phases

Multi-aspect reward optimization is not confined to multiple semantic labels. A broader line of work decomposes reward by where it is attached in the generated object.

Locus of reward	Construction	Representative method
Token level	Entropy-weighted token reward $G$4 or conflict-token masking $G$5	"GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy" (Tan et al., 6 Aug 2025); "GTPO: Trajectory-Based Policy Optimization in LLMs" (Simoni et al., 5 Aug 2025)
Sequence level	Entropy-shaped sequence reward $G$6	"GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy" (Tan et al., 6 Aug 2025)
Prefix / partial trajectory	$G$7 from consecutive-prefix comparisons	"WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning" (Mundada et al., 19 Feb 2026)
Group level	Reward vector $G$8 defined over the whole sampled group	"Group-Aware Reinforcement Learning for Output Diversity in LLMs" (Anschel et al., 16 Nov 2025)
Two-phase sequence	Separate $G$9 and $a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$0 with separate group-relative advantages	"F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking" (Surana et al., 13 May 2026)

At the token and sequence levels, dynamic entropy weighting provides a canonical example. GTPO assigns each token in successful sequences a shaped reward

$a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$1

while GRPO-S uses a sequence-level entropy score

$a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$2

The stated motivation is that token-level uncertainty often peaks at critical decision points in a reasoning chain, so high-entropy tokens in correct answers deserve a larger share of the reward (Tan et al., 6 Aug 2025).

A different token-sensitive mechanism appears in the other GTPO paper, which focuses on conflict tokens and entropy collapse. There, tokens appearing in the same position across completions with opposite rewards are masked so that negative updates are skipped and positive ones amplified, through

$a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$3

The final objective also includes an entropy-based completion filter and a trajectory-level entropy penalty $a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$4, replacing KL-based stabilization with direct entropy control (Simoni et al., 5 Aug 2025).

Prefix-level reward decomposition is developed in WS-GRPO. A Bradley–Terry preference model $a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$5 is trained from outcome-only correctness labels, and then converted into prefix-level signals by comparing consecutive prefixes: $a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$6 which are mixed with the sparse final reward as

$a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$7

This construction turns a single terminal reward into correctness-aware continue/stop guidance over partial trajectories (Mundada et al., 19 Feb 2026).

Group-level reward decomposition appears in GAPO, where the reward for rollout $a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$8 can depend on the entire sampled group $a_1,\dots,a_G\sim\pi_\theta(\cdot\mid s)$9. For a known valid set $r(s,a_j)$0, the frequency-aware reward is

$r(s,a_j)$1

This allows the optimizer to target diversity and uniform coverage, objectives that cannot be expressed by independent per-rollout rewards alone (Anschel et al., 16 Nov 2025).

Phase-specific decomposition is central to F-GRPO. The policy is factorized as

$r(s,a_j)$2

with separate rewards for coverage and ranking quality,

$r(s,a_j)$3

and separate group-relative advantages

$r(s,a_j)$4

The total loss is then $r(s,a_j)$5, avoiding what the paper calls cross-phase gradient contamination (Surana et al., 13 May 2026).

A related two-layer decomposition appears in MGRPO, where standard GRPO is followed by a second GRPO pass trained to identify and correct errors in the initial response. The second layer retains only successful corrections or confirmations, thereby supplying implicit process-level supervision without an explicit dense reward model (Ding et al., 5 Jun 2025). This suggests that “aspect” can refer not only to reward semantics but also to reward placement in a staged reasoning-and-revision process.

4. Preference, uncertainty, causal interaction, and distillation as auxiliary reward aspects

Another major branch of the literature expands the reward signal through auxiliary signals that are not simple semantic labels.

A direct bridge to preference optimization is established by the observation that GRPO can be reframed as contrastive learning. One formulation shows that both GRPO and DPO have gradients of contrastive form, and that GRPO with group size $r(s,a_j)$6 is effectively the same form as DPO up to a choice of weight. The resulting 2-GRPO objective is

$r(s,a_j)$7

The same analysis states that the equations extend verbatim to any scalar reward $r(s,a_j)$8 or learned preference model $r(s,a_j)$9, and explicitly notes that pairwise or groupwise human preference data can be used to define positives and negatives (Wu et al., 1 Oct 2025).

Uncertainty-aware reward modulation is developed in SEED-GRPO, which uses semantic entropy over clusters of semantically equivalent outputs as a prompt-level uncertainty signal. After defining $$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$0 over meaning clusters, the GRPO advantage is rescaled as

$$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$1

Low semantic entropy implies semantically consistent rollouts and therefore larger updates; high semantic entropy implies conflicting meanings and therefore smaller updates. The paper explicitly frames this as dynamic adjustment of policy update magnitudes based on question uncertainty (Chen et al., 18 May 2025).

AMIR-GRPO introduces an implicit DPO-style contrastive regularizer mined directly from within-group reward rankings. For reward-differentiated pairs

$$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$2

it defines

$$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$3

and jointly optimizes

$$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$4

The stated effect is to amplify suppression of low-reward trajectories, attenuate response-level length bias, and transform each rollout group into a denser set of supervision constraints (Yari et al., 7 Jan 2026).

GCPO incorporates semantic interaction through a structural causal model. It argues that candidate responses are not independent once conditioned on a final integrated output, and uses a causally informed weight

$$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$5

together with a second KL penalty to a causally projected reference distribution $$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$6. The full objective therefore contains the usual group-relative term, a standard KL to $$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$7, and a second KL to $$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$8 (Gu et al., 7 Aug 2025). This suggests that multi-aspect reward optimization can be extended to semantic complementarity and contradiction among responses, rather than only to properties of individual responses.

CoDistill-GRPO introduces yet another auxiliary aspect: on-policy knowledge distillation. The small model’s effective reward is

$$\mu_s \;=\;\tfrac1G\sum_{j=1}^G r(s,a_j),\qquad \sigma_s \;=\;\sqrt{\tfrac1G\sum_{j=1}^G\bigl(r(s,a_j)-\mu_s\bigr)^2},\qquad A(s,a_j)\;=\;\frac{r(s,a_j)-\mu_s}{\sigma_s}.$$9

The paper proves that the expected gradient decomposes into a direction toward higher task reward and a direction toward the teacher distribution, thereby turning reward optimization into joint task-and-distillation optimization (Kwon et al., 9 May 2026).

5. Empirical regimes and application domains

In safe and aligned language generation, the explicit four-aspect reward model produced consistent gains across model scales. The reported validation-set average scores were $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$0 for the $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$1B model, $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$2 for the $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$3B model, and $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$4 for the $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$5B model. Safety rose by up to $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$6 for $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$7B, and human annotators preferred the GRPO-fine-tuned outputs $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$8 of the time for the $$\nabla_\theta J(\theta)\;=\;\mathbb{E}_{s\sim\mathcal{D}}\Biggl[\sum_{j=1}^G\nabla_\theta\log\pi_\theta(a_j\mid s)\;A(s,a_j)\Biggr].$$9B model on $$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$0 prompts (Li et al., 26 Mar 2025). These results are the most direct empirical demonstration that explicit multi-label scalarization can improve several alignment metrics simultaneously.

In reasoning, the most visible empirical pattern is the trade-off between correctness and rollout efficiency. WS-GRPO reports that rollout length shrank by $$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$1 in steps and tokens, while accuracy dropped only modestly by $$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$2 points and was often within $$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$3 points of GRPO baselines; ablations further state that weakly supervised prefix rewards with $$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$4 drive the bulk of the gains, while raw length penalties alone underperform (Mundada et al., 19 Feb 2026). This is a clear instance in which the “aspects” being jointly optimized are final correctness and efficient stopping behavior.

For output diversity, GAPO provides the strongest evidence that group-aware reward design can optimize coverage without discarding core task utility. On close-set uniformity tasks, baseline models had $$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$5, whereas GAPO achieved $$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$6. On open-set sampling, Unique$$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$7 increased from $$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$8 to $$\mathbb E \bigl\|\widehat{g}_{\rm GRPO}(x;\theta)-g(x;\theta)\bigr\|^2 =\frac{\mathrm{trace}\bigl[\Sigma_{\rm oracle}(x;\theta)\bigr]}{G} +O\!\bigl(G^{-2}\bigr),$$9 for Qwen2.5 32B. On creative writing, average embedding distance increased from $G^* =\sqrt{\frac{c_3}{c_1}}.$0 to $G^* =\sqrt{\frac{c_3}{c_1}}.$1, and $G^* =\sqrt{\frac{c_3}{c_1}}.$2 increased from $G^* =\sqrt{\frac{c_3}{c_1}}.$3 to $G^* =\sqrt{\frac{c_3}{c_1}}.$4 (Anschel et al., 16 Nov 2025). Here the jointly optimized aspects are validity and diversity, with the reward defined over the output group rather than over isolated sequences.

In unified candidate generation and ranking, F-GRPO reports consistent top-ranked improvements over GRPO and decoupled baselines. On LastFM with Qwen3-4B, Recall@5 improved from $G^* =\sqrt{\frac{c_3}{c_1}}.$5 to $G^* =\sqrt{\frac{c_3}{c_1}}.$6; on MovieLens with Qwen3-4B, Recall@3 improved from $G^* =\sqrt{\frac{c_3}{c_1}}.$7 to $G^* =\sqrt{\frac{c_3}{c_1}}.$8; on MuSiQue with Qwen3-4B, Recall@3 improved from $G^* =\sqrt{\frac{c_3}{c_1}}.$9 to $r_\phi(s,a)\in[0,1]^K$0 (Surana et al., 13 May 2026). These results show that separate group-relative advantages can stabilize an intrinsically multi-objective task in which subset coverage and ranking order are distinct utilities.

In ASR, GRPO with rule-based sequence rewards produced up to $r_\phi(s,a)\in[0,1]^K$1 relative improvement in word error rate, along with reduced hallucinations, increased robustness on out-of-domain datasets, and effective domain adaptation (Shivakumar et al., 2 Sep 2025). Although this work uses WER-like rewards rather than explicit multi-label alignment dimensions, it shows that multi-aspect reward optimization extends naturally to multimodal sequence generation whenever several error components—substitutions, deletions, insertions, hallucinations, and domain robustness—are operationally coupled.

6. Misconceptions, limitations, and open problems

A recurring misconception is that multi-aspect reward optimization necessarily requires large groups, complicated critics, or separate reward models for every sub-signal. Several papers argue otherwise. One result reframes GRPO as contrastive learning and shows that 2-GRPO achieves performance on par with 16-GRPO while using only $r_\phi(s,a)\in[0,1]^K$2 of the rollouts and reducing training time by over $r_\phi(s,a)\in[0,1]^K$3 (Wu et al., 1 Oct 2025). Another derives a universal scaling law for optimal group size, indicating that the choice of $r_\phi(s,a)\in[0,1]^K$4 is governed by prompt variance and residual U-statistic variance rather than by a generic requirement for very large groups (Zhou et al., 1 Mar 2026). This suggests that richer reward composition does not automatically entail larger sampling budgets.

At the same time, explicit scalarization has evident limitations. The multi-label GRPO paper states that uneven weighting or a single-scalar reward model leads to overemphasis on safety at the expense of meaningfulness (Li et al., 26 Mar 2025). That result does not eliminate scalarization bias; it shows only that equal weights worked better than the tested alternatives in that setting. A plausible implication is that weight selection remains an unresolved identification problem whenever aspects are in tension.

Reward shaping at finer granularity introduces its own failure modes. The 2-GRPO analysis notes that zero-advantage cases are discarded when both rollouts succeed or both fail, that discrete $r_\phi(s,a)\in[0,1]^K$5 advantages may limit fine-grained credit assignment, and that reward noise can break group normalization (Wu et al., 1 Oct 2025). The trajectory-based GTPO paper states that its conflict mask focuses only on outer contiguous spans and that filtering high-entropy completions discards information (Simoni et al., 5 Aug 2025). SEED-GRPO explicitly lists dependence on $r_\phi(s,a)\in[0,1]^K$6, $r_\phi(s,a)\in[0,1]^K$7, $r_\phi(s,a)\in[0,1]^K$8, and the clustering method as limitations (Chen et al., 18 May 2025). WS-GRPO identifies the upfront cost of training the preference model and the risk of mis-ranking out-of-distribution patterns (Mundada et al., 19 Feb 2026).

Group-aware and diversity-oriented formulations face additional structural constraints. GAPO’s frequency-aware reward assumes the complete valid set $r_\phi(s,a)\in[0,1]^K$9, which the paper identifies as unrealistic for many open-ended tasks, and extending the method to unknown mode sets requires approximate clustering or online density estimation (Anschel et al., 16 Nov 2025). F-GRPO, while effective in two-phase list-to-rank settings, still depends on RL fine-tuning with $K=4$0 rollouts and a fixed two-phase factorization, leaving more complex multi-phase pipelines open (Surana et al., 13 May 2026).

Finally, the optimization substrate itself can become unstable when the policy class changes. In diffusion LLMs, estimated importance ratios are noisy, standard GRPO clipping can be anomalously bypassed by model-agnostic estimation noise, and fixed group-size normalization amplifies gradient-magnitude fluctuations. StableDRL addresses this with unconditional clipping and self-normalization, indicating that GRPO-style reward optimization is not architecture-agnostic in its raw form (Zhong et al., 6 Mar 2026). This suggests that future multi-aspect reward optimization will likely require joint design of reward composition, granularity, and optimizer mechanics rather than treating them as separable choices.

Taken together, the literature presents multi-aspect reward optimization not as a single algorithm but as a design space. Its main axes are explicit multi-label scalarization, reward placement across structural granularity, auxiliary preference or uncertainty signals, and group-aware or phase-aware objectives. The common trajectory is toward denser supervision without abandoning the computational advantages of group-relative, critic-free optimization.