Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization

Abstract: Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training LLM reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. As a result, training continues to allocate gradient budget to already-solved objectives instead of focusing on those with greater remaining headroom. We introduce \textbf{Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization} (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This dynamically reallocates optimization effort toward under-optimized objectives while empirically maintaining performance on those that are already well satisfied. We further show that saturation-aware reweighting can reverse the sign of an update, rather than merely rescale its magnitude. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to $5\%$ on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by $3.8\%$ on average and up to $9.2 \%$ on AMC23, and on coding benchmarks it improves pass rate by up to $2.3\%$, while in all settings maintaining the easier objectives near their already satisfied levels.

• The paper introduces SA-MRPO, which combines group-relative reward advantages with saturation-based reweighting to reduce updates for objectives near their reward ceilings.
• The method outperforms GDPO in 12 of 15 mathematical benchmark comparisons, including a 5.0-point AIME24 gain and a 9.2-point AMC23 gain in adaptive reasoning.
• The results show practical benefits for reasoning and code generation, but performance depends on reward bounds and tuning the saturation exponent because objective conflicts can still cause regressions.

Problem formulation and motivation

The paper studies multi-reward policy optimization for language-model reasoning under group-relative training schemes such as GRPO. The central claim is that the conventional treatment of multiple rewards is inadequate for two distinct reasons. First, scalarizing reward dimensions before group normalization destroys reward-profile information: different combinations of objective values can yield the same scalar reward and consequently the same rollout advantage. Second, even reward-decoupled methods may continue assigning comparable optimization pressure to objectives that are already close to their attainable maxima. The resulting gradient allocation is insensitive to objective saturation.

This distinction is important in RLVR settings, where correctness is often optimized jointly with auxiliary constraints such as length, output format, safety, executability, or process validity. Auxiliary objectives can become nearly solved early in training, whereas correctness remains substantially under-optimized. If their weights remain fixed, the policy may spend gradient budget improving an objective for which little useful headroom remains.

The paper introduces Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO), which modifies only the construction of the rollout advantage while retaining the clipped GRPO surrogate objective. Its design combines two signals: within-group relative performance for each reward dimension and the batch-level fraction of that objective’s attainable reward range that has already been achieved. The method is positioned as a direct extension of Group reward-Decoupled Policy Optimization (GDPO) (Liu et al., 8 Jan 2026), while reducing to GRPO (Shao et al., 2024) in the single-objective case.

Failure of scalarized advantages

Let $r_k^{(i,j)}$ be the reward for objective $k$ assigned to rollout $j$ for query $i$. Standard multi-objective GRPO first constructs a scalar reward,

$r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$

and then standardizes this scalar across the $G$ rollouts associated with a query. This operation is not injective with respect to the reward vector. Under equal weights, for example, reward profiles $(1,0)$ and $(0,1)$ both produce the same scalar value. They are therefore indistinguishable to the policy update despite representing qualitatively different outcomes.

GDPO addresses this resolution loss by standardizing each reward dimension independently before aggregation. SA-MRPO retains this decoupling but adds a second allocation mechanism based on saturation. The paper’s illustrative four-rollout construction makes the difference operationally explicit.

Figure 1: GRPO collapses distinct reward profiles, GDPO preserves them but can still overemphasize a saturated objective, and SA-MRPO reallocates advantage toward correctness with greater remaining headroom.

In the example, format is already saturated while correctness is not. Two rollouts have identical scalar rewards under GRPO and consequently receive zero advantage, even though their reward decompositions differ. GDPO resolves this ambiguity but can assign a larger positive advantage to a rollout whose correctness reward is zero because its format score dominates after independent normalization. SA-MRPO discounts the saturated format contribution and can assign that rollout a more negative aggregate advantage than another rollout with better correctness. This demonstrates the paper’s stronger claim that saturation-aware weighting can change the sign of an update, not merely rescale it.

SA-MRPO construction

For each objective $k$, SA-MRPO computes a group-relative advantage,

$$A_k^{(i,j)} = \frac{r_k^{(i,j)}-\mu_k^{(i)}}{\sigma_k^{(i)}},$$

where $k$0 and $k$1 are the mean and standard deviation of objective $k$2 over the rollouts for query $k$3. This preserves objective-specific rank information and avoids mixing heterogeneous reward dimensions before normalization.

The method then estimates the batch-level saturation of objective $k$4 using known reward bounds:

$k$5

Here, $k$6 is the mean reward over all rollouts in the current batch. The effective objective weight is

$k$7

where $k$8 controls the strength of saturation-aware reweighting. The unnormalized aggregate advantage is therefore

$k$9

These aggregate advantages are finally normalized over the batch and inserted into the ordinary clipped GRPO objective.

The interpretation is direct. When an objective is far from its upper bound, $j$0 is large and its effective weight remains close to the prescribed value. As the objective approaches saturation, its contribution is attenuated. At $j$1, all saturation factors equal one, and SA-MRPO becomes GDPO. In the single-objective case, the construction reduces further to GRPO.

The exponent also provides a monotonic control over relative allocation. For two objectives $j$2 and $j$3,

$j$4

If objective $j$5 is more saturated than objective $j$6, increasing $j$7 strictly decreases its relative weight. Thus, $j$8 is not merely a generic optimization hyperparameter: it determines how aggressively the update favors nominally under-optimized objectives.

Optimization consequences

The paper emphasizes that SA-MRPO changes the composition of the update before final batch normalization. This matters because final normalization removes global scale but does not remove changes in the relative weighting of objective-specific advantages. Consequently, saturation-aware reweighting can alter both the magnitude and direction of the policy update.

This directional effect is particularly consequential under objective conflict. If $j$9 denotes the policy-gradient contribution of objective $i$0, the aggregate ascent direction is

$i$1

The first-order change in objective $i$2 along this direction contains its self-improvement term and cross-objective alignment terms. If all other gradients have nonnegative inner product with $i$3, the aggregate update cannot decrease objective $i$4 to first order. However, if sufficiently strong negative alignment exists, reducing $i$5 can remove part of the protection provided by its own gradient, allowing another objective to degrade it.

This is an explicit qualification of the method’s objective. SA-MRPO is an adaptive allocation rule, not a constrained Pareto-preservation method. Its intended tradeoff is empirically favorable when saturated-objective degradation is small and under-optimized-objective improvement is substantial, but monotonic retention is not guaranteed.

The paper also distinguishes nominal reward headroom from achievable policy improvement. The saturation statistic measures distance from the prescribed reward bounds, not distance from the best reward realizable by the current model and training distribution. An objective can therefore have low measured saturation while being effectively capacity-limited. The method assumes that the specified reward interval is a useful proxy for optimization potential; the experiments do not establish that this proxy is uniformly reliable.

Mathematical reasoning results

The primary mathematical-reasoning experiments use Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct, with binary correctness, length-compliance, and format rewards. Each prompt produces eight rollouts, and training uses the DeepScaleR-Preview corpus. The comparison is primarily against GDPO, isolating the effect of saturation-aware reweighting from reward decoupling.

Across two- and three-objective configurations, SA-MRPO obtains higher accuracy than GDPO in 12 of 15 benchmark comparisons. For Qwen2.5-7B-Instruct with correctness, length, and format objectives, accuracy rises from $i$6 to $i$7 on AIME24, a gain of $i$8 percentage points, and from $i$9 to $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$0 on MATH500, a gain of $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$1 points. The method also improves Minerva and OlympiadBench, although it falls slightly below GDPO on AMC23.

For Qwen2.5-3B-Instruct, the method improves four of five benchmarks under both two-objective and three-objective training. In the two-objective setting, AIME24 accuracy increases from $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$2 with GDPO to $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$3 with SA-MRPO; in the three-objective setting, it increases from $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$4 to $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$5. On AMC23, the corresponding gains are $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$6 and $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$7 percentage points.

These gains are accompanied by small changes in the fraction of responses exceeding the 4,000-token budget. For example, in the Qwen2.5-7B three-objective setting, AIME24 Exceed changes from $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$8 to $r_{\mathrm{sum}}^{(i,j)}=\sum_k w_k r_k^{(i,j)},$9, while MATH500 changes from $G$0 to $G$1. The results support the narrower claim that correctness can improve without materially abandoning length compliance. They do not show universal improvement on every benchmark: several configurations exhibit small regressions, particularly on OlympiadBench and AMC23.

Adaptive reasoning and coding results

The adaptive-reasoning experiment creates an explicit saturation regime using a graded length reward. Responses at or below 1,024 tokens receive the maximum length reward, while rewards decline linearly until reaching zero at 2,048 tokens. Correctness remains binary and has no analogous early saturation mechanism.

Against GDPO, SA-MRPO improves accuracy on all five mathematical benchmarks. The largest gain is on AMC23, where accuracy increases from $G$2 to $G$3, an improvement of $G$4 percentage points. Average accuracy rises from $G$5 to $G$6, or $G$7 points. The cost is an increase in average response length from 333 to 459 tokens. Because both averages remain below the 1,024-token saturation threshold, the authors interpret this increase as evidence that SA-MRPO relaxes unnecessary pressure for additional shortening once length is already sufficiently optimized. The experiment directly operationalizes the proposed mechanism, although it does not disentangle improved reasoning from other effects of allowing longer generations.

In code generation, Qwen2.5-7B-Instruct is trained with test-case pass rate and executability rewards. Executability is an easier validity criterion, whereas pass rate captures functional correctness. SA-MRPO improves pass rate over GDPO on APPS, CodeContests, and Codeforces by $G$8, $G$9, and $(1,0)$0 percentage points, respectively. On TACO, it is $(1,0)$1 points lower than GDPO. Bug rates remain broadly comparable: SA-MRPO reduces bugs relative to GDPO on CodeContests, matches it on Codeforces, and is slightly worse on APPS and TACO.

The coding results are consistent with the intended allocation pattern: the method generally improves the harder correctness-related reward while preserving the easier executability objective. However, the magnitude of the gains is smaller than in adaptive reasoning, and the TACO regression shows that saturation-based allocation does not eliminate task-specific tradeoffs.

Saturation-strength ablation

The $(1,0)$2 ablation tests whether the control parameter produces the predicted reward tradeoff. As $(1,0)$3 increases, the highly saturated length objective receives less relative weight, while correctness receives more. The training trajectories show increasing correctness reward and declining length reward, particularly at $(1,0)$4 and $(1,0)$5.

Figure 2: Larger saturation exponents shift training reward from the highly saturated length objective toward correctness.

Downstream performance is generally maximized at moderate reweighting. On Qwen2.5-3B-Instruct, every positive $(1,0)$6 value improves average accuracy over $(1,0)$7 across the five mathematical benchmarks. $(1,0)$8 gives the strongest overall balance, including the best AIME24 and AMC23 results among the tested settings. Larger values remain competitive but tend to increase Exceed. On AIME24, accuracy follows a non-monotonic pattern: $(1,0)$9 at $(0,1)$0, $(0,1)$1 at $(0,1)$2, and $(0,1)$3 at $(0,1)$4. This indicates that aggressively suppressing a saturated objective can overcorrect rather than produce continual improvement.

The ablation therefore validates the mechanism while also limiting its interpretation. Saturation strength is controllable, but the optimal value is task- and model-dependent. The best results do not arise from maximizing the suppression of saturated objectives.

Limitations and open questions

The principal assumption is that reward bounds are known and that normalized distance from the upper bound is informative about remaining optimization value. This assumption is reasonable for the paper’s bounded, rule-based rewards, but less straightforward for unbounded, learned, noisy, or distribution-dependent reward functions. Even with valid bounds, nominal headroom may be unattainable because of model capacity, data coverage, or incompatibility between objectives.

The batch-mean saturation estimate also has a relatively coarse temporal and distributional resolution. It treats an objective’s current mean reward as the relevant optimization state, without modeling per-task difficulty, reward variance, subpopulation imbalance, or the rate of recent improvement. Two objectives with the same mean saturation may have very different gradient informativeness or achievable gains.

The theoretical analysis establishes conditions under which objective conflict can cause degradation, but the empirical study does not provide systematic gradient-alignment measurements, Pareto-frontier evaluations, or guarantees on retained-objective performance. Comparisons are also concentrated on GDPO; broader comparisons with variance-adaptive, conflict-aware, or policy-decomposition methods would clarify whether saturation is complementary to those mechanisms. Finally, the experiments use limited model scales, fixed reward definitions, and mostly short training horizons. Whether batch-level saturation remains stable under longer training, changing reward bounds, or highly nonstationary curricula is left open.

Conclusion

SA-MRPO reframes multi-reward policy optimization as an allocation problem governed not only by relative rollout quality but also by remaining reward headroom. Independent objective normalization addresses scalarization-induced information loss, while the factor $(0,1)$5 reduces pressure on objectives that are already near their prescribed maxima. Across mathematical reasoning, adaptive reasoning, and code generation, this produces consistent gains on harder objectives, including a $(0,1)$6-point AIME24 improvement, a $(0,1)$7-point AMC23 improvement, and coding gains up to $(0,1)$8 points over GDPO. The method does not guarantee preservation under conflicting gradients, and its effectiveness depends on the validity of nominal saturation as a proxy for achievable improvement. Within those constraints, the experiments support saturation-aware advantage construction as a practical extension of reward-decoupled group-relative policy optimization.