MAPO: Mixed Advantage Policy Optimization (2509.18849v1)

Abstract: Recent advances in reinforcement learning for foundation models, such as Group Relative Policy Optimization (GRPO), have significantly improved the performance of foundation models on reasoning tasks. Notably, the advantage function serves as a central mechanism in GRPO for ranking the trajectory importance. However, existing explorations encounter both advantage reversion and advantage mirror problems, which hinder the reasonable advantage allocation across different query samples. In this work, we propose an easy but effective GRPO strategy, Mixed Advantage Policy Optimization (MAPO). We reveal that the trajectory appears with different certainty and propose the advantage percent deviation for samples with high-certainty trajectories. Furthermore, we dynamically reweight the advantage function for samples with varying trajectory certainty, thereby adaptively configuring the advantage function to account for sample-specific characteristics. Comparison with related state-of-the-art methods, along with ablation studies on different advantage variants, validates the effectiveness of our approach.

• The paper introduces a mixture of deviation-based and mean-based advantage estimators to address advantage reversion and mirror issues.
• It leverages trajectory certainty through a dynamic reweighting mechanism to deliver stable and accurate policy optimization.
• Empirical evaluations show significant performance gains and robust training stability across mathematical and emotional reasoning benchmarks.

Mixed Advantage Policy Optimization: A Principled Approach to Trajectory Certainty in GRPO

Introduction

The paper "MAPO: Mixed Advantage Policy Optimization" (2509.18849) addresses fundamental limitations in the advantage function design for Group Relative Policy Optimization (GRPO), a @@@@1@@@@ (RL) paradigm widely used for post-training foundation models (FMs) in reasoning tasks. The authors identify two critical issues—advantage reversion and advantage mirror—arising from the uniform application of normalized advantage functions across samples with varying trajectory certainty. MAPO introduces a principled mixture of advantage formulations, leveraging trajectory certainty to dynamically interpolate between deviation-based and mean-based advantage estimators. This approach yields more stable and accurate optimization, as demonstrated empirically on both mathematical and emotional reasoning benchmarks.

Figure 1: Diverse successful trajectory counts $N$ across samples during reinforcement, illustrating the heterogeneity in trajectory certainty.

Motivation and Problem Analysis

GRPO eliminates the need for a learned reward critic by relying on group-based sampling and rule-based reward functions. The standard advantage function, $\hat{A}_i = \frac{r_i - \mu}{\sigma}$, is applied uniformly to all samples, regardless of their trajectory certainty. The authors formalize trajectory certainty as the empirical success ratio $p = N/G$, where $N$ is the number of successful trajectories in a group of size $G$. High-certainty samples (either very easy or very hard) exhibit low prediction variance, while low-certainty samples display high variance.

Two key problems are identified:

• Advantage Reversion: High-certainty samples receive disproportionately large or small advantage allocations due to small $\sigma$, leading to numerical instability and misaligned optimization.
• Advantage Mirror: Symmetric reward distributions (e.g., all successes vs. all failures) yield mirrored advantage scores, failing to distinguish semantically distinct cases.

These phenomena are visualized and discussed in the context of GRPO's trajectory sampling behavior.

Figure 2: MAPO architecture, showing dynamic reweighting of advantage functions based on trajectory certainty.

Methodology

Advantage Percent Deviation (APD)

For high-certainty samples, the standard deviation-based normalization is unstable. MAPO introduces the Advantage Percent Deviation (APD):

$\hat{A}_i^{APD} = \frac{r_i - \mu}{\mu}$

This formulation emphasizes the proportional deviation from the mean, mitigating instability when $\sigma$ is small and preventing mirrored allocations for extreme cases.

Trajectory Certainty Reweight (TCR)

MAPO dynamically interpolates between the deviation-based and mean-based advantage functions using a certainty-aware weighting scheme:

$\lambda(p) = 1 - 4p(1-p)$

$$\hat{A}_i^* = (1-\lambda(p)) \frac{r_i-\mu}{\sigma} + \lambda(p) \frac{r_i-\mu}{\mu}$$

When $p$ is near $0.5$ (high uncertainty), the deviation-based advantage dominates; when $p$ approaches $0$ or $1$ (high certainty), the mean-based advantage is emphasized. This mixture ensures robust and context-sensitive advantage estimation.

Figure 3: Comparative analysis of existing advantage functions and their failure modes in GRPO.

Theoretical Analysis

The authors provide a gradient-level analysis, showing that MAPO amplifies gradients for harder samples ($p < 0.5$) and attenuates them for easier samples ($p > 0.5$):

$$\varrho(p) = (1-\lambda(p)) + \lambda(p) \sqrt{\frac{1-p}{p}}$$

This property aligns with prior findings that emphasizing difficult samples improves reasoning performance, while avoiding entropy collapse associated with overemphasizing easy samples.

Empirical Evaluation

MAPO is evaluated on Qwen2.5-VL-7B-Instruct across mathematics (Geo3K, MathVision, MathVista, MathVerse) and emotion (EmoSet, WEBEmo, Emotion6) datasets. Key findings include:

• Consistent Performance Gains: MAPO outperforms Vanilla, GRPO, and DAPO baselines in both in-domain and out-of-domain settings, with average accuracies of $51.26$ (math) and $66.77$ (emotion) for $G=12$.
• Ablation Studies: Solely replacing the advantage function with APD yields limited improvement; dynamic reweighting via TCR is essential for robust gains.
• Training Stability: MAPO maintains stable training and testing accuracy, while DAPO fails in scenarios with dynamic sampling failure.

  Figure 4: Performance comparison of advantage formulations on geometry tasks, highlighting MAPO's superior accuracy.

  

Figure 5: Training and testing accuracy curves, demonstrating MAPO's stability and DAPO's failure in EmoSet.

Figure 6: Visualization of in-domain and out-of-domain datasets used in experiments.

Figure 7: In-domain case visualization, illustrating MAPO's robust optimization.

Figure 8: Out-of-domain case visualization, showing MAPO's generalization capability.

Implementation Considerations

MAPO is architecture-agnostic and does not require additional model components or hyperparameter tuning. The method is implemented within the EasyR1 RL framework, with batch sizes and rollout parameters consistent with standard GRPO setups. The approach is compatible with both unimodal and multimodal FMs, and can be scaled to larger models and datasets, subject to computational resources.

Limitations and Future Directions

MAPO's reliance on trajectory certainty may reduce to a single advantage function in extreme scenarios where successful trajectories are rare. Further refinement of reward allocation mechanisms and extension to larger-scale models are promising avenues for future research. The method's principled mixture of advantage functions offers a template for adaptive RL strategies in other domains.

Conclusion

MAPO provides a theoretically grounded and empirically validated solution to the advantage reversion and mirror problems in GRPO. By leveraging trajectory certainty to dynamically interpolate between advantage estimators, MAPO achieves stable, accurate, and generalizable optimization for foundation model reasoning tasks. Its architecture independence and lack of hyperparameter requirements make it a practical choice for RL-based post-training of FMs. The approach sets a new standard for advantage function design, with implications for broader RL applications in AI.