Soft Adaptive Policy Optimization (2511.20347v1)

Abstract: Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of LLMs, yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO. When a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence, whereas SAPO selectively down-weights only the offending tokens and preserves the learning signal from the near-on-policy ones, improving sample efficiency. Relative to GRPO, SAPO replaces hard token-level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.

• The paper introduces SAPO, a smooth, temperature-controlled trust region method that overcomes the instability of hard clipping in RL fine-tuning for LLMs.
• It employs a token-level soft gating mechanism using a rescaled sigmoid to selectively attenuate gradients, ensuring sequence coherence and robust training.
• Empirical results on Qwen3 models show SAPO achieves higher training rewards and superior Pass@1 accuracy compared to GSPO and GRPO-R2 while maintaining extended stability.

Soft Adaptive Policy Optimization: A Smooth, Token-Adaptive Approach for RL Fine-Tuning of LLMs

Motivation and Background

Recent progress in LLMs has relied heavily on reinforcement learning (RL)-based fine-tuning to enhance reasoning abilities, particularly for complex tasks involving mathematics, code, and multimodal understanding. Group-based policy optimization, a practical RL paradigm for LLMs, has taken center stage in these developments. In this regime, multiple candidate responses per query are sampled, rewards are normalized group-wise, and policy updates utilize token-level importance ratios to weight learning signals. However, such token-level importance ratios are often high-variance—this is especially pronounced in Mixture-of-Experts (MoE) architectures—leading to unstable updates.

To mitigate this, prior approaches such as Group Relative Policy Optimization (GRPO) and Group Sequence Policy Optimization (GSPO) employ hard clipping, capping gradients outside a fixed trust region. Despite limiting extreme off-policy updates, hard clipping presents a brittle trade-off: stringent clipping starves gradient signal, while loose clipping accentuates instability. This dichotomy hinders both the stability and efficiency of RL-based optimization in LLMs.

The SAPO Algorithm

Soft Adaptive Policy Optimization (SAPO) introduces a smooth, temperature-controlled alternative to hard clipping, employing a token-level soft gating mechanism parameterized via a sigmoid-based function. This soft gate modulates updates by smoothly attenuating gradients as importance ratios deviate from unity, thereby implementing a continuous trust region. Critically, SAPO is both sequence-coherent and token-adaptive. In the predominant regime—small steps, low variance among token log-ratios—SAPO recovers a sequence-level form analogous to GSPO, but achieves smooth suppression rather than abrupt truncation. In heterogeneous or outlier-dominated sequences, SAPO adaptively down-weights only the offending tokens, preserving gradient flow from informative, near on-policy samples.

SAPO further employs asymmetric temperature settings for positive and negative advantage tokens. Analysis shows that negative-token updates—corresponding to disadvantages—diffuse gradients across the entire action space, amplifying instability in large-vocabulary scenarios. Assigning a higher temperature to negative tokens accelerates attenuation, enhancing stability while maintaining exploration.

Figure 1: Comparison of policy-update objectives under positive advantage. The left panel shows SAPO's smooth surrogate objective; the right panel illustrates its soft gradient weighting as a function of the policy ratio.

Theoretical Unification and Analysis

SAPO is formalized within a unified objective structure, parameterized via algorithm-specific gating functions. For SAPO, the gating function is a rescaled sigmoid, yielding a gradient modulator that peaks on-policy and decays exponentially with ratio deviation. Theoretically, under common empirical conditions—small update steps and low intra-sequence dispersion—SAPO's average token-wise gate is well-approximated by a sequence-level gate ($\mathrm{sech}^2$ function), ensuring sequence-level coherence while preserving token-level adaptivity for heterogeneous data.

Empirical validation across both MoE and dense models confirms the central assumptions: token importance ratios tightly cluster around one, and variance within sequences is generally low, especially for dense model architectures. Compared to GSPO, SAPO's advantages lie in continuous trust region enforcement and selective attenuation: it avoids the optimization noise and gradient starvation associated with discontinuous hard clipping, and adaptively retains informative gradients even in sequences that would be globally suppressed by GSPO.

Figure 2: Empirical validation on MoE models shows token ratio concentration around 1 and low intra-sequence log-ratio variance, supporting SAPO’s assumptions.

Figure 3: Dense model analysis indicates even tighter ratio concentration and variance, underscoring SAPO’s theoretical reduction to sequence-level gates in practice.

Empirical Results

SAPO's effectiveness is established via controlled experiments and large-scale RL fine-tuning benchmarks. In cold-start training from Qwen3-30B-A3B-Base on mathematical reasoning, SAPO consistently outperforms GSPO and GRPO-R2 (GRPO with routing replay), yielding both higher training rewards and stronger validation Pass@1 accuracy. Notably, GSPO and GRPO-R2 frequently encounter early-stage training divergence, while SAPO maintains extended periods of stability and realizes higher final performance under identical computational budgets.

Figure 4: SAPO demonstrates both improved training reward progression and superior validation performance over GSPO and GRPO-R2, avoiding early-collapse.

Ablations further confirm the necessity of asymmetric temperature control: assigning higher temperature to negative advantages stabilizes learning, while lower or equal temperatures induce instability. These findings empirically corroborate the theoretical analysis regarding the destabilizing effect of negative-token gradients.

Figure 5: Training with higher temperature for negative tokens maximizes SAPO’s stability; reversing the asymmetry results in instability.

Application to the Qwen3-VL series extends SAPO's utility to practical multi-task, multimodal training over diverse model scales and architectures, including both dense and MoE variants. Here, SAPO maintains its performance premiums over baseline algorithms, delivering consistent training reward improvements and superior validation metrics across a suite of reasoning and coding benchmarks.

Figure 6: SAPO attains steady improvements and consistently outperforms GSPO and GRPO-R2 in large-scale Qwen3-VL training.

Implications and Future Directions

SAPO’s introduction of smooth, temperature-controlled trust regions advances the toolkit for RL-based LLM fine-tuning. Practically, SAPO alleviates engineering overhead by obviating the need for special stabilization mechanisms such as routing replay in MoE settings. The soft gating paradigm enables more robust, stable, and sample-efficient RL training, directly improving the Pass@1 performance on challenging benchmarks.

Theoretically, SAPO provides a framework for interpolating between sequence- and token-level update control, enabling finer granularity in gradient optimization while retaining sequence coherence. The adoption of asymmetric temperature settings to mitigate action-space instability highlights further directions for fine-grained control in large-vocabulary environments.

In future work, exploration of adaptive, data-driven temperature scheduling and extension to offline RL or batch-constrained scenarios could further enhance SAPO’s universality. Additionally, evaluating SAPO's performance on non-rewarded settings, e.g., preference modeling and RLHF alignment objectives, may reveal broader utility across LLM deployment modalities.

Conclusion

SAPO provides a robust, smooth, and token-adaptive alternative to hard-clipped policy optimization in RL-based LLM fine-tuning. Empirical and theoretical analyses demonstrate that SAPO delivers improved training stability, mitigates sample inefficiency arising from brittle hard clipping, and consistently achieves higher task rewards and validation performance. By integrating soft trust regions and asymmetric temperature control, SAPO supplies a reliable and generalizable approach for RL optimization in modern LLM architectures, including both dense and Mixture-of-Experts models (2511.20347).