Think Outside the Policy: In-Context Steered Policy Optimization (2510.26519v1)

Abstract: Existing Reinforcement Learning from Verifiable Rewards (RLVR) methods, such as Group Relative Policy Optimization (GRPO), have achieved remarkable progress in improving the reasoning capabilities of Large Reasoning Models (LRMs). However, they exhibit limited exploration due to reliance on on-policy rollouts where confined to the current policy's distribution, resulting in narrow trajectory diversity. Recent approaches attempt to expand policy coverage by incorporating trajectories generated from stronger expert models, yet this reliance increases computational cost and such advaned models are often inaccessible. To address these issues, we propose In-Context Steered Policy Optimization (ICPO), a unified framework that leverages the inherent in-context learning capability of LRMs to provide expert guidance using existing datasets. ICPO introduces Mixed-Policy GRPO with Implicit Expert Forcing, which expands exploration beyond the current policy distribution without requiring advanced LRM trajectories. To further stabilize optimization, ICPO integrates Expert Region Reject Sampling to filter unreliable off-policy trajectories and Annealed Expert-Bonus Reward Shaping to balance early expert guidance with later autonomous improvement. Results demonstrate that ICPO consistently enhances reinforcement learning performance and training stability on mathematical reasoning benchmarks, revealing a scalable and effective RLVR paradigm for LRMs.

• The paper presents ICPO, a novel framework that integrates in-context learning with mixed-policy GRPO to overcome exploration limitations in RLVR.
• It employs implicit expert forcing, expert region reject sampling, and annealed reward shaping to improve reasoning accuracy and training stability.
• Experimental results on Qwen3 models demonstrate significant performance gains, enhanced trajectory diversity, and robust out-of-distribution generalization.

In-Context Steered Policy Optimization for Large Reasoning Models

Motivation and Limitations of Existing RLVR Approaches

Reinforcement Learning from Verifiable Rewards (RLVR) has become a central paradigm for post-training Large Reasoning Models (LRMs), particularly in mathematical reasoning domains. Group Relative Policy Optimization (GRPO) is a widely adopted RLVR algorithm, but its exploration is fundamentally limited by on-policy sampling, which restricts trajectory diversity and often leads to suboptimal convergence (Figure 1).

Figure 1: GRPO exploration is confined to the current policy’s distribution, limiting trajectory diversity and often leading to suboptimal convergence. ICPO leverages existing datasets for mixed-policy GRPO with implicit expert forcing, eliminating reliance on external expert models.

Recent methods attempt to expand the exploration space by incorporating trajectories from stronger expert LRMs, but this approach is computationally expensive and often infeasible due to limited access to advanced models. Furthermore, indiscriminate use of external trajectories can introduce noise and instability into training.

ICPO: Leveraging In-Context Learning for Expert Guidance

The proposed In-Context Steered Policy Optimization (ICPO) framework addresses these limitations by exploiting the inherent in-context learning (ICL) capability of LRMs. Instead of relying on external expert models, ICPO generates expert-conditioned rollouts using few-shot ICL from existing datasets, thereby expanding exploration beyond the current policy distribution.

Empirical analysis demonstrates that 1-shot ICL consistently improves reasoning accuracy over 0-shot baselines across multiple benchmarks (Figure 2), and enhances trajectory diversity more effectively than temperature-based sampling (Figure 3).

Figure 2: Comparison between 0-shot and 1-shot ICL on reasoning accuracy across benchmark datasets.

Figure 3: 1-shot ICL produces trajectories with larger semantic distribution distances and a higher ratio of flipped-correct generations compared to temperature-based sampling.

The theoretical underpinning is that ICL induces a task vector $\vartheta$ in the transformer’s latent space, steering the rollout distribution toward expert-like regions without explicit parameter updates. ICPO incorporates these expert-conditioned trajectories into RL training via Implicit Expert Forcing (IEF), forming a mixed-policy GRPO objective.

ICPO Framework and Algorithmic Components

The ICPO framework consists of three key components:

1. Mixed-Policy GRPO with IEF: On-policy and ICL-conditioned off-policy rollouts are combined, with group-normalized advantages computed over the mixed set. The off-policy branch uses importance weighting to correct for the distributional shift induced by ICL.
2. Expert Region Reject Sampling (ERRS): Only expert-conditioned trajectories with rewards exceeding a threshold are retained for policy updates, filtering out noisy or low-quality rollouts.
3. Annealed Expert-Bonus Reward Shaping (RS): A step-annealed bonus is added to correct expert-region trajectories, enforcing strong expert guidance early in training and gradually relaxing it to promote autonomous optimization.

The overall ICPO training procedure is illustrated in Figure 4.

Figure 4: ICPO Overall Framework. ICPO performs mixed-policy GRPO using off-policy trajectories generated by the policy model itself via implicit expert forcing.

Experimental Results and Analysis

ICPO was evaluated on Qwen3-1.7B and Qwen3-8B models across multiple mathematical reasoning benchmarks. The main results show that ICPO consistently outperforms vanilla GRPO, with average improvements of up to +4.17 points for Qwen3-1.7B and +2.15 points for Qwen3-8B. The variant ICPO$\dagger$, which incorporates reward shaping, further enhances expert-domain alignment, yielding up to +3.6 point improvements on MATH-500.

ICPO also demonstrates strong generalization on out-of-distribution (OOD) benchmarks, maintaining or improving performance compared to GRPO, especially for larger models. The ablation paper confirms that all components—IEF, ERRS, and RS—are complementary and essential for maximizing ICPO’s effectiveness.

Figure 5: Expert Domain Reward curves of Qwen3-8B over training steps, showing consistent improvements for ICPO variants.

Figure 6: Exploration Dynamics. ICPO drives the policy to deviate further from the reference distribution, increasing entropy and KL divergence, and promoting deeper exploration.

Comparison with LUFFY, which relies on advanced LRM-generated trajectories, shows that ICPO achieves superior performance using only existing datasets, eliminating the need for costly external computation.

Implementation Considerations

• Data Requirements: ICPO requires access to high-quality expert demonstrations for few-shot ICL, but does not depend on external expert models.
• Computational Overhead: The mixed-policy rollout generation and ERRS filtering introduce moderate overhead, but this is offset by the elimination of external model inference.
• Scalability: ICPO is scalable across model sizes and generalizes well to datasets of varying difficulty.
• Deployment: The framework is compatible with standard RLVR pipelines and can be integrated into existing LRM post-training workflows.

Implications and Future Directions

ICPO establishes a scalable and effective RLVR paradigm for LRMs, leveraging in-context learning to provide expert guidance without external dependencies. The framework improves both exploration and training stability, and its modular design allows for further extensions, such as adaptive demonstration selection or more sophisticated reward shaping.

Theoretically, ICPO highlights the utility of latent task vectors induced by ICL for policy steering, suggesting new directions for controllable and interpretable RL in LLMs. Practically, ICPO enables efficient post-training of LRMs in domains where external expert models are unavailable or prohibitively expensive.

Conclusion

ICPO introduces a unified RLVR framework that leverages the in-context learning capability of LRMs to provide expert guidance from existing datasets. By integrating mixed-policy GRPO with implicit expert forcing, expert region reject sampling, and annealed reward shaping, ICPO consistently enhances reasoning performance and training stability. The approach is scalable, efficient, and broadly applicable to post-training large reasoning models, representing a significant advance in RL-based optimization for LLMs.