RL-PLUS: Hybrid-Policy Reinforcement Learning

• RL-PLUS is a hybrid-policy reinforcement learning framework that addresses capability boundary collapse in LLMs by blending on-policy exploitation with off-policy exploration using Multiple Importance Sampling.
• It incorporates an exploration-based advantage function that rescaled updates for rarely sampled yet correct reasoning paths, enhancing diversity in problem solving.
• Experimental results show RL-PLUS achieves state-of-the-art improvements on math reasoning and out-of-distribution tasks, with gains up to 69% across different LLM architectures.

RL-PLUS is a hybrid-policy reinforcement learning framework developed to address the capability boundary collapse observed in LLMs when optimized with conventional Reinforcement Learning with Verifiable Reward (RLVR) methods. The approach systematically combines internal exploitation (“thinking,” i.e., optimized on-policy exploration of reasoning paths by the LLM) with external data-driven learning. RL-PLUS integrates Multiple Importance Sampling and an Exploration-Based Advantage Function to more effectively leverage external demonstrations while actively encouraging the discovery of high-value, rarely explored solution paths. Extensive experiments demonstrate that RL-PLUS achieves state-of-the-art performance across math reasoning and out-of-distribution benchmarks, with robust improvements across model architectures, effectively overcoming the narrowing of the problem-solving scope endemic to standard RLVR.

1. Motivation and Problem of Capability Boundary Collapse

Conventional RLVR methods for LLM reasoning employ strict on-policy optimization with verifiable rewards (e.g., executing predicted solutions to math or code tasks and rewarding correct outputs). This process improves the model’s immediate answer accuracy (pass@1), but leads to "capability boundary collapse": the reinforcement agent increasingly exploits a limited set of familiar reasoning trajectories, resulting in diminished diversity and stagnation (lower pass@k for k > 1). As a consequence, models optimized with RLVR risk a reduction in broader problem-solving capabilities, struggling to generalize to unseen or low-probability correct reasoning paths.

RL-PLUS is designed to disrupt this collapse. The central insight is that a hybrid approach that unites on-policy exploitation with carefully managed off-policy exploration can not only boost immediate performance but also expand the effective reasoning boundary.

2. Multiple Importance Sampling for Off-policy Correction

A core challenge in integrating external knowledge is the distributional mismatch between the target policy (the current LLM) and the external (unknown) policy under which demonstration data were generated. Standard importance sampling approaches estimate target-policy returns from off-policy data but suffer from high variance or bias when the distributional overlap is poor.

RL-PLUS adopts Multiple Importance Sampling (MIS), where at each step the importance weight is estimated as

$$r_{i,t}^m(\theta) = \frac{2\cdot \pi_{\theta}(e_{i,t}|q, e_{i,<t})}{\pi_{\omega}(e_{i,t}|q, e_{i,<t}) + \pi_{\theta_{old}}(e_{i,t}|q, e_{i,<t})}$$

Here, $\pi_{\omega}$ is the density of the (unknown) demonstration policy, and $\pi_{\theta_{old}}$ is a previous iteration policy (kept close to the current $\pi_{\theta}$). This mixture estimator stabilizes the variance and provides a lower bound for the effective sample weight, improving the robustness of hybrid-policy optimization over purely self-generated or purely external rollouts.

Theoretical results establish that as long as one policy in the mixture is a reasonable approximation of the target, MIS yields bounded-variance, and Bayesian analysis justifies the use of mixtures with a uniform reference policy to minimize expected $L_2$ error on unseen actions.

3. Exploration-Based Advantage Function

A distinctive component of RL-PLUS is the explicit encouragement of exploration along underrepresented reasoning paths. Standard policy gradient approaches weight updates according to return-normalized advantage, which can overlook correct but low-probability trajectories. To address this, RL-PLUS rescales the advantage with an exploration-aware weighting factor:

$$A^c_{i,t} = \left(\frac{R_i - \text{mean}(R_1, ..., R_G)}{\text{std}(R_1, ..., R_G)}\right) \cdot C_{i,t}$$

with

$$C_{i,t} = \left(1 - \text{detach}(\pi_{\theta}(e_{i,t}|q, e_{i,<t}))\right)^\gamma$$

where $\gamma$ is a hyperparameter and detach ensures no gradient flows through the estimated probability. This construction amplifies the gradient signal for rarely sampled (low probability under current policy) but correct tokens, guiding the optimizer to systematically expand the LLM’s reasoning support set.

Gradient analyses in the paper show that the update magnitude for a rare but correct action is strictly increased relative to standard advantage weighting, while over-learned (over-confident) actions are suppressed.

4. Experimental Results Across Reasoning and OOD Benchmarks

RL-PLUS is evaluated on a comprehensive suite of six math reasoning tasks, including GSM8K, MATH500, Minerva Math, OlympiadBench, AIME, and AMC, as well as on out-of-distribution (OOD) scenarios covering code (HumanEval, LeetCode, LiveCodeBench) and QA tasks (ARC-c, GPQA-diamond, MMLU-Pro).

Key results:

• RL-PLUS consistently outperforms state-of-the-art RLVR algorithms such as SimpleRL, OpenReasoner, and PRIME.
• Average relative improvements range from 21.1% to 69.2% across diverse LLM families (Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, LLaMA-3.1-8B, DeepSeek-Math-7B).
• RL-PLUS uniquely sustains superior pass@k performance as k increases, indicating a persistent expansion in problem-solving diversity. In contrast, baseline RLVR approaches often exhibit declining or plateauing pass@k curves, underscoring their inability to overcome capability boundary collapse.

Performance Table (excerpted values based on paper data):

Model	Task	RL-PLUS Rel. Gain	Pass@1	Pass@5	Pass@10
Qwen2.5-Math-1.5B	MATH500	+40%	↑	↑	↑↑
LLaMA-3.1-8B	Minerva	+51%	↑	↑	↑↑
DeepSeek-Math-7B	Olympiad	+22%	↑	↑	↑↑

5. Theoretical Justification

The RL-PLUS hybrid-policy estimator is supported by mathematical analysis:

• Variance robustness of the MIS estimator is formally derived, with proofs showing that including at least one “good” (target-like) policy keeps variance bounded across training iterations.
• The optimal Bayesian estimator for the unknown external policy—by convex-combining the most recent policy and a uniform prior—minimizes epistemic risk when integrating demonstration data.
• Exploration-weighted advantage derivations demonstrate explicit gradient amplification for hard-to-discover, correct reasoning paths, analytically justifying improved exploration-exploitation balance.

6. Robustness, Generalizability, and Model Family Coverage

Unlike approaches that tightly couple optimization to a specific policy architecture or base model, RL-PLUS is implemented generically and evaluated across divergent LLM backbones:

• Consistent reductions in overfitting to narrow solution paths, as evidenced by robust pass@k, across both architecture families and task genres (math, code, QA).
• Demonstrated ability to transfer improvements from math reasoning to unrelated OOD code and science benchmarks.
• The method’s stability is maintained regardless of whether demonstrations or internal rollouts dominate the batch; hybridization is controlled through MIS, ensuring general applicability.

7. Outlook and Future Prospects

The RL-PLUS approach substantiates the benefits of hybrid-policy optimization for structured reasoning in LLMs, directly countering critical loss modes encountered in vanilla RLVR. According to the authors, the next research frontiers include developing finer-grained exploration weighting schedules, automating policy pool updates for MIS, and extending beyond static external data to incorporate dynamically sampled or curriculum-generated demonstrations. The anticipated result is a new class of continuously self-improving LLMs, with the RL-PLUS framework as a basis for scalable, robust, and generalizable reasoning enhancement (Dong et al., 31 Jul 2025).