AR3PO: Adaptive Rollout & Response Reuse

• AR3PO is an algorithmic framework that integrates adaptive rollout allocation and response reuse to boost sample efficiency in reinforcement learning with verifiable rewards.
• It adaptively focuses computational effort on challenging prompts by stopping further rollouts once a correct response is achieved, reducing redundant sampling.
• The framework reuses verified responses from a replay buffer to sustain training signals, enhancing stability even under sparse reward conditions.

Adaptive Rollout and Response Reuse Policy Optimization (AR3PO) is an algorithmic framework designed to improve sampling efficiency in reinforcement learning with verifiable rewards (RLVR), particularly in the post-training domain of LLMs. AR3PO integrates two core techniques: adaptive rollout allocation and systematic response reuse, enabling the learning process to concentrate computational effort on challenging prompts while sustaining informative training signals across diverse model states and scales.

1. Algorithmic Foundations and Motivation

AR3PO builds upon RLVR protocols in LLM optimization, specifically addressing the inefficiencies of group relative policy optimization (GRPO). GRPO normalizes rewards within response groups for each prompt but is prone to "vanishing advantage" when all responses in a group receive identical rewards, which stalls gradient-based updates and hampers sample efficiency. AR3PO mitigates this by (i) adaptively allocating model-generated rollouts to prompts based on their verification outcomes and (ii) reusing previously verified correct responses (from a replay buffer) when new rollouts yield no success.

This structure is compatible with both 7B/8B parameter models (Qwen2.5, Llama-3.1) and larger scales (Qwen2.5-32B), and demonstrates empirical superiority over GRPO and competitive parity or better sample efficiency compared to Dynamic Advantage Policy Optimization (DAPO), as shown on mathematical reasoning benchmarks such as Math500, Minerva Math, Olympiad Bench, and AIME 2024 (Zhang et al., 30 Sep 2025).

2. Adaptive Multi-Stage Rollout Mechanism

The adaptive rollout mechanism is an iterative generation procedure divided into $S$ stages. At each stage, a prompt pool $\mathcal{U}$ is maintained. For each prompt $q \in \mathcal{U}$, $k$ responses $\{o_i\}_{i=1}^k$ are sampled from the current policy $\pi_\theta(\cdot | q)$. Each response is passed through a binary verifier, which determines whether it meets the reward criterion ($R_i = 1$ for correct, $0$ for incorrect).

After every stage:

• If at least one response for a prompt is verified as correct, that prompt is removed from $\mathcal{U}$, preventing further rollout allocation.
• Prompts that remain without any correct responses persist in $\mathcal{U}$ for additional sampling in subsequent stages.

This mechanism ensures that "easy" prompts—those for which the policy already produces correct responses—consume minimal rollout resources, while "hard" prompts are adaptively allocated more samples. The process can be formalized as follows:

U = batch_prompts # initial prompt pool for stage in range(1, S+1): for q in U: responses = [sample_response(pi_theta, q) for _ in range(k)] if any(verify(r) for r in responses): U.remove(q)

This is distinct from the uniform sampling of GRPO and the repeated sampling seen in DAPO, providing a direct, data-driven allocation of computational resources. The net effect is a controlled reduction in unnecessary generation, yielding improved sample efficiency—up to $4.2\times$ reduction in rollout cost relative to DAPO on Qwen models.

3. Response Reuse Strategy

In scenarios where adaptive rollout fails to yield any correct response for a given prompt, AR3PO leverages a response reuse module:

• The algorithm maintains a replay buffer $\mathcal{B}$ of previously verified correct responses.
• For each unsuccessful prompt, one correct response $o_c$ is randomly drawn from $\mathcal{B}$ and injected into the group for advantage computation.

This approach ensures that the training signal is not lost due to transient policy deficiencies. The GRPO-style normalized advantage is recalculated:

$$A_i = \frac{R_i - \text{mean}(\{R_i\})}{\text{std}(\{R_i\})}$$

If reused responses originate from an earlier behavior policy $\pi_{\theta_\text{old}}$, the algorithm uses one of two techniques to suppress variance:

• Recomputes token probabilities under the current policy for importance weighting, partially trading variance for bias.
• Stops the gradient for recycled samples, ensuring that only on-policy samples contribute to updates while maintaining reward information for stability.

This dual strategy addresses high variance in importance ratios and maintains learning stability across policy updates.

4. Empirical Evaluation and Comparative Analysis

AR3PO has been systematically benchmarked against GRPO and DAPO across multiple mathematical reasoning tasks and model sizes. Performance and sample efficiency metrics are summarized in the following table (as reported in (Zhang et al., 30 Sep 2025)):

Model	Method	Math500	Minerva	Olympiad	AIME	Avg. Score	Sampled Responses	Speedup vs. DAPO
Qwen2.5-7B	GRPO	77.5	37.4	38.8	15.2	42.2	512 × 8.0	3.0×
Qwen2.5-7B	DAPO	77.2	36.4	41.1	16.9	42.9	1536 × 8.0	1.0×
Qwen2.5-7B	AR3PO	78.8	36.0	39.6	18.0	43.1	512 × 5.7	4.2×

On larger models, such as Qwen2.5-32B, AR3PO sustains similar accuracy curves to DAPO at equivalent training steps, while maintaining the efficiency benefits (average of 5.3 responses per prompt). This indicates the scalability and generality of the adaptive rollout and response reuse protocol.

5. Efficiency, Scalability, and Stability

Two principal sources drive AR3PO's efficiency:

• Adaptive Rollout Savings: Immediate cessation of further sampling once a correct response emerges for a prompt, thus lower aggregate generation.
• Response Reuse Benefit: Ensures that no prompt is discarded without an informative gradient signal, even in cases of temporary policy decline.

By synchronizing computational effort to prompt difficulty and preserving signals from correct outputs, AR3PO reduces overall sampling costs, minimizes redundant computation, and ensures stable learning dynamics across policy iterations and model scales.

6. Algorithmic Summary and Training Procedure

The AR3PO training cycle can be formalized (cf. (Zhang et al., 30 Sep 2025) Algorithm 1) as:

1. Initialize policy $\pi_\theta$, prompt dataset $\mathcal{D}$, replay buffer $\mathcal{B}$.
2. For each training iteration:
   • Sample batch $\mathcal{D}_b \subseteq \mathcal{D}$.
   • Set prompt pool $\mathcal{U} \leftarrow \mathcal{D}_b$.
   • For each stage $s=1,\ldots,S$:
     • For prompts $q \in \mathcal{U}$:
     • Generate $k$ responses $\{o_i\}$ via $\pi_\theta$.
     • Evaluate via binary verifier to update $\mathcal{U}$.
   • For prompts still in $\mathcal{U}$ after all stages:
     • Replace an incorrect response with a buffer-correct response $o_c$ if available.
   • Compute normalized advantage for all responses.
   • For reused responses, either recompute token probabilities or stop gradients.
   • Update policy using collected advantages and add new correct responses to $\mathcal{B}$.

7. Theoretical and Practical Implications

AR3PO's innovations are supported by extensive empirical data showing improved sample efficiency and competitive performance across model scales. Its adaptive allocation and reuse strategy offer a principled approach to overcoming vanishing advantage and computational inefficiency inherent in baseline RLVR methods. In the broader context of policy optimization, AR3PO exemplifies a general trend towards systems that leverage verification feedback and adaptively modulate generation effort in response to ongoing learning signals.

A plausible implication, based on findings in (Zhang et al., 30 Sep 2025) and related work, is that AR3PO-style algorithms may generalize well to other verification-driven RL domains, especially those characterized by sparse reward signals, non-stationarity, or where computational resources are at premium.

References to Related Work

The theoretical underpinnings and sampling efficiency offered by AR3PO are closely related to adaptive rollout and bandit-style sampling in API frameworks (0805.2027), meta-level rollout optimization (Bhatia et al., 2022), context-aware reuse strategies (Li et al., 2018), and multi-agent model-based approaches (Zhang et al., 2021). While the present instantiation is focused on LLM RLVR, future research may draw from these areas to extend AR3PO's scope to transfer learning, policy reuse libraries, and model-based RL.