BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning

Abstract: Reinforcement learning with verifiable rewards has become a standard recipe for improving the reasoning abilities of LLMs. Existing algorithms face a tradeoff between computational efficiency and sample efficiency in value estimation and policy learning. We introduce BASIS, a critic-free post-training algorithm designed to address this tradeoff. At each online training step, BASIS samples only one rollout per prompt, but leverages rich information across prompts in the entire batch to improve value function estimation. Our experiments demonstrate that BASIS reduces MSE in value function estimation by 69% compared to REINFORCE++, a representative single-rollout baseline, and achieves lower MSE with one rollout than group mean estimators with 8 rollouts. This improvement in value estimation translates to better policy optimization: using substantially less training time, BASIS achieves performance close to multi-rollout GRPO-type baselines and often outperforms single-rollout REINFORCE-type baselines.

• The paper introduces BASIS, a critic-free, single-rollout method that efficiently estimates advantages using batch information.
• It combines offline value estimation, batchwise refinement, and online calibration to significantly reduce MSE compared to multi-rollout baselines.
• Empirical results show improved training stability, computational efficiency, and superior policy learning performance on multiple reasoning benchmarks.

BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning

Motivation and Problem Setting

Reinforcement learning with verifiable rewards (RLVR) has become a mainstay for improving reasoning in LLMs, leveraging automated verifiers to assign objective rewards for generated outputs. The standard RLVR pipeline inherits a fundamental tradeoff between computational and sample efficiency in value estimation. Critic-based methods such as PPO introduce heavy compute overhead due to networks trained for value estimation, while multi-rollout schemes like GRPO obtain sample efficiency at the cost of repeated rollouts per prompt. Single-rollout critic-free baselines—though computationally economical—exhibit suboptimal value estimation, resulting in high-variance and unstable policy updates.

BASIS ("Batchwise Advantage Estimation from Single-Rollout Information Sharing") directly addresses this bottleneck, introducing a critic-free algorithm that, at each training iteration, only requires one rollout per prompt yet exploits full-batch information for variance-reduced value and advantage estimates.

Methodology

BASIS operates in three core stages: offline value estimation, batchwise refinement, and online calibration.

1. Offline Value Estimation:

Using preference-based objectives regularized by a reference policy through a KL term, BASIS leverages a closed-form solution to estimate the value of an implicitly optimal policy without sampling or training that policy. Pre-computed reward statistics from reference policy rollouts enable efficient, value estimates at training time.

2. Batchwise Refinement:

For a batch of $B$ prompts, the value for each prompt is refined as a weighted sum of observed rewards from other batch members. The weighting scheme is derived from the best linear unbiased estimator (BLUE) under leave-one-out (LOO) constraints, ensuring the target prompt's reward is never used to estimate its own baseline. The weights are theoretically characterized to minimize MSE, considering prompt-level values and reward variances.

Figure 1: Overview of BASIS advantage estimation; a single completion per prompt leverages all batch rewards, with the weight matrix $W$ minimizing MSE and excluding diagonal (self) entries.

3. Online Calibration:

A hyperparameter $\beta$, controlling the strength of the KL regularization, is adaptively selected at each training step to match the evolving policy relative to the reference. $\beta$ is chosen to minimize the empirical squared error (across the active batch set) between rewards and refined value estimates.

This combination yields an adaptive process where only one rollout per prompt is necessary, but advantage estimation for policy gradient computation is greatly improved, reducing the variance present in naïve single-rollout methods.

Empirical Evaluation

Value Estimation Efficiency and Robustness

Empirical results demonstrate BASIS achieves substantial improvements in estimator quality.

• MSE Reduction: BASIS reduces the MSE of value estimation by 69% compared to REINFORCE++ and, with a single rollout per prompt, surpasses GRPO and RLOO operating on 8 rollouts per prompt.

Figure 2: MSEs of initial and refined estimators at three training checkpoints; batchwise refinement yields consistently lower, $\beta$-robust MSE.

• Batch Heterogeneity and Prompt Difficulty: BASIS maintains uniform estimator quality across both batch reward heterogeneity and prompt difficulty, whereas REINFORCE++'s global baseline degrades with batch heterogeneity and GRPO/RLOO degrade on medium-difficulty (maximum-variance) prompts.

  Figure 3: MSE of various value estimators—aggregated, binned by within-batch value variance, and by prompt difficulty.

• Advantage Informativeness: Unlike GRPO, which frequently outputs degenerate (0 or 1) baselines for binary rewards, leading to zero-advantage updates, BASIS consistently produces informative non-extremal advantage signals.

  Figure 4: MSE and extremal frequency for RLOO value estimators compared with BASIS and REINFORCE++.

Policy Optimization Outcomes

The sample-efficient and robust advantage estimation of BASIS translates to downstream policy learning performance.

• Computational Efficiency: BASIS, coupled with GRPO-type objectives, achieves accuracy competitive with or exceeding multi-rollout baselines, using only half the training budget and wall-clock time. Across seven mathematical reasoning benchmarks, BASIS matches or modestly outperforms GRPO using 8 rollouts per prompt.
• Superiority over Single-Rollout Baselines: BASIS outperforms both single-rollout vanilla and REINFORCE++ (global batch baseline) variants by up to 44.8 percentage points (absolute) across several datasets, using only 50% (versus REINFORCE++) to 33% (versus vanilla) of their computation.
• Training Stability: Vanilla single-rollout methods are prone to collapse, severely degrading in performance mid-course. BASIS’s improved value estimation stabilizes policy optimization and eliminates collapse.

  Figure 5: Qwen3-4B fine-tuning results—BASIS with GRPO, GPG, and GSPO vs vanilla (zero baseline) baselines across seven math benchmarks.

  

  Figure 6: Qwen3-4B fine-tuning results—BASIS with GRPO, GPG, GSPO vs REINFORCE++ (global baseline) single-rollout baselines.

Calibration Dynamics

During training, the calibration parameter $\beta_t$ naturally decreases as the fine-tuned policy shifts from the reference to the reward-optimal policy, tracking the tradeoff between KL-regularized and reward-driven learning.

Figure 7: Trajectories of calibrated $\beta_t$ during BASIS training; all variants decrease and stabilize as training progresses.

Theoretical and Practical Implications

Theoretically, BASIS demonstrates that cross-prompt information sharing in batchwise critic-free advantage estimation yields significant MSE improvements. The estimator is determined by a minimax variance principle, connecting the design to established optimality criteria in linear estimation.

Practically, the methodological design aligns well with large-scale LLM post-training settings where repeated rollouts per prompt are costly. BASIS is especially apt for binary reward regimes typical in math and reasoning benchmarks with automated verifiers, but the weighting strategy and calibration dynamic hold potential for generalization.

Moreover, because BASIS only modifies the advantage estimation module, it is compatible as a plug-in with diverse policy objectives and optimization pipelines—robust across models, tasks, and reward distributions.

Future Directions

While demonstrations focus on RLVR with automated verifiers and predominantly binary rewards, extending BASIS to settings involving partial, noisy, or preference-based rewards is a promising direction. Combining batchwise information sharing (BASIS) with temporal or cross-iteration information sharing (e.g., via Kalman filtering or kernelized approaches) could further enhance sample efficiency and estimator robustness [wang2025kalman, gong2026kernelized].

Investigation into batch construction strategies, alternative weighting schemes, and generalization to rich, structured, or continuous reward formulations will be important for broadening applicability.

Conclusion

BASIS enables high-fidelity, sample-efficient policy gradient optimization for LLM reasoning through batchwise advantage estimation leveraging single rollouts. Empirical results confirm strong gains in estimator quality and policy optimization, robust performance across prompt difficulty and reward heterogeneity, and increased computational efficiency over both single- and multi-rollout baselines. BASIS constitutes a principled, practical improvement in the RL post-training toolkit for LLMs.

---

References:

For further reading and foundational results, see the original paper ["BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning" (2605.27293)], and related works on RLVR, policy optimization, and variance reduction in LLM fine-tuning.