How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors

Abstract: Reinforcement learning with verifiable rewards (RLVR) recently thrives in LLM reasoning tasks. However, the reward sparsity and the long reasoning horizon make effective exploration challenging. In practice, this challenge manifests as the \emph{entropy collapse} phenomenon, where RLVR improves single-rollout accuracy but fails to expand coverage on successful reasoning trajectories. Passive exploration techniques like entropy regularization tend to dismiss generation quality, resulting in noisy rollouts. In response to this issue, we propose an Information-Maximizing Augmented eXploration (IMAX) framework to train a pool of soft prefixes that reshapes the base model's prior over reasoning trajectories. Rather than relying on RL to incentivize exploration on top of the base model, each prefix acts as a trainable control knob that induces a distinct rollout distribution from the same backbone model. To encourage discovery of diverse and task-relevant reasoning behaviors, we derive an Information Maximization (InfoMax) reward to complement the verifiable rewards for RL training. IMAX is in general algorithm-agnostic and can be seamlessly integrated into existing RLVR pipelines. Experiment results have shown that across three backbone scales, IMAX consistently improves reasoning performance over standard RLVR, with gains up to 11.60\% in Pass@4 and 10.57\% in Avg@4.

• The paper demonstrates that using soft prompt pools can decouple exploration from base priors, significantly enhancing reasoning diversity in RLVR.
• It introduces an InfoMax reward with a variational classifier to maintain semantic diversity and improve metrics like Pass@4 and Avg@4 across various LLM backbones.
• The method achieves consistent performance gains and lower compute costs, offering a modular, efficient approach to augment exploration in reinforcement learning from verifiable rewards.

Information-Maximizing Augmented Exploration in RLVR with Prefix-Tuned Priors

Motivation and Problem Statement

Reinforcement learning from verifiable rewards (RLVR) has established itself as a central methodology for enhancing reasoning capabilities in LLMs, particularly within mathematical, symbolic, and multi-step reasoning domains. Nevertheless, RLVR faces a pronounced exploration challenge: reward signals are sparse, reasoning horizons are extended, and policies exhibit rapid entropy collapse, concentrating probability mass on a narrow set of successful trajectories. Empirical analyses indicate that, while RLVR increases single-rollout (Pass@1) correctness, it often narrows exploration and can even degrade performance at higher sample counts (Pass@K) due to collapsed diversity. This collapsed exploration impedes both coverage and the discovery of novel high-quality reasoning strategies.

Current approaches largely focus on modifying RL objectives, reward shaping, or stochastic rollout procedures, but mounting evidence suggests RLVR remains strongly tethered to the backbone model's initial distribution. The inability to decouple exploration from base priors has motivated the consideration of prompt-based conditioning—specifically, leveraging soft prompt (prefix) tuning as a means of exerting lightweight, fine-grained control over the model’s generative trajectory space.

Methodological Contributions

The paper introduces the Information-Maximizing Augmented Exploration (IMAX) framework, targeting the decoupling and expansion of exploration in RLVR through the use of a learnable prefix pool. Each soft prefix comprises a set of continuous virtual tokens prepended to the input that, when combined with a frozen base model, induce distinct conditional distributions over generated reasoning trajectories.

Prefix Pool Conditioning

IMAX initializes a pool of $C$ soft prompts, each mapped from a set of meaningful, strategy-oriented seed text prompts (e.g., explicit variable introduction, case analysis, transformation to simpler form), ensuring maximally diverse priors ab initio. Prefix tuning is executed solely on these soft prompt parameters, maintaining a frozen LLM backbone to constrain model updates and computational cost.

InfoMax Reward and Objective

To explicitly prevent mode collapse in the prefix pool and institutionalize structured diversity, IMAX employs a mutual information maximization objective $I(Z; Y \mid X)$ between the soft prefix identity $Z$ and the model's generated trajectory $Y$, conditioned on the original prompt $X$. This InfoMax reward is implemented using a variational lower bound: a lightweight classifier $q_\phi(z \mid x, y)$ predicts the prefix identity from the generated response, and its log-likelihood is added as an intrinsic reward to the RL objective.

The final training objective is:

$$\max \mathbb{E}_{x, z, y} \left[ R(x, y) + \beta \log q_\phi(z \mid x, y) \right]$$

where $R(x, y)$ is the verifiable correctness reward and $\beta$ weights the InfoMax term.

Alternating Optimization

IMAX employs a coordinate ascent-style optimization, alternating between updating the variational classifier $q_\phi$ and the prefix pool. Group-relative policy optimization (GRPO) or other sequence-level RLVR algorithms are used for credit assignment within each prefix group. This scheme generalizes across RLVR algorithms without modifying their core policy gradients, maintaining algorithmic agnosticism.

Empirical Evaluation

Comprehensive experiments are conducted on mathematical reasoning datasets, with backbones Qwen2.5-1.5B, Qwen3-4B, and Qwen3-8B. The IMAX framework is benchmarked against:

• Standard RLVR (GRPO, DAPO) under both parameter- and prefix-tuning regimes
• Diversity-oriented baselines (MERCI, DIVER)
• Supervised (SFT) and initialization-only prompt pools

Key Findings

1. Performance Gains: IMAX yields consistent improvements in both Pass@4 and Avg@4 metrics across all backbone sizes. For instance, on Qwen3-4B, GRPO+IMAX achieves Pass@4 gains of up to 11.60% and Avg@4 gains up to 10.57% over the base, surpassing both classical RLVR and current diversity-augmented methods.
2. Efficient Exploration: t-SNE visualizations and reasoning component analyses demonstrate that IMAX-trained prefixes cover complementary and semantically meaningful regions of the model’s latent reasoning space—recovering distinct solution schemes (e.g., algebraic manipulation, case analysis) as intended by initialization.
3. Computational Efficiency: Prefix-only tuning maintains 3–6x lower compute cost (in GPU hours) than full-parameter RLVR methods, with scaling advantages growing as backbone size increases.
4. Ablation: The sample efficiency and coverage both scale positively with the number of learned prefixes, confirming the prefix pool's principal role in expanding high-quality exploration.

Theoretical and Practical Implications

IMAX advances the RLVR paradigm by decoupling exploration-inducing mechanisms from the backbone's implicit priors. By leveraging prompt pools as discrete control handles, it operationalizes a lightweight approach to induce structured, non-redundant exploration among reasoning strategies—without incurring the computational overhead of full-parameter updates. The introduction of a variational InfoMax criterion is crucial for maintaining prefix diversity, avoiding token-level entropy regularization pitfalls, and ensuring diversity is semantically aligned with reasoning strategy rather than mere surface form.

The reusability of learned prefixes opens novel applications—not only for improved few-shot/zero-shot inference-time control (strategy selection, ensemble sampling), but also as a front-end for future adaptive or user-directed RLVR post-training. However, the method’s practical efficacy is currently domain-specific: gains are pronounced on reasoning tasks but not observed in general instruction following, and prefix initialization remains dependent on hand-crafted or large LLM-generated strategies, which may require retuning per downstream domain.

Future Directions

Potential research avenues include:

• Automated discovery or search of optimal prefix pools across broader reasoning domains
• Application to multi-domain, multi-modal, or multi-agent settings
• Integration with retrieval-augmented or modular reasoning architectures
• Real-time dynamic strategy selection during inference or RLVR adaptation

Conclusion

The IMAX framework constitutes a significant algorithmic advancement in RLVR for reasoning-centric LLMs by systematically reshaping and expanding the exploration space via prefix-tuned priors. Its InfoMax reward, semantically seeded initialization, and computationally efficient prefix-only adaptation demonstrate strong empirical benefits in structured reasoning diversity and successful trajectory coverage. This research paves the way for prompt-driven, exploration-aware RLVR that leverages the modularity and flexibility of soft prefix pools for both training- and inference-time reasoning enhancements.