ExpRL: Exploratory RL for LLM Mid-Training

Abstract: Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model. In practice, models are often primed for RL through \emph{mid-training} on curated reasoning traces that teach useful primitive skills such as decomposition, verification, or self-correction. Although effective, this strategy requires manually specifying what the model should learn, and it remains unclear whether such primitive coverage is enough for much harder problems, which require combining these skills into broader solution strategies. We study a more automated approach: \emph{RL-based mid-training} using large corpora of human-written question-answer data. Rather than treating reference solutions as targets to imitate, our method, ExpRL, uses them as \emph{reward scaffolds}: references are hidden from the policy and used only to construct problem-specific grading rubrics for judging on-policy reasoning traces. The policy samples from the original problem prompt, while an LLM judge compares the sampled reasoning trace against the reference solution and assigns outcome-level or process-level dense rewards. This lets ExpRL reinforce partial progress, useful intermediate reductions, and productive reasoning behaviors that sparse final-answer rewards often fail to upweight. On challenging math reasoning tasks, ExpRL yields stronger RL priming than SFT, sparse-reward GRPO, and self-distillation, and provides a better initialization for subsequent sparse-reward RL. Additional mixed-domain experiments further suggest that ExpRL can extend beyond the original math-only setting.

• The paper introduces a dense reward, reference-guided RL framework (ExpRL) that significantly broadens exploration of productive reasoning paths in LLM mid-training.
• It presents two methods, ExpRL-Outcome and ExpRL-Process, which provide graded feedback for partial progress and improve local credit assignment.
• Empirical results on math benchmarks show enhanced pass@k metrics and behavioral shifts, validating its superiority over traditional RL and imitation approaches.

Exploratory RL for LLM Mid-Training: Reference-Guided Dense Reward Priming

Motivation and Problem Statement

Sparse-reward reinforcement learning (RL) is an established approach for enhancing LLM reasoning, yet its effectiveness is fundamentally constrained by the coverage of reasoning paths present in the base model distribution. When the base model assigns limited probability to productive or diverse reasoning traces—especially on combinatorially hard math and science problems—traditional RL with sparse feedback is bottlenecked by poor exploration. Prior mid-training approaches, including supervised fine-tuning (SFT) on curated traces and self-distillation, offer incremental improvements but generally focus on imbuing isolated skills or matching token distributions, rather than cultivating diverse, compositional strategy coverage for high-dimensional reasoning tasks.

ExpRL addresses mid-training as an explicit exploration bottleneck. Instead of imitation, ExpRL leverages reference solutions solely to define reward scaffolds: dense grading rubrics, constructed around partial-progress alignment between model rollouts and human solutions, provide process-level or outcome-level signal. This alters the exploration prior, refocusing RL mid-training on broadening coverage of productive solution trajectories before downstream sparse reward RL.

ExpRL Methods and RL Priming Mechanisms

ExpRL introduces two core reference-guided RL priming methods:

• ExpRL-Outcome: Assigns dense reference-alignment reward to entire rollouts, offering graded signal even for incomplete or partially correct generations.
• ExpRL-Process: Slices the sampled rollout into intermediate prefixes (e.g., step-delimited reasoning) and computes local segment-level advantages using reference-alignment judgments. This yields token/segment level reward attribution, improving local credit assignment for partial but promising reasoning strategies.

Rather than exposing reference solutions as demonstrations or conditioning tokens, ExpRL uses the base model or a suitably capable LLM as a judge. The judge scores sampled trajectories relative to the reference under a rubric emphasizing technique overlap, correct decompositions, and strategic alignment, but excluding generative hallucination or off-distribution teacher modes. This approach necessitates no model exposure to solution content during generation, maintaining pure on-policy exploration and avoiding off-policy learning pathologies.

Empirical Analysis: Training Dynamics and Priming Effects

ExpRL is empirically validated on Qwen3-4B-Instruct as the policy backbone, with both math-only and mixed-domain settings. Across challenging benchmarks (AIME-25/26, HMMT, IMO-AnswerBench), ExpRL-Outcome and ExpRL-Process consistently outperform SFT, standard sparse-reward RL (GRPO), and self-distillation—especially at low $k$ in pass@k, where productive reasoning paths are rare and exploration is most challenging.

Figure 1: ExpRL accelerates unlock of previously unsolvable prompts, sustains entropy during RL priming, and produces controlled length increases under process rewards with subsequent stabilization.

ExpRL-Process in particular reduces the number of unsolved prompts faster than comparators, maintains or slightly increases token-level entropy (in contrast to entropy collapse under GRPO), and dynamically regulates response length. This indicates that exploration broadening, not just mode-seeking, is being reinforced. The process-level reward provides finer-grained feedback and facilitates efficient credit assignment for partial progress—a key differentiator in high-complexity reasoning.

Figure 3: ExpRL (both outcome and process) consistently outperforms imitation and sparse-only baselines on pass@$k$ for held-out answer-based math benchmarks across difficult $k$ regimes.

Stage-I RL priming with ExpRL results in stronger initializations for downstream sparse-reward RL, which translates to meaningful downstream gains. Notably, ExpRL-Process attains 63.4% pass@1 after RL on AIME-2026, versus 58.75% from the best comparator. Gains consistently hold across pass@1 and pass@k for most domains.

Analysis of Induced Behavioral Shifts and Trajectory Distribution

ExpRL not only lifts aggregate benchmark performance but also induces substantive behavioral changes in model policy distribution.

Figure 4: ExpRL yields net increases in search-oriented behaviors (verification, self-correction, backtracking), indicating reference-guided RL priming shifts coverage toward productive exploration, not merely final correctness.

Relative to the base and SFT models—which lose some verification and search traits—ExpRL-primed models expand coverage of self-correction, exploration, and adaptive verification. Self-distillation increases some behaviors, but lags ExpRL in downstream RL metrics. This highlights that enriching behavioral coverage is necessary but not sufficient; coverage must align with productive solution paths, not just behavioral diversity for its own sake.

Process Reward Normalization and Robustness to Length Effects

ExpRL-Process assigns segment-wise advantages using the difference in judge scores relative to previous prefixes. This robust delta-centric approach keeps spurious length increases and degenerate learning in check. Additional ablations (not shown here) investigate alternate normalization strategies (referenced in Figure 5), ensuring that reward shaping reflects true progress rather than artifacts of rollout length or delimiter frequency.

Figure 6: Distribution shift in delimiter counts after ExpRL-Process: collapse attributable primarily to length clipping, not process-reward dynamics per se, demonstrating the importance of sound rollout management.

Judge Calibration and Generalization to Mixed-Domain/Scale

ExpRL generalizes to settings with mismatched or smaller LLM judges. Stage-I RL priming with the reference-conditioned judge provides substantive reward discrimination only when judges are sufficiently capable and references are well-matched to the problem. Misplacement rates validate that judge quality and rubric integrity are prerequisites for effective reward-guided exploration. Furthermore, the ExpRL framework extends to science and coding tasks, though in highly automatable domains (e.g., code execution), environmental rewards from execution dominate over reference-scaffolded signals.

Figure 8: LLM judge calibration: reference-guided scoring enables robust discrimination of partial correctness; process-level scores track downstream success trajectories, albeit with higher variance.

Theoretical Implications and Future Directions

ExpRL operationalizes a shift from skill-inducing or imitation mid-training to explicit exploration prior shaping. By capturing the verification-generation gap of LLMs, reference-anchored reward scaffolds act as a mechanism for credit assignment over compositional reasoning dimensions that SFT and self-distillation cannot reach.

Key directions for future development include:

• Augmenting scalar reward with natural language feedback from the judge for more interpretable, actionable signal.
• Combining ExpRL with prefix-conditioned or active exploration strategies to further expand solution trajectory support.
• Systematic study of score calibration, normalization, and judge design for improved reward signal stability and generalization across domains.

Conclusion

ExpRL establishes a dense reward, reference-guided RL priming methodology for LLMs that effectively broadens the exploration prior over productive reasoning paths. This leads to improved pass@$k$ metrics, accelerated unlock of previously hard-to-explore prompts, and a pronounced amplification of verification- and correction-oriented reasoning behaviors. The method circumvents the distribution shift and learning instability issues associated with off-policy distillation and SFT, providing a framework where privileged information is used exclusively for reward shaping without contaminating the generative prior. ExpRL thus represents an advance in automatable, scalable RL mid-training protocols for complex sequence modeling. Its implications for general reasoning, transferability, and RL stability are significant for future AI research (2606.17024).