ExpRL: Exploratory RL for LLM Mid-Training

• ExpRL is a reinforcement-learning mid-training method that uses human-written solutions as hidden reward scaffolds instead of imitation targets.
• It employs an LLM judge to assign dense outcome-level and process-level rewards, reinforcing productive reasoning steps and intermediate improvements.
• The method improves model reasoning coverage and serves as a robust initializer for subsequent sparse-reward RL, yielding higher Pass@1 on benchmark tasks.

ExpRL, short for Exploratory RL for LLM Mid-Training, is a reinforcement-learning-based mid-training method for LLM reasoning that uses human-written reference solutions as reward scaffolds rather than imitation targets. In ExpRL, the reference solution is hidden from the policy and used only to construct a problem-specific grading rubric: the policy samples from the original problem prompt, while an LLM judge compares the sampled reasoning trace against the reference and assigns either outcome-level or process-level dense rewards. The stated purpose is to improve the model’s coverage over productive reasoning paths, yielding a stronger initializer for subsequent sparse-reward RL on harder downstream tasks (Xiang et al., 15 Jun 2026).

1. Conceptual basis and motivation

ExpRL is motivated by two limitations of standard reasoning post-training. First, sparse reward RL with binary final-answer correctness depends critically on the coverage already present in the base model; if the model rarely samples correct or partially correct chain-of-thoughts, the RL signal becomes weak. Second, conventional mid-training on curated reasoning traces can teach useful primitive skills such as decomposition, verification, or self-correction, but it requires manually specifying what the model should learn, and it remains unclear whether such primitive coverage is sufficient for much harder problems that require combining these skills into broader solution strategies (Xiang et al., 15 Jun 2026).

The method therefore studies a more automated form of RL-based mid-training on large corpora of human-written question-answer data. Rather than treating reference solutions as targets to imitate, ExpRL treats them as hidden scaffolds for reward construction. The judge compares an on-policy reasoning trace with the human reference and assigns dense feedback that can reinforce partial progress, useful intermediate reductions, and productive reasoning behaviors that sparse final-answer rewards often fail to upweight (Xiang et al., 15 Jun 2026).

A common misunderstanding is to treat ExpRL as a variant of supervised fine-tuning. Its defining design choice is the opposite: the reference solution is not exposed to the policy as a target sequence. This is intended to avoid the failure mode in which SFT on $(x,y^\star)$ “overshoot[s]” and collapses the model’s own reasoning coverage (Xiang et al., 15 Jun 2026).

2. Formal setup and the role of coverage

ExpRL defines a mid-training dataset

$$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$

where $x$ is a hard question prompt and $y^\star$ is a human-written step-by-step solution. The policy is denoted by $\pi_\theta$, with the untuned base model written as $\pi_0$. For a prompt $x$, a rollout

$y=(y^1,\dots,y^T)$

is the generated reasoning trace ending in a final answer, while the human reference is

$y^\star=(y^{\star,1},\dots,y^{\star,M}).$

The downstream objective is not only immediate accuracy on $\mathcal D_{\text{mid}}$, but producing a better initialization for later sparse-reward RL on a downstream dataset $$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$0 (Xiang et al., 15 Jun 2026).

The paper uses pass@$$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$1 as its operational proxy for coverage. Pass@$$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$2 is measured by sampling $$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$3 independent rollouts and checking whether at least one final answer matches the ground truth. In this setup, high pass@$$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$4 is treated as evidence that the model covers more productive reasoning paths, which is exactly the property sparse-reward RL needs in order to improve efficiently (Xiang et al., 15 Jun 2026).

This setup also sharpens the distinction between ExpRL and standard sparse-reward GRPO. Under a binary reward $$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$5 defined only by final answer correctness, training can fail when the base model seldom reaches correct answers. ExpRL instead supplies graded supervision without turning the human reference into an imitation target (Xiang et al., 15 Jun 2026).

3. Reward scaffolds and dense feedback

The core mechanism is an LLM judge $$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$6 that is prompted to verify rather than solve. Given $$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$7, the judge returns an integer score

$$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$8

under a fixed rubric that can include matching key decomposition steps and correct intermediate reductions. This score is renormalized as

$$\mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N,$$9

The reference solution thus functions as a hidden grading key rather than a demonstrated trajectory (Xiang et al., 15 Jun 2026).

ExpRL extracts two dense reward forms from this score.

The outcome-level dense reward is defined directly as

$x$0

This provides graded feedback even when the final answer is wrong, so long as the sampled reasoning aligns partially with the reference (Xiang et al., 15 Jun 2026).

The process-level dense reward is computed on partial prefixes. The rollout is sliced into $x$1 segments using the delimiter “###”. For each prefix $x$2, the judge score is

$x$3

Segment-wise advantages are then defined as

$x$4

In this formulation, each segment is rewarded according to its relative improvement in alignment with the reference (Xiang et al., 15 Jun 2026).

The distinction between these two variants is substantive. ExpRL-Outcome rewards the quality of the whole rollout, whereas ExpRL-Process attributes credit at the segment level. The paper reports that both substantially improve pass@$x$5, but that on the hardest tasks process-level advantages unlock more prompts faster, while outcome-level ExpRL reaches higher late-training entropy (Xiang et al., 15 Jun 2026).

4. Optimization objective and training pipeline

ExpRL solves the on-policy objective

$x$6

where $x$7 is either the single outcome reward $x$8 or the summed process-level advantages $x$9 (Xiang et al., 15 Jun 2026).

In practice, the method has two concrete variants. ExpRL-Outcome uses GRPO updates with outcome reward group-normalized across $y^\star$0 rollouts per prompt. ExpRL-Process uses a vanilla REINFORCE update at the segment level, substituting $y^\star$1 into

$y^\star$2

with the same KL penalty (Xiang et al., 15 Jun 2026).

The training workflow is explicitly two-stage. In Stage I, ExpRL performs judge-based mid-training on $y^\star$3. In Stage II, the resulting policy is used as the initialization for sparse binary-reward RL. The pseudocode in the paper initializes $y^\star$4 base LLM, $y^\star$5 base LLM, and $y^\star$6 copy(base), then samples rollouts, scores them with the judge, computes either outcome or process rewards, applies the policy gradient with KL regularization, and finally switches to sparse final-answer reward RL after Stage I (Xiang et al., 15 Jun 2026).

The implementation details reported are specific. The policy is Qwen3-4B-Instruct-2507, which “emits chain-of-thought by default,” and the judge is a frozen copy of the 4B policy, with Qwen3-0.6B used in calibration tests. Stage-I uses $y^\star$7 rollouts per prompt, sampling temperature $y^\star$8, maximum response length $y^\star$9 tokens, batch size $\pi_\theta$0 prompts per update, and $\pi_\theta$1 updates. Stage-II uses $\pi_\theta$2 sparse-reward RL updates with batch size $\pi_\theta$3. The KL penalty coefficient $\pi_\theta$4 is set to match GRPO defaults, “e.g. 0.01” (Xiang et al., 15 Jun 2026).

5. Empirical results, priming effects, and behavioral changes

The mid-training data $\pi_\theta$5 consists of InT + POPE hard math problems with approximately 1.5k examples. The downstream benchmarks are AIME ’25, AIME ’26, HMMT Nov 2025, and IMO-AnswerBench (held-out). A separate mixed-domain study uses 4,001 examples spanning Math, SciKnow (MCQ/OE), and LiveCodeBench coding tasks (Xiang et al., 15 Jun 2026).

The baselines are SFT, Sparse GRPO, and Self-distillation. After Stage-II sparse RL, the reported Pass@1 results are:

• Base Qwen3-4B Instruct: 46.46 on AIME25, 51.40 on AIME26, 40.60 on HMMT, 31.37 on IMO-Ans.
• SFT: 26.62, 30.26, 20.09, 21.80.
• GRPO (sparse): 55.99, 58.75, 42.91, 35.28.
• Self-Distillation: 55.59, 58.41, 46.08, 35.18.
• ExpRL-Outcome: 59.07, 61.74, 49.11, 37.85.
• ExpRL-Process: 58.08, 63.41, 48.13, 35.73 (Xiang et al., 15 Jun 2026).

These numbers establish two distinct effects. First, both ExpRL variants outperform SFT, sparse GRPO, and self-distillation overall. Second, the two dense-reward schemes differ by benchmark: ExpRL-Outcome is strongest on AIME25, HMMT, and IMO-AnswerBench, whereas ExpRL-Process is highest on AIME ’26 at 63.41% (Xiang et al., 15 Jun 2026).

The paper also reports a Stage-I priming ablation. After mid-training alone, before Stage-II sparse RL, ExpRL yields higher pass@$\pi_\theta$6 on all benchmarks than SFT, sparse GRPO, and self-distillation. This is used to argue that the method improves coverage directly, rather than only benefiting from downstream optimization (Xiang et al., 15 Jun 2026).

Behaviorally, ExpRL priming changes the model’s reasoning style. Compared to base, SFT, and sparse RL, the model shows net gains in verification steps, self-correction, and backtracking and restarts. The analysis further states that outcome-level ExpRL reaches higher late-training entropy, suggesting broader exploration, while process-level training grows response length (Xiang et al., 15 Jun 2026).

The paper includes two additional diagnostic findings. On math and science tasks, correct-reference judging yields the lowest misplacement rate, while wrong or no reference degrades discrimination. On coding tasks from LiveCodeBench, wrong reference and no reference perform similarly because code execution is the primary judge. It also argues that self-distillation is hindered by KL mismatch: the self-distillation “teacher” lies far outside the KL ball reachable by on-policy RL, which is presented as an explanation for unstable coverage relative to ExpRL (Xiang et al., 15 Jun 2026).

6. Limitations, misconceptions, and position within exploration-oriented RL

ExpRL has two explicit limitations. It requires reference solutions for mid-training, which may not exist in some domains, and it relies on a capable LLM judge (≥ 4 B parameters) for reliable reward. The paper also lists several extensions: extracting the judge’s natural-language critiques and training on them, combining ExpRL priming with oracle prefix insertion during downstream RL, and systematic study of rubric design, length normalization, and KL annealing (Xiang et al., 15 Jun 2026).

The method is best understood as part of a broader family of exploration-oriented RL methods for reasoning, but its mechanism is distinct. EFRame augments GRPO with additional rollouts, online filtering, and experience replay (Wang et al., 27 Jun 2025). ExPO generates self-explanations by conditioning on the ground-truth answer and integrates them into DPO or GRPO updates (Zhou et al., 3 Jul 2025). ExTra adds a correctness-gated embedding-based novelty reward after GRPO normalization and uses entropy-guided prefix regeneration on all-incorrect groups (Hu et al., 23 Jun 2026). XRPO combines adaptive rollout allocation, in-context seeding, and novelty-aware advantage sharpening (Bamba et al., 8 Oct 2025). ReLaX introduces Dynamic Spectral Dispersion and augments GRPO with a latent-exploration regularizer derived from Koopman spectral analysis (Zhang et al., 8 Dec 2025). ExpLang instead expands the action space by allowing on-policy thinking-language selection in multilingual RL post-training (Gao et al., 25 Feb 2026).

This comparison clarifies a second common misconception: ExpRL is not merely another rollout-exploration heuristic inside GRPO. Its defining move is earlier in the pipeline. It uses human references to produce dense, judge-mediated reward during a mid-training stage, and then uses the resulting policy as a better initializer for later sparse-reward RL (Xiang et al., 15 Jun 2026).

A plausible implication is that ExpRL occupies a complementary position in the exploration literature. The reported results show that it improves both coverage and final downstream Pass@1, while the related methods above focus on exploration–exploitation tradeoffs during sparse-reward post-training itself. Within that landscape, ExpRL’s distinctive contribution is to turn abundant human solutions into a reward source without turning them into imitation targets (Xiang et al., 15 Jun 2026).