--- title: 'ExpRL: Exploratory RL for LLM Mid-Training' url: https://www.emergentmind.com/topics/exprl type: topic --- # ExpRL: Exploratory RL for LLM Mid-Training ExpRL, short for **Exploratory RL for LLM Mid-Training**, is a reinforcement-learning-based mid-training method for large language model reasoning that uses human-written reference solutions as **reward scaffolds** rather than imitation targets. In ExpRL, the reference solution is hidden from the policy and used only to construct a problem-specific grading rubric: the policy samples from the original problem prompt, while an LLM judge compares the sampled reasoning trace against the reference and assigns either outcome-level or process-level dense rewards. The stated purpose is to improve the model’s **coverage** over productive reasoning paths, yielding a stronger initializer for subsequent sparse-reward RL on harder downstream tasks [2606.17024]. ## 1. Conceptual basis and motivation ExpRL is motivated by two limitations of standard reasoning post-training. First, sparse reward RL with binary final-answer correctness depends critically on the coverage already present in the base model; if the model rarely samples correct or partially correct chain-of-thoughts, the RL signal becomes weak. Second, conventional mid-training on curated reasoning traces can teach useful primitive skills such as decomposition, verification, or self-correction, but it requires manually specifying what the model should learn, and it remains unclear whether such primitive coverage is sufficient for much harder problems that require combining these skills into broader solution strategies [2606.17024]. The method therefore studies a more automated form of **RL-based mid-training** on large corpora of human-written question-answer data. Rather than treating reference solutions as targets to imitate, ExpRL treats them as hidden scaffolds for reward construction. The judge compares an on-policy reasoning trace with the human reference and assigns dense feedback that can reinforce partial progress, useful intermediate reductions, and productive reasoning behaviors that sparse final-answer rewards often fail to upweight [2606.17024]. A common misunderstanding is to treat ExpRL as a variant of supervised fine-tuning. Its defining design choice is the opposite: the reference solution is not exposed to the policy as a target sequence. This is intended to avoid the failure mode in which SFT on $(x,y^\star)$ “overshoot[s]” and collapses the model’s own reasoning coverage [2606.17024]. ## 2. Formal setup and the role of coverage ExpRL defines a mid-training dataset $$ \mathcal D_{\text{mid}}=\{(x_i,y_i^\star)\}_{i=1}^N, $$ where $x$ is a hard question prompt and $y^\star$ is a human-written step-by-step solution. The policy is denoted by $\pi_\theta$, with the untuned base model written as $\pi_0$. For a prompt $x$, a rollout $$ y=(y^1,\dots,y^T) $$ is the generated reasoning trace ending in a final answer, while the human reference is $$ y^\star=(y^{\star,1},\dots,y^{\star,M}). $$ The downstream objective is not only immediate accuracy on $\mathcal D_{\text{mid}}$, but producing a better initialization for later sparse-reward RL on a downstream dataset $\mathcal D'$ [2606.17024]. The paper uses **pass@\(\!k\)** as its operational proxy for coverage. Pass@\(\!k\) is measured by sampling $k$ independent rollouts and checking whether at least one final answer matches the ground truth. In this setup, high pass@\(\!k\) is treated as evidence that the model covers more productive reasoning paths, which is exactly the property sparse-reward RL needs in order to improve efficiently [2606.17024]. This setup also sharpens the distinction between ExpRL and standard sparse-reward GRPO. Under a binary reward $r(x,y)\in\{0,1\}$ defined only by final answer correctness, training can fail when the base model seldom reaches correct answers. ExpRL instead supplies graded supervision without turning the human reference into an imitation target [2606.17024]. ## 3. Reward scaffolds and dense feedback The core mechanism is an LLM judge $J$ that is prompted to **verify rather than solve**. Given $(x,y,y^\star)$, the judge returns an integer score $$ \tilde s\in\{1,2,3,4,5\}, $$ under a fixed rubric that can include matching key decomposition steps and correct intermediate reductions. This score is renormalized as $$ s(x,y,y^\star)=\frac{\tilde s-1}{4}\in[0,1]. $$ The reference solution thus functions as a hidden grading key rather than a demonstrated trajectory [2606.17024]. ExpRL extracts two dense reward forms from this score. The **outcome-level dense reward** is defined directly as $$ r_{\mathrm{outcome}}(x,y,y^\star)=s(x,y,y^\star). $$ This provides graded feedback even when the final answer is wrong, so long as the sampled reasoning aligns partially with the reference [2606.17024]. The **process-level dense reward** is computed on partial prefixes. The rollout is sliced into $T$ segments using the delimiter “###”. For each prefix $y_{\le t}$, the judge score is $$ s_t=s(x,y_{\le t},y^\star),\quad t=1,\dots,T. $$ Segment-wise advantages are then defined as $$ A_t= \begin{cases} s_t-s_{t-1}, & t>1,\\ s_1-s_T, & t=1. \end{cases} $$ In this formulation, each segment is rewarded according to its relative improvement in alignment with the reference [2606.17024]. The distinction between these two variants is substantive. ExpRL-Outcome rewards the quality of the whole rollout, whereas ExpRL-Process attributes credit at the segment level. The paper reports that both substantially improve pass@\(\!k\), but that on the hardest tasks process-level advantages unlock more prompts faster, while outcome-level ExpRL reaches higher late-training entropy [2606.17024]. ## 4. Optimization objective and training pipeline ExpRL solves the on-policy objective $$ \max_\theta\;\mathbb E_{(x,y^\star)\sim\mathcal D_{\text{mid}}}\Bigl[\mathbb E_{y\sim\pi_\theta(\cdot\mid x)}[R(x,y,y^\star)]-\beta\,\mathrm{KL}(\pi_\theta(\cdot\mid x)\,\|\,\pi_0(\cdot\mid x))\Bigr], $$ where $R$ is either the single outcome reward $s(x,y,y^\star)$ or the summed process-level advantages $\sum_t A_t$ [2606.17024]. In practice, the method has two concrete variants. **ExpRL-Outcome** uses GRPO updates with outcome reward group-normalized across $n$ rollouts per prompt. **ExpRL-Process** uses a vanilla REINFORCE update at the segment level, substituting $A_t$ into $$ \nabla_\theta J= \frac1N\sum_{i=1}^N\frac1n\sum_{j=1}^n\sum_{t=1}^{T_{ij}} A_t^{(i,j)}\;\nabla_\theta\log\pi_\theta\!\bigl(y_{ij}^t\mid y_{ij}^{