RL-Based Mid-Training Methods

• RL-based mid-training is an intermediate phase that incorporates reinforcement learning signals to bridge generic pre-training and specialized downstream RL optimization.
• It improves efficiency by reweighting next-token predictions and restructuring model representations, leading to significant gains on out-of-distribution and reasoning benchmarks.
• Methodologies include RL-guided token reweighting, online RL priming, and hybrid objectives that integrate curriculum control and rollout interventions to optimize final performance.

RL-based mid-training denotes a family of intermediate training interventions positioned between broad pre-training and later post-training, in which reinforcement learning, RL-derived signals, or RL-compatible curriculum and rollout design are used to shape a model before its final downstream optimization. The term is not fully standardized. In the survey literature, mid-training is defined primarily as a continuation of next-token prediction on curated data, with RL treated mostly as post-training; later papers broaden the notion to include RL-guided reweighting during annealing, online RL stages that prepare models for later sparse-reward optimization, and explicitly named “reinforcement mid-training” frameworks (Tu et al., 27 Oct 2025, Huang et al., 3 Feb 2026, Tian et al., 29 Sep 2025).

1. Definition and stage boundaries

The most conservative definition comes from the survey literature. There, mid-training is a stage between pre-training and post-training that uses intermediate computational demands, targeted large-scale data utilization, and the same next-token prediction objective, while post-training adopts specialized objectives for SFT and RL (Tu et al., 27 Oct 2025). Under that definition, RL-based mid-training is not yet a mature methodological category; it is mostly a boundary case, or a preparatory phase for later RL rather than RL itself.

Subsequent work relaxes that boundary in three distinct ways. First, RL may remain external to mid-training but provide a guidance signal, as in RL-guided token reweighting during the annealing phase. Second, RL may itself become the intermediate stage, but with objectives designed to improve later sparse-reward post-training rather than to serve as the final optimization target. Third, prompt- or rollout-level interventions can alter the effective RL regime after supervised fine-tuning without changing the RL objective, making the “mid” property operational rather than formal. This suggests that the field now uses “RL-based mid-training” in both a strict sense—an intermediate RL optimization stage—and a looser sense—any intermediate intervention designed to improve later RL.

A useful editorial distinction is between canonical mid-training, which preserves pre-training-style next-token prediction, and RL-conditioned mid-training, which injects RL signals, RL priors, or RL-compatible behavior into that intermediate stage. The survey record supports the former; most recent method papers occupy the latter category (Tu et al., 27 Oct 2025).

2. Why intermediate stages matter for later RL

Several studies argue that RL effectiveness depends less on the optimizer in isolation than on the model state from which RL begins. In a controlled synthetic framework, pre-training installs primitives, mid-training bridges the distribution gap, and RL yields true capability gains only when data target the model’s “edge of competence”; under fixed compute, mid-training plus RL outperforms RL alone by +10.8% on OOD-hard tasks (Zhang et al., 8 Dec 2025). This suggests that RL is most productive when it amplifies partially formed capabilities rather than when it is asked to create them from scratch.

Large-scale empirical studies make the same point in open-model settings. PRISM reports that mid-training on approximately 27B high-quality tokens yields consistent gains of +15 to +40 points on math, +5 to +12 points on code, and +6 to +13 points on science while preserving general performance; the full PRISM-to-RL pipeline lifts macro-average across six reasoning benchmarks from under 12 to 29–42, whereas RL applied directly to most base models leaves AIME scores near zero (Runwal et al., 17 Mar 2026). PRISM further reports that mid-training densely restructures over 90% of model weights, whereas RL makes sparse refinements to approximately 5% of parameters and preserves mid-training’s representational geometry with over 0.998 CKA, indicating that RL largely refines a configuration created earlier rather than replacing it (Runwal et al., 17 Mar 2026).

OctoThinker reaches a related conclusion from the perspective of RL compatibility. It shows that high-quality mathematical corpora such as MegaMath-Web-Pro improve both base-model and downstream RL performance, that QA-style data and instruction data further improve RL outcomes, and that long chain-of-thought data increases reasoning depth but can also induce verbosity and RL instability (Wang et al., 25 Jun 2025). A plausible implication is that RL-based mid-training is not only about adding reward optimization earlier, but also about engineering the pre-RL data distribution, format prior, and reasoning style that determine whether RL scales at all.

3. Main methodological families

One major family keeps the mid-training objective as weighted next-token prediction, but derives the weights from an RL-tuned model. ReMiT uses a frozen RL-tuned reference to compute a token-level discrepancy and then rescales the standard autoregressive loss during the annealing phase. For sequence $x_{1:T}$, standard NTP is modified into

$$\mathcal L_{\text{ReMiT}}(\theta)= -\,\frac{1}{\sum_i T_i}\sum_{i=1}^{N}\sum_{t=1}^{T_i} w^{(i)}_{t} \log p_\theta\!\left(x^{(i)}_{t}\mid x^{(i)}_{<t}\right).$$

The method does not run PPO, GRPO, DPO, or reward maximization inside mid-training itself; instead, RL acts as a source of token-level importance. Empirically, ReMiT reports an average improvement of about 3% on 10 pre-training benchmarks and sustains gains of over 2% throughout later post-training, while also supporting a second iterative cycle, ReMiT$^2$ (Huang et al., 3 Feb 2026).

A second family treats the intermediate stage itself as online RL, but uses reference solutions only as hidden reward scaffolds rather than imitation targets. ExpRL defines a Stage-I RL objective

$$\max_\theta \; \mathbb{E}_{(x,y^\star)\sim\mathcal{D}_\text{mid}} \left[ \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)} \big[ R(x,y,y^\star) \big] - \beta\,\mathrm{KL}\!\left(\pi_\theta(\cdot\mid x)\,\|\,\pi_0(\cdot\mid x)\right) \right],$$

where the policy never sees the reference during generation; an LLM judge uses the reference to assign dense outcome-level or process-level rewards. After this Stage-I RL priming, standard sparse-reward RL is applied. On held-out math benchmarks, Stage-II sparse-reward RL initialized from ExpRL is stronger than initialization from sparse-reward GRPO, SFT, or self-distillation; for example, AIME25 pass@1 reaches 59.07 for ExpRL-Outcome versus 55.99 for GRPO, and AIME26 pass@1 reaches 63.41 for ExpRL-Process (Xiang et al., 15 Jun 2026).

A third family explicitly defines “reinforcement mid-training” as a hybrid objective over large unlabeled corpora. RMT partitions a token sequence $\mathcal S=\{\tau_1,\tau_2,\ldots,\tau_L\}$ into an RL subset $\Phi_{\text{RL}}$ and an NTP subset $$\Phi_{\text{NTP}}=\mathcal S\setminus\Phi_{\text{RL}}$$, with $|\Phi_{\text{NTP}}|\gg |\Phi_{\text{RL}}|$. It couples a dynamic token budget, curriculum-based adaptive sampling, token-selective GRPO, and masked NTP:

$$\mathcal L = \mathcal L_{\text{RL}}(\theta) + \lambda \cdot \mathcal L_{\text{NTP}}(\theta).$$

RMT reports up to +64.91% performance improvement with only 21% of the reasoning length in language modeling, and checkpoints after reinforcement mid-training improve subsequent post-training by up to +18.76% in the mathematical domain (Tian et al., 29 Sep 2025).

4. Diversity, curriculum, and rollout-control interventions

A recurring thesis is that RL benefits from broader pre-RL support over valid reasoning trajectories. One approach is to generate multiple correct solution variants before RL. In a study based on George Pólya’s heuristics, each question is paired with multiple self-generated correct traces, and the mid-training objective averages the supervised log-likelihood over those variants. The resulting models show stronger downstream RL than vanilla RL and STaR+RL; the average pass@64 rises to 48.09 at $n=16$ variants, versus 44.21 for vanilla RL and 45.69 for STaR+RL, and the same study reports gains on HumanEval and MuSR (RRV et al., 8 May 2026). The paper’s theoretical interpretation is that multimodal pre-RL token distributions are less collapse-prone and more responsive to policy-gradient updates.

A second line of work alters the effective RL regime by modifying rollout prompting rather than the optimizer. Mid-Think identifies token-level triggers such as a leading “Okay” and the newline pattern after </think> as the main drivers of Think/No-think mode switching. It composes the suppressive and activating cues in a single template, > \n\n\n\n<reason>\nOkay..., to induce intermediate-budget reasoning during RL after SFT. In Qwen3-8B post-SFT RL with GRPO and the verl framework, this reduces training time from 54 hours to 46 hours while improving final Think-mode evaluation from 69.8% to 72.4% on AIME and from 58.5% to 61.1% on GPQA (Yang et al., 11 Jan 2026). The paper is explicit that this is not a new RL objective; it is a rollout-format intervention that changes exploration, entropy, and trajectory length.

Bridge methods blur the line between mid-training and post-training even further. DYPO is framed as post-training, but it operationally resembles RL-based mid-training because it replaces the strict SFT-then-RL boundary with unified optimization, dynamic routing by rollout difficulty, multi-teacher supervision on all-fail cases, and a bounded contrastive Group Alignment Loss on mixed-success groups. It reports an average improvement of 4.8% on complex reasoning benchmarks and 13.3% on out-of-distribution tasks over traditional sequential pipelines (Zhu et al., 10 Apr 2026). This suggests that some of the design logic of RL-based mid-training—competence-aware mixing, curriculum, and bounded exploratory updates—may be realized either before or during the nominal post-training phase.

5. Systems, stability, and training dynamics

As RL-based mid-training moves toward longer trajectories and larger clusters, systems questions become first-order. SortedRL targets the rollout bottleneck in long-chain-of-thought RL, where rollouts can consume 70% of total training time at 16K maximum generation length. It reorders rollouts online by completion length, enabling large rollout batches, flexible update batches, and a near on-policy micro-curriculum. The paper reports bubble-ratio reductions of over 50% and performance gains of 3.9% to 18.4% over baseline given the same amount of data (Zhang et al., 24 Mar 2026).

A related stability problem is training–inference mismatch during long RL runs. Beyond Precision argues that mismatch is not merely a static precision artifact but a dynamic optimization failure that escalates with gradient noise and response length. The proposed remedy is a response-length-triggered learning-rate scheduler, with the practical heuristic

$$\mathcal L_{\text{ReMiT}}(\theta)= -\,\frac{1}{\sum_i T_i}\sum_{i=1}^{N}\sum_{t=1}^{T_i} w^{(i)}_{t} \log p_\theta\!\left(x^{(i)}_{t}\mid x^{(i)}_{<t}\right).$$0

followed by repeated halving down to a floor. On Qwen3-4B-Base and Qwen3-8B-Base, this suppresses mismatch and stabilizes RL more reliably than importance-sampling corrections alone (Zhang et al., 2 Feb 2026).

At cluster scale, MindSpeed RL treats RL training as a distributed dataflow problem with two core flows: sample flow and resharding flow. Its distributed transfer dock replaces centralized replay-buffer dispatch with controllers and warehouses, and its allgather–swap strategy reduces redundant memory during transitions between update-time and generation-time sharding layouts. Across Qwen2.5-Dense-7B/32B, Qwen3-MoE-30B, and DeepSeek-R1-MoE-671B, the framework reports throughput gains of 1.42 ~ 3.97 times on a 384-NPU Ascend superpod (Feng et al., 25 Jul 2025).

Long-horizon agent settings introduce a different systems-and-optimization pathology: the environment itself becomes rollout-dependent. Memory-R2 shows that in multi-session memory-augmented agents, different rollouts inherit different memory states, so standard group-relative comparisons become unfair. Its LoGo-GRPO combines global trajectory rewards with local rerollouts from matched intermediate memory states, and a progressive curriculum from 8 to 16 to 32 sessions. On LoCoMo, LoGo-GRPO improves F1 from 46.62 to 49.67 and reduces M-Fail from 10.20% to 6.72% relative to plain GRPO (Yan et al., 20 May 2026). This broadens RL-based mid-training from reasoning traces to persistent-state agents.

6. Limitations, retention, and open questions

The literature remains fragmented. The survey record still treats RL-based mid-training largely as a boundary case rather than a mature taxonomy, and many method papers occupy narrow empirical regimes—reasoning-heavy corpora, specific model families, or relatively small scales (Tu et al., 27 Oct 2025). This suggests that the field has not yet converged on a single formal definition or a standardized benchmark suite.

One open problem is retention under later post-training. A value-oriented study shows that values instilled during a compassion-oriented mid-training stage can be differentially degraded by later SFT or GRPO depending on the domain of post-training data. On the Animal Harm Benchmark, helpfulness-domain SFT scores 35.7% versus 65.2% for coding-domain SFT, and helpfulness-domain GRPO scores 18.7% versus 32.0% for coding-domain GRPO (Brazilek et al., 30 Apr 2026). A plausible implication is that RL-based mid-training cannot be evaluated independently of the later RL or post-training stage that may overwrite it.

A related concern appears in cross-domain agent training. An SFT warmup or mid-training phase before RL helps prevent catastrophic forgetting for domains included in its datamix, but undermines generalization to domains that are not included. In the reported setup, domains covered by the warmup degrade much less under later RL, while the excluded Sokoban domain degrades substantially more (Liu et al., 26 Jan 2026). This establishes a central tension: intermediate shaping can preserve covered competencies yet narrow open-ended transfer.

The larger open questions are therefore structural rather than incremental. What should count as RL-based mid-training under a unified taxonomy? Which signals are best injected at the intermediate stage—online rewards, frozen RL priors, token budgets, diverse self-generated trajectories, or prompt-level rollout control? When does intermediate RL enlarge a model’s coverage over productive reasoning paths, and when does it merely specialize or destabilize the model? Current results show that the stage can improve efficiency, raise downstream RL ceilings, and make later post-training easier; they also show that the benefits depend heavily on data composition, reward design, rollout formatting, and the precise capabilities one wishes to preserve.