---
title: RL-Based Mid-Training Methods
url: https://www.emergentmind.com/topics/rl-based-mid-training
type: topic
---

# RL-Based Mid-Training Methods

RL-based mid-training denotes a family of intermediate training interventions positioned between broad pre-training and later post-training, in which reinforcement learning, RL-derived signals, or RL-compatible curriculum and rollout design are used to shape a model before its final downstream optimization. The term is not fully standardized. In the survey literature, mid-training is defined primarily as a continuation of next-token prediction on curated data, with RL treated mostly as post-training; later papers broaden the notion to include RL-guided reweighting during annealing, online RL stages that prepare models for later sparse-reward optimization, and explicitly named “reinforcement mid-training” frameworks [2510.23081][2602.03075][2509.24375].

## 1. Definition and stage boundaries

The most conservative definition comes from the survey literature. There, mid-training is a stage between pre-training and post-training that uses intermediate computational demands, targeted large-scale data utilization, and the same next-token prediction objective, while post-training adopts specialized objectives for SFT and RL [2510.23081]. Under that definition, RL-based mid-training is not yet a mature methodological category; it is mostly a boundary case, or a preparatory phase for later RL rather than RL itself.

Subsequent work relaxes that boundary in three distinct ways. First, RL may remain external to mid-training but provide a guidance signal, as in RL-guided token reweighting during the annealing phase. Second, RL may itself become the intermediate stage, but with objectives designed to improve later sparse-reward post-training rather than to serve as the final optimization target. Third, prompt- or rollout-level interventions can alter the effective RL regime after supervised fine-tuning without changing the RL objective, making the “mid” property operational rather than formal. This suggests that the field now uses “RL-based mid-training” in both a strict sense—an intermediate RL optimization stage—and a looser sense—any intermediate intervention designed to improve later RL.

A useful editorial distinction is between **canonical mid-training**, which preserves pre-training-style next-token prediction, and **RL-conditioned mid-training**, which injects RL signals, RL priors, or RL-compatible behavior into that intermediate stage. The survey record supports the former; most recent method papers occupy the latter category [2510.23081].

## 2. Why intermediate stages matter for later RL

Several studies argue that RL effectiveness depends less on the optimizer in isolation than on the model state from which RL begins. In a controlled synthetic framework, pre-training installs primitives, mid-training bridges the distribution gap, and RL yields true capability gains only when data target the model’s “edge of competence”; under fixed compute, mid-training plus RL outperforms RL alone by **+10.8% on OOD-hard tasks** [2512.07783]. This suggests that RL is most productive when it amplifies partially formed capabilities rather than when it is asked to create them from scratch.

Large-scale empirical studies make the same point in open-model settings. PRISM reports that mid-training on approximately **27B high-quality tokens** yields consistent gains of **+15 to +40 points** on math, **+5 to +12 points** on code, and **+6 to +13 points** on science while preserving general performance; the full PRISM-to-RL pipeline lifts macro-average across six reasoning benchmarks from **under 12** to **29–42**, whereas RL applied directly to most base models leaves AIME scores near zero [2603.17074]. PRISM further reports that mid-training densely restructures **over 90%** of model weights, whereas RL makes sparse refinements to approximately **5%** of parameters and preserves mid-training’s representational geometry with **over 0.998 CKA**, indicating that RL largely refines a configuration created earlier rather than replacing it [2603.17074].

OctoThinker reaches a related conclusion from the perspective of RL compatibility. It shows that high-quality mathematical corpora such as MegaMath-Web-Pro improve both base-model and downstream RL performance, that QA-style data and instruction data further improve RL outcomes, and that long chain-of-thought data increases reasoning depth but can also induce verbosity and RL instability [2506.20512]. A plausible implication is that RL-based mid-training is not only about adding reward optimization earlier, but also about engineering the pre-RL data distribution, format prior, and reasoning style that determine whether RL scales at all.

## 3. Main methodological families

One major family keeps the mid-training objective as weighted next-token prediction, but derives the weights from an RL-tuned model. ReMiT uses a frozen RL-tuned reference to compute a token-level discrepancy and then rescales the standard autoregressive loss during the annealing phase. For sequence $x_{1:T}$, standard NTP is modified into
$$
\mathcal L_{\text{ReMiT}}(\theta)= -\,\frac{1}{\sum_i T_i}\sum_{i=1}^{N}\sum_{t=1}^{T_i} w^{(i)}_{t} \log p_\theta\!\left(x^{(i)}_{t}\mid x^{(i)}_{<t}\right).
$$
The method does not run PPO, GRPO, DPO, or reward maximization inside mid-training itself; instead, RL acts as a source of token-level importance. Empirically, ReMiT reports an average improvement of about **3%** on **10** pre-training benchmarks and sustains gains of **over 2%** throughout later post-training, while also supporting a second iterative cycle, ReMiT\(^2\) [2602.03075].

A second family treats the intermediate stage itself as online RL, but uses reference solutions only as hidden reward scaffolds rather than imitation targets. ExpRL defines a Stage-I RL objective
$$
\max_\theta \; \mathbb{E}_{(x,y^\star)\sim\mathcal{D}_\text{mid}}
\left[
\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)} \big[ R(x,y,y^\star) \big]
- \beta\,\mathrm{KL}\!\left(\pi_\theta(\cdot\mid x)\,\|\,\pi_0(\cdot\mid x)\right)
\right],
$$
where the policy never sees the reference during generation; an LLM judge uses the reference to assign dense outcome-level or process-level rewards. After this Stage-I RL priming, standard sparse-reward RL is applied. On held-out math benchmarks, Stage-II sparse-reward RL initialized from ExpRL is stronger than initialization from sparse-reward GRPO, SFT, or self-distillation; for example, AIME25 pass@1 reaches **59.07** for ExpRL-Outcome versus **55.99** for GRPO, and AIME26 pass@1 reaches **63.41** for ExpRL-Process [2606.17024].

A third family explicitly defines “reinforcement mid-training” as a hybrid objective over large unlabeled corpora. RMT partitions a token sequence $\mathcal S=\{\tau_1,\tau_2,\ldots,\tau_L\}$ into an RL subset $\Phi_{\text{RL}}$ and an NTP subset $\Phi_{\text{NTP}}=\mathcal S\setminus\Phi_{\text{RL}}$, with $|\Phi_{\text{NTP}}|\gg |\Phi_{\text{RL}}|$. It couples a dynamic token budget, curriculum-based adaptive sampling, token-selective GRPO, and masked NTP:
$$
\mathcal L = \mathcal L_{\text{RL}}(\theta) + \lambda \cdot \mathcal L_{\text{NTP}}(\theta).
$$
RMT reports up to **+64.91%** performance improvement with only **21%** of the reasoning length in language modeling, and checkpoints after reinforcement mid-training improve subsequent post-training by up to **+18.76%** in the mathematical domain [2509.24375].

## 4. Diversity, curriculum, and rollout-control interventions

A recurring thesis is that RL benefits from broader pre-RL support over valid reasoning trajectories. One approach is to generate multiple correct solution variants before RL. In a study based on George Pólya’s heuristics, each question is paired with multiple self-generated correct traces, and the mid-training objective averages the supervised log-likelihood over those variants. The resulting models show stronger downstream RL than vanilla RL and STaR+RL; the average pass@64 rises to **48.09** at \(n=16\) variants, versus **44.21** for vanilla RL and **45.69** for STaR+RL, and the same study reports gains on HumanEval and MuSR [2605.08472]. The paper’s theoretical interpretation is that multimodal pre-RL token distributions are less collapse-prone and more responsive to policy-gradient updates.

A second line of work alters the effective RL regime by modifying rollout prompting rather than the optimizer. Mid-Think identifies token-level triggers such as a leading “Okay” and the newline pattern after `</think>` as the main drivers of Think/No-think mode switching. It composes the suppressive and activating cues in a single template,
`<think>\n\n</think>\n\n<reason>\nOkay...`,
to induce intermediate-budget reasoning during RL after SFT. In Qwen3-8B post-SFT RL with GRPO and the verl framework, this reduces training time from **54 hours** to **46 hours** while improving final Think-mode evaluation from **69.8%** to **72.4%** on AIME and from **58.5%** to **61.1%** on GPQA [2601.07036]. The paper is explicit that this is not a new RL objective; it is a rollout-format intervention that changes exploration, entropy, and trajectory length.

Bridge methods blur the line between mid-training and post-training even further. DYPO is framed as post-training, but it operationally resembles RL-based mid-training because it replaces the strict SFT-then-RL boundary with unified optimization, dynamic routing by rollout difficulty, multi-teacher supervision on all-fail cases, and a bounded contrastive Group Alignment Loss on mixed-success groups. It reports an average improvement of **4.8%** on complex reasoning benchmarks and **13.3%** on out-of-distribution tasks over traditional sequential pipelines [2604.08926]. This suggests that some of the design logic of RL-based mid-training—competence-aware mixing, curriculum, and bounded exploratory updates—may be realized either before or during the nominal post-training phase.

## 5. Systems, stability, and training dynamics

As RL-based mid-training moves toward longer trajectories and larger clusters, systems questions become first-order. SortedRL targets the rollout bottleneck in long-chain-of-thought RL, where rollouts can consume **70%** of total training time at **16K** maximum generation length. It reorders rollouts online by completion length, enabling large rollout batches, flexible update batches, and a near on-policy micro-curriculum. The paper reports bubble-ratio reductions of **over 50%** and performance gains of **3.9% to 18.4%** over baseline given the same amount of data [2603.23414].

A related stability problem is training–inference mismatch during long RL runs. Beyond Precision argues that mismatch is not merely a static precision artifact but a dynamic optimization failure that escalates with gradient noise and response length. The proposed remedy is a response-length-triggered learning-rate scheduler, with the practical heuristic
$$
T_{\text{decay}} \approx 1.8 \times (\text{surge onset step}),
$$
followed by repeated halving down to a floor. On Qwen3-4B-Base and Qwen3-8B-Base, this suppresses mismatch and stabilizes RL more reliably than importance-sampling corrections alone [2602.01826].

At cluster scale, MindSpeed RL treats RL training as a distributed dataflow problem with two core flows: sample flow and resharding flow. Its distributed transfer dock replaces centralized replay-buffer dispatch with controllers and warehouses, and its allgather–swap strategy reduces redundant memory during transitions between update-time and generation-time sharding layouts. Across Qwen2.5-Dense-7B/32B, Qwen3-MoE-30B, and DeepSeek-R1-MoE-671B, the framework reports throughput gains of **1.42 ~ 3.97 times** on a **384**-NPU Ascend superpod [2507.19017].

Long-horizon agent settings introduce a different systems-and-optimization pathology: the environment itself becomes rollout-dependent. Memory-R2 shows that in multi-session memory-augmented agents, different rollouts inherit different memory states, so standard group-relative comparisons become unfair. Its LoGo-GRPO combines global trajectory rewards with local rerollouts from matched intermediate memory states, and a progressive curriculum from **8** to **16** to **32** sessions. On LoCoMo, LoGo-GRPO improves F1 from **46.62** to **49.67** and reduces M-Fail from **10.20%** to **6.72%** relative to plain GRPO [2605.21768]. This broadens RL-based mid-training from reasoning traces to persistent-state agents.

## 6. Limitations, retention, and open questions

The literature remains fragmented. The survey record still treats RL-based mid-training largely as a boundary case rather than a mature taxonomy, and many method papers occupy narrow empirical regimes—reasoning-heavy corpora, specific model families, or relatively small scales [2510.23081]. This suggests that the field has not yet converged on a single formal definition or a standardized benchmark suite.

One open problem is retention under later post-training. A value-oriented study shows that values instilled during a compassion-oriented mid-training stage can be differentially degraded by later SFT or GRPO depending on the domain of post-training data. On the Animal Harm Benchmark, helpfulness-domain SFT scores **35.7%** versus **65.2%** for coding-domain SFT, and helpfulness-domain GRPO scores **18.7%** versus **32.0%** for coding-domain GRPO [2606.26102]. A plausible implication is that RL-based mid-training cannot be evaluated independently of the later RL or post-training stage that may overwrite it.

A related concern appears in cross-domain agent training. An SFT warmup or mid-training phase before RL helps prevent catastrophic forgetting for domains included in its datamix, but undermines generalization to domains that are not included. In the reported setup, domains covered by the warmup degrade much less under later RL, while the excluded Sokoban domain degrades substantially more [2601.18217]. This establishes a central tension: intermediate shaping can preserve covered competencies yet narrow open-ended transfer.

The larger open questions are therefore structural rather than incremental. What should count as RL-based mid-training under a unified taxonomy? Which signals are best injected at the intermediate stage—online rewards, frozen RL priors, token budgets, diverse self-generated trajectories, or prompt-level rollout control? When does intermediate RL enlarge a model’s coverage over productive reasoning paths, and when does it merely specialize or destabilize the model? Current results show that the stage can improve efficiency, raise downstream RL ceilings, and make later post-training easier; they also show that the benefits depend heavily on data composition, reward design, rollout formatting, and the precise capabilities one wishes to preserve.

Source: https://www.emergentmind.com/topics/rl-based-mid-training