---
title: 'Agentic Mid-training: Bridging Imitation and RL'
url: https://www.emergentmind.com/topics/agentic-mid-training-9c3d9472-15ac-480f-a4db-06347d46aa03
type: topic
---

# Agentic Mid-training: Bridging Imitation and RL

Agentic mid-training is a paradigm and methodology designed to bridge the gap between language modeling, imitation learning, and fully agentic reinforcement learning (RL) for large language models (LLMs) and agentic systems. By abstracting high-level actions and learning temporally-extended behaviors through mid-stage training protocols, agentic mid-training enables models to acquire compact and generalizable action spaces, facilitating robust online RL and accelerating the acquisition of complex reasoning, planning, and tool-use capabilities.

## 1. Formal Foundations and Motivation

Agentic mid-training formalizes the transition from sequence-level imitation to action-abstraction-driven RL. Tasks are modeled as Markov Decision Processes (MDPs) $M = (S, A, R, \gamma)$, where $S$ denotes the state space (e.g., code prefixes), $A$ the atomic action space (e.g., token selections), $R$ is the reward, and $\gamma$ the discount factor. Standard next-token prediction (NTP) corresponds to imitation learning:

\[
J_{\mathrm{NTP}}(\pi) = \mathbb{E}_{(s_{0:T}, a_{0:T}) \sim D_E}\left[ \sum_{t=0}^T \log \pi(a_t | s_t) \right]
\]

Agentic mid-training departs from pure NTP by introducing a compact, temporally-extended action space $Z$, where each $z \in Z$ may implement subpolicies or macro-actions spanning multiple atomic steps (duration $\tau$). The objective is to select a minimal yet sufficient subset $\widehat{Z} \subset Z$ such that subsequent online RL can operate effectively within this reduced space, facilitating policy optimization over options, rationales, or latent abstractions rather than primitive tokens [2509.25810].

## 2. Theoretical Characterization of Performance

The efficacy of agentic mid-training is characterized by its impact on regret decomposition following post-training RL:

\[
\text{Regret}(\pi) = \mathbb{E}_{M \sim p(M)}[V_M^*(s_0) - V_M^\pi(s_0)]
\]
\[
\mathbb{E}[V^* - V^\pi] = \mathbb{E}[\Delta(M, Z')] + \mathbb{E}[V^*_{M_{Z'}} - V^\pi_{M_{Z'}}]
\]
where $\Delta(M, Z') = V^*_M(s_0) - V^*_{M_{Z'}}(s_0)$ quantifies the value-approximation (pruning) error induced by restricting actions to $Z'$, and the second term is the RL error within the pruned space.

Key formal results:
- The minimal $\varepsilon$-optimal $Z'$ controls sample complexity: with $|D_E| = \Theta(|\overline{Z}| \log(|Z|/\delta)/\sigma)$ expert demonstrations, all $(\varepsilon, \sigma)$-suboptimal actions are pruned with probability $1-\delta$, yielding $\mathbb{E}[\Delta(M, \widehat{Z})]\leq \varepsilon$.
- RL convergence in the abstracted MDP $M_{\widehat{Z}}$ is governed by effective discount $\overline{\gamma} = \sup_{s,z} \mathbb{E}[\gamma^\tau|s,z]$, with contraction rate $(1-\overline{\gamma})$ and sample complexity $N \geq \frac{1}{1-\overline{\gamma}} \ln\left(\frac{R_{\max}}{\varepsilon(1-\gamma)}\right)$ [2509.25810].

These results demonstrate that pruning efficiency ($|\overline{Z}|$) and horizon-shortening via longer-temporal abstractions ($\overline{\gamma} \ll \gamma$) synergistically reduce expert data and RL updates.

## 3. Agentic Mid-Training Methodologies and Algorithms

### 3.1 Latent-Option Extraction and RA3

The Reasoning as Action Abstractions (RA3) algorithm exemplifies agentic mid-training [2509.25810]. RA3 frames mid-training as an EM algorithm alternating between:

- **E-Step (RL Bootstrap):** Discover temporally-consistent latent variables $z_{0:T}$ by solving an RL problem over expert trajectories with reward $r_t = \log \pi_i(a_t|s_t, z_t) - c \cdot \mathbf{1}[z_t \neq \langle act \rangle]$, where $c$ penalizes unnecessary rationale switches.
- **M-Step (Supervised Fine-tuning):** Optimize $\pi_{i+1}$ via next-token prediction on bootstrapped $(s_t, z_t, a_t)$ trajectories.

This process is underpinned by the temporal ELBO:

\[
J_{\mathrm{NTP}}(\pi) \geq J(\pi, q) = \mathbb{E}\left[ \sum_{t=0}^T \log \pi(a_t | s_t, z_t) - \mathrm{KL}[q(z_t | s_t, z_{0:t-1})\, ||\, p(z_t | s_t, z_{0:t-1})] \right]
\]

The prior $p(z_t | s_t, z_{t-1}) = \alpha \delta(z_{t-1}) + (1-\alpha)\mathrm{Uniform}(Z)$ encourages persistence in high-level reasoning states.

### 3.2 Data-Centric and RL Approaches

Empirical recipes for agentic mid-training, as in [2510.11701], stress the importance of initializing supervised fine-tuning (SFT) with real, end-to-end agentic trajectories (e.g., tool-use with verified recovery and reflection) over synthetic stitched data. High-diversity, model-aware RL datasets—stratifying by difficulty and domain—sustain exploration and accelerate RL-driven refinement using GRPO/PPO-style objectives with exploration-encouraging entropy and shaped rewards.

### 3.3 Curriculum and Massive-Scale Pipelines

In curriculum-driven setups, stagewise data distribution shifts, as in Youtu-LLM, employ initial commonsense/STEM pre-training, followed by an agentic mid-training phase dominated by structured, multi-domain agentic trajectories (e.g., planning, code, tool use), leveraging masking and input-formatting strategies to prevent noise propagation. This enables even lightweight LLMs to internalize high-level agentic schemas [2512.24618].

## 4. Data Construction and Action Abstraction Granularity

Agentic mid-training critically depends on constructing action/trajectory spaces that balance expressivity and compactness:

- **Action abstraction:** Temporally-extended options, rationales, or macro-actions enable RL over skills rather than primitives, reducing the effective planning horizon and sample complexity [2509.25810].
- **Trajectory diversity:** Datasets span code, mathematics, deep research, tool-use, and reflection, formatted as XML/JSON segments with structured stages (e.g., <Analysis>, <Plan>, <Action>, <Reflection>, <Summary>). Data may be generated by strong teacher models or via adversarial User/Assistant LLMs with rigorous error-checking and negative augmentation [2512.24618].
- **Abstraction control:** Hyperparameters such as the KL penalty $c$ (or prior persistence $\alpha$) provide a direct handle for tuning granularity—higher $c$ biases toward fewer, longer options; lower $c$ permits finer-grained reasoning [2509.25810].

## 5. Empirical Effects, Benchmarks, and Architectural Integration

Agentic mid-training consistently yields substantial performance and efficiency improvements across multiple agentic domains:

- **Code generation:** RA3 improves average pass@1 by ≈4–8 points over NTP and base models on HumanEval, MBPP, and derived datasets; cross-entropy loss is also consistently lower [2509.25810].
- **Reasoning benchmarks:** With high-diversity data and calibrated RL protocols, compact models (4B) achieve >70% average@32 on AIME24/25, rivaling or exceeding 32B-parameter agents [2510.11701].
- **Long-context and lightweight models:** Agentic mid-training (e.g., 200B tokens on Youtu-LLM) produces up to +42.7% improvement on SWE-Bench-Verified (k=1), +13.7% on APTBench, and enables strong planning, reflection, and tool-use even in sub-2B models [2512.24618].
- **Architectural adaptation:** Dense multi-latent attention (MLA), XL-context support (128k+), and STEM-oriented tokenizers are used in conjunction with mid-training, with masking and prefix-sharing (tree training) to minimize computational overhead [2512.24618], [2511.00413].

Empirical results support the sufficiency of relatively small expert datasets, provided the action space is aggressively abstracted and pruned, and the RL curriculum exploits domain-specific granularity.

## 6. Integration with RL Pipelines and Infrastructure

Agentic mid-training is increasingly realized as part of large-scale distributed training architectures, exemplified by frameworks such as AWorld [2508.20404]:

- **System design:** Two-tiered orchestration with high-concurrency rollout-executors (sandboxed agent/environment pairs) and separate training clusters, enabling near-linear scaling to 16-32 parallel pods.
- **Distributed RL:** Experience collection and online RL are decoupled; rollouts are streamed, and rewards (including sparse and stepwise variants) are computed asynchronously.
- **Practical workflow:** Mid-training is initiated with SFT on real trajectories, followed by GRPO/PPO-based RL, with careful tuning of batch sizes, learning rates, and exploration strategies.

Such infrastructure is critical for rendering mid-training tractable at the scale required for highly agentic benchmarks (GAIA, LiveCodeBench, etc.).

## 7. Practical Guidelines and Open Directions

Standardized best practices for agentic mid-training include:
- Begin with real, multi-turn agentic SFT data; avoid exclusively synthetic "stitched" trajectories [2510.11701].
- Employ curriculum learning, with mid- and late-training data emphasizing agentic and high-variance trajectories.
- Employ action abstraction and pruning to compactify the decision space; tune abstraction granularity with hyperparameters (KL penalty, $\alpha$).
- Integrate tree training to reuse shared prefixes for efficiency [2511.00413].
- In multi-domain setups, use per-domain RL specialization, model merge (SCE), or joint RL with schedule balancing [2511.05951].
- Monitor policy entropy and use exploration-enhancing PPO/GRPO schedules.

Emerging directions include partial cross-GPU shared-prefix reuse, more adaptive abstraction-discovery protocols, and refined environment-aware reward modeling.

---

**Key References:**

- "Learning to Reason as Action Abstractions with Scalable Mid-Training RL" [2509.25810]
- "Demystifying Reinforcement Learning in Agentic Reasoning" [2510.11701]
- "AWorld: Orchestrating the Training Recipe for Agentic AI" [2508.20404]
- "Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models" [2512.24618]
- "Tree Training: Accelerating Agentic LLMs Training via Shared Prefix Reuse" [2511.00413]
- "Agent Fine-tuning through Distillation for Domain-specific LLMs in Microdomains" [2510.00482]
- "Klear-AgentForge: Forging Agentic Intelligence through Posttraining Scaling" [2511.05951]

Source: https://www.emergentmind.com/topics/agentic-mid-training-9c3d9472-15ac-480f-a4db-06347d46aa03