Scaling Agents via Continual Pre-training (2509.13310v1)

Abstract: LLMs have evolved into agentic systems capable of autonomous tool use and multi-step reasoning for complex problem-solving. However, post-training approaches building upon general-purpose foundation models consistently underperform in agentic tasks, particularly in open-source implementations. We identify the root cause: the absence of robust agentic foundation models forces models during post-training to simultaneously learn diverse agentic behaviors while aligning them to expert demonstrations, thereby creating fundamental optimization tensions. To this end, we are the first to propose incorporating Agentic Continual Pre-training (Agentic CPT) into the deep research agents training pipeline to build powerful agentic foundational models. Based on this approach, we develop a deep research agent model named AgentFounder. We evaluate our AgentFounder-30B on 10 benchmarks and achieve state-of-the-art performance while retains strong tool-use ability, notably 39.9% on BrowseComp-en, 43.3% on BrowseComp-zh, and 31.5% Pass@1 on HLE.

• The paper demonstrates that introducing an Agentic CPT layer significantly improves agentic behaviors and overcomes post-training optimization conflicts.
• The methodology uses First-order and High-order Action Synthesis to generate diverse, scalable agentic data for improved training efficiency.
• Empirical results show that AgentFounder-30B outperforms existing agents with state-of-the-art tool-use, long-context reasoning, and robust generalization.

Scaling Agentic LLMs via Continual Pre-training

Introduction and Motivation

The paper "Scaling Agents via Continual Pre-training" (2509.13310) addresses a critical bottleneck in the development of agentic LLMs: the inability of post-training methods (SFT, RL) to endow models with robust, generalizable agentic behaviors, especially in open-source settings. The authors identify that post-training on general-purpose foundation models forces simultaneous acquisition of agentic capabilities and alignment, leading to optimization conflicts and suboptimal performance in complex, tool-augmented, multi-step reasoning tasks. To resolve this, the paper introduces Agentic Continual Pre-training (Agentic CPT) as an intermediate scaling layer, producing pre-aligned agentic foundation models that facilitate more effective downstream fine-tuning.

Agentic Continual Pre-training: Pipeline and Data Synthesis

The proposed agentic training pipeline augments the standard pre-training $\rightarrow$ post-training paradigm by inserting an Agentic CPT stage between them. This stage is designed to imbue the model with agentic inductive biases and behavioral priors before any alignment or task-specific fine-tuning.

Figure 1: Agentic Training Pipeline.

Data Synthesis: First-Order and High-Order Action Synthesis

Agentic CPT leverages two scalable data synthesis methodologies:

• First-order Action Synthesis (FAS): Generates (question, planning, action) tuples from diverse, domain-agnostic knowledge sources. FAS includes:
  • Knowledge-to-Question Transformation: Converts unstructured web and factual data into entity-anchored open-world memory, then synthesizes multi-style questions (factual, numerical, multi-hop, synthesis) to simulate real-world agentic scenarios.
  • 

  • Figure 2: Multi-Style Question-Answer Generation Based on Scalable Information Sources.

  • Planning Action Synthesis: For each question, LLMs generate diverse problem decompositions and first-step action predictions (tool calls or direct answers) without incurring API costs. Diversity is enhanced by generating actions for multiple questions sharing the same knowledge context.

    Figure 3: Planning Action Synthesis.

  • Reasoning Action Synthesis: LLMs decompose questions into sub-questions, generate speculative answers, and then refine them using mapped knowledge, simulating logical deduction without tool invocation.
• High-order Action Synthesis (HAS): Reuses and augments post-training trajectories by expanding the action space at each step. For each trajectory step, LLMs generate $N$ alternative reasoning-action candidates, transforming the original trajectory into a decision process with explicit feedback, thus enabling step-wise decision learning rather than mere trajectory imitation.

  Figure 4: Comparison of high-order action synthesis data and the original trajectory.

Progressive Two-Stage Training

Agentic CPT is implemented in two stages:

• Stage 1: 200B tokens of FAS and short HAS data with 32K context, focusing on basic agentic behaviors.
• Stage 2: 100B tokens of high-quality HAS data with 128K context, targeting long-horizon planning and complex action spaces.

Empirical Results: Performance, Scaling, and Analysis

State-of-the-Art Performance

The resulting model, AgentFounder-30B, is evaluated on 10 benchmarks, including BrowseComp-en/zh, GAIA, HLE, and xbench-DeepSearch. It achieves 39.9% on BrowseComp-en, 43.3% on BrowseComp-zh, 72.8% on GAIA, 31.5% on HLE, and 73.0% on xbench-DeepSearch, surpassing all open-source and several closed-source commercial agents.

Figure 5: Performance comparison between AgentFounder and state-of-the-art deep research agents.

Scaling Laws and Data Efficiency

Agentic CPT exhibits strong scaling law behavior with respect to both model size and data volume. Performance increases logarithmically with training tokens, and larger models benefit more from agentic pre-training than from naive scaling alone. Notably, AgentFounder-30B outperforms larger models such as DeepSeek-V3.1 and Kimi-K2, indicating superior scaling efficiency.

Figure 6: Scaling Law Exploration for Agentic Capabilities. (a) Model size scaling. (b) Data volume scaling.

Training Dynamics and Loss Convergence

Agentic CPT substantially improves fine-tuning efficiency, as evidenced by lower SFT loss and faster convergence compared to baseline models. The monotonic decrease in loss with increased CPT data volume and the benefit of mixing FAS and HAS data are empirically validated.

Figure 7: Training loss evolution showing superior convergence of AgentFounder models compared to baseline.

Tool Use and Behavioral Adaptation

AgentFounder demonstrates adaptive tool usage strategies, with heavy-tailed tool call distributions for complex research tasks and conservative usage for structured tasks. This indicates learned calibration of exploration depth based on task complexity.

Figure 8: Tool call distribution comparison.

Data Quality and Filtering

The FAS data generation pipeline incorporates weakly supervised filtering, raising retained trajectory accuracy from 50% to 82% by removing semantically inconsistent or logically discontinuous samples.

Figure 9: Filtering performance and representative low-quality outputs for weakly supervised filtering in first-order action synthesis.

Solution Diversity and Robustness

Pass@N metrics on BrowseComp-en show that AgentFounder maintains high solution diversity, with Pass@16 reaching 75.8%, indicating robust exploration of the solution space.

Figure 10: Pass@N Scaling on BrowseComp-en.

Task Difficulty and Generalization

Performance on GAIA degrades with increasing task difficulty, but remains competitive even at the highest levels, demonstrating generalization beyond retrieval to complex reasoning.

Figure 11: Pass rate on different levels of the GAIA dataset.

MoE Utilization and Long-Context Stability

MoE router activations become more balanced after CPT, reducing expert collapse and improving training stability for long-context tasks.

Figure 12: MoE activations illustration on BrowseComp-zh dataset. Top-Middle-Bottom: Baseline-Our model-Difference.

Tool Call Efficiency and Success Rate

Analysis of accuracy versus tool call turns reveals that efficient planning (fewer tool calls) correlates with higher success rates, but the model maintains non-trivial accuracy even in high-turn, complex scenarios.

Figure 13: Accuracy distribution across tool call turns for BrowseComp-en, BrowseComp-zh, GAIA, and Xbench-DeepResearch, where darker colors indicate the number of tools used.

Implications and Future Directions

The introduction of Agentic CPT as an intermediate scaling layer fundamentally alters the agentic LLM training paradigm. By decoupling the acquisition of agentic behaviors from alignment, the approach resolves optimization conflicts inherent in post-training-only pipelines. The demonstrated scaling laws suggest that agentic capabilities can be efficiently acquired and transferred, and that data synthesis strategies (FAS, HAS) are critical for sample efficiency and behavioral diversity.

Practically, AgentFounder-30B establishes a new open-source SOTA for deep research agents, with robust tool-use, long-context reasoning, and generalization across domains. The methodology is extensible to multi-agent and multi-modal settings, and the data synthesis pipeline is compatible with offline, large-scale generation, reducing reliance on expensive APIs.

Theoretically, the work motivates further paper of agentic inductive biases, the interplay between pre-training and alignment, and the design of data curricula for emergent agentic behaviors. The observed improvements in MoE utilization and long-context stability also suggest architectural implications for future agentic LLMs.

Conclusion

"Scaling Agents via Continual Pre-training" demonstrates that agentic continual pre-training is a critical missing layer for scaling agentic LLMs. By synthesizing diverse, scalable agentic data and employing a progressive training strategy, the approach yields models that outperform both open-source and commercial baselines on a wide range of deep research tasks. The results highlight the necessity of pre-aligned agentic foundation models and provide a blueprint for future research in scalable, general-purpose agentic systems.