Agent Foundation Model Training

• Agent foundation model training is the process of developing large-scale, Transformer-based neural policies that empower agents with reasoning, tool use, and autonomous decision-making.
• It integrates supervised multitask learning, reinforcement learning, and expert distillation to optimize agent behaviors across domains like robotics, web interaction, and healthcare.
• Key strategies include multi-modal data curation, synthetic augmentation, and a blend of cross-entropy and RL-based losses that drive robust performance in complex, interactive settings.

Agent foundation model training refers to the process of instantiating and tuning large-scale neural policies—often based on Transformer architectures—to serve as generalist agents capable of reasoning, goal-directed behavior, tool use, and robust performance across multiple interactive domains. This paradigm builds on the foundation model concept, extending it to the agentic setting where models are not simple predictors but autonomous problem-solvers interfacing with complex environments. Training agent foundation models (AFMs) subsumes approaches from supervised multitask learning and behavior cloning, hierarchical and curriculum reinforcement learning, imitation from multi-agent systems, and multi-modal transfer with expert knowledge distillation.

1. Architectural and Algorithmic Foundations

Agent foundation models are typically constructed as sequence models that ingest mixed-modality inputs—natural language, visual observations, action traces, and occasionally structured environment feedback—and autoregressively predict either the next action, subpolicy, or full agent output at each interaction step. Prominent architectural motifs include:

• Modular Transformer-based policies: Integrating visual encoders (e.g., CLIP ViT variants), action transformers (OPT, Qwen, LLaMA), language adapters, and multi-modal cross-attention layers. Examples include the multimodal transformer in "An Interactive Agent Foundation Model" (Durante et al., 8 Feb 2024), "CPathAgent" (Sun et al., 26 May 2025), and agentic variants based on Qwen3, LLaVA, or BERT/BLIP-style encoders.
• Multi-agent orchestration: Single models simulating or actually orchestrating multiple roles (e.g., planning, tool selection, tool execution), as in "Chain-of-Agents" (Li et al., 6 Aug 2025), "Cognitive Kernel-Pro" (Fang et al., 1 Aug 2025), and multi-agent RL distillation frameworks.
• Foundation model inheritance and adaptation: Starting from generalist pre-trained models and further adapting via domain-specific multitask objectives, hierarchical loss weighting, and RL with reward shaping.

The training paradigms unify a spectrum of learning objectives:

• Supervised learning: Masked language modeling, next-action prediction, or multi-agent behavior cloning across diverse trajectory datasets.
• Reinforcement learning: Proximal Policy Optimization (PPO), GRPO, DAPO, or step-wise RL methods, applying on-policy gradient optimization in goal-conditioned MDPs.
• Multi-agent distillation and transfer: Knowledge distillation from heterogeneous expert models or multi-agent systems via weighted loss functions and trajectory masking.

2. Data Curation, Synthetic Augmentation, and Preprocessing

AFM training hinges on the assembly and preprocessing of large, high-quality, and multi-domain datasets. Key strategies and practices include:

• Aggregation of agentic trajectories from diverse environments: Robotics (Language-Table, CALVIN), gaming (Minecraft, Bleeding Edge), web interaction (WebVoyager, WebArena), physiological data (ICU video), and domain-specialized QA or coding benchmarks (Durante et al., 8 Feb 2024, Zhou et al., 17 Dec 2024, Fang et al., 1 Aug 2025, Li et al., 6 Aug 2025, Sun et al., 26 May 2025).
• Data homogenization and balancing: Standardizing inputs into unified formats (prompt, action/observation pairs, tool calls), applying domain- and task-weighted loss normalization, and carefully balancing data across domains for robust generalization.
• Synthetic data generation: Automated trajectory synthesis using powerful generative agents or LLMs. For instance, LightAgent’s GUI data uses Gemini-2.5-pro to generate CoT explanations and Qwen3-32B to label function calls (Jiang et al., 24 Oct 2025); Cognitive Kernel-Pro generates web and file reasoning trajectories with model-based explorer agents (Fang et al., 1 Aug 2025).
• Negative sampling and hallucination prevention: Inclusion of negative examples to mitigate tool-use and format hallucinations during tuning (Chen et al., 19 Mar 2024).

A typical data pipeline involves iterative agent/teacher rollouts, rejection sampling, data cleaning (label consistency, class balancing), and augmentation (color jittering for vision, CoT injection for language, backtranslation or synonym replacement for text).

3. Training Objectives, Losses, and Optimization Strategies

Agent foundation model training leverages a blend of cross-entropy, distillation, and RL-based losses:

• Unified multitask and multimodal objectives: For example, the Interactive Agent Foundation Model combines masked image reconstruction, causal language modeling, and next-action prediction with equal weighting, normalized by token count (Durante et al., 8 Feb 2024). A general abstract formula:

$$L(S) = \frac{L_{\text{lang}}(S) + L_{\text{MAE}}(S) + L_{\text{act}}(S)}{|W| + \sum_{t=1}^T(|V_t| + |A_t|)}$$

• Capability-decomposed and weighted losses: Agent-FLAN decomposes data into instruction-following, reasoning, retrieval, and understanding, applying empirically determined weights (e.g., w_R : w_U : w_Ret : w_IF ≈ 1.0 : 0.75 : 0.25 : 0.1) and constructing the total loss as a weighted sum (Chen et al., 19 Mar 2024).
• Distillation and expert trajectory masking: Multi-agent distillation losses are often used, with explicit masking of tool output tokens to avoid learning artifacts, as in Chain-of-Agents and Cognitive Kernel-Pro (Li et al., 6 Aug 2025, Fang et al., 1 Aug 2025).
• RL-based policy optimization: AFM RL fine-tuning is cast in terms of episode-level or step-level return maximization. Prototypical actor-critic objectives are $$J(\theta) = \mathbb{E}_{\pi_\theta}\left[\sum_{t=0}^{T}\gamma^t r_t\right]$$ with clipped PPO or DAPO surrogate losses. Chain-of-Agents and Agent-R1 both use trajectory-level and token-level masks in RL gradients to handle partial credit assignment and interface with external tools (Cheng et al., 18 Nov 2025).
• Domain-specific or task-specific reward shaping: For instance, ML-Agent translates feedback from ML experiments (error, performance, or OOM) into consistent scalar rewards (Liu et al., 29 May 2025).
• Difficulty-aware, label-guided optimization: Agentar-Fin-R1 and related works use automated weight estimation and attribution systems based on pass@k difficulty measures to direct more computation to challenging tasks (Zheng et al., 22 Jul 2025).

4. System Implementations, Infrastructure, and Orchestration

Advanced AFM training systems typically integrate modular, distributed infrastructures:

• Multi-agent frameworks: Orchestrating agents for planning, data processing, model training, and deployment as disaggregated services (TrainerAgent (Li et al., 2023), Cognitive Kernel-Pro (Fang et al., 1 Aug 2025)).
• Agent observability and standardized finetuning interfaces: Through OpenTelemetry-based tracing and OpenAI-compatible endpoints, as in Agent Lightning, enabling arbitrary agent codebases to be wrapped for RL without code modification (Luo et al., 5 Aug 2025).
• Device-cloud orchestration for deployment constraints: LightAgent assigns tasks on-device or offloads to the cloud in real-time, leveraging complexity assessment functions for mobile efficiency (Jiang et al., 24 Oct 2025).
• Test-time reflection and ensemble voting: Cognitive Kernel-Pro performs LLM-based trajectory reflection and N-run voting to enhance robustness and amplify answer reliability; this approach improves pass@1 by ~6% absolute (Fang et al., 1 Aug 2025).

5. Benchmarking, Evaluation, and Empirical Insights

AFMs are validated on a suite of demanding agentic benchmarks:

• Web and GUI interaction: GAIA, WebVoyager, WebArena, AndroidLab.
• Tool-augmented QA and code reasoning: HotpotQA, 2WikiMultihopQA, LiveCodeBench.
• Financial reasoning: Fineva, FinEval, FinanceIQ, Finova.
• Healthcare, robotics, and vision-language transfer: PathMMU-HR², RASS, CALVIN.

Common evaluation metrics include exact match (EM), pass@k, BLEU-4 (action prediction), balanced accuracy, FID (generative quality), and ablation comparisons to non-agentic baselines or non-RL variants.

Empirically, practices such as step-wise RL, multi-agent distillation, reflection voting, and label-guided weighting substantially improve generalization and stability versus monolithic or pipeline approaches. Noteworthy findings include:

• Agent-FLAN yields >3.5% overall gain in pass@k; negative samples reduce hallucination rates from ~15% to <10% (Chen et al., 19 Mar 2024).
• ML-Agent’s 7B policy surpasses a 671B DeepSeek-R1 model using step-level RL and exploration-enriched SFT (Liu et al., 29 May 2025).
• Cognitive Kernel-Pro’s CK-Pro-8B model outperforms previous free 7B agents by >6 pp on GAIA text-only pass@1 (Fang et al., 1 Aug 2025).
• Reflection and voting yield 6–15 pp gain in test accuracy/pass@k (Fang et al., 1 Aug 2025).
• IM-based context-aware task proposals and filtered behavior cloning (e.g., PAE (Zhou et al., 17 Dec 2024)) facilitate OOD transfer and new skill discovery.

6. Open Problems and Design Insights

Critical unresolved challenges and design lessons include:

• Scalable and transparent credit assignment: Existing RL approaches (e.g., PPO, GRPO) mostly use identical or naive token credit; long-horizon environments demand more nuanced or learnable credit mechanisms (Luo et al., 5 Aug 2025, Cheng et al., 18 Nov 2025).
• Generalist vs. specialist tradeoffs: Multidomain AFMs (web, code, file, reasoning) achieve reasonable coverage but may still trail SOTA on hard in-domain tasks; two-stage pipelines (broad SFT then targeted RL/SFT) mitigate this (Zheng et al., 22 Jul 2025).
• Hallucination, overfitting, and action safety: Negative sampling, dialogue-aligned corpora, and explicit reflection blocks are effective, but open-ended tool use remains brittle without dynamic assessment (Chen et al., 19 Mar 2024, Fang et al., 1 Aug 2025).
• Inference and deployment constraints: Efficient summarization and memory, decoupled device-cloud orchestration, and LoRA or adapter tuning are key for practical AFM deployment (Jiang et al., 24 Oct 2025).
• Trustworthiness and data governance: Three-layer frameworks for domain knowledge curation, multi-agent synthesis, and automated validation are essential for regulated domains (e.g., finance, healthcare) (Zheng et al., 22 Jul 2025).

The field is actively exploring integration with dynamic curriculum learning, formal-reasoning augmentations, advanced off-policy and exploration methods, and more interpretable behavior architectures. The modularity and scalability of current AFM frameworks imply easy transfer to new domains, as demonstrated by domain-adapted variants across finance, medicine, robotics, and web interaction.

---

• (Durante et al., 8 Feb 2024) An Interactive Agent Foundation Model
• (Chen et al., 19 Mar 2024) Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for LLMs
• (Li et al., 2023) TrainerAgent: Customizable and Efficient Model Training through LLM-Powered Multi-Agent System
• (Zhou et al., 17 Dec 2024) Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents
• (Fang et al., 1 Aug 2025) Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training
• (Luo et al., 5 Aug 2025) Agent Lightning: Train ANY AI Agents with Reinforcement Learning
• (Liu et al., 29 May 2025) ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering
• (Cheng et al., 18 Nov 2025) Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning
• (Sun et al., 26 May 2025) CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic
• (Zheng et al., 22 Jul 2025) Agentar-Fin-R1: Enhancing Financial Intelligence through Domain Expertise, Training Efficiency, and Advanced Reasoning
• (Guo et al., 16 Oct 2024) TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration
• (Zeng et al., 17 Sep 2025) Behavior Foundation Model for Humanoid Robots
• (Li et al., 6 Aug 2025) Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL
• (Jiang et al., 24 Oct 2025) LightAgent: Mobile Agentic Foundation Models
• (Bhattacharjya et al., 2 Feb 2024) Foundation Model Sherpas: Guiding Foundation Models through Knowledge and Reasoning