Agent Fine-Tuning Overview

• Agent fine-tuning is the process of adapting pretrained agentic models through supervised trajectories, domain data, and reward optimization to align behavior with specific tasks.
• It encompasses methods like hybrid instruction tuning, domain adaptation, and reinforcement fine-tuning to bridge gaps between general language modeling and practical agent performance.
• Successful fine-tuning relies on high-quality, well-curated data and tailored reward strategies to balance enhanced task execution with maintained general abilities.

Agent fine-tuning is the post-pretraining adaptation of an agentic model—most often a LLM used for planning, tool use, memory, reasoning, or interaction, but also learned policies for multi-agent control—through supervised trajectories, domain corpora, mixed instruction data, or closed-loop reward optimization so that behavior aligns with a target environment and task distribution. In current research, the term spans hybrid instruction-tuning of interaction traces, domain-specific supervised fine-tuning, reinforcement fine-tuning in simulators and GUI environments, and active correction of decentralized policies (Zeng et al., 2023, Chen et al., 2024, Pei et al., 28 Sep 2025, Andreychuk et al., 30 Jun 2025).

1. Definition and problem setting

In the LLM-agent literature, fine-tuning is motivated by a recurring gap between strong general language modeling and weak agent behavior. Open-source models are reported to lag behind larger commercial systems on planning, long-term memory, and tool utilization, even when prompting methods such as CoT or ReAct are available (Zhou et al., 2024, Zeng et al., 2023). Fine-tuning is used to internalize these behaviors into model parameters rather than carrying them entirely in prompts or external orchestration.

A second, distinct motivation arises in offline-to-online control and simulation. Here, an agent is first learned from logged data and then adapted online, but naive fine-tuning can induce severe early degradation because exploration overrides the offline policy. This problem is explicit in offline-to-online RL and in behavior models for autonomous driving, where distribution shift between supervised training and closed-loop deployment degrades reliability (Wang et al., 1 May 2025, Peng et al., 2024).

Across these strands, agent fine-tuning usually targets one or more of four goals. The first is capability acquisition, such as learning Thought/Action/Observation trajectories, tool calls, or decomposition strategies. The second is domain adaptation, as in protein production, JP1 middleware, neuron-kernel generation, and GUI control. The third is closed-loop robustness, where reward-driven updates correct rollout drift. The fourth is general-skill preservation, since many works explicitly try to improve agent behavior without compromising broader language ability (Zeng et al., 2023, Chen et al., 2024).

This scope also extends to multi-agent systems. In LLM-based multi-agent settings, fine-tuning must address asynchronous communication, role-conditioned prompts, dynamically changing dependencies, and token-sequence action spaces. In classical multi-agent control, fine-tuning may instead target coordination quality, realism, or safe exploration under deployment shift (Liao et al., 21 Apr 2025, Pei et al., 28 Sep 2025).

2. Supervised formulations and agent representations

The dominant supervised objective is next-token cross-entropy over agent trajectories, code traces, or structured outputs. Representative formulations include standard causal language modeling,

$$\mathcal{L}_{CE}(\theta) = -\sum_{t=1}^{T}\log p_\theta(x_t \mid x_{<t}, c),$$

and mixed-data objectives that interpolate agent-specific and general instruction data (Deng et al., 2024, Zeng et al., 2023). In domain-specialized code generation, NKI-Agent uses

$$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$

with prompts serialized from multi-turn conversations into standard causal language modeling examples (Tang et al., 5 Jul 2026).

Training data are usually structured as interaction trajectories rather than isolated question-answer pairs. AgentTuning’s AgentInstruct keeps sequences $(u_1,a_1,\dots,u_n,a_n)$ with rewards, while low-parameter agent tuning constructs three-role dialogues among a Question Generator, an Action Maker, and an Environment Agent (Zeng et al., 2023, Zhou et al., 2024). In microdomain adaptation for JP1 middleware, the target is explicitly agentic: GPT-4-generated ReAct and CoT traces in Japanese are fine-tuned so that procedural knowledge such as “check status → if unhealthy, restart service → verify logs” becomes encoded in the model parameters (Xue et al., 1 Oct 2025).

Architecturally, many systems preserve the base model and tune either all parameters or adapters. AgentTuning applies full-parameter fine-tuning to Llama 2-chat 7B, 13B, and 70B (Zeng et al., 2023). MIMIR exposes both full and LoRA modes in a one-click workflow and balances domain and general competencies through a mixed loss with default $\alpha=0.5$ (Deng et al., 2024). The 7B/13B agent-capability study uses LoRA with rank $r=8$, $\alpha=16$, and mixes $D_{\text{agent}}$ with $D_{\text{general}}$ at $\lambda \approx 0.1$–$0.3$ to avoid over-specialization (Zhou et al., 2024). NKI-Agent uses LoRA adapters with rank $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$0, $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$1, adding $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$2 M trainable parameters $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$3 on Qwen3-Coder-30B-A3B (Tang et al., 5 Jul 2026).

Some agent fine-tuning schemes also modify the effective conditioning context. RAISE augments an LLM with short-term scratchpad memory and a long-term example pool, and trains on samples containing system prompt, history, query, CoT, and response (Liu et al., 2024). This suggests that in agent settings, the tuned object is often not merely a decoder but a decoder embedded in a reasoning-and-memory scaffold.

3. Data curation, protocol design, and negative supervision

Data construction is often the central bottleneck. AgentTuning begins with 35,341 raw interactions across six tasks and retains 1,866 high-quality trajectories after filtering for perfect success, except Mind2Web where the threshold is $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$4; the surviving traces average 5.24 turns (Zeng et al., 2023). The low-parameter agent study likewise generates agent-specific dialogues with GPT-4 and then manually screens trajectories to remove illogical or inconsistent interactions (Zhou et al., 2024).

Several works explicitly argue that agent data should be decomposed, balanced, or standardized rather than merely scaled. Agent-FLAN identifies an entanglement between format following and agent reasoning, then chat-aligns structured ReAct examples into multi-turn chat and decomposes the corpus into instruction-following, reasoning, retrieval, and understanding subsets with different weights (Chen et al., 2024). It also adds negative samples to teach “when not to act,” reducing $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$5 from 15.6% to 9.9% and $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$6 from 13.5% to 11.9%, with $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$7 rising from 84.5% to 89.1% (Chen et al., 2024).

Domain-specific fine-tuning frequently relies on carefully constructed positives and negatives. In the sustainable protein system, the information-extraction agent is fine-tuned on approximately 80 positive and approximately 80 negative examples; negatives are split equally across four “wrong” categories: wrong species but correct domain, correct species but wrong topic, unrelated microbes, and entirely unrelated fields (Kalian et al., 25 Jun 2025). This is paired with preprocessing steps such as manual PDF download, text extraction via pdfminer.six, reference-section removal, token-count reduction, and assembly into prompt–ideal output pairs (Kalian et al., 25 Jun 2025).

At larger scale, standardization becomes a systems problem. The Agent Data Protocol (ADP) introduces a unified Action/Observation schema, converts 13 public datasets into ADP format, and yields approximately 1.3 M trajectories after balanced sampling for SFT (Song et al., 28 Oct 2025). Its stated engineering advantage is a Raw→ADP→SFT hub-and-spoke design whose effort scales as $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$8 rather than $$L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),$$9, with automated validators enforcing constraints such as valid function names, JSON arguments, and trajectory termination patterns (Song et al., 28 Oct 2025). MIMIR addresses a related problem for personalized tuning by combining local private uploads with 520 domain-specific corpora and copyright checks, then converting raw text or JSON into instructional trajectories via GPT-4 prompts (Deng et al., 2024).

A recurring empirical lesson is that data quality dominates optimizer detail. NKI-Agent reports that each of four SFT dataset iterations yielded larger $(u_1,a_1,\dots,u_n,a_n)$0 than any tuning of learning rate, batch size, or LoRA rank, with one iteration fixing wrong function names that were 64% misnamed and another correcting incorrect rank errors such as 3D versus 2D (Tang et al., 5 Jul 2026).

4. Reinforcement and online fine-tuning

Reinforcement fine-tuning (RFT) is used when supervised imitation alone does not provide satisfactory closed-loop behavior. In SMART-R1, closed-loop simulation is treated as an MDP and optimized with Metric-Oriented Policy Optimization (MPO), using a simplified advantage $(u_1,a_1,\dots,u_n,a_n)$1 with $(u_1,a_1,\dots,u_n,a_n)$2 and a KL penalty with $(u_1,a_1,\dots,u_n,a_n)$3 against a fixed reference model (Pei et al., 28 Sep 2025). The full post-training pipeline alternates SFT–RFT–SFT, and on WOSAC 2025 SMART-R1 reaches a Realism Meta score of 0.7858, ranking first on the leaderboard at the time of submission (Pei et al., 28 Sep 2025).

In GUI agents, reinforcement fine-tuning is typically grounded in rule-based rewards over syntax, action type, and spatial correctness. MagicGUI defines

$(u_1,a_1,\dots,u_n,a_n)$4

and optimizes a PPO variant called Dual-Filtering Group Relative Policy Optimization (DF-GRPO), with group size $(u_1,a_1,\dots,u_n,a_n)$5, static filtering, dynamic filtering, and KL anchoring to the CPT policy (Tang et al., 19 Jul 2025). On Magic-RICH low-level Chinese tasks, RFT changes Routine subset SR from 97.2% to 97.5%, Instruction SR from 92.9% to 94.0%, Complex SR from 72.9% to 74.1%, and Exception SR from 93.2% to 92.1% (Tang et al., 19 Jul 2025). AgentCPM-GUI likewise uses GRPO for mobile GUI interaction, improving Exact-Match from 83.1 to 90.2 on AC-Low, from 66.7 to 75.0 on GUI-Odyssey, and from 61.1 to 76.4 on AITZ, while changes on AC-High and CAGUI are marginal or neutral (Zhang et al., 2 Jun 2025).

Closed-loop RL also appears in autonomous driving behavior models. Peng et al. pre-train MotionLM with behavioral cloning and then apply on-policy REINFORCE with reward

$(u_1,a_1,\dots,u_n,a_n)$6

For the 1 M model, the reported WOSAC composite score improves from 0.490 to 0.597, collision from 0.544 to 0.863, and ADE from 6.332 to 2.436 after fine-tuning (Peng et al., 2024). The underlying claim is not that RL replaces supervised learning, but that a strong supervised warm start is necessary because RL-only from scratch performs poorly (Peng et al., 2024).

Outside LLMs, safe online fine-tuning remains a separate research thread. Automatic Jump-Start (AJS) combines a conservative guide policy updated by InAC with a fast exploration policy updated by SAC, and uses Fitted Q Evaluation to decide when to expand exploration. Averaged over nine D4RL settings, SAC fine-tuning shows $(u_1,a_1,\dots,u_n,a_n)$7 and $(u_1,a_1,\dots,u_n,a_n)$8, InAC shows $(u_1,a_1,\dots,u_n,a_n)$9 and $\alpha=0.5$0, and AJS shows $\alpha=0.5$1 and $\alpha=0.5$2 (Wang et al., 1 May 2025). A plausible implication is that “agent fine-tuning” includes not only improving final return but also controlling transient degradation during adaptation.

Sparse-reward RL can also fail. In NKI-Agent, GRPO with a binary compile+verify reward underperforms pure SFT on both single-shot and agent settings, and the authors attribute this to the inability of sparse binary signals to distinguish “almost correct” kernels (Tang et al., 5 Jul 2026).

5. Domain-specific and multi-agent deployments

Recent work treats agent fine-tuning less as a generic post-training step and more as a domain-binding procedure that joins a base model to a concrete environment, corpus, and evaluation protocol. In the sustainable protein framework, a two-agent RAG system contains a literature search agent and an information extraction agent; the latter extracts four fields—protein % dry mass, trophic mechanism, reported substrate, and substrate class—and fine-tuning raises mean cosine similarity scores to consistently $\alpha=0.5$3, versus prompt engineering at $\alpha=0.5$4 (Kalian et al., 25 Jun 2025). In Hitachi’s JP1 middleware, continual pre-training on roughly 3 million tokens from manuals is followed by SFT on approximately 55 k examples and agentic prompting with RAG; the reported overall average gain on certification MCQs is approximately 14 percentage points over the base model (Xue et al., 1 Oct 2025).

NKI-Agent moves the same paradigm into program synthesis for AWS Trainium and Inferentia. It curates 6,000 NKI kernel generation tasks, distills 647 high-quality SFT episodes, and combines domain-specific SFT with a compile-verify-fix loop on real hardware. On a 60-task subset, the SFT-trained Qwen3-Coder-30B-A3B reaches a 25.0% pass rate at roughly 1/100th the per-token inference cost of Claude Opus 4.8, while Opus 4.8 with NKI-Agent and a rank-aware system prompt reaches 77.3% on the 150-task NKIBench (Tang et al., 5 Jul 2026).

Fine-tuning is equally prominent in non-language multi-agent control. MAPF-GPT-DDG continues training a 2 M-parameter decentralized MAPF policy on mixed batches of original expert data and delta-generated correction data, with 25% of each minibatch drawn from the generated set. The method reports operation on obstacle-free $\alpha=0.5$5 grids with up to $\alpha=0.5$6 agents, independent success $\alpha=0.5$7, and per-agent decision latency of approximately $\alpha=0.5$8 (Andreychuk et al., 30 Jun 2025). In multi-agent traffic simulation, SMART-R1 uses RFT to align closed-loop behavior with the Realism Meta metric rather than only logged trajectories (Pei et al., 28 Sep 2025). MARFT generalizes the LLM-based case by formalizing a Flexible Partially Observable Markov Decision Process (Flex-POMDP) with agent profiles, dependency functions, and PPO-style LoRA adaptation for multi-agent systems (Liao et al., 21 Apr 2025).

Domain	Fine-tuning regime	Reported outcome
Sustainable protein production	GPT-4.1 fine-tuning on balanced extraction data	mean cosine similarity consistently $\alpha=0.5$9
JP1 middleware	CPT + SFT + agentic prompting	overall average gain $r=8$0 pp
NKI kernel generation	domain-specific SFT + compile-verify-fix	25.0% pass rate at 1/100th cost on 60 tasks
Multi-agent pathfinding	DDG fine-tuning of MAPF-GPT	up to $r=8$1 agents, independent success $r=8$2

These examples show that “agent fine-tuning” is now applied to knowledge extraction, technical troubleshooting, code generation, GUI control, traffic simulation, and multi-robot planning, with task-specific rewards and datasets determining the practical meaning of adaptation.

6. Trade-offs, limitations, and open questions

A central misconception is that fine-tuning uniformly dominates prompting. The sustainable protein study reports that prompt engineering also improved mean cosine similarity by up to 25% and achieved universally $r=8$3, with lower statistical uncertainties than fine-tuning in the reported tests (Kalian et al., 25 Jun 2025). That paper further recommends a staged workflow: prototype with prompt engineering, then apply lightweight fine-tuning if higher accuracy is needed, while maintaining negative-example sets and monitoring calibration (Kalian et al., 25 Jun 2025).

Another recurring issue is catastrophic specialization. AgentTuning addresses this with a hybrid regime that mixes AgentInstruct and ShareGPT at best $r=8$4, and reports that general NLP performance for AgentLM-70B remains effectively unchanged overall, from 0.95 for Llama 2-70B to 0.96 for AgentLM-70B (Zeng et al., 2023). Agent-FLAN pursues the same goal through chat alignment, capability decomposition, and negative supervision, while the 7B/13B agent-capability study reports that agent-specific SFT reduces hallucination and formatting errors by approximately 30–50% on code- and dialog-heavy tasks (Chen et al., 2024, Zhou et al., 2024).

The literature also consistently emphasizes cost and operational complexity. Fine-tuning closed-source GPT-4.1 via API incurs token-level and epoch-based usage fees, whereas prompt engineering requires only inference calls (Kalian et al., 25 Jun 2025). Full-parameter tuning of large models, such as JP1 adaptation on 24 × H100 GPUs, is compute-intensive (Xue et al., 1 Oct 2025). Even when using parameter-efficient adapters, NKI-Agent shows that data curation and grounded tool feedback, rather than hyperparameter sweeps alone, drive most of the gains (Tang et al., 5 Jul 2026).

Autonomous fine-tuning systems remain an open frontier rather than a solved problem. FT-Dojo frames end-to-end fine-tuning as an interactive search problem over data curation, tool use, training configuration, and evaluation diagnosis, and FT-Agent achieves the best performance on 10 out of 13 tasks across five domains (Li et al., 2 Mar 2026). Yet the same work reports failure cases where the agent cannot identify the root cause of poor performance and instead produces generic remedies, exposing a causal-reasoning limitation (Li et al., 2 Mar 2026).

Finally, not all fine-tuning methods come with strong theory. Automatic Jump-Start is justified empirically and explicitly provides no formal regret or convergence bounds (Wang et al., 1 May 2025). This suggests that a mature theory of agent fine-tuning—especially one covering mixed supervised/RL regimes, adaptive tool use, and multi-agent dependencies—remains incomplete. The empirical record nonetheless establishes several robust patterns: high-quality trajectories matter more than raw volume, negative supervision is often necessary to suppress hallucination, closed-loop reward is useful when deployment shift is severe, and mixed-data training is the main mechanism currently used to preserve general ability while improving agent behavior.