---
title: Agent Fine-Tuning Overview
url: https://www.emergentmind.com/topics/agent-fine-tuning
type: topic
---

# Agent Fine-Tuning Overview

Agent fine-tuning is the post-pretraining adaptation of an agentic model—most often a large language model used for planning, tool use, memory, reasoning, or interaction, but also learned policies for multi-agent control—through supervised trajectories, domain corpora, mixed instruction data, or closed-loop reward optimization so that behavior aligns with a target environment and task distribution. In current research, the term spans hybrid instruction-tuning of interaction traces, domain-specific supervised fine-tuning, reinforcement fine-tuning in simulators and GUI environments, and active correction of decentralized policies [2310.12823] [2403.12881] [2509.23993] [2506.23793].

## 1. Definition and problem setting

In the LLM-agent literature, fine-tuning is motivated by a recurring gap between strong general language modeling and weak agent behavior. Open-source models are reported to lag behind larger commercial systems on planning, long-term memory, and tool utilization, even when prompting methods such as CoT or ReAct are available [2403.19962] [2310.12823]. Fine-tuning is used to internalize these behaviors into model parameters rather than carrying them entirely in prompts or external orchestration.

A second, distinct motivation arises in offline-to-online control and simulation. Here, an agent is first learned from logged data and then adapted online, but naive fine-tuning can induce severe early degradation because exploration overrides the offline policy. This problem is explicit in offline-to-online RL and in behavior models for autonomous driving, where distribution shift between supervised training and closed-loop deployment degrades reliability [2505.00913] [2409.18343].

Across these strands, agent fine-tuning usually targets one or more of four goals. The first is **capability acquisition**, such as learning Thought/Action/Observation trajectories, tool calls, or decomposition strategies. The second is **domain adaptation**, as in protein production, JP1 middleware, neuron-kernel generation, and GUI control. The third is **closed-loop robustness**, where reward-driven updates correct rollout drift. The fourth is **general-skill preservation**, since many works explicitly try to improve agent behavior without compromising broader language ability [2310.12823] [2403.12881].

This scope also extends to multi-agent systems. In LLM-based multi-agent settings, fine-tuning must address asynchronous communication, role-conditioned prompts, dynamically changing dependencies, and token-sequence action spaces. In classical multi-agent control, fine-tuning may instead target coordination quality, realism, or safe exploration under deployment shift [2504.16129] [2509.23993].

## 2. Supervised formulations and agent representations

The dominant supervised objective is next-token cross-entropy over agent trajectories, code traces, or structured outputs. Representative formulations include standard causal language modeling,
$$
\mathcal{L}_{CE}(\theta) = -\sum_{t=1}^{T}\log p_\theta(x_t \mid x_{<t}, c),
$$
and mixed-data objectives that interpolate agent-specific and general instruction data [2404.04285] [2310.12823]. In domain-specialized code generation, NKI-Agent uses
$$
L_{\mathrm{SFT}}(\theta) = - \sum_{(x,y)} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x),
$$
with prompts serialized from multi-turn conversations into standard causal language modeling examples [2607.04395].

Training data are usually structured as interaction trajectories rather than isolated question-answer pairs. AgentTuning’s AgentInstruct keeps sequences \((u_1,a_1,\dots,u_n,a_n)\) with rewards, while low-parameter agent tuning constructs three-role dialogues among a Question Generator, an Action Maker, and an Environment Agent [2310.12823] [2403.19962]. In microdomain adaptation for JP1 middleware, the target is explicitly agentic: GPT-4-generated ReAct and CoT traces in Japanese are fine-tuned so that procedural knowledge such as “check status → if unhealthy, restart service → verify logs” becomes encoded in the model parameters [2510.00482].

Architecturally, many systems preserve the base model and tune either all parameters or adapters. AgentTuning applies full-parameter fine-tuning to Llama 2-chat 7B, 13B, and 70B [2310.12823]. MIMIR exposes both full and LoRA modes in a one-click workflow and balances domain and general competencies through a mixed loss with default \(\alpha=0.5\) [2404.04285]. The 7B/13B agent-capability study uses LoRA with rank \(r=8\), \(\alpha=16\), and mixes \(D_{\text{agent}}\) with \(D_{\text{general}}\) at \(\lambda \approx 0.1\)–\(0.3\) to avoid over-specialization [2403.19962]. NKI-Agent uses LoRA adapters with rank \(32\), \(\alpha=64\), adding \(26.7\) M trainable parameters \((0.09\%)\) on Qwen3-Coder-30B-A3B [2607.04395].

Some agent fine-tuning schemes also modify the effective conditioning context. RAISE augments an LLM with short-term scratchpad memory and a long-term example pool, and trains on samples containing system prompt, history, query, CoT, and response [2401.02777]. This suggests that in agent settings, the tuned object is often not merely a decoder but a decoder embedded in a reasoning-and-memory scaffold.

## 3. Data curation, protocol design, and negative supervision

Data construction is often the central bottleneck. AgentTuning begins with 35,341 raw interactions across six tasks and retains 1,866 high-quality trajectories after filtering for perfect success, except Mind2Web where the threshold is \(r \ge 2/3\); the surviving traces average 5.24 turns [2310.12823]. The low-parameter agent study likewise generates agent-specific dialogues with GPT-4 and then manually screens trajectories to remove illogical or inconsistent interactions [2403.19962].

Several works explicitly argue that agent data should be decomposed, balanced, or standardized rather than merely scaled. Agent-FLAN identifies an entanglement between format following and agent reasoning, then chat-aligns structured ReAct examples into multi-turn chat and decomposes the corpus into instruction-following, reasoning, retrieval, and understanding subsets with different weights [2403.12881]. It also adds negative samples to teach “when not to act,” reducing \(H_{\mathrm{ReAct}}\) from 15.6% to 9.9% and \(H_{\mathrm{General}}\) from 13.5% to 11.9%, with \(H_{\mathrm{Score}}\) rising from 84.5% to 89.1% [2403.12881].

Domain-specific fine-tuning frequently relies on carefully constructed positives and negatives. In the sustainable protein system, the information-extraction agent is fine-tuned on approximately 80 positive and approximately 80 negative examples; negatives are split equally across four “wrong” categories: wrong species but correct domain, correct species but wrong topic, unrelated microbes, and entirely unrelated fields [2506.20598]. This is paired with preprocessing steps such as manual PDF download, text extraction via `pdfminer.six`, reference-section removal, token-count reduction, and assembly into prompt–ideal output pairs [2506.20598].

At larger scale, standardization becomes a systems problem. The Agent Data Protocol (ADP) introduces a unified Action/Observation schema, converts 13 public datasets into ADP format, and yields approximately 1.3 M trajectories after balanced sampling for SFT [2510.24702]. Its stated engineering advantage is a Raw→ADP→SFT hub-and-spoke design whose effort scales as \(O(D + A)\) rather than \(O(D \times A)\), with automated validators enforcing constraints such as valid function names, JSON arguments, and trajectory termination patterns [2510.24702]. MIMIR addresses a related problem for personalized tuning by combining local private uploads with 520 domain-specific corpora and copyright checks, then converting raw text or JSON into instructional trajectories via GPT-4 prompts [2404.04285].

A recurring empirical lesson is that data quality dominates optimizer detail. NKI-Agent reports that each of four SFT dataset iterations yielded larger \(\Delta\) than any tuning of learning rate, batch size, or LoRA rank, with one iteration fixing wrong function names that were 64% misnamed and another correcting incorrect rank errors such as 3D versus 2D [2607.04395].

## 4. Reinforcement and online fine-tuning

Reinforcement fine-tuning (RFT) is used when supervised imitation alone does not provide satisfactory closed-loop behavior. In SMART-R1, closed-loop simulation is treated as an MDP and optimized with Metric-Oriented Policy Optimization (MPO), using a simplified advantage \(A = r - \alpha\) with \(\alpha = 0.77\) and a KL penalty with \(\beta = 0.04\) against a fixed reference model [2509.23993]. The full post-training pipeline alternates **SFT–RFT–SFT**, and on WOSAC 2025 SMART-R1 reaches a Realism Meta score of 0.7858, ranking first on the leaderboard at the time of submission [2509.23993].

In GUI agents, reinforcement fine-tuning is typically grounded in rule-based rewards over syntax, action type, and spatial correctness. MagicGUI defines
$$
R(s,a) = R_{\mathrm{format}}(s,a) + R_{\mathrm{acc}}(s,a) + R_{\mathrm{dist}}(s,a),
$$
and optimizes a PPO variant called Dual-Filtering Group Relative Policy Optimization (DF-GRPO), with group size \(G=8\), static filtering, dynamic filtering, and KL anchoring to the CPT policy [2508.03700]. On Magic-RICH low-level Chinese tasks, RFT changes Routine subset SR from 97.2% to 97.5%, Instruction SR from 92.9% to 94.0%, Complex SR from 72.9% to 74.1%, and Exception SR from 93.2% to 92.1% [2508.03700]. AgentCPM-GUI likewise uses GRPO for mobile GUI interaction, improving Exact-Match from 83.1 to 90.2 on AC-Low, from 66.7 to 75.0 on GUI-Odyssey, and from 61.1 to 76.4 on AITZ, while changes on AC-High and CAGUI are marginal or neutral [2506.01391].

Closed-loop RL also appears in autonomous driving behavior models. Peng et al. pre-train MotionLM with behavioral cloning and then apply on-policy REINFORCE with reward
\[
r_{t,i} = -\|Pos_{t,i} - GT_{t,i}\|_2 - \lambda \cdot Coll_{t,i}.
\]
For the 1 M model, the reported WOSAC composite score improves from 0.490 to 0.597, collision from 0.544 to 0.863, and ADE from 6.332 to 2.436 after fine-tuning [2409.18343]. The underlying claim is not that RL replaces supervised learning, but that a strong supervised warm start is necessary because RL-only from scratch performs poorly [2409.18343].

Outside LLMs, safe online fine-tuning remains a separate research thread. Automatic Jump-Start (AJS) combines a conservative guide policy updated by InAC with a fast exploration policy updated by SAC, and uses Fitted Q Evaluation to decide when to expand exploration. Averaged over nine D4RL settings, SAC fine-tuning shows \(\Delta_{\min}\approx -48\%\) and \(\Delta_{\mathrm{final}}\approx +32\%\), InAC shows \(0\%\) and \(+6\%\), and AJS shows \(-4\%\) and \(+21\%\) [2505.00913]. A plausible implication is that “agent fine-tuning” includes not only improving final return but also controlling transient degradation during adaptation.

Sparse-reward RL can also fail. In NKI-Agent, GRPO with a binary compile+verify reward underperforms pure SFT on both single-shot and agent settings, and the authors attribute this to the inability of sparse binary signals to distinguish “almost correct” kernels [2607.04395].

## 5. Domain-specific and multi-agent deployments

Recent work treats agent fine-tuning less as a generic post-training step and more as a domain-binding procedure that joins a base model to a concrete environment, corpus, and evaluation protocol. In the sustainable protein framework, a two-agent RAG system contains a literature search agent and an information extraction agent; the latter extracts four fields—protein % dry mass, trophic mechanism, reported substrate, and substrate class—and fine-tuning raises mean cosine similarity scores to consistently \(\ge 0.94\), versus prompt engineering at \(\ge 0.89\) [2506.20598]. In Hitachi’s JP1 middleware, continual pre-training on roughly 3 million tokens from manuals is followed by SFT on approximately 55 k examples and agentic prompting with RAG; the reported overall average gain on certification MCQs is approximately 14 percentage points over the base model [2510.00482].

NKI-Agent moves the same paradigm into program synthesis for AWS Trainium and Inferentia. It curates 6,000 NKI kernel generation tasks, distills 647 high-quality SFT episodes, and combines domain-specific SFT with a compile-verify-fix loop on real hardware. On a 60-task subset, the SFT-trained Qwen3-Coder-30B-A3B reaches a 25.0% pass rate at roughly 1/100th the per-token inference cost of Claude Opus 4.8, while Opus 4.8 with NKI-Agent and a rank-aware system prompt reaches 77.3% on the 150-task NKIBench [2607.04395].

Fine-tuning is equally prominent in non-language multi-agent control. MAPF-GPT-DDG continues training a 2 M-parameter decentralized MAPF policy on mixed batches of original expert data and delta-generated correction data, with 25% of each minibatch drawn from the generated set. The method reports operation on obstacle-free \(2048\times 2048\) grids with up to \(2^{20}\) agents, independent success \(\ge 99.9\%\), and per-agent decision latency of approximately \(160\,\mu s\) [2506.23793]. In multi-agent traffic simulation, SMART-R1 uses RFT to align closed-loop behavior with the Realism Meta metric rather than only logged trajectories [2509.23993]. MARFT generalizes the LLM-based case by formalizing a Flexible Partially Observable Markov Decision Process (Flex-POMDP) with agent profiles, dependency functions, and PPO-style LoRA adaptation for multi-agent systems [2504.16129].

| Domain | Fine-tuning regime | Reported outcome |
|---|---|---|
| Sustainable protein production | GPT-4.1 fine-tuning on balanced extraction data | mean cosine similarity consistently \(\ge 0.94\) |
| JP1 middleware | CPT + SFT + agentic prompting | overall average gain \(\approx 14\) pp |
| NKI kernel generation | domain-specific SFT + compile-verify-fix | 25.0% pass rate at 1/100th cost on 60 tasks |
| Multi-agent pathfinding | DDG fine-tuning of MAPF-GPT | up to \(2^{20}\) agents, independent success \(\ge 99.9\%\) |

These examples show that “agent fine-tuning” is now applied to knowledge extraction, technical troubleshooting, code generation, GUI control, traffic simulation, and multi-robot planning, with task-specific rewards and datasets determining the practical meaning of adaptation.

## 6. Trade-offs, limitations, and open questions

A central misconception is that fine-tuning uniformly dominates prompting. The sustainable protein study reports that prompt engineering also improved mean cosine similarity by up to 25% and achieved universally \(\ge 0.89\), with lower statistical uncertainties than fine-tuning in the reported tests [2506.20598]. That paper further recommends a staged workflow: prototype with prompt engineering, then apply lightweight fine-tuning if higher accuracy is needed, while maintaining negative-example sets and monitoring calibration [2506.20598].

Another recurring issue is catastrophic specialization. AgentTuning addresses this with a hybrid regime that mixes AgentInstruct and ShareGPT at best \(\eta=0.2\), and reports that general NLP performance for AgentLM-70B remains effectively unchanged overall, from 0.95 for Llama 2-70B to 0.96 for AgentLM-70B [2310.12823]. Agent-FLAN pursues the same goal through chat alignment, capability decomposition, and negative supervision, while the 7B/13B agent-capability study reports that agent-specific SFT reduces hallucination and formatting errors by approximately 30–50% on code- and dialog-heavy tasks [2403.12881] [2403.19962].

The literature also consistently emphasizes cost and operational complexity. Fine-tuning closed-source GPT-4.1 via API incurs token-level and epoch-based usage fees, whereas prompt engineering requires only inference calls [2506.20598]. Full-parameter tuning of large models, such as JP1 adaptation on 24 × H100 GPUs, is compute-intensive [2510.00482]. Even when using parameter-efficient adapters, NKI-Agent shows that data curation and grounded tool feedback, rather than hyperparameter sweeps alone, drive most of the gains [2607.04395].

Autonomous fine-tuning systems remain an open frontier rather than a solved problem. FT-Dojo frames end-to-end fine-tuning as an interactive search problem over data curation, tool use, training configuration, and evaluation diagnosis, and FT-Agent achieves the best performance on 10 out of 13 tasks across five domains [2603.01712]. Yet the same work reports failure cases where the agent cannot identify the root cause of poor performance and instead produces generic remedies, exposing a causal-reasoning limitation [2603.01712].

Finally, not all fine-tuning methods come with strong theory. Automatic Jump-Start is justified empirically and explicitly provides no formal regret or convergence bounds [2505.00913]. This suggests that a mature theory of agent fine-tuning—especially one covering mixed supervised/RL regimes, adaptive tool use, and multi-agent dependencies—remains incomplete. The empirical record nonetheless establishes several robust patterns: high-quality trajectories matter more than raw volume, negative supervision is often necessary to suppress hallucination, closed-loop reward is useful when deployment shift is severe, and mixed-data training is the main mechanism currently used to preserve general ability while improving agent behavior.

Source: https://www.emergentmind.com/topics/agent-fine-tuning