Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories

Abstract: LLM agents are commonly trained from expert trajectories using supervised fine-tuning (SFT), which treats multi-turn agent behavior as ordinary text imitation. This recipe is simple and low-cost, but it only learns to imitate the sequence of expert actions, rather than training the agent to choose the right action against plausible mistakes at each state. Existing methods to mitigate this problem include preference learning or reinforcement learning, but they usually need high-cost environment rollouts and reward models. We propose Agentic-DPO, a lightweight offline agent policy optimization method that turns expert trajectories into state-conditioned preference supervision. At each expert action state, Agentic-DPO samples a one-step action from the current state, treats plausible wrong actions as negatives, and contrasts them with the expert action using a DPO-style preference objective. To avoid mixing both policy and schema in preference learning, we introduce Policy-Preserving Augmentation (PPA), which renders the same latent trajectory under multiple schemas while keeping the expert policy fixed. Agentic-DPO requires no online environment rollout, reward model, or full-trajectory student exploration. We conduct experiments across StableToolBench, tau-bench retail, and Mind2Web, where Agentic-DPO consistently improves agents at different model scales beyond imitation. In particular, it raises tau-bench accuracy from 21.7% (SFT) to 41.4% for a 9B model, matching online GRPO under the same backbone with only step-level rollouts and without environment interaction during gradient steps. The results suggest that expert trajectories can support low-cost agentic policy optimization when converted from demonstrations into state-level action preferences. Code for Agentic-DPO is released at https://github.com/Schuture/Agentic-DPO.

• The paper introduces a contrastive, agentic approach for optimizing policies offline using expert trajectories.
• It leverages state-conditioned negative sampling with Policy-Preserving Augmentation to mitigate formatting biases and enhance robustness.
• Empirical results show significant improvements in accuracy and efficiency, nearly matching online RL performance with reduced compute costs.

Agentic-DPO: A Framework for Offline State-Conditioned Policy Optimization From Expert Trajectories

Problem Formulation and Methodological Innovations

Agentic-DPO addresses the ineffectiveness of standard SFT for multi-turn LLM agents, which only provides token-level supervision and does not train the policy to assert correct actions against its own likely mistakes. Instead, Agentic-DPO reframes agent supervision using a preference-based approach applied at the action level: for each expert state-action pair, a candidate set of negative (student) actions is sampled from the model's own policy under the same state, and the student is trained to assign higher preference to the expert action than to its own plausible errors. The learning signal is provided by a DPO-style loss applied to these state-anchored action preference pairs.

This is operationalized as follows:

• Given expert trajectories $\mathcal{D}_{\mathrm{exp}} = \{(s_t, u^+_t)\}$, at each state $s_t$ the algorithm samples $K$ candidate student actions and selects hard negatives (non-expert, highest student log-probability).
• The objective is a length-normalized DPO-style preference loss, contrasting the expert action against the selected negative under the current student and a frozen reference policy (the SFT anchor).
• Training proceeds in rounds, refreshing negatives to remain close to the student’s evolving mistake distribution.

To avoid overfitting to superficial action string formats and to stabilize the training signal, Agentic-DPO introduces Policy-Preserving Augmentation (PPA), whereby the same latent decision is rendered under multiple syntactic schemas (ReAct, JSON, tool renaming, context rewrites), thus ensuring the preference signal is attached to the underlying policy decision rather than formatting artifacts. Additionally, an SFT anchor term in the loss ensures the optimization remains in the support of valid actions and mitigates DPO-specific likelihood displacement [see also “Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization”, (Razin et al., 2024)].

The key methodological distinction is visualized in the introductory schematic:

Figure 1: Comparison between SFT (token imitation) and Agentic-DPO (state-conditioned action-level preference learning via expert-vs-student contrast).

Empirical Evaluation and Strong Claims

Agentic-DPO is evaluated extensively on StableToolBench, $\tau$-bench retail, and Mind2Web using the Qwen3.5 and Gemma3 backbones. Results indicate highly consistent improvements in canonical and perturbed accuracy, success rate, and held-out generalization over SFT, PPA+SFT, DFT, ETO, and GRPO.

On StableToolBench (Qwen3.5-9B):

• SFT: 78.5% (canonical) $\to$ Agentic-DPO: 94.1%
• PPA+SFT (without contrast): 92.2%
• ETO: 92.5%
• Online RL baseline (GRPO): 88.6%

On $\tau$-bench retail (Qwen3.5-9B):

• SFT: 21.7% $\to$ Agentic-DPO: 41.4%
• Online RL: 40.0%

On Mind2Web (Qwen3.5-9B):

• SFT: 45.6% $\to$ Agentic-DPO: 64.4%

Strong claims are made: Agentic-DPO matches or exceeds the accuracy of online RL with only step-level offline rollouts and no environment interaction during gradient steps. It demonstrates improved robustness to surface-level prompt perturbations (Table~\ref{tab:robustness_holdout}), and, via PPA, out-of-distribution generalization to unseen syntactic encodings (BFCL-v3 benchmark).

Analysis: Data Efficiency, Sampled Negative Scaling, and Ablations

Agentic-DPO's sample efficiency is notable. Saturation is reached at $K=2-4$ negatives per state, increasing $K$ beyond this does not appreciably improve performance.

Figure 2: Left: Performance gain plateaus beyond $s_t$0 negatives. Right: Agentic-DPO exhibits higher sample efficiency than PPA+SFT, with comparable performance using only 25% of the expert data.

Ablation studies confirm that both SFT anchoring and PPA are critical. Removing SFT-warmup collapses training; omitting either action-rendering or context-rewrite PPA widens the performance gap on perturbation/generalization benchmarks, highlighting that contrastive preference gradients alone are not reliably robust without multi-view data augmentation.

Training Dynamics: Refresh Rounds and Model Scale

The effect of negative-refresh rounds is strongly dependent on model scale. Small/base models benefit from multiple rounds; larger models converge with a single refresh, indicating that model capacity enables more effective error boundary discovery even with limited negative mining.

Figure 3: Canonical accuracy as a function of negative-refresh rounds for Qwen3.5-2B/4B/9B. Larger models require fewer refreshes for optimal performance.

Practical and Theoretical Implications

From a practical standpoint, Agentic-DPO strikes a compelling cost-accuracy Pareto: per-step compute is approximately 1.6$s_t$1 that of SFT, orders of magnitude less than online RL methods (\emph{e.g.}, GRPO at $s_t$2). Ablation and robustness analyses demonstrate that using expert-only trajectories (no reward models, no human feedback, no additional state exploration) is sufficient for high-quality policy shaping when paired with contrastive state-anchored learning and policy-preserving schema augmentation.

Theoretically, the work formalizes conditions under which local DPO gradients are aligned with SFT gradients, and that SFT anchoring ensures monotonic expert KL reduction at initialization. Furthermore, it provides a framework for distinguishing latent policy from schema-induced spurious gradients, a source of overfitting in earlier preference-based approaches.

Limitations and Future Directions

A fundamental constraint remains: Agentic-DPO, as an offline protocol, cannot optimize for novel (student-induced) states that are absent from expert traces. Thus, for tasks requiring extensive generalization via long-horizon exploration, online RL or hybrid on-policy distillation remains necessary. However, the major practical implication is that with moderately-coverage expert data, agentic policy optimization is tractable without environment rollouts, reward modeling, or massive human-in-the-loop feedback.

Extensions could incorporate modest online rollouts, iterative data augmentation, or state exploration into the Agentic-DPO pipeline, blending the cost benefits of offline preference optimization with selective online improvement. The general principle of state-conditioned, policy-aware offline contrastive learning is extensible to a large class of agent-LLM alignment settings.

Conclusion

Agentic-DPO establishes a new empirical benchmark for offline LLM agent training, demonstrating that conversion of expert sequences into state-anchored action preferences—with schema-robust augmentation—yields robust, high-quality, and data-efficient policies. It closes a significant portion of the performance gap to full online RL at a fraction of the cost, substantially advancing the methodology for learning robust agentic policies from expert data alone (2607.10601).