On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment

Abstract: Tool-using LLM agents fail through trajectories rather than only final responses, as they may execute unsafe tool calls, follow injected instructions, comply with harmful requests, or over-refuse benign tasks despite producing a seemingly safe answer. Existing safety-alignment signals are largely response-level or off-policy, and often incur a safety-utility trade-off: improving agent safety comes at the cost of degraded task performance. Such sparse and single-objective rewards severely limit real-world usability. To bridge this gap, we propose FATE, an on-policy self-evolving framework that transforms verifier-scored failures into repair supervision without expert demonstrations. For each failure, the same policy proposes repair candidates, which are then re-scored by verifiers and filtered across security, utility, over-refusal control, and trajectory validity. This dense trajectory-level information is then used as a supervision signal for agent self-evolution. During this process, we further introduce Pareto-Front Policy Optimization (PFPO), combining supervised warmup with Pareto-aware policy optimization to preserve safety-utility trade-offs. Experiments on AgentDojo, AgentHarm, and ATBench show that FATE improves safety across different models and scales while preserving useful behavior. Compared with strong baselines, FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%. These results suggest that failed trajectories can provide structured repair supervision for safer self-evolving agents.

• The paper introduces FATE, a framework that mines and repairs agent failure trajectories to balance safety and utility in LLM agents.
• It employs multi-objective verifier filtering with Pareto-front selection to guide policy refinement via supervised fine-tuning and PFPO.
• Empirical results show significant reductions in harmful compliance and attack success, boosting task performance and diagnostic metrics.

On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment

Motivation and Problem Setting

The increasing deployment of tool-using LLM agents, which interact with external environments over multi-step trajectories, presents unique safety challenges. Traditional response-level safety alignment fails to address trajectory-level failures where agents may execute unsafe tool calls, comply with harmful requests, follow injected instructions, or over-refuse benign tasks. Such failures are prevalent in environments probed by recent benchmarks (e.g., AgentDojo, AgentHarm, ATBench). Existing methods—especially those relying on RLHF, DPO, or single-objective rewards—induce a pronounced safety-utility trade-off, often triggering broad refusals that degrade agent utility.

FATE: Failure-Trajectory Evolution Framework

The paper introduces FATE (FAilure-Trajectory Evolution), an on-policy self-evolving framework for trajectory-level agentic safety alignment that leverages the agent's own failed trajectories as the source of dense, multi-objective supervision. This process involves the following stages:

1. Mining Failures: The current policy is rolled out in the environment to collect a distribution of verifier-scored failure trajectories.
2. Repair Proposal: For each failure, the same policy is prompted—using the failure context and verifier feedback—to generate multiple candidate repairs tailored to its own failure distribution.
3. Verifier Filtering and Pareto Selection: Candidates are re-scored on four axes: security, utility, over-refusal control, and trajectory validity. Candidates are filtered by feasibility thresholds, and Pareto-front selection identifies non-dominated repairs.
4. Policy Refinement: The selected repairs form a supervision dataset for two-stage updates: a supervised fine-tuning (SFT) warmup followed by Pareto-Front Policy Optimization (PFPO), which sharpens preferences among high-quality completions.

   Figure 1: Overview of FATE. The current policy mines its own failures, proposes same-policy repairs, and converts them into verifier-filtered Pareto repair supervision. The selected repairs are internalized by SFT and PFPO, producing the next policy for another self-evolution round.

This iterative protocol, termed self-evolution, updates the policy using each new set of failures and associated filtered repairs, progressively shifting both its behavior and the nature of observed errors.

Theoretical and Algorithmic Contributions

FATE’s core methodological contribution is its policy-coupled dense supervision mechanism:

• On-policy Repair Proposals: By generating repairs using the current policy, the repair process targets real, recurring failure modes of the incumbent agent.
• Verifier-Guided Pareto Front: The use of trajectory-level multi-objective verifiers and Pareto-front selection prevents collapse into degenerate solutions—such as broad refusal or safety-only optimization at the expense of utility.
• KL-Regularized Supervision: The construction of the supervision distribution is shown to be a KL-regularized projection onto the verifier-filtered Pareto front, with front-only tie-breaking ensuring balanced improvements across objectives.
• PFPO Algorithm: PFPO extends standard RL algorithms by assigning group-relative advantages using the Pareto rewards over sampled repairs, penalizing completions that are safe only via refusal or remain unsafe.

A series of ablations confirm the necessity of verifier re-scoring, over-refusal control, and Pareto-front filtering. Removing these elements results in clear detractions in trade-off calibration, undesirable agent refusals, or insufficient repair behaviors.

Empirical Results

Comprehensive evaluations are provided across AgentDojo (indirect prompt injection), AgentHarm (harmful request compliance), and ATBench (long-horizon trajectory diagnosis) tasks, using various open-source LLM backbones covering a range of model scales.

The principal results demonstrate:

• Substantial Reductions in Attack/Harm Rates: FATE achieves a 33.5% reduction in attack success rate (ASR), an 82.6% reduction in harmful compliance, and a 26.0% increase in task success rate under attack relative to the strongest baselines.
• Improved External Diagnostic Scores: On ATBench, FATE-outfitted agents attain notably higher accuracy, precision, recall, and fine-grained risk/failure/harm scores, surpassing open-source and even some closed-source models.
• Iterative Self-Evolution Gains: Repeated FATE rounds progressively lower ASR and HCR while elevating TSR and VRR, with diminishing returns stabilizing after several iterations.

  Figure 2: FATE progressively reduces ASR and HCR while improving TSR and VRR over successive self-evolution rounds on held-out test tasks.

These improvements replicate across model scales (0.6B to 32B parameters in the Qwen3 family) and architectures (Qwen3, Llama-3.1, Ministral-3, Gemma-3, Phi-4), indicating framework robustness.

Implications and Theoretical Insights

FATE provides a paradigm for aligning agentic LLMs via on-policy trajectory repair and multi-objective supervision, directly addressing the chronic deficiency of dense, trajectory-level signals in RLHF-style pipelines. Its reliance on verifier-filtered self-generated repairs—rather than external experts or static datasets—facilitates continual adaptation as the policy evolves.

This approach mitigates exposure bias and overfitting to static objective distributions, and by leveraging agent-specific, dynamically observed failures, more effectively targets the most severe and persistent failure modes encountered in operational deployments.

A key implication is the feasibility of integrating safety alignment and utility preservation at scale without depending on human or expert-written repairs—provided that verifiers are sufficiently expressive and are paired with rigorous multi-objective selection strategies.

Limitations and Future Directions

While FATE demonstrates substantial improvement in trajectory-level safety and utility trade-offs, several limitations pertain:

• Quality and coverage of verifiers fundamentally constrain the efficacy of supervision.
• For weak base policies, on-policy repair proposals may lack sufficient diversity or correctness, potentially requiring hybridization with external teacher models or curriculum learning.
• The current benchmarks, while challenging, do not encompass the full range of real-world agentic safety challenges, notably longer horizons, richer tool interfaces, and human feedback integration.

Future research should consider extending FATE to semi- or fully online settings with human-in-the-loop corrections, richer verifier architectures (potentially learned), and applications in environments with hard-to-simulate or ambiguous failure signals.

Conclusion

FATE advances the landscape of agentic alignment by systematically converting verifier-scored failed trajectories into dense, multi-objective repair supervision via on-policy, Pareto-front filtering, and group-relative optimization. The resultant self-evolving agents consistently demonstrate enhanced trajectory-level safety and utility, with strong empirical support across models and environments. This work substantiates a practical route toward scalable, data-efficient agent safety alignment and highlights the centrality of failure-driven repair in the development of robust, agentic LLMs (2605.11882).