Tracing Agentic Failure from the Flow of Success

Abstract: Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000 $\times$ faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.

• The paper introduces Oat, an unsupervised one-class learning approach that trains on successful trajectories to accurately identify error-contributing steps in agentic systems.
• It employs Neural CDEs with a gated control path to model temporal dynamics, achieving notable improvements in F1 and AUPRC while drastically reducing computational latency.
• The study demonstrates practical implications for real-time debugging and auditability, validated through extensive benchmarks on both in-domain and out-of-distribution tasks.

Tracing Agentic Failure from the Flow of Success

Problem Motivation and Formulation

LLM-based agentic systems operating on complex, long-horizon tasks are susceptible to multifaceted failure modes. Diagnosing which steps in a task execution trajectory are causal for overall failure is essential for debugging, auditability, and reliability of these systems. The annotation bottleneck – step-level error labeling on failure trajectories – makes supervised approaches costly and impractical for real-world deployment. Previous prompting-based solutions incur substantial computational overhead and latency, depend on frontier LLM inference, and scale poorly.

This paper introduces the notion of unsupervised failure attribution: a model is trained solely on successful trajectories and tasked, at inference, with identifying error-contributing steps within a failure trajectory. This eliminates the dependence on annotated failure data, leveraging abundant successful executions from normal operation.

Methodological Contributions

The proposed method, Oat, frames failure attribution as one-class learning in LLM representation space, using Neural Controlled Differential Equations (Neural CDEs) to model the temporal dynamics of successful agentic trajectories as latent paths. Each trajectory step is represented as an aggregated hidden state from an LLM layer, typically via mean pooling over tokens. Trajectories are normalized into a continuous-time latent path by cubically interpolating discrete step representations, enabling the model to capture the time evolution of agent behavior.

Figure 1: Oat models agent trajectories as continuous latent paths with Neural CDEs, reconstructing hidden representations and identifying anomaly steps via deviation.

The Neural CDE is augmented with a gated control path. This gating function, parameterized as a neural network, adaptively suppresses distributional shifts in trajectory derivatives that arise under out-of-distribution (OOD) contexts, improving robustness and generalizability without sacrificing in-domain fit.

Figure 2: Gated control path enhances Oat's generalizability with minimal trade-off in in-domain performance.

At inference, Oat computes per-step anomaly scores via reconstruction error between actual step representations and predicted normal flow. Two detection strategies are implemented: top-$k$ selection and conformal prediction calibration for adaptive thresholding, which offers formal coverage guarantees and automatic precision-recall tradeoff.

Empirical Evaluation

Oat is evaluated on the MCP-Atlas benchmark and Who{content}When, using failures from single-agent tool-use and multi-agent settings, respectively. The model is trained with <100 successful trajectories annotated only at the trajectory level. Across all metrics, Oat exhibits strong numerical results:

• In-domain (MCP-Atlas): Oat achieves +20% F1 improvement and +10% AUPRC over frontier prompting-based baselines, including GPT-5.
• OOD (Who{content}When): Oat yields +7% F1 improvement and +5% AUPRC despite substantial distributional shift (agent paradigm, generation model, task structure).
• Efficiency: Oat operates with zero token cost and 200–5000$\times$ lower latency than prompting-based alternatives, with <1 GB VRAM requirements for deployment.

Figure 3: Neural CDE outperforms RNN baselines across all attribution metrics.

Oat demonstrates consistent qualitative behaviors in case studies. Error-originating steps such as hallucinations, faulty code, or unfaithful assumptions are flagged with elevated anomaly scores; downstream error propagation manifests as decaying anomaly scores. Hard thresholding via conformal prediction sometimes suppresses marginally causal steps, indicating room for improvement via adaptive strategies.

Figure 4: Oat correctly flags hallucination at step 7 and error propagation at step 9.

Figure 5: Oat trades precision for recall as detection thresholds increase.

Case analyses highlight that Oat consistently surfaces meaningful indicators, even in missed ground-truth steps, assigning increasing scores to suboptimal plans and explicit failure verbalizations. Limiting factors include coverage gaps for latent reasoning failures and threshold boundary issues.

Figure 6: Oat identifies repeated useless actions and inconsistent reasoning in a successful attribution case.

Figure 7: Visualization tool supports error tracing and trajectory annotation efforts.

Ablations and Robustness

Several ablation studies support methodological choices:

• Neural CDE vs ODE: Continuous conditioning via control paths enables dynamic sensitivity to trajectory structure, outperforming deterministic ODE flows.
• Gated Control Path: Substantially boosts OOD performance (+0.172 AUROC) with minor in-domain trade-off.
• Layer selection: Later-layer LLM representations (e.g., 64th layer in Qwen3.5-27B) yield higher precision and recall; earlier layers lack task-level semantic abstraction.
• Proxy LLM setting: Oat maintains stable performance when representations are extracted via different models, demonstrating robustness.

Figure 8: Oat shows stable attribution performance across proxy LLM settings.

Practical and Theoretical Implications

Oat introduces a resource-efficient direction for agentic system failure tracing, suggesting that one-class learning on successful trajectories is both label-efficient and computationally scalable, bypassing the annotation and inference bottlenecks of prior work. The adoption of Neural CDEs for agentic trajectory modeling is novel and well-suited to irregular, long-horizon sequences common in multi-agent AI systems.

Practically, real-time deployment of attribution becomes feasible, enabling scalable monitoring, debugging, and audit of agentic systems in high-stakes domains. The unsupervised attribution paradigm improves transferability to unseen tasks and systems, as successful trajectories are readily available and can be used to recalibrate models prior to deployment.

Theoretically, the paradigm can be extended via continuous normalizing flows for density-based anomaly scoring, attention mechanisms for control path refinement, or stochastic models (Neural SDEs) to capture inherent randomness in LLM output and agent strategies.

Conclusion

This paper presents Oat, a continuous-time one-class learning method for unsupervised failure attribution in LLM-based agentic systems. By leveraging Neural CDEs and training solely on successful trajectories, Oat achieves robust, efficient error step identification outperforming prompting-based baselines across both in-domain and OOD benchmarks. Future research may explore density estimation extensions, more adaptive thresholding, and integration with stochastic trajectory modeling—potentially generalizing attribution in even more challenging agentic environments.