From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulation

Abstract: Although robotic manipulation has made significant progress, reliable execution remains challenging because task failures are inevitable in dynamic and unstructured environments. To handle such failures, existing frameworks typically follow a stepwise detect-reason-recover pipeline, which often incurs high latency and limited robustness due to delayed reasoning and reactive planning. Inspired by the human capability to anticipate and proactively plan for potential failures, we introduce AgentChord, an agentic system that models a manipulation task as a directed task graph. Before execution, this graph is enriched with anticipatory recovery branches that specify context-aware corrective behaviors, enabling immediate and targeted responses when failures occur. Specifically, AgentChord operates through a choreography of specialized agents: a composer that structures the nominal task graph, an arranger that augments the graph with anticipatory recovery branches, and a conductor that compiles and coordinates executable transitions using low-latency monitors to detect deviations and trigger pre-compiled recoveries without re-planning. Empirical studies on diverse long-horizon bimanual manipulation tasks demonstrate that AgentChord substantially improves success rates and execution efficiency, advancing the reliability and autonomy of real-world robotic systems. The project page is available at: https://shengxu.net/AgentChord/.

• The paper introduces AgentChord, a novel system that proactively augments task graphs with recovery branches to anticipate failures in robotic manipulation.
• The methodology employs three agents to structure tasks, orchestrate recovery, and compile executable commands using constraint-aware transitions and real-time monitors.
• Empirical evaluations demonstrate higher success rates and reduced execution times in both simulation and real-world robotic tasks compared to reactive approaches.

Proactive Failure Recovery in Robotic Manipulation: An Analysis of AgentChord

Motivation and Context

Persistent unreliability in robotic manipulation arises from inherent task failures encountered in real-world settings, especially in dynamic and unstructured environments. Conventional frameworks largely employ reactive pipelines (“detect-reason-recover”) that suffer from delayed response and limited robustness due to inefficiency in re-planning and imprecise grounding. Recent integration of MLLMs has advanced the detection of failures by treating it as a VQA problem but has introduced bottlenecks (latency, lack of fine-grained spatial grounding) and failed to fundamentally address recovery latency or regressive resets. The paper "From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulation" (2605.11951) introduces AgentChord, an agentic manipulation system advancing failure recovery by proactive anticipation.

AgentChord: System Architecture

AgentChord models a manipulation task as a directed task graph, with semantic sub-goals (nodes) and constraint-aware transitions (edges). The framework is based on three agents:

• Task Structuring Agent: Interprets high-level task instructions and sensory input to synthesize nominal task graphs with explicit sub-goals and constraints.
• Recovery Orchestration Agent: Anticipates likely failure modes for each edge and augments the graph before execution with recovery branches and nodes, specifying context-aware corrective behaviors.
• Execution Compilation Agent: Compiles both nominal and recovery transitions into executable, interruptible programs with monitors and triggers leveraging hierarchical constrained solvers.

During execution, multimodal monitors continuously evaluate constraint violations and instantly invoke the corresponding pre-compiled recovery branch, preventing the latency of re-invoking full task planning.

Figure 1: AgentChord system architecture involves task graph construction, augmentation with anticipatory recovery branches, and compilation into executable programs with low-latency monitors.

Recovery-Augmented Task Graph

The critical innovation is the proactive, forward-moving recovery paradigm: before execution, the task graph is augmented with recovery nodes and branches representing anticipated failures. Recovery is guaranteed to be forward-moving (never regressive), enforced by shortest-path constraints that ensure any triggered recovery branch reduces or preserves task progression cost to the goal node.

Upon the activation of a monitor (e.g., persistent violation of geometric, proprioceptive, or relational constraints), execution immediately switches to a recovery node, executes corrective actions (such as re-grasp, re-align, or re-approach), and resumes nominal progress without regressive resets or re-planning.

Implementation: Perception and Execution

AgentChord utilizes open-vocabulary segmentation and pose estimation for scene understanding (SAM3, AnyGrasp, FoundationPose), and atomic manipulation skills from a library for action realization. Feature extraction combines scene representation (point clouds, principal axes, boundary points) and robot state (joint configuration, gripper metrics) into $z_t$ for constraint and failure monitoring.

Hierarchical constrained solvers perform both node-level (sub-goal realization) and edge-level (transition planning) synthesis, optimizing for collision avoidance, reachability, smoothness, and bimanual coordination. Monitors are compiled as scalar functions over $z_t$, triggering recovery only upon persistent violation (robust to transient noise).

Figure 2: Hardware system with AgileX CobotMagic dual-arm robot and stereo RGB-D cameras, facilitating robust bimanual manipulation and perception.

Empirical Evaluation

Simulation Experiments

AgentChord was evaluated on long-horizon tasks in simulation with stochastic disturbances (object drop). Results demonstrate:

• Success Rate (Average): AgentChord 99.2%, outperforming reactive and backtracking-based baselines (IM, DRM, ReKep, CaM).
• Execution Efficiency: Lowest average execution time (41.5s vs. CaM 78.1s) and episode steps.
• Ablation Study: AgentChord’s proactive recovery graph yields higher success and lower execution time than backtracking variants.

Real-World Experiments

Six real-world tasks spanning single-arm, bimanual, rigid, and deformable objects were conducted under controlled disturbances. AgentChord achieved the highest success rate (77.5%) and the lowest execution time (92.2s) across all tasks. Recovery is immediate and efficient; the system adapts to diverse objects, poses, and disturbance profiles.

Figure 3: Failure detection and recovery in dual-arm pour water—upon detection, execution switches to pre-compiled recovery branch and rejoins nominal task graph.

Figure 4: Simulation tasks demonstrate AgentChord’s ability to detect failures and execute recovery branches seamlessly.

Figure 5: Real-world task success across trials with varied object types and disturbances—AgentChord consistently completes tasks.

Figure 6: AgentChord executes pour water task robustly in cluttered environments, demonstrating strong generalization.

Policy Learning

The trajectories generated by AgentChord, including failure-recoverable actions, were shown to improve policy learning for robotic manipulation. Policies fine-tuned with recovery trajectories exhibited substantially higher robustness in failure scenarios, indicating the utility of structured recovery-augmented data for downstream autonomous skill acquisition.

Failure Modes, Limitations, and Analysis

Failures in AgentChord were primarily attributed to imperfect anticipation, reasoning inaccuracies in long-horizon settings, kinematic infeasibility during corrective action synthesis, and perception noise-induced errors. The modular feature extractor ($\Phi$) allows integration of stronger perceptual models to enhance robustness. Rare, compound, or unanticipated failures may be missed—future systems can overlay MLLM-based diagnostics at task boundaries or timeout events to synthesize novel recovery branches online.

Implications and Future Directions

AgentChord’s recovery-augmented task graph represents a practical, extensible framework for integrating proactive failure anticipation and efficient recovery within interpretable agentic robotic systems. The ability to immediately activate recovery paths without regressive planning substantially advances autonomy, execution efficiency, and reliability. The approach suggests that proactive, graph-based scheduling of failure handling may generalize to other embodied domains, and the recovery data can fuel robust policy learning.

Future directions include leveraging stronger multimodal reasoning models for improved anticipation coverage, integrating richer geometric-semantic features, adapting the framework to highly deformable or articulated objects, and applying online augmentation for unforeseen failures.

Conclusion

AgentChord operationalizes anticipatory failure recovery in robotic manipulation through recovery-augmented task graphs, agentic orchestration, and low-latency execution. Empirical evidence demonstrates substantially improved success rates and efficiency over state-of-the-art baselines in both simulation and real-world scenarios. The structured recovery paradigm enables robust execution, effective data generation for policy learning, and sustained progress in autonomous manipulation. This paradigm establishes a new baseline for integrating anticipation, monitoring, and recovery in agentic robotic systems.