Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulation

Abstract: Post-training algorithms based on deep reinforcement learning can push the limits of robotic models for specific objectives, such as generalizability, accuracy, and robustness. However, Intervention-requiring Failures (IR Failures) (e.g., a robot spilling water or breaking fragile glass) during real-world exploration happen inevitably, hindering the practical deployment of such a paradigm. To tackle this, we introduce Failure-Aware Offline-to-Online Reinforcement Learning (FARL), a new paradigm minimizing failures during real-world reinforcement learning. We create FailureBench, a benchmark that incorporates common failure scenarios requiring human intervention, and propose an algorithm that integrates a world-model-based safety critic and a recovery policy trained offline to prevent failures during online exploration. Extensive simulation and real-world experiments demonstrate the effectiveness of FARL in significantly reducing IR Failures while improving performance and generalization during online reinforcement learning post-training. FARL reduces IR Failures by 73.1% while elevating performance by 11.3% on average during real-world RL post-training. Videos and code are available at https://failure-aware-rl.github.io.

• The paper introduces the FARL framework which integrates task and recovery policies with a world-model-based safety critic to minimize failure episodes.
• It achieves up to a 65.8% reduction in simulated failures and a 73.1% decrease in real-world failure state-action pairs while maintaining strong task performance.
• The formal analysis guarantees improved policy rewards under risky actions by leveraging self-recovery mechanisms during online fine-tuning across diverse manipulation tasks.

Failure-Aware RL: A Formal Analysis and Empirical Study of Reliable Offline-to-Online Reinforcement Learning with Self-Recovery

Introduction and Motivation

The deployment of offline-to-online RL in high-stakes, real-world robotic manipulation faces a major impediment: exploration-driven failure states that require costly human intervention. Current offline-to-online RL methods, such as Uni-O4, inherently embrace exploration for policy refinement, but this exposes systems to irreversible or unrecoverable errors (e.g., pushing objects beyond workspace boundaries or breaking fragile items), presenting a critical safety bottleneck. The work "Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulation" (2601.07821) systematically addresses this challenge by proposing the Failure-Aware RL (FARL) paradigm and empirically validating its advantages for safe, high-performance RL-based manipulation.

The FARL Framework

FARL is an offline-to-online RL approach that minimizes intervention-requiring failures during policy adaptation. The core pipeline (Figure 1) consists of two phases:

Figure 1: The entire training pipeline consists of pre-training (task policy, recovery policy, world model) and a safe online fine-tuning phase.

Offline Phase:

• Task policy is initialized by behavior cloning and offline RL from sub-optimal human demonstrations.
• Recovery policy is learned via behavior cloning and offline RL, from trajectories that specifically demonstrate escapes from near-failure states.
• World Model is trained on both success and failure trajectories, capturing system dynamics and predicting the likelihood of imminent constraint violations via an auxiliary constraint prediction head.

Online Phase:

• The task policy is fine-tuned in the real (or simulated) environment under CMDP constraints.
• At every time step, the world model estimates near-term constraint violation risk for the action under consideration.
• If risk exceeds a threshold, the recovery policy intervenes, overriding the task policy to return the system to a safe region. Otherwise, the task policy proceeds.

This design instantiates a selective action correction principle, providing provable benefit under realistic assumptions about failure frequency, recovery policy efficacy, and safe-vs-risky action value difference.

Theoretical Analysis

A formal analysis is presented, showing that—under states where the baseline task policy may sample risky (constraint-violating) actions—FARL's gated action selection improves expected policy performance provided that:

• Dangerous actions occur with non-trivial probability,
• The recovery policy produces safe actions reliably ($\epsilon_{rec}$ small),
• The advantage value for safe actions exceeds that of risky actions by at least $\delta > 0$.

This yields the improvement guarantee:

$$\Delta J_{FARL} \geq \Delta J_{baseline} + \mathbb{E}_{s \sim \rho_{\pi_{task}}}\left[ p_{risk}(s) \cdot \delta \cdot (1-\epsilon_{rec}) \right] - O(\epsilon_{rec})$$

indicating that FARL achieves both safety (through risk-averse action selection) and improved task reward, particularly in environments with frequent and costly failure modes.

FailureBench: A Benchmark for Failure-Aware RL

To systematically evaluate both failure avoidance and policy improvement, the authors introduce FailureBench, a suite of robot manipulation environments instrumented with frequent, realistic failure states requiring external intervention. Representative tasks include Bounded Push, Bounded Soccer, Fragile Push Wall, and Obstructed Push (Figure 2), each testing distinct aspects of constraint satisfaction and recovery in RL.

Figure 2: FailureBench tasks: Bounded Push, Bounded Soccer, Fragile Push Wall, and Obstructed Push, modeling different forms of real-world constraint violations.

Empirical Results

Simulation

Across all FailureBench environments, FARL consistently yields dramatic reductions in failure rates during online fine-tuning compared to the Uni-O4 baseline—demonstrating an average 43.6% reduction in failure episodes, with a maximum observed reduction of 65.8% in the most challenging environments (Figure 3).

Figure 3: FARL (red) exhibits considerably fewer failure episodes during fine-tuning compared to Uni-O4 (blue) across the FailureBench suite.

Crucially, this safety gain does not come at the expense of task reward. Instead, FARL matches or surpasses baseline task performance (Figure 4) and outperforms traditional safe RL methods (e.g., PPO-Lagrangian, CPO, P3O) by over 800% in normalized returns—a strong empirical claim supported by systematic experimentation.

Figure 4: FARL achieves superior or comparable average returns while reducing failures, as measured before and after fine-tuning (normalized to expert policy).

Ablation studies confirm the necessity of both the world-model-based safety critic and a learned recovery policy. Replacing either component with simpler alternatives such as a Q-critic or MPPI planning results in increased failures and degraded performance (Figure 5).

Figure 5: FARL's integrated approach (red) outperforms ablation variants, maintaining superior returns after online adaptation.

Real-World Robotic Validation

Deployment to a Franka Emika Panda robot on tasks mirroring simulation settings further substantiates the framework's practical value. Over 50 training episodes per task, FARL reduces the total number of failure state-action pairs by 73.1% and elevates performance by 11.3% (Figure 6). These reductions directly translate to decreased human oversight and downtime—a critical requirement for robust, scalable robot learning in physical domains.

Figure 6: Total failure state-action pairs are sharply reduced for FARL across three real-world manipulation environments.

Practical and Theoretical Implications

FARL illustrates that model-based safety critics and recovery policies—trained solely from offline demonstration—enable safe, reward-efficient RL policy refinement in environments where failures are costly or irrecoverable. Notably, the method demonstrates strong generalization across diverse manipulation tasks, obviating the need for inefficient online resets and minimizing distributional shift pitfalls faced by standard safe RL algorithms.

Theoretically, the presented action-correction analysis provides a rigorous foundation for the efficacy of recovery policy intervention, bridging the gap between safe RL and practical offline-to-online learning. These findings point to several promising research directions:

• Scaling to richer modalities: Incorporation of high-dimensional sensory inputs (e.g., 3D vision, tactile feedback) for improved model accuracy and robustness.
• Robust cross-domain failure prediction: Pre-training large-scale world models to generalize failure detection and recovery skills across task and embodiment boundaries.
• Adaptive recovery mechanics: Online adaptation of recovery policies as new forms of failures are encountered in the wild.

Conclusion

This work advances failure-aware reinforcement learning for real-world manipulation by combining offline world-modeling of constraint violations, demonstration-driven recovery policy design, and rigorous theoretical and empirical evaluation. FARL sets a new standard for safe, reliable, and high-performing offline-to-online RL in robotics, highlighting the need for explicit failure modeling and intervention mechanisms in the progression toward robust autonomous learning systems.