ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies

Abstract: Generative robot policies fail unpredictably at deployment: they hesitate at critical moments, drift off-task, or commit to unrecoverable actions. Existing online failure detectors either require white-box access to policy internals or add runtime overhead through resampling and observation-side signals. Our empirical analysis shows that emitted action chunks themselves already carry strong predictive signal for impending failures in generative robot policies. Motivated by this observation, we introduce ActProbe, a lightweight, pure action-space detector that uses two compact signals available from a single forward pass: Temporal Consistency Error (TCE) between consecutive action chunks and Action Chunk Magnitude (ACM) of the current chunk. ActProbe maps these signals to per-step failure probabilities with a task-conditioned LSTM-MLP architecture. Across a diverse suite of generative robot policies and benchmarks, ActProbe raises alerts before failures become visually recognizable, improving the accuracy (F1)-timeliness Pareto frontier of failure detection by an average hypervolume gain of +12.7% over both internal- and external-feature baselines, with a +9.0% early-detection ROC-AUC lead on unseen tasks. ActProbe further transfers to deployment, predicting failures on unseen real-robot pick tasks and accelerating RL fine-tuning (PPO) with 2.9x fewer environment interactions.

• The paper introduces ActProbe, a novel failure detection method using Temporal Consistency Error and Action Chunk Magnitude to anticipate robotic policy failures early.
• It employs a bridged LSTM–MLP architecture integrated with language embeddings to achieve robust cross-task and cross-policy generalization.
• Experimental evaluations show significant improvements in F1 accuracy, early detection latency, and RL sample efficiency compared to baseline methods.

ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies

Introduction and Motivation

The deployment of generative robot policies—such as Vision-Language-Action models (VLAs) and World-Action Models (WAMs)—in real-world manipulation tasks exposes systems to unpredictable failures due to the inherent stochasticity and coverage gaps in the learned artifacts. Failures often materialize as hesitation, drift, or unrecoverable unintended actions, necessitating robust and timely failure detection to enable system-level interventions or safety fallbacks. Existing online failure detectors either require intrusive, white-box access to proprietary model internals or are computationally heavy, leveraging observation-based or resampling-based uncertainty signals, leading to significant overhead and limitations in generalization.

This work introduces ActProbe, a lightweight, fully action-space-based failure detector that operates solely on two low-dimensional action-space signals, eschews observation-side fusion or policy introspection, and is readily deployable in black-box settings (Figure 1).

Figure 1: Overview of ActProbe, which triggers alarms preemptively using action-space signals, ahead of temporal windows where failures are visually apparent.

Methodology

Action-Space Signal Design

ActProbe leverages two informative signals derived from robot policy outputs:

1. Temporal Consistency Error (TCE): Captures the mean squared error between overlapping segments of consecutive action chunks, acting as a surrogate indicator of trajectory instability or policy indecision across replanning steps.
2. Action Chunk Magnitude (ACM): Quantifies the L2 norm of the policy's issued action chunk; large deviations are empirically correlated with imminent failure events, consistent with motor control literature.

Both features are directly accessible from emitted action sequences without additional forward passes or intrusive access. Empirical analysis demonstrates that these features sharply distinguish successful from failed rollouts and subsume detection coverage relative to existing detectors such as SAFE-MLP and RND-based external probes (Figure 2).

Figure 2: TCE and ACM diverge pre-failure (left), and ActProbe’s detection coverage robustly subsumes prior detectors (right).

Model Architecture

ActProbe introduces a bridged LSTM–MLP architecture. The recurrent pathway (LSTM) aggregates historical context of TCE and ACM, encoding temporally integrated patterns, while a direct skip (bridge) from the current feature vector to the classification head retains sensitivity to instantaneous anomalies. The probe is conditioned on language embeddings of the task specification via Qwen3, supporting robust multi-task and out-of-distribution transfer.

Training and Calibration

Supervised training is performed using binary cross-entropy over episodic success/failure signals, with model inference rate synchronized to the policy’s replan frequency. Deployment thresholding uses split-conformal quantile calibration, targeting statistically controlled false-positive rates. ActProbe’s minimal (~24K) parameterization enables efficient, robust deployment across a variety of policy classes.

Experimental Evaluation

End-to-End Benchmarking

Evaluations span five major policy–environment pairs (OpenVLA, $\pi_0$, $\pi_{0.5}$, GR00T; LIBERO and RoboCasa benchmarks) under both seen-task and unseen-task splits. Relative to both white-box (SAFE-LSTM, SAFE-MLP, Cosine $k$-NN) and black-box (LogpZO, STAC-Single, RND-OE) baselines, ActProbe consistently advances the Pareto frontier of F1 accuracy versus detection latency by an average hypervolume gain of +12.7% and achieves a +9.0% early-detection ROC-AUC lead on unseen tasks.

Figure 3: F1 versus average detection time curve. ActProbe consistently dominates baselines across all benchmarks and data regimes.

Signal and Architecture Ablation

Systematic ablation over feature computation and probe architecture establishes that native overlap MSE-based TCE is superior to K-resampled or alternative distances, and the LSTM–MLP bridge architecture yields additional early detection gains versus standalone MLP or LSTM variants. Robustness is maintained across both data-poor (few-shot) and data-rich regimes.

Cross-Task and Cross-Policy Generalization

Experimental results show that:

• Task-level transfer: Policy failures in behaviorally rich categories (pick-and-place, turning appliances) transfer well to entirely unseen task configurations. Language embedding conditioning via Qwen3 supports such generalization; even same-category language substitution preserves high detection AUC (Figure 4).
• Policy-level transfer: ActProbe trained on GR00T generalizes zero-shot to $\pi_{0.5}$, producing AUC in the 58–70% range.

  Figure 4: Cumulative generalization as the number of training tasks increases. High AUC is achieved in new categories after limited exposure to behaviorally diverse sources.

Data Scaling

Scaling the number of training rollouts per task from 50 to 200 produces significant improvements; ActProbe’s accuracy–timeliness tradeoff frontier expands, with the gap over SAFE-MLP widening at higher data regimes (Figure 5).

Figure 5: ActProbe's F1–timeliness frontier expands with more data, outpacing SAFE-MLP.

Deployment: Real Robot and RL Acceleration

On hardware (Unitree G1-D, two unseen pick tasks), ActProbe outperforms both SAFE-MLP and STAC-Single without retraining, demonstrating robustness to perception noise and dynamics shift (Figure 6).

Figure 6: Real-robot deployment environment for pick tasks.

In RL fine-tuning (sparse reward, LIBERO-Object), ActProbe-enabled early failure termination accelerates training, achieving baseline-level performance with 2.9× fewer environment interactions, a direct benefit for sample efficiency in RL pipelines (Figure 7).

Figure 7: RL fine-tuning learning curve. ActProbe-based early termination yields significant interaction efficiency gains.

Per-$\alpha$ Operating Characteristics

Detailed analysis shows that ActProbe consistently yields favorable TNR/TPR/balanced accuracy curves as a function of calibrated FPR $\alpha$, maintaining high true positive rate (TPR) with only modest degradation in TNR even under unseen-task evaluation (Figure 8).

Figure 8: Per-$\alpha$ TNR/TPR/balanced accuracy plots. ActProbe curves are stable across seen/unseen splits.

Theoretical and Practical Implications

This research underscores that high-fidelity failure anticipation in generative policy deployment is achievable with purely action-space signals, supporting robust deployment in black-box, resource-constrained, and policy-agnostic settings. The proposed methodology is both generalizable—requiring minimal task-specific adaptation—and scalable as more data supports increasingly fine-grained predictive boundaries. The RL acceleration results suggest substantial potential for automatic sample-efficiency improvements in safety-critical RL pipelines via automatic early termination strategies.

From a theoretical perspective, the work supports the hypothesis that performance failures in high-level BC/IL policies causally propagate to low-level action emissions well in advance of visually recognizable errors, providing new foundations for learning robust, interpretable early warning signals using purely behavioral traces.

Conclusion

ActProbe demonstrates that compact, action-space signals—TCE and ACM—are sufficient for timely, accurate, and generalizable early failure detection in generative robot policies. By avoiding the limitations of hidden-state and observation-resampling-based detectors, ActProbe sets a new state-of-the-art in action-side failure detection. Its robust cross-task, cross-policy, and real-robot transfer characteristics, coupled with RL sample-efficiency gains, position it as a practical and theoretically compelling solution for safety and reliability in large-scale embodied AI deployment.

(2606.08508)