Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring

Abstract: Vision-Language-Action (VLA) models enable robots to follow natural language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise reliability in real-world deployment. Detecting such failures during execution is therefore critical for the robust deployment of embodied systems. Existing failure detection methods either rely on expensive action resampling or external models, while alternatives propagate trajectory-level labels uniformly across every timestep, obscuring localized failure signals. In this paper, we propose \textbf{Hide-and-Seek}, a framework that formulates VLA failure detection as a coarsely supervised learning problem. By combining inter-trajectory and intra-trajectory contrastive objectives, Hide-and-Seek localizes failure-indicative actions and induces temporally structured failure signals from trajectory-level supervision alone, without any step-level annotation. We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies: OpenVLA, $π0$, and $π{0.5}$.Our method achieves state-of-the-art multi-task failure detection performance with a practical accuracy--timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.

• The paper introduces a novel method that leverages inter- and intra-trajectory contrastive losses to pinpoint precise failure signals in VLA policies.
• It employs a lightweight LSTM-based detector to differentiate between failure and success trajectories, significantly reducing label noise.
• Experiments reveal up to +11.7% bACC improvement and >2000x lower latency compared to baselines, affirming its robust real-world application.

Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring

Overview and Motivation

Vision-Language-Action (VLA) policies have recently emerged as a central paradigm for robot control, enabling natural language-conditioned policies capable of operating across a wide range of manipulation environments and tasks. However, the absence of effective, low-overhead failure detection remains a practical barrier for robust real-world deployment. Precise annotation of failure moments across long-horizon, stochastic policies is not scalable, while current approaches—whether uniform trajectory-level label propagation, heavy action resampling, or visual-only external judging—offer suboptimal accuracy, temporal precision, or computational efficiency.

This work introduces Hide-and-Seek, a methodology for failure detection in VLA policies. The key innovation is framing runtime failure detection as a coarsely supervised learning problem: only trajectory-level labels (success/failure) are provided, and the method adaptively discovers temporally localized, failure-indicative actions. Through novel inter- and intra-trajectory contrastive objectives, the approach is capable of inferring when a failure is actually manifesting, even without step-level annotation, and does so in a computationally efficient architecture.

Hide-and-Seek: Methods

The principal insight of Hide-and-Seek is that the failure signal is typically localized—a small set of actions leading to task failure, often hidden among predominantly normal behavior in a trajectory. Uniformly assigning a failure label to all actions in a failed trajectory (as in prior work) creates significant label noise and dilutes the discriminative power of classifiers.

Instead, Hide-and-Seek enforces contrastive constraints at two levels:

• Inter-trajectory contrast: Ensures that the most failure-indicative action within any failure trajectory is scored higher than the most failure-resembling action in any success trajectory. This drives the model to focus on discriminative moments rather than treating all actions equally.
• Intra-trajectory contrast: Induces a sharp temporal separation. Within a failure trajectory, it identifies the step with the sharpest increase in score as a proxy for failure onset and enforces that post-onset actions consistently score higher than pre-onset actions. This provides temporal regularization without step-level supervision.

  Figure 1: Hide-and-Seek identifies localized critical failures from only trajectory-level labels, discovering the failure phase onset and critical moment via coarse supervision.

  

  Figure 2: Framework schematic—failure/success trajectory pairs are scored by the detector; both inter- and intra-trajectory contrastive losses guide the per-timestep failure signal shaping.

The detector is instantiated as a lightweight context-aware sequential model (e.g., LSTM), operating on representations extracted from the VLA policy backbone.

Experimental Results and Analysis

Extensive experiments are presented across three domains: simulation (LIBERO-10, VLABench) and real-world (UFactory xArm 6), and across three VLA policies (OpenVLA, $\pi_0$, $\pi_{0.5}$). The evaluation protocol is rigorous: both seen and unseen task generalization are analyzed, and a large suite of state-of-the-art baselines is included (OOD detection methods, multi-sample uncertainty, action embedding classifiers, and VLM-based video judges).

Key findings include:

• State-of-the-art accuracy: Hide-and-Seek consistently outperforms all baselines, with up to +11.7% bACC and +15.0% TWA improvement over the closest competitor on unseen real-robot tasks.
• Robust generalization: OOD detectors perform comparably on seen tasks but degrade on unseen task splits; Hide-and-Seek maintains high performance across splits.
• Timeliness vs. accuracy: The conformal prediction-based alarm threshold allows explicit control of the accuracy–timeliness tradeoff, with Hide-and-Seek achieving the earliest and most accurate detection curves.

  Figure 3: Promptness–accuracy operating curves on LIBERO-10; Hide-and-Seek dominates baselines in both promptness and balanced accuracy.

• Efficiency: Compared to VLM-based runtime monitors (Qwen3-VL-8B-Instruct), Hide-and-Seek achieves >2000x lower latency with +13.1% higher bACC.

Figure 4: Comparison of performance under uniform (baseline) versus Hide-and-Seek labeling and learning; Hide-and-Seek yields a substantial accuracy lift.

• Failure Mode Discovery: Qualitative visualizations show that the estimated failure onset aligns with semantic failure moments (stopped movement, object slippage) with no access to manual step-level labeling.

Figure 5: Failure trajectories—robot fails to grasp or drifts away after incomplete task completions, with Hide-and-Seek’s failure score correctly rising at the critical point.

Figure 6: Additional failure examples—with both subtle and gross failure modes, the method exhibits sensitivity to actual event onset.

Ablations

Further studies dissect the contributions of architecture choice, loss components, and conformal thresholding. LSTM and GRU variants outperform simple MLPs, evidencing the importance of temporal context in identifying critical transition points. Ablating either the inter- or intra-trajectory loss reduces performance, highlighting the necessity of both constraints for sharp, temporally localized detection.

Classifier-based competitors (SAFE-MLP, SAFE-LSTM) using uniform labeling perform significantly worse due to label noise, validating Hide-and-Seek’s core design—the explicit discovery and separation of the true failure event within weakly annotated data.

Figure 7: Visualization of score separation between success and failure trajectories on seen tasks.

Figure 8: tSNE projection of embeddings with uniform labeling—substantial overlap between mislabeled and true failure states.

Practical and Theoretical Implications

From a practical perspective, Hide-and-Seek enables scalable, real-time failure monitoring for generalist and specialist VLA policies without requiring detailed annotation. The computational overhead is negligible compared to the VLA policy and greatly reduced relative to VLM video judges or action resampling. The framework’s label efficiency and architecture-agnostic design position it for integration with corrective or interactive VLA deployment pipelines, where timely alarms can trigger pause, recovery, or help request mechanisms.

Theoretically, this work draws an explicit connection between weakly/partially supervised learning (multiple instance learning, weakly supervised anomaly detection) and embodied policy failure detection. The construction of loss functions that translate trajectory-level supervision into temporally structured per-action signals can be generalized to other sequential domains (e.g., fault diagnosis, medical event detection) with sparse ground-truth events.

Future Directions

Key open research directions include integrating action-level representation aggregation with kinematics or multi-modal cues, expanding the methodology to handle long chains of interacting failures, and unifying detection and recovery, such that the model not only signals failure but triggers specific policy corrections or human-in-the-loop interventions. The approach is also relevant to scalable dataset curation, policy improvement via early aborts, and the study of robust autonomous systems.

Conclusion

Hide-and-Seek provides a robust, efficient, and generalizable framework for runtime failure detection in VLA models, introducing principled contrastive objectives that enable discovery of critical failures with only trajectory-level labeling. Empirical results substantiate its superiority in terms of accuracy, timeliness, and computational viability. The method establishes a foundation for further developments in autonomous system safety, weakly supervised sequence learning, and reliable robot deployment.