ForeSight: Long-Horizon Failure Detection

• ForeSight is a failure detection framework for long-horizon robotic manipulation that leverages action‐conditioned world model latents to detect trajectory deviations early.
• It employs a causal sequence model and functional conformal prediction to calibrate time-varying thresholds for reliable failure monitoring.
• The framework is policy-agnostic and has been validated in simulations and real-robot experiments on state-of-the-art vision-language-action tasks.

ForeSight is a failure detection framework for long-horizon robotic manipulation that monitors manipulation trajectories using latent representations from an action-conditioned world model. It is trained using only final task-level success or failure labels, without dense temporal annotations, and uses functional conformal prediction to calibrate time-varying detection thresholds. In the reported evaluation, the framework is applied to state-of-the-art vision-language-action policies in simulation on LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K, and on real robots with three long-horizon tasks on a ReactorX-200 arm and one task on a Franka arm. The reported results suggest that action-conditioned world-model embeddings provide a scalable representation for reliable failure monitoring in long-horizon manipulation (Zhang et al., 22 Jun 2026).

1. Problem formulation and scope

ForeSight is designed for long-horizon robotic manipulation tasks in which failure onset is often ambiguous and dense temporal annotations are typically unavailable. This operating regime is central to real-world deployment: tasks may span hundreds to thousands of steps, and partial observations can leave the distinction between transient deviation and irreversible failure unclear. The framework addresses this setting by treating failure detection as runtime monitoring over the trajectory rather than as post hoc classification of a completed rollout.

A defining property of the method is that it is policy-agnostic. It does not rely on policy internals, and it is intended to function across different policies and robot platforms as long as visual observations and actions are recorded. This design separates the monitoring problem from the control policy itself. A direct implication is that failure detection can be trained and deployed even when the monitored policy changes, provided the observation-action interface remains available.

Another defining choice is supervision at the trajectory level. Each rollout carries only a final success or failure label, $y \in \{0,1\}$, and every step in the rollout inherits that final label during training. This removes the need for dense failure annotations while still permitting a stepwise failure score at runtime. The framework further encourages early detection through special loss weighting so that alarms are raised before failures actually occur. This distinguishes ForeSight from methods that require temporally localized labels or policy-specific uncertainty signals (Zhang et al., 22 Jun 2026).

2. Action-conditioned world-model latents

At runtime, ForeSight uses latent features from a pretrained action-conditioned world model, notably V-JEPA 2-AC. The input at step $t$ consists of an image observation context $c_t$ and a policy-predicted action chunk

$A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$

The visual context is encoded as

$z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$

and the action-conditioned predictor produces

$$z_t^p = \mathrm{Pool}\left(g_\psi(z_t^h, A_t)\right).$$

The distinction between $z_t^h$ and $z_t^p$ is central. The latent $z_t^h$ is the current observed latent, whereas $z_t^p$ is the predicted latent for the next action chunk and encodes how the world should look if the planned actions are successful. ForeSight then constructs monitoring tokens

$t$0

where $t$1 is a linear projection and $t$2 is a sinusoidal positional encoding. The token sequence up to time $t$3,

$t$4

is processed by a causal sequence model,

$t$5

where $t$6 is the failure score at step $t$7, with higher values indicating that the trajectory is more likely to be on a path to failure.

The framework evaluates MLP, LSTM, and Transformer detectors, with the causal Transformer reported as best. The action-conditioned predictor is trained with the sum of teacher-forcing and short-horizon autoregressive rollout $t$8 losses, normalized, while the failure detector uses binary cross-entropy loss, possibly weighted to encourage early failure detection. The reported ablations indicate that action-conditioning—using $t$9 rather than $c_t$0—improves results, and that sequence modeling is necessary, with frame-level MLPs being insufficient. This suggests that the monitoring signal is not merely visual anomaly detection, but a mismatch between intended action progression and predicted future state (Zhang et al., 22 Jun 2026).

3. Functional conformal prediction and threshold calibration

ForeSight does not use a fixed global alarm threshold. Instead, it applies functional conformal prediction to construct a time-varying, statistically controlled decision band from held-out successful rollouts. For each timestep $c_t$1, the calibration procedure computes a mean score

$c_t$2

and a standard deviation term $c_t$3. For each calibration rollout, it defines a nonconformity score

$c_t$4

Let $c_t$5 denote the $c_t$6-quantile of the set $c_t$7. The bandwidth is then

$c_t$8

which yields the time-varying threshold

$c_t$9

The alarm rule is

$A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$0

This construction gives the detector an adaptive threshold that varies over rollout time rather than assuming a stationary score distribution. The reported statistical guarantee is that, with significance level $A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$1, the probability of false positives—raising an alarm on an actually successful rollout—is controlled across all timesteps. In practical terms, this is intended to make detection thresholds compatible with the fact that long-horizon manipulation trajectories can have different score profiles at different phases of execution (Zhang et al., 22 Jun 2026).

4. Evaluation regime and comparative setting

The empirical study spans both simulation and real-robot deployment. The simulation benchmarks are LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K. The real-robot experiments use a ReactorX-200 arm in tabletop settings and a Franka arm. Policies evaluated include OpenVLA, SmolVLA, $A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$2-FAST, $A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$3, ACT, and GR00T N1.5. Baselines include FAIL-Detect, SAFE (MLP/LSTM), RND (ReDiffuser), and Gauge. All methods use the same data splits and calibration protocol for fair comparison (Zhang et al., 22 Jun 2026).

Setting	Environment or platform	Reported scale
Simulation	LIBERO-Long	$A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$4 steps/rollout
Simulation	ManiSkill-Long	$A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$5 steps/rollout
Simulation	BEHAVIOR-1K	$A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$6 steps/rollout
Real robot	ReactorX-200 arm	multiple policies
Real robot	Franka arm	GR00T N1.5

The evaluation uses two metrics. ROC-AUC measures discriminative ability, using the maximum over $A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$7 for each rollout. Balanced Accuracy is

$A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$8

calibrated at the best $A_t = (a_{t|t}, a_{t+1|t}, \dots, a_{t+H-1|t}).$9. This pairing separates ranking quality from thresholded decision quality. Because the benchmarks differ substantially in rollout length and embodiment, the experimental setup is aimed at testing both horizon scaling and transfer across control policies and robot platforms.

5. Reported results, ablations, and transfer properties

The Transformer-based variant of ForeSight is reported as achieving the best or among the best ROC-AUC and balanced accuracy across the long-horizon simulation benchmarks. The reported values are ROC-AUC $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$0 and BalAcc $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$1 on LIBERO-Long, ROC-AUC $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$2 and BalAcc $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$3 on ManiSkill-Long, and ROC-AUC $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$4 and BalAcc $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$5 on BEHAVIOR-1K. The BEHAVIOR-1K result is described as having the largest performance gap versus baselines. The method is reported to outperform Gauge, SAFE, RND, and FAIL-Detect.

On real robots, the framework maintains consistently high ROC-AUC on real-robot rollouts, with reported values up to $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$6 for ACT and $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$7 for GR00T on Franka. Cross-policy transfer is also reported: the detector can be trained on one policy and tested on another. The paper notes, however, that performance depends on the training policy's behavioral diversity. This is important because the method is explicitly positioned as policy-agnostic, but policy-agnosticism does not imply policy-invariant data coverage.

The ablation results support two technical conclusions. First, action-conditioning is crucial for catching failures where the scene looks normal but is inconsistent with intended actions. Second, temporal sequence modeling is necessary; Transformer and LSTM detectors outperform frame-level MLPs. These findings indicate that the predictive latent is not simply a compressed visual observation, but a representation of action-conditioned future consistency.

The implementation profile is also reported. The world model uses a ViT-Giant encoder and 24 transformer layers for the action predictor. The action-conditioned predictor has $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$8 million parameters. End-to-end inference time is approximately $z_t^h = \mathrm{Pool}\left(f_\phi(c_t)\right),$9 ms per replan interval, described as well within a typical replan window. Runtime cost is dominated by the frozen world model, while the detection head itself is fast. This suggests that deployment feasibility depends less on the detector architecture than on the availability of world-model inference at the required control cadence (Zhang et al., 22 Jun 2026).

6. Limitations, deployment considerations, and nomenclature

Two limitations are stated explicitly. First, the computational overhead of large world models may be problematic for on-device, real-time deployment on highly dynamic tasks. Second, the statistical guarantees of functional conformal prediction rely on the calibration set matching test-time conditions. These caveats bound the scope of the method: its thresholding is statistically controlled only under the calibration regime, and its runtime profile is manageable for moderate-horizon tasks but may become restrictive when the perception-control loop tightens.

A common misunderstanding would be to treat ForeSight as a general-purpose anomaly detector. The reported design is narrower and more structured: it monitors trajectories through predictive embeddings from an action-conditioned world model and calibrates alarms against successful rollouts. A plausible implication is that its strongest use case is failure monitoring when the intended action sequence itself carries indispensable information about whether an apparently normal scene evolution is acceptable.

The name “ForeSight” or “Foresight” is also used for unrelated systems in other subfields. Examples in the cited literature include a foundation world model centric planning framework for autonomous driving (Zhang et al., 8 May 2026), a joint detection and trajectory forecasting framework for vision-based 3D perception in autonomous vehicles (Papais et al., 9 Aug 2025), a multimodal interleaved reasoning framework for vision-LLMs (Wu et al., 27 Apr 2026), a test-time framework for open-world mapless navigation (Zhang et al., 10 Jun 2026), an adaptive layer-reuse technique for text-to-video generation (Adnan et al., 31 May 2025), and visualization recommender systems for exploratory data analysis (Demiralp et al., 2017). In the robotics literature considered here, however, ForeSight denotes the long-horizon manipulation failure detection framework built around action-conditioned world-model latents and functional conformal prediction (Zhang et al., 22 Jun 2026).