- The paper introduces SPR, a progress-aware VLA framework that combines milestone-based spatial planning, anomaly detection, and three-step rewind recovery without additional data collection or inference models.
- SPR achieves 91.8% average success on LIBERO and 71.8% on LIBERO-Plus, showing stronger long-horizon performance and the smallest robustness degradation among compared methods.
- Real-robot tests show gains over MolmoAct on pickup, table tidying, and Push-T tasks, while failures reveal remaining challenges in precise actions, physical stuck-state detection, and planning–execution consistency.
Motivation and Problem Statement
Vision-language-action (VLA) models trained on large-scale demonstration data have achieved strong in-distribution manipulation performance, yet they remain fragile when execution deviates from the training distribution: errors accumulate, failed grasps propagate, and models persist in out-of-distribution (OOD) states without any mechanism for detection or correction. The paper attributes this fragility to a missing capability it formalizes as progress awareness — the ability to measure task execution against a sequence of concrete, actionable milestones, rather than relying on abstract linguistic plans or binary success flags. Existing approaches to this problem carry substantial costs: FailSafe requires dedicated failure-data collection, REFLECT depends on hand-engineered LLM prompting, and methods such as ECoT produce progress signals that lack spatial grounding for action generation.
The paper introduces See, Plan, Rewind (SPR), a progress-aware VLA framework built on the MolmoAct architecture that closes this gap through a three-phase execution cycle: Seeing the current state and remaining milestones, Planning a 2D trajectory toward the next spatial waypoint, and Rewinding to a recoverable configuration when progress monitoring detects sustained anomalies. Notably, both the supervision pipeline and the recovery mechanism require no additional training data collection, no auxiliary inference-time models, and no extra computational overhead beyond the 7B baseline.
The See-Plan-Rewind Framework
Spatial Subtask Planning
SPR extends MolmoAct's autoregressive action reasoning with a structured token-generation factorization. Given an RGB observation I and instruction T, the model generates: depth perception tokens d; a remaining-subtask count n; subtask specifications s=(s1,…,sn), each pairing a semantic description with 2D completion coordinates; trajectory tokens τ containing up to five waypoints from the current gripper position to the next subtask goal; and finally action tokens a. Planning toward the next subtask rather than the final goal is a deliberate design choice: for long-horizon tasks, the final goal may be spatially irrelevant or misleading before intermediate milestones are complete.
Automated Data Curation
The framework's practicality hinges on a supervision pipeline that extracts all signals from existing demonstrations. Subtask boundaries are detected task-adaptively: gripper open/close transitions serve as reliable boundaries for pick-and-place tasks (used for all LIBERO data, with DeepSeek-R1 prompted to generate segment descriptions given the detected count), while Gemini-3 video annotation handles continuous-contact tasks such as Push-T where gripper state is uninformative. Spatial grounding is obtained without task-specific training by combining DINOv3 patch-feature matching for coarse gripper localization with SAM-based segmentation refinement; the final 2D coordinates fuse the x-coordinate from SAM's bounding-box center with the y-coordinate from DINOv3 detection, discretized to [0,255]. Trajectories are smoothed via outlier interpolation and a temporal median filter that preserves subtask boundaries.
Progress-Driven Rewind
Recovery is implemented entirely through joint training on synthesized rewind data: forward demonstrations are temporally reversed from the first subtask waypoint back to the initial position, with action tokens negated and the instruction replaced by "return to initial position." At inference, a FIFO state recorder monitors two anomaly criteria over sliding windows: a subtask count anomaly (predicted count increases across two consecutive 4-step windows, indicating regression) and progress stagnation (identical planned 2D trajectories across 8 consecutive steps, indicating the robot is trapped in an OOD state). Both require persistence over multiple timesteps to filter transient prediction noise. Upon detection, the instruction is swapped to the rewind command for T0 steps — empirically optimal, as fewer steps provide insufficient clearance and more steps drive the arm outside the camera's field of view — after which normal execution resumes.
Experimental Results
On LIBERO, SPR reaches 90.6% average success with per-subset models (+3.8% over the reproduced MolmoAct baseline) and 91.8% with a jointly trained model (+5.0%), exceeding OpenVLA, T1-fast, GRAPE, and ThinkAct. The largest gains appear on LIBERO-Long (+5.6% and +8.2% respectively), consistent with the hypothesis that milestone-based planning benefits multi-step tasks most. Importantly, joint training improves rather than degrades performance, which the authors interpret as evidence that SPR learns generalizable progress-aware reasoning rather than overfitting to a task distribution.
Out-of-Distribution Robustness
The most striking results come from LIBERO-Plus, which perturbs backgrounds, robot initial states, language phrasings, object layouts, and lighting across thousands of variants. SPR achieves the highest average success rate (71.8%) with the smallest average degradation (−18.8%), compared to −27.0% for OpenVLA-OFT and −37.5% for UniVLA. Per-perturbation results:
| Method |
Background |
Robot |
Language |
Layout |
Light |
Avg |
| OpenVLA-OFT |
83.6% |
30.6% |
83.6% |
73.2% |
91.6% |
70.6% |
| T2-fast |
67.7% |
24.8% |
63.3% |
70.3% |
73.0% |
58.4% |
| UniVLA |
80.0% |
50.3% |
71.8% |
34.3% |
59.1% |
57.7% |
| SPR (Ours) |
86.0% |
47.7% |
78.5% |
69.6% |
85.0% |
71.8% |
SPR attains the smallest degradation on Language (−12.1%), Background (−4.6%), Light (−5.6%), and Robot (−42.9%) perturbations. The Robot-perturbation robustness is attributed directly to the Rewind mechanism's ability to restore recoverable gripper poses, while the Language robustness reflects semantics-based progress grounding.
Real-Robot Evaluation
Across three real tasks (10 trials each), SPR outperforms the MolmoAct baseline on all: 70% vs. 50% on single-object pick-up, 30% vs. 0% on long-horizon table tidying, and 40% vs. 0% on Push-T. The Push-T result is significant because it demonstrates the framework extends beyond gripper-transition-delineated tasks: SPR decomposes continuous-contact pushing into five phases (approach, adjust, push, align, fine-tune) via the VLM annotation path. A scaling analysis shows SPR degrades gracefully from 1 to 4 objects in the tidying task, whereas the baseline collapses entirely at 3 objects — evidence that subtask-level planning becomes increasingly critical with task complexity.
Ablations
Removing the Rewind mechanism costs 1.0% on LIBERO average (89.6% → 90.6%), but the Rewind contribution is larger on LIBERO-Plus Language and Robot variants and manifests as faster task completion at extended episode lengths (980 steps): progress-aware models continue completing tasks after the baseline plateaus, and the full model recovers from failed subtasks more quickly than the w/o-Rewind variant, validating the design rationale of elevating the arm for operational clearance. Removing semantic descriptions while retaining 2D coordinates costs 3.4% on LIBERO-Long and degrades OOD robustness, establishing that spatial coordinates and semantics play complementary roles.
Limitations and Open Questions
The supplementary failure analysis is candid about three failure modes. First, discrete action tokens limit precision: quantization errors cause failures on millimeter-level placement tasks, such as placing a mug at a plate's edge rather than its center. Second, the Rewind mechanism fails when the robot is physically stuck — the anomaly detector senses logical inconsistency (unchanged subtask counts) but has no direct physical-state awareness, so a mug lodged at a microwave edge cannot be recovered. Third, and most concerning, the model sometimes exhibits a disconnect between planning and execution: after a successful rewind, predicted 2D trajectories and waypoints can be correct while generated actions deviate from them, indicating that spatial awareness alone does not guarantee action consistency.
Additional constraints are implicit in the method design: the fixed rewind horizon T3 is tuned empirically and risks the arm exiting the camera view; the anomaly thresholds (4-step count window, 8-step trajectory window) are hand-set; and subtask annotation for non-pick-and-place tasks depends on Gemini-3's video understanding, introducing reliance on an external VLM at the curation stage. The paper leaves open how to close the planning–execution consistency gap and how to extend progress awareness to physical contact states rather than purely visual progress signals.
Conclusion
SPR demonstrates that explicit, spatially grounded progress awareness can be injected into a standard autoregressive VLA with modest architectural change and no additional data collection, yielding consistent in-distribution gains (+5.0% over MolmoAct on LIBERO), state-of-the-art OOD robustness on LIBERO-Plus (−18.8% average degradation versus −27.0% to −57.8% for competitors), and substantial real-robot improvements on long-horizon and continuous-contact tasks where the baseline fails entirely. The framework's principal contribution is reframing error recovery as an executable consequence of progress monitoring rather than a separately trained or prompted subsystem — while its documented failure modes identify action-execution fidelity and physical-state awareness as the remaining obstacles to fully autonomous recovery.