Long-Horizon Manipulation via Trace-Conditioned VLA Planning

Abstract: Long-horizon manipulation remains challenging for vision-language-action (VLA) policies: real tasks are multi-step, progress-dependent, and brittle to compounding execution errors. We present LoHo-Manip, a modular framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM. The manager is decoupled from the executor and is invoked in a receding-horizon manner: given the current observation, it predicts a progress-aware remaining plan that combines (i) a subtask sequence with an explicit done + remaining split as lightweight language memory, and (ii) a visual trace -- a compact 2D keypoint trajectory prompt specifying where to go and what to approach next. The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control by following the trace. Crucially, predicting the remaining plan at each step yields an implicit closed loop: failed steps persist in subsequent outputs, and traces update accordingly, enabling automatic continuation and replanning without hand-crafted recovery logic or brittle visual-history buffers. Extensive experiments spanning embodied planning, long-horizon reasoning, trajectory prediction, and end-to-end manipulation in simulation and on a real Franka robot demonstrate strong gains in long-horizon success, robustness, and out-of-distribution generalization. Project page: https://www.liuisabella.com/LoHoManip

• The paper presents LoHo-Manip that decouples high-level planning from low-level control to mitigate error accumulation in extended robotic tasks.
• It employs a task manager to generate sub-task decompositions and visual trace predictions, guiding a short-horizon VLA executor for precise movement.
• Experimental results demonstrate state-of-the-art BLEU scores and enhanced spatial trajectory accuracy across diverse simulation and real-world benchmarks.

Long-Horizon Manipulation via Trace-Conditioned VLA Planning: A Technical Perspective

Introduction and Problem Context

Robotic manipulation in real-world environments requires bridging the gap between single-step decision making and robust, long-horizon task execution under continuous feedback and failure recovery. Existing Vision-Language-Action (VLA) models excel at short-horizon, atomic tasks but are fundamentally limited by error accumulation, weak temporal abstraction, and inability to recover from drift when deployed in open-ended multi-step contexts. Furthermore, most contemporary architectures entangle high-level planning and low-level control within monolithic models, severely restricting extensibility, robustness to distribution shift, and cross-embodiment reusability.

This work introduces LoHo-Manip, a modular, hierarchical framework that explicitly decouples high-level task management from low-level sensorimotor control. The architecture operationalizes long-horizon manipulation as a receding-horizon composition of (i) progress-aware symbolic sub-task plans and (ii) spatial visual trace prediction, which guides a downstream short-horizon VLA executor. The task manager, instantiated as a vision-LLM, reasons over the current observation and explicit progress summary, emitting both an updated to-do list and actionable trajectory prompt, thus establishing an implicit closed feedback loop for robust, multi-step execution and recovery.

Figure 1: The task manager supports high-level planning via sub-task decomposition and visual trace prediction decoupled from low-level VLA execution.

System Architecture and Methodology

Hierarchical Modular Decomposition

The LoHo-Manip pipeline consists of two principal modules:

• Task Manager: A VLM-based planner conditions on the current visual observation and textual progress memory. It outputs: (a) a sequence of remaining sub-task descriptions, and (b) a 2D visual trace—a normalized sequence of waypoints (end-effector image coordinates)—grounded to the intended interaction target.
• Executor: A short-horizon VLA policy (e.g., $\pi_{0.5}$) is fine-tuned to condition on the rendered visual trace (and optionally the sub-task text) superimposed onto the observation. This converts long-horizon reasoning into repeated, spatially-guided local control.

The closed-loop protocol invokes the task manager intermittently, obtaining new plans and traces after each sub-task (or fixed intervals), enabling implicit progress tracking, continual re-planning under execution errors, and robust task-state bookkeeping without history windowing or bespoke failure detectors.

Figure 2: Framework overview—task manager predicts the next sub-task and visual trace from current observation; the executor solves the locally-conditioned short-horizon control, closing the loop.

Data Pipeline and Supervision Extraction

To construct dense supervisory labels, the pipeline processes RGB demonstration videos with foundation vision-LLMs for frame-wise object detection, captioning, and event grounding.

• Temporal segmentation yields atomic interaction primitives (e.g., grasp, insert, pour), annotated with frame indices.
• Robot end-effector positions per frame are extracted, producing compact 2D visual traces describing execution trajectories.
• Object instance grounding allows tight coupling between sub-task language and spatial intent for training the task manager with (sub-task, trace) pairs.

  Figure 3: Automated data pipeline—segmentation, grounding, and trace-label extraction from raw manipulation videos.

Progress-Aware Plan Representation and Memory Interface

At each step, the task manager generates a textual split of the completed and remaining sub-tasks, serving as a lightweight language-based task memory. Conditioning on only the current observation and explicit "done/remaining" lists ensures robustness to distribution drift and provides a stable interface invariant to rollout length or prior execution failures.

The visual trace $\tau_t$ is rendered as an overlay, acting as an actionable, spatial prompt for the executor. This decomposes complex instruction following into a sequence of trace-conditioned, local motion policies, circumventing the need for lengthy, error-prone visual history buffers.

Training Paradigm

Manager and executor are trained independently:

• The manager is fine-tuned (frozen vision encoder, language-model head) with supervised learning to predict (plan, trace) from (observation, instruction, progress summary).
• The executor is fine-tuned to follow rendered traces and sub-task descriptions for short-horizon control, facilitating interface-agnostic reusability.
• Synthetic failure-recovery scenarios augment the manager’s exposure to error cases, improving implicit recovery performance.

Experimental Validation

Generalization, Reasoning, and Trajectory Prediction

LoHo-Manip achieves state-of-the-art results on RoboVQA and EgoPlan-Bench2, outperforming recent proprietary and open-source VLMs and embodied models in both BLEU-based semantic metrics for reasoning and accuracy for human-level planning.

• LoHo-Manip-4B achieves BLEU-4 of 53.5 on RoboVQA, and 56.7% on EgoPlan-Bench2.
• ShareRobot-T and VABench-V results indicate lower path deviation (DFD, HD, RMSE), demonstrating precise spatial trajectory generation.

Embodied Planning on Discrete Command Benchmarks

On EmbodiedBench (EB-Alfred, EB-Habitat), LoHo-Manip-4B outperforms leading multimodal models—even those of larger parameter count—across all splits, particularly on categories requiring complex temporal abstraction and spatial reasoning.

End-to-End Manipulation: Simulation and Real Robots

On VLABench and LIBERO, the framework exhibits clear gains over monolithic baselines:

• For VLABench, LoHo-Manip obtains an average success of 0.39 compared to 0.24 for the best prior VLA baseline.
• On LIBERO, average score reaches 97.5, surpassing published models in all tracks.

  Figure 4: Semantic instruction execution—predicted sub-tasks and visual trace for complex VLABench task.

Real-world robot experiments (Franka arm; wrist and overhead camera) show robust multi-step completion and zero-shot generalization to unseen objects/categories and OOD language combinations, where tightly coupled VLAs degrade.

Figure 5: Multi-step robot executions demonstrating successful handling of in-distribution and OOD scenarios using the hierarchical closed-loop architecture.

Figure 6: Quantitative robot results. LoHo-Manip exhibits significant OOD performance gains over finetuned monolithic VLA baselines.

Cross-Embodiment Generalization

The manager's sub-task and trace outputs generalize to new robot embodiments and datasets, decoupled from specific executors—key for scalable deployment.

Figure 7: Trace prediction generalizes across manipulators, scenes, and downstream executor architectures.

Qualitative Results and Error Correction

The task manager identifies semantic execution errors (e.g., wrong object grasped), triggers appropriate corrective sub-task (e.g., "drop the sushi") and recovery trace, and resumes the overall goal—without handcrafted logic.

Figure 8: Sub-task decomposition, trajectory grounding, and error recovery via manager’s semantic plan and trace update.

Implications and Theoretical Significance

LoHo-Manip advances the state of hierarchical embodied planning by demonstrating that:

• Decoupling high-level plan generation from low-level control is critical for scaling long-horizon, progress-dependent tasks, robust error recovery, and systematic generalization.
• Conditioning short-horizon executor rollouts on spatial visual traces effectively externalizes spatial intent, allowing robust task execution on novel objects, instructions, and under embodiment/domain shifts.
• Explicit, lightweight language-based progress memory mitigates distributional drift and prevents interface instability under failed or deviating executions.

This paradigm provides an extensible interface: future advances in VLMs can immediately strengthen the task manager, while advances in local control policies are readily composable without retraining the manager. For practical robotics, systems can thus adapt to new hardware, task families, and instructions by training/fine-tuning only the required modular component.

Future Directions

Key future topics include:

• Extending visual traces to richer spatial-temporal guidance (e.g., 3D paths, force/interaction profiles) to enable non-tabletop, dexterous, or multi-agent tasks.
• Generalizing to hierarchical or multi-modal progress memories and integrating explicit world state estimation for additional robustness.
• Scaling to non-stationary, lifelong learning environments and heterogeneous agent morphologies.

Conclusion

This work establishes a rigorous, scalable approach to long-horizon robotic manipulation using trace-conditioned vision-language-action planning. LoHo-Manip substantially elevates robustness, sub-task reasoning, trajectory generation, and generalization over monolithic VLA baselines. Its modular, closed-loop architecture provides a clear pathway for the integration of next-generation foundation models, and robust, adaptive control policies, marking a significant advance in practical and theoretical embodied AI.