NovaPlan: Hierarchical Robotic Manipulation

• NovaPlan is a hierarchical robotic manipulation framework for zero-shot long-horizon tasks, integrating vision-language planning, video generation, and geometric grounding.
• It deploys a closed-loop planning loop where a high-level language model decomposes tasks and a low-level module extracts precise robot motions using object and hand flow.
• The system achieves improved multi-stage assembly performance through adaptive recovery strategies, including non-prehensile corrections when execution errors occur.

Searching arXiv for papers on NovaPlan and closely related planning frameworks. NovaPlan is a hierarchical robotic manipulation framework for zero-shot long-horizon manipulation that combines closed-loop vision-language planning, video generation, and geometrically grounded robot execution. Its central design is to let a high-level vision-LLM decompose a task into language sub-goals and monitor execution, while a low-level module converts generated videos into executable robot motions using task-relevant object keypoints and human hand poses as kinematic priors. The system is presented as a way to couple semantic planning, imagined future outcomes, and physically grounded control without demonstrations, task-specific training, or finetuning on the evaluated tasks (Fu et al., 23 Feb 2026).

1. Problem setting and conceptual scope

NovaPlan targets zero-shot long-horizon manipulation, meaning previously unseen multi-step tasks are solved directly from task instructions and scene observations, with no task-specific demonstrations or adaptation. In the reported formulation, long-horizon difficulty arises from the mismatch between semantic planning and physical execution: a planner may know the next sub-goal, but not the millimeter-accurate trajectory needed for insertion, alignment, grasping, or placement; a generated video may depict a plausible interaction, but not a robot-executable one; and small execution errors compound over multiple stages (Fu et al., 23 Feb 2026).

The framework is explicitly positioned against three limitations. First, VLM-only planners can decompose tasks and reason about ordering, but do not provide precise 3D manipulation trajectories or physically grounded contact behavior. Second, video-generation-only or open-loop systems can imagine interactions, but generated motion is not directly executable and may exhibit hallucination, temporal inconsistency, geometric warping, or embodiment mismatch. Third, policy-learning or VLA approaches can predict actions directly, but generally require large robot datasets and do not address the paper’s zero-shot objective in the same way. NovaPlan’s stated alternative is to use pretrained perception, video generation, and LLMs off the shelf, while grounding execution through geometry, tracking, and control (Fu et al., 23 Feb 2026).

The benchmarked tasks make the intended scope concrete. Reported examples include four-layer stacking, color sorting into constrained containers, hidden-object search with conditional branching, and multi-stage assembly from the Functional Manipulation Benchmark (FMB). The paper’s operational definition of zero-shot is narrow and technical: no task-specific demonstrations, no task-specific policy training, and no finetuning on the evaluated tasks. This does not imply absence of pretrained foundation models; rather, it excludes adaptation specific to the downstream manipulation episodes (Fu et al., 23 Feb 2026).

2. Hierarchical architecture and planning loop

NovaPlan is organized as a closed-loop hierarchy in which language proposes, video imagines, geometry grounds, the robot executes, and a VLM verifies the resulting state. The high-level planner takes a task instruction $g$, the current scene image $I_{real,t}$, and action history, then proposes candidate macro-actions or sub-goals in natural language, such as grasping, pulling, inserting, or lowering an object. Each candidate also identifies the track object that should be monitored visually (Fu et al., 23 Feb 2026).

For each candidate, a video model generates multiple rollout videos conditioned on the current image and the action description. Those rollouts are then ranked by a VLM according to whether the correct object moved, whether the motion is physically plausible, whether the flow matches the intended action, and whether the final frame matches the desired sub-goal result. The selected rollout is not executed directly. Instead, at each execution step, NovaPlan re-generates a new video reference $V_t$ from the updated real observation $I_{real,t}$ and the chosen action $a_t$, thereby re-grounding the reference in the current state (Fu et al., 23 Feb 2026).

Component	Input	Role
High-level planner	Task instruction, current image, action history	Proposes language sub-goals and tracked objects
Video model	Current image, sub-goal text	Generates rollout videos as imagined futures
VLM evaluator	Candidate rollouts	Scores plausibility, object motion, and final result
Low-level planner	Selected reference video	Extracts object flow or hand flow for execution
VLM critic	Start image, post-execution image, target image	Verifies success and triggers recovery if needed

The planning loop is described procedurally as a generate-then-verify tree search. The planner seeks a sequence of validated actions

$A = \{a_1, \dots, a_h\},$

where $h$ is the planning horizon. At each expansion step, the VLM proposes $N_c$ candidates, the video model generates $L$ rollouts per candidate, the VLM ranks those rollouts, and the process continues recursively until a plan of length $h$ is obtained. The horizon may be used in two modes: greedy mode with $I_{real,t}$0 for reactive or exploratory tasks, and strategic mode with $I_{real,t}$1 for coupled tasks in which order matters. In the appendix, this is described as a beam-search-like procedure in which the final beam is selected by score (Fu et al., 23 Feb 2026).

Sub-goals are represented primarily in language, but they are evaluated through visual rollout outcomes. Prompted reasoning explicitly enforces ordering constraints for interacting objects, awareness of completed objects, distinction between valid placements and obstructions, “no telekinesis” physical realism, and single-hand action descriptions. This means NovaPlan’s planning representation is neither purely symbolic nor purely generative: it is a multimodal loop in which language structures search and videos supply physically interpretable future states (Fu et al., 23 Feb 2026).

3. Geometric grounding and low-level action generation

The low-level planner extracts executable motion from the selected generated video using two complementary motion representations: object flow and hand flow. Object flow is the 3D trajectory of tracked object keypoints,

$I_{real,t}$2

where $I_{real,t}$3. Hand flow is the generated human hand sequence,

$I_{real,t}$4

where $I_{real,t}$5 denotes the MANO hand mesh at frame $I_{real,t}$6 (Fu et al., 23 Feb 2026).

The object-flow pipeline estimates depth from the generated RGB video using MoGe2, optionally refines temporal consistency with Consistent Video Depth, affine-calibrates generated depth to metric scale using the first frame and the real depth sensor, segments the target object, samples object keypoints, and tracks them in 3D with TAPIP3D. Depth calibration is written as

$I_{real,t}$7

with affine parameters estimated by RANSAC under an inlier threshold $I_{real,t}$8. Rigid object motion is then recovered frame by frame: rotation is estimated with the Kabsch algorithm,

$I_{real,t}$9

and translation by

$V_t$0

This yields an $V_t$1 trajectory that can be transferred to the end effector through a fixed object-to-end-effector transform after grasp selection (Fu et al., 23 Feb 2026).

Hand flow is introduced as a fallback when object tracking becomes unreliable under self-occlusion, large rotations, or depth inaccuracy. NovaPlan estimates a 3D hand mesh sequence from the generated video using HaMeR, then applies a dual-anchor calibration to correct scale inconsistency and projective drift. Contact onset is detected from mask change relative to the initial object mask, with $V_t$2 in grasp mode and $V_t$3 in non-prehensile mode. At contact onset, candidate fingertip scales are computed and the selected scale is

$V_t$4

For non-prehensile recovery, scale is computed as

$V_t$5

with an additional translation term enforcing fingertip-object contact (Fu et al., 23 Feb 2026).

The switching mechanism is based on trajectory smoothness. If the recovered object rotation between adjacent frames, written in axis-angle form as $V_t$6, contains any $V_t$7, the system switches to hand flow, with $V_t$8 in experiments. The paper describes this as a per-step choice made before execution rather than continuous per-control-cycle switching. Hand trajectories are also rejected if more than 30% of the hand mask goes out of view in a critical frame or if no valid contact fingertip/scale can be found; in that case the current generation step is treated as a failure and the system regenerates video (Fu et al., 23 Feb 2026).

A common misconception is that NovaPlan directly imitates a generated human motion. The reported mechanism is narrower: hand motion is used as a kinematic prior. Translation is taken from the designated contact fingertip, rotation is derived from a palm frame, and the resulting $V_t$9 sequence is tracked by the robot through a joint impedance controller. The robot platform itself is a Franka Research 3 arm with a Robotiq 2F-85 gripper and an Intel RealSense D455 RGB-D camera (Fu et al., 23 Feb 2026).

4. Closed-loop verification, replanning, and recovery

After each executed step, NovaPlan uses a VLM critic to compare the start image $I_{real,t}$0, the post-execution image $I_{real,t}$1, and the target image $I_{real,t}$2, defined as the last frame of the generated reference video. The critic asks whether the intended object moved, whether action-specific physical constraints are satisfied, and whether the current state is sufficiently close to the target state. If the verifier returns success = false, replanning is triggered (Fu et al., 23 Feb 2026).

This verification layer is intended to catch both no-change failures and incorrect-change failures. Reported failure modes include grasp slippage, partial insertion, wrong object displacement, and local misalignment near the goal. Recovery is then conditioned on the current real observation, the target state from the failed step, the original goal or action context, and the VLM’s failure analysis. The recovery strategy is chosen from

$I_{real,t}$3

If the discrepancy can be repaired by a small perturbation, the system may choose non-prehensile recovery rather than a full re-grasp (Fu et al., 23 Feb 2026).

Non-prehensile recovery is one of the paper’s distinctive mechanisms. In that case, the VLM selects a single contact point, annotates the current image with a red star, writes a structured prompt describing exactly one poke or push, and generates a recovery video conditioned on the annotated current frame and the target image as the last frame: $I_{real,t}$4 Hand-flow grounding is then used to execute the corrective motion. This is intended for cases such as slight insertion jams or local assembly misalignment, where a finger poke or nudge is preferable to grasp-based correction (Fu et al., 23 Feb 2026).

Another common misconception is that generated videos are executed open-loop. The paper explicitly states that NovaPlan does not simply generate one rollout and imitate it once. Instead, the framework repeatedly re-generates a visual reference from the current real observation, executes a grounded trajectory, verifies the resulting state, and synthesizes a corrective action when necessary. Recovery is therefore not merely “try again”; it can be a qualitatively different action selected by the VLM and grounded through a separate generated video (Fu et al., 23 Feb 2026).

5. Experimental evaluation and reported performance

The reported system uses GPT-5.2 for VLM reasoning, Wan 2.2 and Veo 3.1 for video generation, MoGe2 for depth estimation, SAM3 for segmentation, TAPIP3D for 3D point tracking, HaMeR for hand tracking, and GraspGen for grasp proposals. For a 41-frame 720p video, runtime is approximately 40 seconds end-to-end with parallel object and hand extraction. The breakdown is 30 seconds for Wan or Veo, 3 seconds for MoGe2, 90 seconds for Consistent Video Depth if used, 3.5 seconds for SAM3, 3.5 seconds for TAPIP3D, and 0.8 seconds per frame for HaMeR. The system is therefore described as a deliberative planning-and-execution framework rather than a low-latency real-time controller (Fu et al., 23 Feb 2026).

The main long-horizon evaluations cover four-layer block stacking, color sorting, hidden-object search, and FMB Multi-Object Multi-Stage Assembly 1. Ten trials are reported for the long-horizon tasks. Baselines include NovaFlow as a representative video-based planning baseline, $I_{real,t}$5 as a representative VLA model, and MOKA as a representative VLM-based planner. Because some baselines do not inherently perform long-horizon decomposition, some are given an oracle task decomposition module for a fairer execution comparison (Fu et al., 23 Feb 2026).

Setting	Reported result
Four-layer block stacking	NovaPlan succeeds in 7/10 trials
Hidden object search	NovaPlan succeeds in all trials
Four-layer stacking baseline reference	NovaFlow: 70% on third block, 30% on fourth block
Runtime reference	Approximately 40 seconds end-to-end for a 41-frame 720p video

In four-layer block stacking, $I_{real,t}$6 can stack up to two layers but fails beyond that. NovaFlow achieves 70% success on stacking the third block and 30% on the fourth block. NovaPlan succeeds in 7/10 trials overall, and the paper attributes this improvement to stability from switching to hand flow when object-centric tracking becomes unreliable (Fu et al., 23 Feb 2026).

In hidden-object search, NovaPlan and NovaFlow both succeed in all trials. The reported contrast is instead with $I_{real,t}$7, which is stronger on drawer opening than on object retrieval, and with MOKA, which fails to find a correct grasp pose for the horizontal drawer handle. In color sorting, all methods experience difficulty in the low-tolerance yellow-block insertion case; for NovaPlan, failures are attributed mainly to depth estimation error, which degrades pose extraction (Fu et al., 23 Feb 2026).

The FMB experiments are reported mainly qualitatively. The paper states that none of the VLA or VLM baselines could complete even a single FMB step in the reported zero-shot setup, whereas NovaPlan can solve complex FMB assembly sequences and a variant requiring a recovery step. Veo 3.1 is described as necessary for FMB, because Wan 2.2 and Veo 3.1 were sufficient for earlier tasks but only Veo could generate feasible videos across all FMB stages, and even then with low success rate. Recovery videos are also reported to be significantly harder to generate than nominal insertion videos. These experiments are used to support the claim that NovaPlan can exhibit dexterous corrective behaviors, including non-prehensile finger-poke recovery, in complex assembly settings (Fu et al., 23 Feb 2026).

6. Limitations, interpretation, and relation to adjacent planning frameworks

The reported limitations are concentrated in three components: video generation, geometry estimation, and grasping. Video generation can fail through physically implausible rollouts, poor recovery motion synthesis, difficulties with irregular shapes, and weak performance under single-view conditions. Depth estimation errors degrade both object flow and hand-scale calibration, especially in low-tolerance insertions. GraspGen can fail on irregular shapes or cluttered failure states. Reorientation tasks are described as especially challenging and often unsolved in the current pipeline (Fu et al., 23 Feb 2026).

The FMB-specific discussion sharpens these limitations. In recovery situations with very small displacement, both keypoint flow and hand pose can be dominated by noise. The paper suggests that non-prehensile recovery works better than grasp-based recovery in some such cases, and also notes that a multi-view or moving-view video generator would likely help significantly. This suggests that NovaPlan’s current capability envelope depends strongly on the quality of generated visual foresight and on having enough geometric signal to extract stable motion references (Fu et al., 23 Feb 2026).

A second interpretive point concerns the meaning of “closed loop.” NovaPlan is closed loop at the level of stepwise monitoring and replanning, not as a high-frequency reactive controller. The reported runtime of tens of seconds per step places it in the category of high-level deliberative manipulation systems. A plausible implication is that NovaPlan is best suited to tasks where semantic sequencing, verification, and recovery matter more than millisecond-scale servo reactivity (Fu et al., 23 Feb 2026).

Within the broader planning literature in the provided corpus, NovaPlan occupies a distinct niche. UniPlan, for example, unifies navigation and manipulation into a single PDDL problem over a visual-topological map for long-horizon mobile manipulation in indoor environments (Ye et al., 9 Feb 2026). NovaPlan instead targets robot-arm manipulation and assembly by using language proposals, generated videos as imagined futures, and geometric extraction for execution. This suggests two different lines of integration between high-level reasoning and embodiment: a symbolic-topological route in UniPlan, and a video-grounded geometric route in NovaPlan (Ye et al., 9 Feb 2026).

Taken together, the paper presents NovaPlan less as a monolithic policy than as a layered manipulation architecture with redundancy and verification. Its reported strengths are closed-loop replanning, complementary hand/object motion grounding, and recovery that can switch from grasp-based to non-prehensile correction. Its reported weaknesses are dependence on current video and depth models, limited real-time feasibility, and remaining brittleness on irregular objects, reorientation, and difficult recovery synthesis. The work is therefore best understood as a technically explicit attempt to connect semantic task decomposition with physically grounded long-horizon execution under zero-shot constraints (Fu et al., 23 Feb 2026).