ExploreVLM: Closed-Loop Robot Task Planning

• ExploreVLM is a framework for robot exploration that organizes task execution as a sequential loop of perception, planning, execution, and validation.
• It employs an object-centric spatial relation graph and a dual-stage task planner with self-reflection to generate logically coherent action sequences.
• The execution validator detects discrepancies after each action, enabling real-time adaptation and significantly enhancing task success rates.

Searching arXiv for ExploreVLM and closely related VLM exploration/planning papers to ground the article. ExploreVLM is a closed-loop robot exploration task planning framework powered by Vision-LLMs (VLMs) for real-world embodied tasks that are interactive, partially observed, and exploration-dependent. Its central premise is that many VLM-based robot systems fail because they rely on static perception, produce single-shot plans, and lack fine-grained execution feedback; ExploreVLM instead organizes task execution as a step-wise loop of perception, planning, execution, validation, feedback, and re-planning. The framework combines an object-centric spatial relation graph, a dual-stage task planner with self-reflection, and an execution validator that operates after every primitive action, and it is evaluated on real-robot tasks involving hidden objects, drawers, cabinets, refrigerators, and obstacle clearing (Lou et al., 16 Aug 2025).

1. Problem setting and conceptual scope

ExploreVLM addresses tasks in which a robot cannot complete the instruction from an initial observation alone. A representative example is “Put the fruits into the drawer containing only fruits.” Solving such a task requires identifying which objects are fruits, exploring the environment to locate candidate drawers, opening drawers and inspecting contents, determining which drawer contains only fruits, detecting obstacles blocking access, removing obstacles if necessary, and updating the plan after each interaction (Lou et al., 16 Aug 2025).

The framework is motivated by a failure mode common in prior VLM-based embodied systems. The paper contrasts ExploreVLM with methods such as VILA, ReplanVLM, SayCan, VoxPoser, Inner Monologue, and RoboExp, and identifies several recurring weaknesses: weak scene understanding, no robust exploration strategy, weak real-time adaptation, illogical action sequences, and poor robustness to noise. These weaknesses appear most clearly in environments that are only partially observed and that change after every interaction, where a single initial plan is often invalidated by newly revealed state (Lou et al., 16 Aug 2025).

A common misconception is to equate “exploration” in embodied VLM systems with frontier exploration or geometric navigation alone. In ExploreVLM, exploration is task-conditional information acquisition inside manipulation-centered workflows. The system explores not merely to cover space, but to reveal hidden state, verify category membership, inspect containment relations, and expose preconditions for later actions. This distinguishes its target regime from open-scene navigation systems such as NavVLM (Yin et al., 2024) and from frontier-selection systems that use VLMs for strategic map coverage (Aitha et al., 22 May 2026).

2. Closed-loop architecture and control flow

The framework takes an initial RGB image $I_0$, a language task goal $g$, and primitive skills $\Pi$. At each step $t$, it constructs a scene graph $G_t = \text{ScenePerception}(I_t, g)$, generates an exploration-stage plan and sub-goal $(P_E, g_E)$ together with a completion-stage plan and sub-goal $(P_C, g_C)$, selects one primitive action, executes that primitive skill, re-perceives the scene as $G_{t+1}$, and produces validator feedback $F_t$ for the next planning cycle. If $P_E = g_E = P_C = g_C = None$, the task terminates (Lou et al., 16 Aug 2025).

The decision rule is deliberately local. If an exploration plan is available, the system executes the first exploration primitive; otherwise it executes the first completion primitive. The robot therefore performs only one primitive action at a time, then re-perceives and re-plans. This one-step horizon at execution time is the mechanism by which the framework remains responsive to newly revealed facts, manipulation failures, or changing accessibility constraints (Lou et al., 16 Aug 2025).

Formally, the loop can be summarized with the framework’s core state variables:

$g$0

Here $g$1 is the current object-centric spatial relation graph, $g$2 and $g$3 are the exploration and completion outputs of the planner, $g$4 is the currently executed primitive action and its sub-goal, and $g$5 is the validator’s feedback. The important architectural point is not the notation itself, but the enforced alternation between action and verification after every primitive step (Lou et al., 16 Aug 2025).

This design also clarifies what ExploreVLM does not do. It does not learn low-level control inside the planning loop; the study focuses on high-level planning, and low-level primitives are treated as existing motion primitives. Nor is it an open-loop planner that commits to a long action script. The closed loop is intrinsic to the method rather than an auxiliary recovery mechanism (Lou et al., 16 Aug 2025).

3. Object-centric spatial relation graph

Scene perception in ExploreVLM is centered on an object-centric spatial relation graph. Nodes correspond to detected objects and include semantic attributes such as category, color, and state, for example open or closed. Directed edges encode spatial relations: above, below, left, right, front, and behind. The result is a structured, interpretable, language-grounded scene representation designed for planning rather than a flat list of detections (Lou et al., 16 Aug 2025).

The graph is constructed in two stages. First, GroundedSAM2 performs instance segmentation using common object names and produces object masks and class labels. Second, the segmented image, together with textual summary cues such as object names and bounding box colors and the task instruction, is fed into the VLM, which infers the spatial relations and outputs the graph. This pipeline converts visual input into relational structure that the planner can query for accessibility, obstruction, and state-dependent task logic (Lou et al., 16 Aug 2025).

The significance of the representation lies in what it makes explicit. The planner can reason about what objects exist, where they are, what blocks access to what, which objects are relevant to the instruction, and whether a planned action is feasible. The paper’s examples of baseline failure—failing to detect an eggplant blocking a plate, confusing Sprite with Pepsi, or missing a cup obstructing a drawer—are precisely the kinds of errors the graph is intended to reduce by moving from image-level impressions to object-level relational structure (Lou et al., 16 Aug 2025).

A second misconception follows from treating scene graphs as merely descriptive. In ExploreVLM the graph is operational: it is the intermediate representation used by both planning and validation. This role places it closer to the semantic mapping used in embodied question answering systems that combine geometry and VLM-driven semantic scoring (Ren et al., 2024), though ExploreVLM’s graph is object- and relation-centric rather than frontier- and map-value-centric.

4. Dual-stage planning and self-reflection

The planning core decomposes task execution into two phases. The exploration stage is invoked when relevant information is missing, such as the location of a target object, the contents of a drawer, or the state of a container. The completion stage is used once exploration has gathered enough information to make the task executable. The planner consumes the current graph $g$6, task goal $g$7, and validator feedback $g$8, and uses CoT prompting with an LLM to generate $g$9 (Lou et al., 16 Aug 2025).

The two-stage decomposition enforces an “explore-then-complete” logic. In the fruit-and-drawer example, the planner can identify banana as fruit while not yet knowing which drawer is fruit-only. It therefore creates an exploration goal to inspect drawers and a completion plan that temporarily contains a placeholder for the target drawer. Once exploration identifies the correct drawer, the completion plan is updated accordingly. This staging is not simply a decomposition convenience; it is the mechanism by which the system avoids acting on incomplete state (Lou et al., 16 Aug 2025).

A distinctive component is self-reflection. Before execution, the planner checks three classes of error. First, it verifies the validity of stage goals: whether the exploration sub-goal is achievable, whether accomplishing it provides the missing information, and whether the completion stage then truly solves the original task. Second, it checks logical consistency of the action sequence, including rules such as GRASP before PLACE, empty gripper before GRASP, and OPENed objects should eventually be CLOSEd. Third, it performs obstacle handling by using the spatial relation graph to detect objects in front of a manipulation target that must be removed first; the paper notes that this is especially important because the robot arm mainly operates from the front (Lou et al., 16 Aug 2025).

These reflection checks are the part of the system most directly targeted at illogical action sequences. The planner explicitly corrects failures such as removing a cup blocking a drawer or inserting a missing GRASP before PLACE. A plausible implication is that ExploreVLM’s gains on exploration-centric tasks are not only due to better information gathering, but also to stricter enforcement of symbolic action preconditions within the task plan. That interpretation is consistent with the reported ablations, where removing the dual-stage planner with self-reflection sharply reduces performance (Lou et al., 16 Aug 2025).

5. Execution validation and step-wise adaptation

The execution validator is the mechanism that makes the framework closed-loop in a strict sense. After each primitive action $\Pi$0, the validator compares the pre-action and post-action graphs, $\Pi$1 and $\Pi$2, checks whether the observed change matches the expected result for the current sub-goal $\Pi$3, and outputs either "yes" if successful or "no" plus a failure reason $\Pi$4. The reported failure reasons include primitive action failure, need for extra steps, plan inconsistency, and obstacle not handled (Lou et al., 16 Aug 2025).

This validator prevents delayed error accumulation. Rather than waiting until a full plan or subtask is completed, the framework verifies each primitive action and immediately feeds the result back into planning as $\Pi$5. If exploration succeeds, the planner uses the discovered information to specialize the completion plan. If an action fails, the planner can re-execute, change the order of actions, insert missing actions, or change the overall strategy. If the goal is already achieved, planning terminates (Lou et al., 16 Aug 2025).

The practical importance of this design becomes clearer under injected execution noise. Because validation occurs after every primitive action, the system is able to detect failed low-level outcomes promptly and prevent the planner from proceeding as though success had occurred. This differs from pipelines in which feedback arrives only after a larger stage ends. In such open-loop or delayed-feedback settings, a failed OPEN or GRASP can propagate downstream into logically impossible states and cascading errors (Lou et al., 16 Aug 2025).

In that respect, ExploreVLM is adjacent to other VLM systems that use semantic feedback for recovery or iterative adaptation. The embodied visual tracking framework of Wang et al. uses a VLM only after failure detection and augments recovery with memory and self-reflection (Wu et al., 27 May 2025). ExploreVLM differs in that feedback is not exceptional and recovery-only; it is the default operating mode of the planner.

6. Experiments, ablations, and position within the literature

The experiments were conducted on a real robot platform with a UR5 robot arm, a Robotiq 2F-85 gripper, an Intel RealSense D435 RGB-D camera, GPT-4o as the VLM, and four primitive actions: GRASP, PLACE, OPEN, and CLOSE. Five real-world tasks were designed, increasing in complexity from basic execution and feedback to complex exploration and interaction. To test robustness, the authors injected a 50% random failure rate into GRASP and OPEN. Each task was run 10 times, and the primary metric was task success rate (%) (Lou et al., 16 Aug 2025).

The five tasks progressively stress different parts of the framework: hidden-object discovery under a blue towel; task logic requiring removal of an eggplant before placing bread; exploration and identification to find Pepsi rather than Sprite across a cabinet and a refrigerator; commonsense selection of the object that requires refrigeration; and complex exploration with drawers, obstacle removal, and category verification for fruit-only containment. This task suite is designed to expose failures in static perception, exploration, action ordering, and noise recovery rather than only simple pick-and-place competence (Lou et al., 16 Aug 2025).

Task	ReplanVLM / VILA	ExploreVLM
Task 1	50% / 60%	100%
Task 2	50% / 30%	100%
Task 3	10% / 50%	100%
Task 4	0% / 10%	90%
Task 5	0% / 0%	80%
Average	22% / 30%	94%

The headline empirical result is a 94% average success rate, compared with 22% for ReplanVLM and 30% for VILA. The separation is largest on the exploration-heavy tasks, particularly Tasks 4 and 5, where the baselines nearly fail outright. The paper attributes this to better scene understanding, explicit exploration capability, improved action logic, and immediate feedback-based recovery (Lou et al., 16 Aug 2025).

The ablation study was conducted on Task 5, the most difficult scenario.

Configuration	Success
ExploreVLM	80%
w/o Object-centric Spatial Relation Graph	30%
w/o Dual-stage Task Planner with Self-reflection	10%
w/o Execution Validator	0%

These ablations support a component-wise reading of the method. Without the object-centric spatial relation graph, the system misses the obstacle cup, fails to confirm drawer contents, and may misidentify objects. Without the dual-stage task planner with self-reflection, it cannot handle exploration well and generates logically flawed plans. Without the execution validator, the feedback loop disappears and the system cannot adapt after actions or recover from noise (Lou et al., 16 Aug 2025).

Within the broader VLM exploration literature, ExploreVLM occupies a specific niche. NavVLM studies zero-shot open-set navigation with a lightweight VLM as a cognitive core for directional guidance and stopping (Yin et al., 2024). “Explore until Confident” studies embodied question answering with semantic frontier weighting and conformal stopping (Ren et al., 2024). OVExp learns goal-conditioned open-vocabulary exploration over top-down VLM-aligned maps (Wei et al., 2024). ScreenExplorer trains a GUI VLM with GRPO, world-model curiosity, and experience-stream distillation for open-ended desktop exploration (Niu et al., 25 May 2025). “Planning with the Views via Scene Self-Exploration” studies view planning in ScanNet scenes using iterative self-exploration and view-graph distillation (Wang et al., 28 May 2026). This comparison suggests that the term “exploration” in VLM research now spans at least manipulation-centered information gathering, open-world navigation, embodied question answering, GUI interaction, and active 3D view planning; ExploreVLM’s contribution lies specifically in closed-loop, step-wise task planning for interactive manipulation under partial observability (Lou et al., 16 Aug 2025).