VLMPlanner: Hybrid Vision-Language Planning

Updated 7 July 2026

VLMPlanner is a planning architecture that integrates vision-language models for semantic scene understanding while delegating precise control to explicit planning substrates.
It underpins diverse applications such as mobile GUI agents, autonomous driving, UAV missions, and kinodynamic task planning by separating semantic guidance from low-level control.
The design leverages methods like BFS, PDDL solving, MPC, and optimization strategies to enhance long-horizon consistency and improve empirical performance over end-to-end approaches.

VLMPlanner denotes a family of planning architectures in which a vision-LLM is coupled to an explicit planning substrate rather than being asked to produce low-level behavior in an unconstrained end-to-end manner. In the literature, the term is used for mobile GUI agents, kinodynamic task and motion planning, socially aware navigation, autonomous driving, UAV mission generation, and symbolic mobile manipulation. Across these settings, the recurrent pattern is a division of labor: the VLM contributes scene understanding, semantic reasoning, grounding, scoring, or world modeling, while search, control, or trajectory generation is delegated to EFSM traversal, PDDL solving, MPC, DWA, candidate fusion, or learned motion planners (Mo et al., 20 May 2025, Kwon et al., 30 Oct 2025, Tang et al., 27 Jul 2025, Azirar et al., 20 Apr 2026, Chen et al., 2 Sep 2025).

1. Conceptual scope and architectural pattern

A common architectural template is semantic guidance plus constrained execution. In SPlanner for mobile GUI agents, a user instruction is parsed into target app EFSMs and target primary functions, a BFS solver traverses the EFSM(s) to obtain feasible execution paths, and an LLM refines those paths into a concise natural-language plan appended to the VLM executor’s prompt; no model fine-tuning or architectural change is required (Mo et al., 20 May 2025). In the autonomous-driving system explicitly named VLMPlanner, the VLM does not emit trajectories directly; instead, LLaVA-v1.5-7B produces hidden-state guidance that is injected into GameFormer or PlanTF through an Adaptive Injection Block with a trainable gate, while the real-time planner remains the trajectory generator (Tang et al., 27 Jul 2025). In latency-resilient sidewalk navigation, the VLM is reduced to a discrete selector over planner proposals, and continuous control remains in the fast planner through Score Fusion or Probability Fusion (Peng et al., 18 Jun 2026). In SYMBOLIZER, the VLM is used purely as a sensor model that grounds typed symbolic states, and domain-independent heuristic search performs the actual planning (Azirar et al., 20 Apr 2026).

This division is not merely an implementation convenience. It reflects a broad consensus in the cited works that VLMs are comparatively strong at semantic interpretation, commonsense cues, and open-vocabulary grounding, but weaker at exact geometric control, long-horizon consistency, or hard feasibility enforcement. This suggests that “VLMPlanner” is best understood not as a single algorithm but as an interface pattern between multimodal reasoning and an external planning mechanism.

System	Planning substrate	VLM role
SPlanner	EFSM + BFS	instruction parsing and plan refinement
VLMFP	PDDL generation and repair	simulation and domain/problem induction
VLM-Social-Nav	DWA with social cost	categorical social guidance
VLMPlanner	GameFormer or PlanTF	hidden-state guidance over raw images
LWDrive	candidate refinement and score head	coarse intent and layer-wise foresight
SYMBOLIZER	GBFS with goal-count and width	typed state grounding

2. Symbolic and formal planning substrates

One major lineage of VLMPlanner systems grounds planning in explicit symbolic formalisms. In SPlanner, each mobile application is modeled as an extended finite state machine

$\epsilon = (S, E, A, V, T, s_0),$

where states correspond to screens, events are natural-language GUI operations, primary functions encode app capabilities, variables capture configurations such as VideoMode or LoggedIn, and transitions have the form

$t = (s, e, a, g(V), u(V), s').$

A BFS-based solver finds a path that starts at $s_0$ , satisfies the target primary functions in order, respects guard conditions, and applies variable updates. The resulting path is then verbalized into an actionable plan for the VLM executor (Mo et al., 20 May 2025).

A second lineage uses PDDL. VLMFP introduces a dual-VLM loop in which SimVLM simulates action consequences from an image and a natural-language domain description, while GenVLM generates and iteratively repairs both PDDL domain and problem files. The formal objects are a domain $D=\langle T,P,A\rangle$ and a problem $\Pi=\langle O,s_0,G\rangle$ , and refinement is driven by disagreement between symbolic execution and visual simulation, summarized by the Exploration Walk score

$m_{EW}(f_d,f_p)=2(\alpha^{-1}+\beta^{-1})^{-1}.$

This mechanism targets a specific failure mode of earlier visual-to-PDDL approaches: acceptable problem-file generation with weak or hand-authored domain models (Hao et al., 3 Oct 2025).

UniPlan extends this symbolic direction to long-horizon mobile manipulation. It programmatically expands a tabletop PDDL domain to include mobility, door traversal, bimanual coordination, and action costs. The resulting domain adds predicates such as robot_at_node, object_at_node, connected, has_door, hand_free ?r ?h, and holding ?r ?h ?o, together with functions such as (travel_cost ?n1 ?n2) and (total-cost), and solves plans under (:metric minimize (total-cost)) over a compressed, task-relevant visual-topological graph (Ye et al., 9 Feb 2026).

SYMBOLIZER removes symbolic action models entirely. It assumes only lifted predicates and typed objects, uses a VLM to ground observations into well-formed symbolic states, and searches over the induced state space with domain-independent heuristics. Its core heuristics are goal count,

$h_{gc}(s)=|G\setminus P(s)|,$

and novelty,

$n(s)=\min\{k:\text{some }k\text{-tuple of atoms true in }s\text{ has not been seen before}\}.$

The transition model is supplied by a simulator or low-level controllers, so planning is “model-free” in the symbolic sense while remaining explicit in its state representation (Azirar et al., 20 Apr 2026).

Taken together, these systems show that VLMPlanner often becomes most stable when visual grounding is converted into discrete structure before long-horizon reasoning is attempted.

Another branch uses the VLM inside continuous planning loops rather than in purely symbolic search. In kinodynamic task and motion planning, the VLM-guided and interleaved sampling framework constructs a hybrid planning problem

$P \equiv \langle \mathcal{O}, \mathcal{H}, \mathcal{A}, \mathcal{T}, h_0, \mathcal{G} \rangle,$

with hybrid states $h=(s,x)$ that combine symbolic and continuous components. A top- $t = (s, e, a, g(V), u(V), s').$ 0 set of symbolic skeletons from Fast-Downward and K* is organized into a discrete guide graph, while a hybrid tree is expanded by sampling continuous parameters, checking IK and motion feasibility with RRT-Connect, validating execution in Genesis, and using a VLM to choose successors and backtracking points from rendered views and structured failure reports (Kwon et al., 30 Oct 2025). Here the VLM does not replace TAMP search; it biases exploration within a feasibility-checked pipeline.

In socially aware ground navigation, VLM-Social-Nav inserts a VLM-derived social cost into Dynamic Window Approach. GPT-4V is queried only when YOLO detects humans, gestures, or doorway contexts, and returns a constrained textual preference such as “Move right with slow down.” That output is mapped to a desired control $t = (s, e, a, g(V), u(V), s').$ 1 and added to the planner objective

$t = (s, e, a, g(V), u(V), s').$ 2

with

$t = (s, e, a, g(V), u(V), s').$ 3

The design keeps DWA responsible for feasibility while allowing the VLM to bias behavior toward social conventions (Song et al., 2024).

BehAV uses a related but more structured formulation for outdoor robot navigation. GPT-4 decomposes the instruction into navigation guidelines and behavioral guidelines; CLIPSeg grounds behavioral targets into segmentation maps, GPT-4o localizes landmarks, and the resulting behavioral cost map is fused with LiDAR-derived obstacle costs inside an unconstrained MPC objective,

$t = (s, e, a, g(V), u(V), s').$ 4

A further mechanism reduces the planner’s upper speed bound when highly undesirable actions such as “stop for” are inferred beyond the immediate horizon (Weerakoon et al., 2024).

Aerial systems adapt the same idea to large-scale or sparse-data settings. UAV-VLA maps a georeferenced satellite image and a text instruction to geodetic waypoints and action sets through GPT-based goal extraction, Molmo-7B-D grounding of image-plane points, and MAVProxy export for ArduPilot/Mission Planner (Sautenkov et al., 9 Jan 2025). UAV-VLPA* couples VLM-based target and obstacle extraction from satellite imagery with 2-opt TSP ordering and A* obstacle avoidance on a 5×5 pixel occupancy grid, explicitly separating mission-scale sequencing from segment-level routing (Sautenkov et al., 4 Mar 2025). FM-Planner benchmarks multiple LLM and VLM variants for drone path planning and then deploys an integrated LLM-Vision planner with YOLOv8, RealSense D435i depth, waypoint interpolation, and PX4 offboard control, reporting that Llama-3.1-8B-Instruct is the strongest textual planner among the tested models (Xiao et al., 27 May 2025).

These systems share a characteristic compromise: the VLM supplies semantic or topological structure, while safety-critical motion feasibility remains in a classical controller, search routine, or simulator.

4. Autonomous-driving trajectory planning

Driving-oriented VLMPlanner systems are distinguished by their attempt to preserve raw visual context while avoiding direct language-model trajectory generation. BEV-VLM replaces perspective-image prompting with a unified BEV-HD map derived from BEVFusion, PCA compression to a 180×180 RGB BEV feature map, and HD map alignment over a $t = (s, e, a, g(V), u(V), s').$ 5 m ego-centric raster. Qwen2.5-VL-3B or Qwen2.5-VL-7B then predicts six waypoint tokens at 0.5 s intervals for a 3 s horizon (Chen et al., 27 Sep 2025). This redesign addresses modality isolation and geometric inconsistency in raw-image VLM planning by presenting the model with a metric, sensor-fused, map-aware abstraction rather than unaligned perspective views.

The driving framework explicitly titled VLMPlanner uses raw multi-view images and vectorized map features instead of a single BEV raster. CLIP image features are projected through a 3D-aware module with learnable 3D queries and 3D positional encodings, map vectors are encoded with GameFormer’s MapEncoder, and the VLM hidden state is adapted and injected into the planner decoder through

$t = (s, e, a, g(V), u(V), s').$ 6

where $t = (s, e, a, g(V), u(V), s').$ 7 is a trainable gate. A Context-Adaptive Inference Gate then modulates VLM query frequency according to scene complexity, using either an EfficientNet-B0 ordinal classifier or a rule-based score (Tang et al., 27 Jul 2025).

A different solution to planner–VLM interaction under real-time constraints appears in “Slow Brain, Fast Planner.” There, the fast anchor-based planner proposes 64 candidates per tick, a VLM asynchronously selects an index among the top- $t = (s, e, a, g(V), u(V), s').$ 8 displayed trajectories, and the stale VLM choice is converted into a real-time bias over fresh candidates through

$t = (s, e, a, g(V), u(V), s').$ 9

with

$s_0$ 0

This interface is explicitly motivated by the mismatch between a 5–20 Hz planner loop and 1–3 s VLM latency (Peng et al., 18 Jun 2026).

LWDrive pushes VLM-based driving planning toward internal world modeling. Qwen2.5-VL-3B generates an intent-aware coarse trajectory and layer-wise hidden states supervised by future-frame generation in latent diffusion space. A Foresight Cascade Planner then refines a proposal set over six stages using selected VLM action layers, Action-Query memory, ego-state context, and current-frame multi-view BEV features, before a score head chooses the final candidate (Yang et al., 29 Jun 2026). The architectural thesis is that VLM output should be treated as a coarse prior rather than a final trajectory.

Across these driving systems, the VLM rarely acts as the sole planner. Instead, it contributes either compact semantic features, intent priors, or candidate preferences, while geometry is enforced by a motion planner, candidate refiner, or trajectory scorer.

5. Learning, world models, and reasoning mechanisms

The learning strategies behind VLMPlanner systems vary from prompt engineering to large-scale world-model training. VLWM is the most explicit attempt to recast planning as language-based world modeling. Given visual context, it predicts a goal, a detailed goal interpretation, and an autoregressive sequence of interleaved actions and world-state changes,

$s_0$ 1

Training targets are extracted from natural videos by Tree of Captions compression followed by iterative LLM Self-Refine, and planning can be performed either by direct policy decoding (“system-1”) or by critic-guided cost minimization over hypothetical roll-outs (“system-2”) (Chen et al., 2 Sep 2025). The learned critic is trained with ranking-based supervision over good, bad, and shuffled trajectories, turning semantic goal distance into an optimization signal.

LaViPlan addresses a different problem: misalignment between a VLM’s textual reasoning and its low-level path output in autonomous driving. It keeps the model output textual, with a > block and a 20-point <answer> trajectory in image coordinates, but optimizes it with Reinforcement Learning with Verifiable Rewards. The reward is

$s_0$ 2

where

$s_0$ 3

and policy optimization uses a GRPO objective with KL regularization to a supervised reference model (Oh, 17 Jul 2025). The training signal is fully verifiable from the labeled dataset and does not require a learned reward model.

Other systems remain prompt-based and modular. “A Navigation Framework Utilizing Vision-LLMs” uses a frozen Qwen2.5-VL-7B-Instruct model, two consecutive RGB frames, structured reflection history, and strict JSON output over the discrete action set $s_0$ 4, explicitly separating multimodal understanding from lightweight planning logic (Duan et al., 11 Jun 2025). OpenVLN combines a VLM waypoint policy with a separate value model and a verifiable similarity reward, using KL-regularized PPO and potential-based shaping

$s_0$ 5

to improve long-horizon aerial navigation under limited-data conditions (Lin et al., 9 Nov 2025).

This range of training paradigms shows that VLMPlanner is not tied to a single learning doctrine. It can be instantiated as zero-shot prompting, LoRA adaptation, reward-driven post-training, or full world-model learning, depending on whether the bottleneck is grounding, reasoning, or trajectory accuracy.

6. Empirical performance, limitations, and research directions

Empirical results across the literature indicate that VLMPlanner designs are most effective when they improve long-horizon consistency or resolve semantic ambiguity that baseline planners handle poorly. On AndroidWorld, SPlanner paired with Qwen2.5-VL-72B achieves a 63.8% task success rate and improves over Qwen2.5-VL-72B without planning assistance by 28.8 percentage points (Mo et al., 20 May 2025). In kinodynamic TAMP, the hybrid state-tree framework reports 32.14%–1166.67% increased average success rates compared to traditional and LLM-based TAMP planners (Kwon et al., 30 Oct 2025). VLMFP reaches 70.0% valid plans for unseen instances in seen appearances and 54.1% in unseen appearances, substantially above direct plan generation baselines (Hao et al., 3 Oct 2025). BEV-VLM reports a 44.8% reduction in displacement error relative to the best purely visual baseline and 0% collision rate for the 7B model on nuScenes validation (Chen et al., 27 Sep 2025). LWDrive reaches 92.0 on NAVSIM and 89.6 on NAVSIM-v2 (Yang et al., 29 Jun 2026). OpenVLN reports gains of up to 4.34% in Success Rate, 6.19% in Oracle Success Rate, and 4.07% in Success weighted by Path Length over baseline methods while using 25% of the TravelUAV training data (Lin et al., 9 Nov 2025). SYMBOLIZER reports state-of-the-art results on ProDG and ViPlan by searching over VLM-grounded symbolic states without handcrafted action models (Azirar et al., 20 Apr 2026).

The limitations are equally consistent across domains. SPlanner depends on manual EFSM modeling, with a typical effort of approximately 1–2 hours per app and a maintenance burden for updated applications (Mo et al., 20 May 2025). Driving and navigation systems repeatedly identify latency as a structural issue: VLMs respond on the order of seconds, which is incompatible with planner control loops unless gated, fused, or queried asynchronously (Peng et al., 18 Jun 2026, Tang et al., 27 Jul 2025). BEV-based planners depend on accurate BEVFusion outputs and HD-map alignment; stale or misaligned maps can misguide trajectory generation (Chen et al., 27 Sep 2025). World-model approaches such as VLWM and the kinodynamic TAMP framework do not provide formal completeness or optimality guarantees, and their search cost grows with rollout width, horizon, or simulation effort (Chen et al., 2 Sep 2025, Kwon et al., 30 Oct 2025). Prompt-driven modular systems remain vulnerable to parser failures, limited context, and weak environmental priors in unseen settings (Duan et al., 11 Jun 2025).

The forward direction suggested by these papers is therefore not a wholesale replacement of planners by VLMs. Instead, it is tighter integration: automated EFSM induction and AI-assisted tooling for GUI agents, stronger runtime re-planning and plan-adherence mechanisms, adaptive VLM querying, richer BEV or multi-view world-model supervision, improved symbolic grounding under appearance shift, and closer coupling between semantic reasoning and feasibility checks (Mo et al., 20 May 2025, Peng et al., 18 Jun 2026, Yang et al., 29 Jun 2026, Hao et al., 3 Oct 2025). A plausible implication is that future VLMPlanner systems will continue to move toward hybrid designs in which VLMs supply high-value semantic priors while explicit planners retain responsibility for search, control, and safety.