VLM Planner: Multimodal Task Planning

• VLM Planners are systems that integrate visual and linguistic inputs to generate, refine, and evaluate complex task plans across robotics and agent-based applications.
• They employ hierarchical, monolithic, and hybrid architectures, leveraging multimodal perception to decompose tasks and synthesize actionable trajectories.
• These planners utilize advanced algorithms including MILP, A*-based searches, and RL-based imitation to optimize performance and ensure context-sensitive decision-making.

A Vision-LLM (VLM) Planner is a class of planning system that leverages foundation models capable of joint visual and linguistic reasoning to generate, refine, or evaluate action sequences, subgoal decompositions, or trajectories in robotics, reinforcement learning, visual task planning, or mission generation. VLM Planners fuse multimodal perception—most commonly images, 3D sensor data, and natural language instructions—with model-based or data-driven planning, enabling context-sensitive decision-making and efficient execution of complex tasks across a wide range of embodied, agent-based, and reasoning domains.

1. Architectural Principles and Taxonomies

VLM Planners are situated within a broader taxonomy of foundation model–assisted planning systems distinguished by the explicit use of vision-LLMs as high-level task decomposers, route planners, or semantic evaluators. The predominant architectures include:

• Hierarchical VLA (Vision-Language-Action) Models: Separate a high-level VLM-based planner that produces interpretable intermediate representations (e.g., subtasks, waypoints, programs) from a downstream executor/policy that implements these steps, as seen in MaP-AVR (Guo et al., 22 Dec 2025), RDD (Yan et al., 16 Oct 2025), and PIGEON (Peng et al., 17 Nov 2025). Hierarchical planners support explicit, explainable planning interfaces but require tight planner–executor alignment (Shao et al., 18 Aug 2025).
• Monolithic Models: Fuse perception, planning, and action prediction into a joint, often non-interpretable, end-to-end network, with the VLM directly decoding low-level actions (e.g., RT-2, OpenVLA, discussed in (Shao et al., 18 Aug 2025)). While simplifying execution, these approaches generally lack explicit plan outputs.
• Hybrid or Modular Planners: Couple a conventional or learning-based planner (e.g., GameFormer, PlanTF) with an auxiliary VLM module that injects semantic corrections or context, mediating via a gating or injection interface, as in VLMPlanner (Tang et al., 27 Jul 2025).
• Formal Planning via VLM–PDDL Translation: Employ the VLM as a bridge between visual scenarios and symbolic planning languages such as PDDL, sometimes in a dual-VLM structure (SimVLM + GenVLM) for robust domain and problem file synthesis (Hao et al., 3 Oct 2025).

A salient property across these taxonomies is that the VLM can be queried in different regimes: as a full plan generator (comprehensive rollout), as an incremental subgoal generator, as a high-level semantic scorer, or as a planning rule synthesizer.

2. Core Methodologies and Algorithms

The core functionality of a VLM Planner typically involves one or more of the following algorithmic primitives:

• Multimodal Perception and State Representation: VLMs process a combination of raw sensory data (RGB images, depth, LiDAR, semantic maps), associated spatial metadata, and unstructured language instructions, encoding them into fused joint representations via architectures such as Qwen2.5-VL, CLIP-BERT, or multimodal transformers (Shao et al., 18 Aug 2025, Chen et al., 27 Sep 2025).
• Subgoal or Action Sequence Generation: Given observed context and task goals, the VLM outputs intermediate subgoals (text, tokens, keypoints, or pseudo-code) that segment long-horizon tasks, enabling sample-efficient or robust downstream RL or classical planning. This is formalized in RL as:

$$\max_{g_{1:K}\in\mathcal G^K}\;\mathbb E\Bigl[\sum_t \gamma^t\,R(s_t,a_t)\Bigr] \quad\text{s.t. each subgoal }g_i\text{ is achieved in sequence}$$

(Schoepp et al., 21 Feb 2025).

• Guided Trajectory Optimization and Route Planning: For spatial tasks (inspection, aerial navigation, driving), VLM Planners parse images and natural language, extract waypoints or POIs, and structure the planning as trajectory optimization or TSP/A*-based global-local search (Sun et al., 3 Jun 2025, Sautenkov et al., 4 Mar 2025). For instance, Molmo-7B-O is used to extract points and obstacles from satellite images, then route is optimized via a TSP solver and refined by A*-based local search (Sautenkov et al., 4 Mar 2025).
• VLM-in-the-Loop Constraint Satisfaction and Validation: Planners leverage VLM-generated semantic risk maps, collision groupings, or constraint scores to guide or scale optimization (as in CoDriveVLM (Liu et al., 10 Jan 2025), where chain-of-thought extracted risk indicators steer both dispatching MILPs and ADMM-based motion planning).
• Retrieval-Augmented and RL-Finetuned Planning: Retrieval-based demonstration aligners (e.g., RDD (Yan et al., 16 Oct 2025)) segment demonstrations into maximally policy-consistent subtasks using visual representations, while RL-based planners (e.g., PIGEON (Peng et al., 17 Nov 2025), OpenVLN (Lin et al., 9 Nov 2025)) employ verifiable or value-shaped dense rewards for sample-efficient fine-tuning of VLM-driven decision policies.
• Programmatic and Scripted Plan Synthesis: Some VLM Planners generate step-wise programmatic visual reasoning scripts (e.g., LOC, CROP, VQA modules in VLAgent (Xu et al., 9 Jun 2025)), which are parsed, repaired, and executed component-wise for compositional interpretability and robustness.

The following table organizes representative workflow stages for selected VLM Planner paradigms:

Planner Paradigm	Perception & Input	VLM Role	Plan Output Type	Downstream Policy
Hierarchical	Images, language, map	Subgoal decomposition	Subtasks, keypoints, program	RL/executor
Monolithic	Images (+proprio/scene)	End-to-end action decoding	Direct action(s)	Implicit/NN decoder
TSP+Local Search	Satellite/BEV image, text	Waypoint/obstacle extraction	Ordered waypoint sequence	TSP+A*, path smoothing
Programmatic Reason	Images, question	Modular script generation	Pseudocode script	Script interpreter
RL-based	Images, language	Policy πθ, reward shaping	Action proposal/trajectory	PPO/VLN-CE hybrid

3. Mathematical Formalization and Losses

VLM Planner frameworks articulate algorithms at multiple levels of abstraction, frequently employing:

• Hierarchical MDP Formalisms: Where the state space $\mathcal S$ includes both low-level proprioceptive state $o$ and multimodal perception $I$, with expanded action sets $\mathcal A \cup \mathcal G$ (the latter output by VLM as subgoals), and reward functions temporally decomposed by subgoal achievement (Schoepp et al., 21 Feb 2025).
• Mixed-Integer Linear Programs (MILPs)/ADMM: For urban dispatching, VLM outputs instantiate MILP cost matrices (incorporating semantic risk scores), and ADMM updates iterate over decentralized vehicle state and control trajectories, as in CoDriveVLM (Liu et al., 10 Jan 2025).
• TSP and A*-based Objective Functions: Global tour minimization,

$\min_{\pi} \sum_{k=1}^{n} d_{\pi(k)\pi(k+1)}$

is coupled to A* obstacle avoidance with risk maps derived from VLM segmentations (Sautenkov et al., 4 Mar 2025).

• Contrastive and InfoNCE Losses: For demonstration decomposition, similarity retrieval is trained by infoNCE or cross-modal alignment losses,

$$L_{retr} = -\,\sum_{i=1}^N \log \frac{\exp(S(I_i, D_{j(i)})/\tau)}{\sum_{j'}\exp(S(I_i, D_{j'})/\tau)}$$

(Yan et al., 16 Oct 2025).

• Supervised and RL-based Imitation: Subgoal, waypoint, or program planners learn via token-level cross-entropy (for text/command decoding) or L2 waypoint regression losses, often augmented with PPO or value-based RL components when acting as policies (Shao et al., 18 Aug 2025, Lin et al., 9 Nov 2025).
• Program Syntax and Semantic Repair: For programmatic planners, syntax/semantic verifiers catch and repair plan step errors (module name validation, argument-type checks, logic corrections), with fallback to direct visual QA if plans cannot be repaired (Xu et al., 9 Jun 2025).

4. Empirical Evaluation and Benchmarking

VLM Planner systems have been rigorously evaluated across a spectrum of simulated, real-world, and benchmarked settings spanning navigation, manipulation, mission generation, and visual reasoning. Key findings include:

• Performance Benchmarks:
  • PIGEON (Peng et al., 17 Nov 2025) achieves 79.2% success rate (SR) and 36.8% SPL on HM3Dv2 for object navigation—an 11-point SR gain over prior zero-shot methods.
  • CoDriveVLM (Liu et al., 10 Jan 2025) reduces average task response times by up to 20% versus best heuristic baselines, and achieves perfect safety (DP_k values never near 1.0).
  • BEV-VLM (Chen et al., 27 Sep 2025) attains a 44.8% relative reduction in average displacement error and 0.00% collision rate versus the best vision-only baseline on the nuScenes trajectory planning benchmark.
  • RDD (Yan et al., 16 Oct 2025) delivers up to 72.3% end-to-end success in simulated long-horizon manipulation, outperforming temporal or CPD-based segmenters by over 15 points.
  • MaP-AVR (Guo et al., 22 Dec 2025) increases end-to-end daily-living task success from 11.3% (w/o ICL) to 43.1% (w/ICL) on OmniGibson.
  • In video generation, VLIPP (Yang et al., 30 Mar 2025) demonstrates +11–16% relative gains in physical plausibility scores by integrating VLM planning with motion-conditional diffusion.
• Ablation Studies:
  • Disabling VLM-based selection or guidance causes significant performance reductions (e.g., −4.8 SR, −5.0 SPL in PIGEON (Peng et al., 17 Nov 2025); −23.33% SR in TAMP (Kwon et al., 30 Oct 2025)).
  • Programmatic planners' performance drops without syntax-semantics repair modules (Xu et al., 9 Jun 2025).
  • Retrieval or RL-based subgoal decomposers show a 7–15 point drop without retrieval loss or alignment objective (Yan et al., 16 Oct 2025).
• Real-World Transfer:
  • CoDriveVLM and RDD validate on CARLA, real-world robotic arms, and AndroidWorld for GUI agents (Liu et al., 10 Jan 2025, Yan et al., 16 Oct 2025, Mo et al., 20 May 2025).
  • Social navigation VLM planners outperform behavioral cloning and classic DWA by 36.4% in average success rates and markedly improve user-rated social compliance (Song et al., 2024).

5. Implementation Strategies, Challenges, and Limitations

VLM Planner design introduces new engineering and research challenges:

• Planner–Policy Alignment: Ensuring generated plans align with the downstream policy’s affordances remains a consistent challenge. Retrieval-based decomposition and RAG-style in-context learning (Guo et al., 22 Dec 2025) mitigate subgoal–policy drift, but grounding failures and hallucination persist (Schoepp et al., 21 Feb 2025, Yan et al., 16 Oct 2025).
• Computational Cost and Latency: VLM inferences are non-trivial in cost (e.g., ~9 s per LLM-planner invocation in FM-Planner (Xiao et al., 27 May 2025)), and prompt engineering is needed for zero or few-shot transfer without fine-tuning (Song et al., 2024, Peng et al., 17 Nov 2025). Gating mechanisms (CAI-Gate in VLMPlanner (Tang et al., 27 Jul 2025)) enable dynamic balance between compute and inference but introduce scheduling overhead.
• Generalization and Scalability: Retrieval-augmented and memory-buffered VLM planners self-augment databases for lifelong adaptation (Guo et al., 22 Dec 2025), but real-time, open-world robustness for complex or moving environments is an open area. Formal-planning VLM hybrids (Hao et al., 3 Oct 2025) generalize across visual and rulespace variation, achieving 70% plan validity on unseen instances, yet remain limited by symbolic model coverage and perceptual errors.
• Handling Long-Horizon and Multimodal Complexity: Hierarchical task/motion planners leveraging interleaved VLM guidance avoid wasted sampling and improve success rates, but simulation cost and full observability assumptions can limit real-world scaling (Kwon et al., 30 Oct 2025).

6. Emerging Directions and Future Developments

Immediate and long-term research frontiers for VLM Planners, as identified across surveys and systems, include:

• Memory and History Integration: Introducing persistent memory or snapshot archives to revisit prior cues during long-horizon planning (Shao et al., 18 Aug 2025, Peng et al., 17 Nov 2025).
• 4D/Spatial-Temporal Scene Understanding: Extending perception and planning beyond 2D/3D imagery to continuous 3D point clouds and dynamic, time-varying environments (Shao et al., 18 Aug 2025), supporting planning under occlusions, partial observability, and dynamic actor interactions.
• Model Efficiency and Distillation: Developing lightweight or quantized VLMs for edge deployment on resource-constrained platforms, employing techniques such as dynamic token pruning (Tang et al., 27 Jul 2025, Shao et al., 18 Aug 2025).
• Formal-Neuro-Symbolic Hybridization: Systematic integration of VLM-based perceptual grounding with symbolic planning languages and rule-based policies (e.g., dual-VLM or VLM-in-the-loop PDDL generation) (Hao et al., 3 Oct 2025).
• Multi-Agent and Socially Aware Planning: Extending planners to reason about coordination, communication, and social compliance in human–robot and multi-robot contexts (Song et al., 2024, Liu et al., 10 Jan 2025).
• Lifelong and Continual Learning: Self-augmentation of planner databases and RL-with-verifiable/retrievable reward shaping to support open-ended, lifelong learning in unstructured domains (Guo et al., 22 Dec 2025, Peng et al., 17 Nov 2025).
• Formal Verification and Safety: Leveraging dense value-based or alignment-verifiable rewards (as in OpenVLN (Lin et al., 9 Nov 2025)) and integrating output-verification modules into visual task planning (Xu et al., 9 Jun 2025).

VLM Planners represent an essential advance toward integrating structured multimodal reasoning with robust task execution at scale, with growing empirical validation across simulation and real-world domains and a substantial trajectory of open challenges for scalable, adaptive, and safe embodied intelligence.