Formal Planning via VLM–PDDL Translation

Updated 1 March 2026

The paper introduces an innovative integration of vision–language models with PDDL translation to bridge perceptual inputs and symbolic planning.
It details a modular architecture that parses visual data and natural language to generate syntactically correct and semantically accurate PDDL domain and problem files.
Empirical evaluations demonstrate improved planning success rates and scalability in managing real-world, open-world robotic tasks despite perceptual challenges.

Formal planning via VLM–PDDL translation refers to the process of converting raw perceptual and natural-language inputs—such as images, videos, and linguistic instructions—into Planning Domain Definition Language (PDDL) problem and domain files using Vision–LLMs (VLMs). This integration bridges continuous perception and symbolic reasoning, enabling the application of sound, verifiable planning algorithms to high-level tasks in visually grounded environments. Recent research formalizes, systematizes, and critically evaluates this translation pipeline across classic, open-world, and large-scale robotic domains (Ye et al., 9 Feb 2026, Hao et al., 3 Oct 2025, He et al., 25 Sep 2025, Dang et al., 29 Jan 2025, Zhang et al., 2024).

1. Problem Definition and Motivation

VLM–PDDL translation addresses the need for robust, generalizable symbolic planning in real-world environments where perception is noisy, objects and relations are not predefined, and task goals are given in multimodal form. The objective is to generate:

A PDDL domain file $\mathcal{D}$ : Specifies types, predicates, and action schemas.
A PDDL problem file $\mathcal{P}$ : Specifies a set of objects $E$ , an initial state $s_0$ as a set of grounded predicate facts, and a goal $\psi_{\text{goal}}$ expressed as a formula over facts.

Inputs include:

Visual observations: A single image, a sequence of images, or multimodal (visual + textual) descriptions.
Task specification: Natural-language instruction or goal description.
Optional pre-defined or inferred PDDL domain schema.

Solving the planning problem $(\mathcal{D}, \mathcal{P})$ allows formal planners (such as Fast Downward) to produce (sound or optimal) action sequences, overcoming the limitations of end-to-end VLM planning, which fails for long-horizon or occluded tasks (He et al., 25 Sep 2025, Hao et al., 3 Oct 2025).

2. Core Architectures and Pipelines

Multiple research lines have converged on modular architectures, each decomposing the translation into subproblems and enforcing syntactic and semantic correctness:

a. Perceptual Parsing and State Extraction

Visual state parsing: VLMs are prompted to extract object inventories and binary relations from images or videos, yielding a structured representation (e.g., a scene graph, set of grounded predicates, or standardized relation list) (Dang et al., 29 Jan 2025, He et al., 25 Sep 2025, Zhang et al., 2024, Ye et al., 9 Feb 2026).
Goal parsing: The target state is derived from a goal image or language instruction, sometimes via direct few-shot prompted VLM extraction and sometimes via template-based grounding.

b. Formalization and Problem Construction

Direct PDDL generation: VLMs are prompted (few-shot or one-shot) with well-formed PDDL examples and domain files to fill in :objects, :init, and :goal sections (Dang et al., 29 Jan 2025, He et al., 25 Sep 2025).
Intermediate representations: Pipelines leverage captions or scene graphs as structured bottlenecks, improving consistency and reducing grounding errors (He et al., 25 Sep 2025).
Dual-VLM feedback: In VLMFP, a simulation VLM (SimVLM) checks the generated PDDL via counterfactual action sequence rollouts, providing human-interpretable feedback and guiding iterative refinement of both domain and problem files (Hao et al., 3 Oct 2025).

c. Closed-Loop Execution

Systems such as DKPrompt tightly couple VLM state queries with classical planning, prompting VLMs before and after each robot action to dynamically re-ground the symbolic state, re-plan if discrepancies are detected, and thus remain robust to perception errors and execution drift (Zhang et al., 2024).

3. Translation Algorithms and Data Flow

The canonical translation pipeline comprises several algorithmic stages (not all present in every system):

Stage	Input	Output
Perceptual Parsing	Image(s), Domain Schema	List of objects, Relations (state $S_0$ )
Goal Parsing	Goal image or text	Target state $S_g$ (predicates/goals)
PDDL File Filling	Domain, Initi. & Goal states, Example	PDDL Problem File ( $\mathcal{P}$ )
(Optional) Refinement	Simulator feedback	Improved domain/problem (via VLM loop)

Example pseudocode from VLMFP (Hao et al., 3 Oct 2025): $\mathcal{P}$ 0 The pipeline continues until a syntactically valid, semantically accurate, executable plan is found or a failure is declared.

4. PDDL Formalization and Domain Expansion

Accurate PDDL encoding is essential for leveraging formal solvers. UniPlan and VLMFP generalize traditional domains via algorithmic domain expansion:

Domain augmentation: Base domains (e.g., blocksworld, tabletop manipulation) are programmatically rewritten to incorporate spatial topology (navigation nodes), bimanual and device-specific affordances, numeric cost fluents, and connectivity constraints (Ye et al., 9 Feb 2026).
Grounding visual instances: VLM-extracted symbolic objects and relations are mapped into the :objects and :init of the PDDL problem, while goals are derived from language and visual cues.

Typical domain schemas include unary predicates for type membership, binary/ternary relations for spatial and functional connections, and action schemas parameterized by both object and location arguments. Cost modeling relies on the inclusion of numeric fluents (e.g., (total-cost)) and stepwise accumulation via action effects.

5. Evaluation, Benchmarks, and Empirical Results

Evaluation frameworks emphasize two orthogonal criteria:

Syntactic correctness: Rate at which VLM-generated problems pass Fast Downward parsing and type/signature checking. For Image2PDDL (Dang et al., 29 Jan 2025) and VLMFP (Hao et al., 3 Oct 2025), syntax errors are near zero across hundreds of diverse scenarios.

Content correctness: Alignment of grounded facts (:init, :goal) and object identities with ground-truth environments. Precision and recall (per predicate/object) are the standard metrics (He et al., 25 Sep 2025): $\mathrm{Precision} = \frac{|\hat X \cap X^*|}{|\hat X|}, \quad \mathrm{Recall} = \frac{|\hat X \cap X^*|}{|X^*|}$ For initial-state recall, values remain a critical bottleneck: recall ≈ 40–50%, precision 70–80%, with errors traceable to missed relations or occluded/blurred objects in input imagery (He et al., 25 Sep 2025).

Planning and execution success: Plan success rates measure the frequency at which generated PDDL leads to valid, goal-achieving plans in simulation or hardware. In VLMFP, plans generated for unseen instances and visual appearances achieve 70.0% and 54.1% success rates, respectively—well above prior GPT-style baselines (Hao et al., 3 Oct 2025). Simulation fidelity can be further quantified using the Exploration Walk (EW) score, assessing alignment between learned simulators and PDDL executability.

Empirical performance in complex domains (kitchen, shoe-box teaching, large-scale indoor navigation) exhibits robust scaling in syntax, modestly declining accuracy with increasing domain complexity, and significant improvement over direct action sequence generation (Dang et al., 29 Jan 2025, Ye et al., 9 Feb 2026).

6. Limitations, Open Problems, and Future Directions

Despite rapid progress, several technical bottlenecks remain:

Vision bottleneck: Omission of spatial and relational facts due to inadequate object detection, especially in cluttered, occluded, or low-quality views. This is the primary source of plan failure (He et al., 25 Sep 2025, Ye et al., 9 Feb 2026).
Semantic alignment: Ensuring state schemas and relation vocabularies are mutually consistent across perception, problem, and domain files.
Domain generalization: Reliance on hand-written domain schemas limits adaptation to unseen rule sets; autonomous domain synthesis (as in VLMFP) is nascent.
Scalability: End-to-end VLM–PDDL pipelines can be slow or fail without map compression, candidate pruning, or iterative grounding (Ye et al., 9 Feb 2026).

Future research directions highlighted include:

Open-vocabulary, multi-view perception (e.g., GLIP, Grounding DINO) for improved scene completeness.
Hybrid neuro-symbolic loops: using planner feedback to prompt VLMs for missed or conflicting facts.
Data-driven grounding of scene-graph generators to PDDL-style relations.
Real-time, dynamic environment handling, and online replanning (He et al., 25 Sep 2025, Ye et al., 9 Feb 2026).
Plug-and-play modularity for task definition, e.g., enabling non-technical users to instantiate tailored domains for robot assistants or educators (Dang et al., 29 Jan 2025).

7. Applications and Impact

VLM–PDDL translation has led to the following practical advances:

Robot-assisted teaching: Autonomous generation of PDDL planning problems from student workspace imagery, with symbolic plan translation into robot guidance or prompts. In autism spectrum disorder (ASD) classrooms, Image2PDDL lowers technical barriers for lesson creation (Dang et al., 29 Jan 2025).
Large-scale mobile manipulation: UniPlan demonstrates robust long-horizon planning in real-world buildings by combining multi-room navigation, object grounding, and task-specific plan synthesis in a unified PDDL abstraction (Ye et al., 9 Feb 2026).
Open-world task execution: DKPrompt handles unforeseen environmental changes by continually re-grounding vision-derived predicate sets and re-invoking symbolic planning upon detection of execution divergence, supporting robust execution in complex, unpredictable environments (Zhang et al., 2024).

The convergence of perception and symbolic planning via VLM–PDDL translation establishes a scalable, modular paradigm for embodied AI, enabling formal guarantees of soundness and optimality in environments that previously resisted symbolic abstraction. Limitations regarding perceptual recall, object/relation vocabulary, and dynamic adaptation define the central open challenges for the field (Hao et al., 3 Oct 2025, He et al., 25 Sep 2025, Ye et al., 9 Feb 2026, Zhang et al., 2024, Dang et al., 29 Jan 2025).