---
title: Formal Planning via VLM–PDDL Translation
url: https://www.emergentmind.com/topics/formal-planning-via-vlm-pddl-translation
type: topic
---

# Formal Planning via VLM–PDDL Translation

Formal Planning via VLM–PDDL Translation

Formal planning via VLM–PDDL translation refers to the process of converting raw perceptual and natural-language inputs—such as images, videos, and linguistic instructions—into Planning Domain Definition Language (PDDL) problem and domain files using Vision–Language Models (VLMs). This integration bridges continuous perception and symbolic reasoning, enabling the application of sound, verifiable planning algorithms to high-level tasks in visually grounded environments. Recent research formalizes, systematizes, and critically evaluates this translation pipeline across classic, open-world, and large-scale robotic domains [2602.08537][2510.03182][2509.21576][2501.17665][2406.17659].

## 1. Problem Definition and Motivation

VLM–PDDL translation addresses the need for robust, generalizable symbolic planning in real-world environments where perception is noisy, objects and relations are not predefined, and task goals are given in multimodal form. The objective is to generate:
- A PDDL **domain file** $\mathcal{D}$: Specifies types, predicates, and action schemas.
- A PDDL **problem file** $\mathcal{P}$: Specifies a set of objects $E$, an initial state $s_0$ as a set of grounded predicate facts, and a goal $\psi_{\text{goal}}$ expressed as a formula over facts.

Inputs include:
- Visual observations: A single image, a sequence of images, or multimodal (visual + textual) descriptions.
- Task specification: Natural-language instruction or goal description.
- Optional pre-defined or inferred PDDL domain schema.

Solving the planning problem $(\mathcal{D}, \mathcal{P})$ allows formal planners (such as Fast Downward) to produce (sound or optimal) action sequences, overcoming the limitations of end-to-end VLM planning, which fails for long-horizon or occluded tasks [2509.21576][2510.03182].

## 2. Core Architectures and Pipelines

Multiple research lines have converged on modular architectures, each decomposing the translation into subproblems and enforcing syntactic and semantic correctness:

### a. Perceptual Parsing and State Extraction

- **Visual state parsing**: VLMs are prompted to extract object inventories and binary relations from images or videos, yielding a structured representation (e.g., a scene graph, set of grounded predicates, or standardized relation list) [2501.17665][2509.21576][2406.17659][2602.08537].
- **Goal parsing**: The target state is derived from a goal image or language instruction, sometimes via direct few-shot prompted VLM extraction and sometimes via template-based grounding.

### b. Formalization and Problem Construction

- **Direct PDDL generation**: VLMs are prompted (few-shot or one-shot) with well-formed PDDL examples and domain files to fill in :objects, :init, and :goal sections [2501.17665][2509.21576].
- **Intermediate representations**: Pipelines leverage captions or scene graphs as structured bottlenecks, improving consistency and reducing grounding errors [2509.21576].
- **Dual-VLM feedback**: In VLMFP, a simulation VLM (SimVLM) checks the generated PDDL via counterfactual action sequence rollouts, providing human-interpretable feedback and guiding iterative refinement of both domain and problem files [2510.03182].

### c. Closed-Loop Execution

- Systems such as DKPrompt tightly couple VLM state queries with classical planning, prompting VLMs before and after each robot action to dynamically re-ground the symbolic state, re-plan if discrepancies are detected, and thus remain robust to perception errors and execution drift [2406.17659].

## 3. Translation Algorithms and Data Flow

The canonical translation pipeline comprises several algorithmic stages (not all present in every system):

| **Stage**                | **Input**                             | **Output**                                 |
|--------------------------|---------------------------------------|--------------------------------------------|
| Perceptual Parsing       | Image(s), Domain Schema               | List of objects, Relations (state $S_0$)   |
| Goal Parsing             | Goal image or text                    | Target state $S_g$ (predicates/goals)      |
| PDDL File Filling        | Domain, Initi. & Goal states, Example | PDDL Problem File ($\mathcal{P}$)          |
| (Optional) Refinement    | Simulator feedback                    | Improved domain/problem (via VLM loop)     |

Example pseudocode from VLMFP [2510.03182]:
```python
# n_d: NL domain, i_p: initial image
n_p = SimVLM.describe(n_d, i_p)
f_d, f_p = GenVLM.generate(n_d, i_p, n_p)
for t in range(T_max):
    if not valid_PDDL(f_d, f_p):
        f_d, f_p = GenVLM.generate(n_d, i_p, n_p)
        continue
    for π in sampled_action_sequences:
        e_sim = SimVLM.simulate(n_d, i_p, π)
        e_pddl = exec_in_PDDL(f_d, f_p, π)
        if e_sim ≠ e_pddl:
            feedback.append(describe_mismatch(π, e_sim, e_pddl))
    if not feedback:
        π_plan = run_pddl_planner(f_d, f_p)
        if π_plan: return f_d, f_p, π_plan
    f_d, f_p = GenVLM.refine(n_d, i_p, n_p, feedback, f_d, f_p)
```
The pipeline continues until a syntactically valid, semantically accurate, executable plan is found or a failure is declared.

## 4. PDDL Formalization and Domain Expansion

Accurate PDDL encoding is essential for leveraging formal solvers. UniPlan and VLMFP generalize traditional domains via algorithmic domain expansion:
- **Domain augmentation**: Base domains (e.g., blocksworld, tabletop manipulation) are programmatically rewritten to incorporate spatial topology (navigation nodes), bimanual and device-specific affordances, numeric cost fluents, and connectivity constraints [2602.08537].
- **Grounding visual instances**: VLM-extracted symbolic objects and relations are mapped into the :objects and :init of the PDDL problem, while goals are derived from language and visual cues.

Typical domain schemas include unary predicates for type membership, binary/ternary relations for spatial and functional connections, and action schemas parameterized by both object and location arguments. Cost modeling relies on the inclusion of numeric fluents (e.g., (total-cost)) and stepwise accumulation via action effects.

## 5. Evaluation, Benchmarks, and Empirical Results

Evaluation frameworks emphasize two orthogonal criteria:

**Syntactic correctness:** Rate at which VLM-generated problems pass Fast Downward parsing and type/signature checking. For Image2PDDL [2501.17665] and VLMFP [2510.03182], syntax errors are near zero across hundreds of diverse scenarios.

**Content correctness:** Alignment of grounded facts (:init, :goal) and object identities with ground-truth environments. Precision and recall (per predicate/object) are the standard metrics [2509.21576]:
\[
\mathrm{Precision} = \frac{|\hat X \cap X^*|}{|\hat X|}, \quad
\mathrm{Recall} = \frac{|\hat X \cap X^*|}{|X^*|}
\]
For initial-state recall, values remain a critical bottleneck: recall ≈ 40–50%, precision 70–80%, with errors traceable to missed relations or occluded/blurred objects in input imagery [2509.21576].

**Planning and execution success:** Plan success rates measure the frequency at which generated PDDL leads to valid, goal-achieving plans in simulation or hardware. In VLMFP, plans generated for unseen instances and visual appearances achieve 70.0% and 54.1% success rates, respectively—well above prior GPT-style baselines [2510.03182]. Simulation fidelity can be further quantified using the Exploration Walk (EW) score, assessing alignment between learned simulators and PDDL executability.

Empirical performance in complex domains (kitchen, shoe-box teaching, large-scale indoor navigation) exhibits robust scaling in syntax, modestly declining accuracy with increasing domain complexity, and significant improvement over direct action sequence generation [2501.17665][2602.08537].

## 6. Limitations, Open Problems, and Future Directions

Despite rapid progress, several technical bottlenecks remain:

- **Vision bottleneck:** Omission of spatial and relational facts due to inadequate object detection, especially in cluttered, occluded, or low-quality views. This is the primary source of plan failure [2509.21576][2602.08537].
- **Semantic alignment:** Ensuring state schemas and relation vocabularies are mutually consistent across perception, problem, and domain files.
- **Domain generalization:** Reliance on hand-written domain schemas limits adaptation to unseen rule sets; autonomous domain synthesis (as in VLMFP) is nascent.
- **Scalability:** End-to-end VLM–PDDL pipelines can be slow or fail without map compression, candidate pruning, or iterative grounding [2602.08537].

Future research directions highlighted include:
- Open-vocabulary, multi-view perception (e.g., GLIP, Grounding DINO) for improved scene completeness.
- Hybrid neuro-symbolic loops: using planner feedback to prompt VLMs for missed or conflicting facts.
- Data-driven grounding of scene-graph generators to PDDL-style relations.
- Real-time, dynamic environment handling, and online replanning [2509.21576][2602.08537].
- Plug-and-play modularity for task definition, e.g., enabling non-technical users to instantiate tailored domains for robot assistants or educators [2501.17665].

## 7. Applications and Impact

VLM–PDDL translation has led to the following practical advances:
- **Robot-assisted teaching:** Autonomous generation of PDDL planning problems from student workspace imagery, with symbolic plan translation into robot guidance or prompts. In autism spectrum disorder (ASD) classrooms, Image2PDDL lowers technical barriers for lesson creation [2501.17665].
- **Large-scale mobile manipulation:** UniPlan demonstrates robust long-horizon planning in real-world buildings by combining multi-room navigation, object grounding, and task-specific plan synthesis in a unified PDDL abstraction [2602.08537].
- **Open-world task execution:** DKPrompt handles unforeseen environmental changes by continually re-grounding vision-derived predicate sets and re-invoking symbolic planning upon detection of execution divergence, supporting robust execution in complex, unpredictable environments [2406.17659].

The convergence of perception and symbolic planning via VLM–PDDL translation establishes a scalable, modular paradigm for embodied AI, enabling formal guarantees of soundness and optimality in environments that previously resisted symbolic abstraction. Limitations regarding perceptual recall, object/relation vocabulary, and dynamic adaptation define the central open challenges for the field [2510.03182][2509.21576][2602.08537][2406.17659][2501.17665].

Source: https://www.emergentmind.com/topics/formal-planning-via-vlm-pddl-translation