---
title: 'Image2PDDL: Vision-Language to PDDL'
url: https://www.emergentmind.com/topics/image2pddl
type: topic
---

# Image2PDDL: Vision-Language to PDDL

Image2PDDL is a framework that leverages Vision-Language Models to automatically convert images of initial states and descriptions of goal states into PDDL problems. In its 2025 formulation, the framework assumes a hand-crafted PDDL domain and focuses on problem-instance generation rather than domain learning, with the explicit aim of bridging perceptual understanding with symbolic planning, reducing the expertise required to create structured problem instances, and improving scalability across tasks of varying complexity [2501.17665]. Formally, it treats the task as a mapping \(f:(I,G,D)\mapsto P\), where \(I=\{I_1,\dots,I_n\}\) is a set of one or more images representing the initial state, \(G\) is a goal specification given either as an image or as text, \(D\) is a hand-crafted PDDL domain, and \(P\) is a fully grounded PDDL problem file comprising objects, instantiated predicates in `INIT`, and a conjunction of goal predicates [2501.17665].

## 1. Formal problem statement

The central abstraction in Image2PDDL is the conversion of multimodal task specifications into a grounded symbolic planning instance. The output problem file is written as
\[
P=\bigl(o,(\alpha_1,\dots,\alpha_k),\mathit{INIT},\mathit{GOAL}\bigr),
\]
where \(o\) is the set of objects, \(\{\alpha_i\}\) are the instantiated predicates in `INIT`, and `GOAL` is the conjunction of goal predicates [2501.17665]. This formulation makes the framework a perception-to-instance compiler: it does not infer action schemas, predicate vocabularies, or typing rules from raw data, but instead relies on a supplied symbolic domain.

A defining feature of the formulation is that the goal specification is heterogeneous. The initial state is represented by one or more images, while the goal may be either an image or a text description. This design allows the same symbolic target format to be reached from either visual or linguistic goal input, provided that the resulting goal specification can be normalized into the same state-template format used for the initial state [2501.17665].

This separation of concerns places Image2PDDL in a specific segment of the neural-symbolic planning landscape. It addresses instance construction under a fixed domain model, rather than full action-model acquisition. A common misconception is therefore that it performs end-to-end domain induction; the stated formulation instead presupposes a hand-crafted PDDL domain \(D\) [2501.17665].

## 2. Pipeline architecture

Image2PDDL’s pipeline comprises three blocks: vision-language parsing of the initial state, parsing of the goal state, and PDDL problem synthesis [2501.17665]. In the first block, an image \(I\) is processed by a vision encoder \(\phi_v\), which produces visual features \(\phi_v(I)\). These features are then provided through a structured prompt to ChatGPT4o, and the output is a normalized text “state template” \(S_{\text{init}}\) [2501.17665].

The second block parses the goal state. If the goal is an image \(G_{\text{img}}\), the framework applies the analogous prompting process using \(\phi_v(G_{\text{img}})\). If the goal is text \(G_{\text{txt}}\), the system applies a text-prompting function \(\psi_t:G_{\text{txt}}\to S_{\text{goal}}\). In both cases, the result is a state template \(S_{\text{goal}}\) that matches the format of \(S_{\text{init}}\) [2501.17665].

The third block performs PDDL problem synthesis. A final ChatGPT4o prompt concatenates the domain \(D\), the initial-state template \(S_{\text{init}}\), the goal-state template \(S_{\text{goal}}\), and an example problem. The model then outputs a syntactically correct PDDL problem file \(P\), after which a cleanup stage strips annotation or highlighting [2501.17665]. This decomposition suggests a modular design in which perceptual recognition is first normalized into an intermediate textual state description before symbolic serialization into PDDL.

The architecture also makes explicit use of prompting as a control mechanism. The quality of the generated problem file depends not only on visual recognition but also on the robustness of the state-format examples and domain hints supplied in the prompts [2501.17665]. This prompt dependence is presented as a limitation rather than as an incidental engineering detail.

## 3. Symbolic representation and PDDL synthesis

Image2PDDL generates a PDDL problem file against a supplied domain schema. The grammar template provided to ChatGPT4o includes the standard components of a PDDL problem:

```lisp
(define (problem <prob-name>)
  (:domain <domain-name>)
  (:objects <obj-list>)
  (:init
    <predicate>*
  )
  (:goal (and
    <predicate>*
  ))
)
```

This template-based generation process is illustrated in the paper with a Blocksworld domain containing predicates such as `(on ?x - block ?y - block)`, `(ontable ?x - block)`, `(clear ?x - block)`, `(holding ?x - block)`, and `(arm-empty)`, together with actions such as `pick-up` defined by explicit `:precondition` and `:effect` clauses [2501.17665].

The generated problems are fully grounded. In the Blocksworld “Easy” example, the problem file lists five block objects, initializes each as `ontable` and `clear`, sets `(arm-empty)`, and specifies a conjunctive goal requiring `(on b2 b1)`, `(on b3 b2)`, and `(on b5 b4)` [2501.17665]. The same general mechanism is applied across other benchmark domains, including sliding-tile puzzles and a 3D kitchen domain.

The representational significance of this design is that Image2PDDL externalizes symbolic structure rather than embedding it only in latent variables. The generated PDDL is intended to be grammar-correct and executable by standard planners, and the evaluation therefore distinguishes between syntax correctness and content correctness [2501.17665]. In effect, the framework treats symbolic problem generation as a controlled text generation task conditioned on visual and textual scene information.

## 4. Empirical evaluation

The evaluation is conducted on various domains, including standard planning domains like Blocksworld and sliding tile puzzles, using datasets with multiple difficulty levels [2501.17665]. Performance is assessed on two metrics: syntax correctness, which ensures grammar and executability, and content correctness, which verifies accurate state representation in generated PDDL problems [2501.17665]. The reported summary covers three domains under an Images→Image goals setting, with 50 scenarios at each difficulty level.

| Domain / difficulty | SyntaxAccuracy | ContentAccuracy |
|---|---:|---:|
| Blocksworld Easy (5 blocks) | 100% | 49/50 = 98% |
| Blocksworld Medium (6 blocks) | 100% | 47/50 = 94% |
| Blocksworld Hard (7 blocks) | 100% | 47/50 = 94% |
| Sliding-Tile Easy (8 tiles) | 100% | 48/50 = 96% |
| Sliding-Tile Medium (15 tiles) | 100% | 49/50 = 98% |
| Sliding-Tile Hard (24 tiles) | 100% | 49/50 = 98% |
| Kitchen Easy | 100% | 43/50 = 86% |
| Kitchen Medium | 100% | 45/50 = 90% |
| Kitchen Hard | 100% | 41/50 = 82% |

Across all reported settings, syntax accuracy is 100%, while content accuracy varies by domain and difficulty [2501.17665]. The sliding-tile results remain high even at larger puzzle sizes, whereas the 3D Kitchen domain shows a clearer degradation as visual complexity increases. Using text goals slightly improves `ContentAccuracy` in visually complex domains, with Kitchen reaching approximately 94% for all levels under text goals [2501.17665].

These results indicate that grammar-constrained synthesis is easier than exact scene interpretation. A plausible implication is that, in the reported setup, failure modes are concentrated more heavily in state extraction than in PDDL serialization. The paper’s own error analysis is consistent with this interpretation: it reports systematic block-stack reversal in Blocksworld, swapped tiles in Sliding-Tile, and cases in Kitchen where “stove” is mis-typed as item versus location [2501.17665].

## 5. Robot-assisted teaching use case

A distinctive application discussed for Image2PDDL is robot-assisted teaching of students with Autism Spectrum Disorder. The motivating context is that students with Autism Spectrum Disorder respond well to visual, consistent cues, and the paper situates the framework in TEACCH® shoebox tasks, where a student must place objects into labeled compartments in a box [2501.17665].

In this use case, the domain model \(D_{\text{shoebox}}\) defines elements such as balls and bowls, locations, and `put_in` actions. For each student instance, Image2PDDL parses one or more camera snapshots to produce a PDDL problem \(P\), automatically identifying current state and goal. A planner such as Fast Downward then generates an action sequence \(A=(a_1,\dots,a_k)\), and a humanoid robot such as NAO or Codey Rocky executes the sequence or supervises the student by pointing, handing objects, or giving verbal instructions [2501.17665].

The initial experiments on a “one-to-one correspondence task” report that a single snapshot often sufficed for state extraction, while two snapshots in a video-snapshot category improved object-count accuracy from approximately 90% to greater than 95%; `SyntaxAccuracy` remained 100% [2501.17665]. The paper attributes remaining misclassifications primarily to object occlusion and proposes cross-snapshot comparison together with a requirement that each object appear in at least one state [2501.17665].

Ongoing work includes collaborating with Autism Spectrum Disorder experts to choose pedagogical strategies such as “model-prompt-fade,” mapping abstract PDDL actions to robot primitives such as move-arm, point-to, and hand-over, and conducting user studies measuring engagement and skill transfer in small Autism Spectrum Disorder cohorts [2501.17665]. This use case places Image2PDDL within assistive robotics rather than only within benchmark planning.

## 6. Relation to adjacent image-to-symbolic planning research

Image2PDDL belongs to a broader line of work that connects raw perception to symbolic planners, but its position within that line is specific. “Neural-Symbolic Descriptive Action Model from Images: The Search for STRIPS” introduces Double-Stage Action Model Acquisition (DSAMA), which obtains a descriptive PDDL action model with explicit preconditions and effects over propositional variables unsupervized-learned from images, trains Random Forest rule-based classifiers, and compiles them into logical formulae in PDDL [1912.05492]. DSAMA thereby addresses action-model acquisition, whereas Image2PDDL assumes the domain model and concentrates on grounding a problem instance.

Latplan similarly operates at the level of learned symbolic models, but it does so through unsupervised latent-space abstraction. “Classical Planning in Deep Latent Space” learns a complete propositional PDDL action model of the environment from unlabeled image pairs, using a State AutoEncoder and a Bidirectional Cube-Space AutoEncoder, and then performs planning in symbolic latent space [2107.00110]. By contrast, Image2PDDL does not infer predicates or actions from transitions; it compiles visually and linguistically specified states into a grounded problem under a supplied domain.

Deep Planning Domain Learning (DPDL) occupies an intermediate position. “Transferable Task Execution from Pixels through Deep Planning Domain Learning” learns a high-level model that predicts values for logical predicates from RGB images and robot joint readings, generates a PDDL problem from the predicted predicates and a user-specified goal, and uses a separate low-level policy to translate symbolic operators into executable robot actions in a photorealistic kitchen scenario [2003.03726]. This makes DPDL closer to Image2PDDL in its reliance on explicit symbolic predicates during planning, although DPDL includes execution-level grounding through learned control.

The term also appears in a broader robotic planning setting in “UniPlan: Vision-Language Task Planning for Mobile Manipulation with Unified PDDL Formulation.” There, an Image2PDDL pipeline retrieves task-relevant nodes from a visual-topological map, grounds anchored images into task-relevant objects and their PDDL states using a VLM, reconnects the nodes into a compressed topological map with connectivity and costs represented in PDDL, and then invokes off-the-shelf PDDL solvers for long-horizon mobile manipulation [2602.08537]. This suggests that “Image2PDDL” can denote not only a specific 2025 framework but also a more general design pattern in which multimodal perception is compiled into symbolic planning instances.

## 7. Limitations and future directions

The limitations of Image2PDDL are described in terms of scalability, error patterns, and prompt dependence. Performance degrades on very cluttered scenes, with Kitchen-Hard content accuracy reported at approximately 82% [2501.17665]. The paper identifies systematic block-stack reversal in Blocksworld, swapped tiles in Sliding-Tile, and the “stove” item-versus-location typing error in Kitchen as characteristic failure modes [2501.17665].

The framework is also explicitly dependent on prompt engineering: output quality hinges on providing robust state-format examples and domain hints [2501.17665]. This is a substantive methodological constraint, since the symbolic correctness of the generated instance depends on both the perceptual model and the prompt-level specification of the target formalism. Another common misconception is that 100% syntax correctness implies semantically reliable planning input; the reported content-accuracy variation across domains shows that grammar and state fidelity are separable properties [2501.17665].

The future directions proposed for the framework include integrating explicit object detectors or CLIP-based grounding for more accurate \(\phi_v\), hierarchical prompting to handle very large numbers of objects, joint vision-language fine-tuning on paired \((\text{image},\text{PDDL})\) datasets, extension to dynamic and continuous domains such as moving obstacles and temporal conditions, and fully automated domain induction from demonstrations for new shoebox tasks [2501.17665]. The last point is especially significant because it identifies a boundary of the current system: domain induction remains outside the scope of the reported pipeline.

Taken together, these limitations clarify the current status of Image2PDDL within neural-symbolic planning. It achieves near-perfect syntax and strong content accuracy in classic benchmarks, and it demonstrates preliminary applicability in robot-assisted teaching for Autism Spectrum Disorder, but it does so under a fixed symbolic domain and with performance that remains sensitive to visual complexity and prompt design [2501.17665].

Source: https://www.emergentmind.com/topics/image2pddl