Image2PDDL: Vision-Language to PDDL

• Image2PDDL is a framework that leverages Vision-Language models to automatically translate images of initial states and textual or visual goal descriptions into formal PDDL planning problems.
• It employs a modular three-block pipeline that parses visual and textual inputs, normalizes state templates, and synthesizes fully grounded, executable PDDL files using a supplied domain model.
• Evaluations across domains like Blocksworld and Kitchen demonstrate near-perfect syntax accuracy and competitive content accuracy, while highlighting challenges related to prompt dependence and visual complexity.

Image2PDDL is a framework that leverages Vision-LLMs to automatically convert images of initial states and descriptions of goal states into PDDL problems. In its 2025 formulation, the framework assumes a hand-crafted PDDL domain and focuses on problem-instance generation rather than domain learning, with the explicit aim of bridging perceptual understanding with symbolic planning, reducing the expertise required to create structured problem instances, and improving scalability across tasks of varying complexity (Dang et al., 29 Jan 2025). Formally, it treats the task as a mapping $f:(I,G,D)\mapsto P$, where $I=\{I_1,\dots,I_n\}$ is a set of one or more images representing the initial state, $G$ is a goal specification given either as an image or as text, $D$ is a hand-crafted PDDL domain, and $P$ is a fully grounded PDDL problem file comprising objects, instantiated predicates in INIT, and a conjunction of goal predicates (Dang et al., 29 Jan 2025).

1. Formal problem statement

The central abstraction in Image2PDDL is the conversion of multimodal task specifications into a grounded symbolic planning instance. The output problem file is written as

$$P=\bigl(o,(\alpha_1,\dots,\alpha_k),\mathit{INIT},\mathit{GOAL}\bigr),$$

where $o$ is the set of objects, $\{\alpha_i\}$ are the instantiated predicates in INIT, and GOAL is the conjunction of goal predicates (Dang et al., 29 Jan 2025). This formulation makes the framework a perception-to-instance compiler: it does not infer action schemas, predicate vocabularies, or typing rules from raw data, but instead relies on a supplied symbolic domain.

A defining feature of the formulation is that the goal specification is heterogeneous. The initial state is represented by one or more images, while the goal may be either an image or a text description. This design allows the same symbolic target format to be reached from either visual or linguistic goal input, provided that the resulting goal specification can be normalized into the same state-template format used for the initial state (Dang et al., 29 Jan 2025).

This separation of concerns places Image2PDDL in a specific segment of the neural-symbolic planning landscape. It addresses instance construction under a fixed domain model, rather than full action-model acquisition. A common misconception is therefore that it performs end-to-end domain induction; the stated formulation instead presupposes a hand-crafted PDDL domain $D$ (Dang et al., 29 Jan 2025).

2. Pipeline architecture

Image2PDDL’s pipeline comprises three blocks: vision-language parsing of the initial state, parsing of the goal state, and PDDL problem synthesis (Dang et al., 29 Jan 2025). In the first block, an image $I$ is processed by a vision encoder $I=\{I_1,\dots,I_n\}$0, which produces visual features $I=\{I_1,\dots,I_n\}$1. These features are then provided through a structured prompt to ChatGPT4o, and the output is a normalized text “state template” $I=\{I_1,\dots,I_n\}$2 (Dang et al., 29 Jan 2025).

The second block parses the goal state. If the goal is an image $I=\{I_1,\dots,I_n\}$3, the framework applies the analogous prompting process using $I=\{I_1,\dots,I_n\}$4. If the goal is text $I=\{I_1,\dots,I_n\}$5, the system applies a text-prompting function $I=\{I_1,\dots,I_n\}$6. In both cases, the result is a state template $I=\{I_1,\dots,I_n\}$7 that matches the format of $I=\{I_1,\dots,I_n\}$8 (Dang et al., 29 Jan 2025).

The third block performs PDDL problem synthesis. A final ChatGPT4o prompt concatenates the domain $I=\{I_1,\dots,I_n\}$9, the initial-state template $G$0, the goal-state template $G$1, and an example problem. The model then outputs a syntactically correct PDDL problem file $G$2, after which a cleanup stage strips annotation or highlighting (Dang et al., 29 Jan 2025). This decomposition suggests a modular design in which perceptual recognition is first normalized into an intermediate textual state description before symbolic serialization into PDDL.

The architecture also makes explicit use of prompting as a control mechanism. The quality of the generated problem file depends not only on visual recognition but also on the robustness of the state-format examples and domain hints supplied in the prompts (Dang et al., 29 Jan 2025). This prompt dependence is presented as a limitation rather than as an incidental engineering detail.

3. Symbolic representation and PDDL synthesis

Image2PDDL generates a PDDL problem file against a supplied domain schema. The grammar template provided to ChatGPT4o includes the standard components of a PDDL problem:

$G$8

This template-based generation process is illustrated in the paper with a Blocksworld domain containing predicates such as (on ?x - block ?y - block), (ontable ?x - block), (clear ?x - block), (holding ?x - block), and (arm-empty), together with actions such as pick-up defined by explicit :precondition and :effect clauses (Dang et al., 29 Jan 2025).

The generated problems are fully grounded. In the Blocksworld “Easy” example, the problem file lists five block objects, initializes each as ontable and clear, sets (arm-empty), and specifies a conjunctive goal requiring (on b2 b1), (on b3 b2), and (on b5 b4) (Dang et al., 29 Jan 2025). The same general mechanism is applied across other benchmark domains, including sliding-tile puzzles and a 3D kitchen domain.

The representational significance of this design is that Image2PDDL externalizes symbolic structure rather than embedding it only in latent variables. The generated PDDL is intended to be grammar-correct and executable by standard planners, and the evaluation therefore distinguishes between syntax correctness and content correctness (Dang et al., 29 Jan 2025). In effect, the framework treats symbolic problem generation as a controlled text generation task conditioned on visual and textual scene information.

4. Empirical evaluation

The evaluation is conducted on various domains, including standard planning domains like Blocksworld and sliding tile puzzles, using datasets with multiple difficulty levels (Dang et al., 29 Jan 2025). Performance is assessed on two metrics: syntax correctness, which ensures grammar and executability, and content correctness, which verifies accurate state representation in generated PDDL problems (Dang et al., 29 Jan 2025). The reported summary covers three domains under an Images→Image goals setting, with 50 scenarios at each difficulty level.

Domain / difficulty	SyntaxAccuracy	ContentAccuracy
Blocksworld Easy (5 blocks)	100%	49/50 = 98%
Blocksworld Medium (6 blocks)	100%	47/50 = 94%
Blocksworld Hard (7 blocks)	100%	47/50 = 94%
Sliding-Tile Easy (8 tiles)	100%	48/50 = 96%
Sliding-Tile Medium (15 tiles)	100%	49/50 = 98%
Sliding-Tile Hard (24 tiles)	100%	49/50 = 98%
Kitchen Easy	100%	43/50 = 86%
Kitchen Medium	100%	45/50 = 90%
Kitchen Hard	100%	41/50 = 82%

Across all reported settings, syntax accuracy is 100%, while content accuracy varies by domain and difficulty (Dang et al., 29 Jan 2025). The sliding-tile results remain high even at larger puzzle sizes, whereas the 3D Kitchen domain shows a clearer degradation as visual complexity increases. Using text goals slightly improves ContentAccuracy in visually complex domains, with Kitchen reaching approximately 94% for all levels under text goals (Dang et al., 29 Jan 2025).

These results indicate that grammar-constrained synthesis is easier than exact scene interpretation. A plausible implication is that, in the reported setup, failure modes are concentrated more heavily in state extraction than in PDDL serialization. The paper’s own error analysis is consistent with this interpretation: it reports systematic block-stack reversal in Blocksworld, swapped tiles in Sliding-Tile, and cases in Kitchen where “stove” is mis-typed as item versus location (Dang et al., 29 Jan 2025).

5. Robot-assisted teaching use case

A distinctive application discussed for Image2PDDL is robot-assisted teaching of students with Autism Spectrum Disorder. The motivating context is that students with Autism Spectrum Disorder respond well to visual, consistent cues, and the paper situates the framework in TEACCH® shoebox tasks, where a student must place objects into labeled compartments in a box (Dang et al., 29 Jan 2025).

In this use case, the domain model $G$3 defines elements such as balls and bowls, locations, and put_in actions. For each student instance, Image2PDDL parses one or more camera snapshots to produce a PDDL problem $G$4, automatically identifying current state and goal. A planner such as Fast Downward then generates an action sequence $G$5, and a humanoid robot such as NAO or Codey Rocky executes the sequence or supervises the student by pointing, handing objects, or giving verbal instructions (Dang et al., 29 Jan 2025).

The initial experiments on a “one-to-one correspondence task” report that a single snapshot often sufficed for state extraction, while two snapshots in a video-snapshot category improved object-count accuracy from approximately 90% to greater than 95%; SyntaxAccuracy remained 100% (Dang et al., 29 Jan 2025). The paper attributes remaining misclassifications primarily to object occlusion and proposes cross-snapshot comparison together with a requirement that each object appear in at least one state (Dang et al., 29 Jan 2025).

Ongoing work includes collaborating with Autism Spectrum Disorder experts to choose pedagogical strategies such as “model-prompt-fade,” mapping abstract PDDL actions to robot primitives such as move-arm, point-to, and hand-over, and conducting user studies measuring engagement and skill transfer in small Autism Spectrum Disorder cohorts (Dang et al., 29 Jan 2025). This use case places Image2PDDL within assistive robotics rather than only within benchmark planning.

6. Relation to adjacent image-to-symbolic planning research

Image2PDDL belongs to a broader line of work that connects raw perception to symbolic planners, but its position within that line is specific. “Neural-Symbolic Descriptive Action Model from Images: The Search for STRIPS” introduces Double-Stage Action Model Acquisition (DSAMA), which obtains a descriptive PDDL action model with explicit preconditions and effects over propositional variables unsupervized-learned from images, trains Random Forest rule-based classifiers, and compiles them into logical formulae in PDDL (Asai, 2019). DSAMA thereby addresses action-model acquisition, whereas Image2PDDL assumes the domain model and concentrates on grounding a problem instance.

Latplan similarly operates at the level of learned symbolic models, but it does so through unsupervised latent-space abstraction. “Classical Planning in Deep Latent Space” learns a complete propositional PDDL action model of the environment from unlabeled image pairs, using a State AutoEncoder and a Bidirectional Cube-Space AutoEncoder, and then performs planning in symbolic latent space (Asai et al., 2021). By contrast, Image2PDDL does not infer predicates or actions from transitions; it compiles visually and linguistically specified states into a grounded problem under a supplied domain.

Deep Planning Domain Learning (DPDL) occupies an intermediate position. “Transferable Task Execution from Pixels through Deep Planning Domain Learning” learns a high-level model that predicts values for logical predicates from RGB images and robot joint readings, generates a PDDL problem from the predicted predicates and a user-specified goal, and uses a separate low-level policy to translate symbolic operators into executable robot actions in a photorealistic kitchen scenario (Kase et al., 2020). This makes DPDL closer to Image2PDDL in its reliance on explicit symbolic predicates during planning, although DPDL includes execution-level grounding through learned control.

The term also appears in a broader robotic planning setting in “UniPlan: Vision-Language Task Planning for Mobile Manipulation with Unified PDDL Formulation.” There, an Image2PDDL pipeline retrieves task-relevant nodes from a visual-topological map, grounds anchored images into task-relevant objects and their PDDL states using a VLM, reconnects the nodes into a compressed topological map with connectivity and costs represented in PDDL, and then invokes off-the-shelf PDDL solvers for long-horizon mobile manipulation (Ye et al., 9 Feb 2026). This suggests that “Image2PDDL” can denote not only a specific 2025 framework but also a more general design pattern in which multimodal perception is compiled into symbolic planning instances.

7. Limitations and future directions

The limitations of Image2PDDL are described in terms of scalability, error patterns, and prompt dependence. Performance degrades on very cluttered scenes, with Kitchen-Hard content accuracy reported at approximately 82% (Dang et al., 29 Jan 2025). The paper identifies systematic block-stack reversal in Blocksworld, swapped tiles in Sliding-Tile, and the “stove” item-versus-location typing error in Kitchen as characteristic failure modes (Dang et al., 29 Jan 2025).

The framework is also explicitly dependent on prompt engineering: output quality hinges on providing robust state-format examples and domain hints (Dang et al., 29 Jan 2025). This is a substantive methodological constraint, since the symbolic correctness of the generated instance depends on both the perceptual model and the prompt-level specification of the target formalism. Another common misconception is that 100% syntax correctness implies semantically reliable planning input; the reported content-accuracy variation across domains shows that grammar and state fidelity are separable properties (Dang et al., 29 Jan 2025).

The future directions proposed for the framework include integrating explicit object detectors or CLIP-based grounding for more accurate $G$6, hierarchical prompting to handle very large numbers of objects, joint vision-language fine-tuning on paired $G$7 datasets, extension to dynamic and continuous domains such as moving obstacles and temporal conditions, and fully automated domain induction from demonstrations for new shoebox tasks (Dang et al., 29 Jan 2025). The last point is especially significant because it identifies a boundary of the current system: domain induction remains outside the scope of the reported pipeline.

Taken together, these limitations clarify the current status of Image2PDDL within neural-symbolic planning. It achieves near-perfect syntax and strong content accuracy in classic benchmarks, and it demonstrates preliminary applicability in robot-assisted teaching for Autism Spectrum Disorder, but it does so under a fixed symbolic domain and with performance that remains sensitive to visual complexity and prompt design (Dang et al., 29 Jan 2025).