---
title: Simulation-to-Rules (VLMFP)
url: https://www.emergentmind.com/topics/simulation-to-rules-vlmfp
type: topic
---

# Simulation-to-Rules (VLMFP)

Simulation-to-Rules (VLMFP) describes a vision–language model (VLM) guided paradigm for autonomously formalizing visual planning scenarios into symbolic, rule-based representations. The approach is instantiated in the Dual-VLM framework VLMFP, which achieves reliable translation of image- and language-conditioned tasks into Planning Domain Definition Language (PDDL) files, thereby enabling formal symbolic planning directly from visual inputs. This method addresses the challenge of automatically extracting all necessary planning rules, rather than relying solely on problem-specific files that require human input or environment access, and demonstrates generalized planning and simulation capabilities across diverse domains [2510.03182].

## 1. Formal Problem Setting and Notation

Simulation-to-Rules (VLMFP) is designed to convert visual and linguistic domain descriptions into executable formal planning rules. A visual planning scenario is specified by:
- A natural language description $n_d$ (defining rules, actions, and constraints)
- An image $i_p$ encoding the spatial configuration (e.g., a grid layout)

The target output consists of:
- $f_d$: the PDDL domain file (defining predicates, actions, and transition rules)
- $f_p$: the PDDL problem file (specifying initial state and goal conditions)

Optimal solutions are formal plans $\pi$ such that a symbolic planner operating on $(f_d,f_p)$ computes a valid action sequence to achieve the designated goal state. The central technical challenge lies in generating both $f_d$ and $f_p$ from $n_d$ and $i_p$, including accurate symbolic representations of general rules, not just problem instances.

The VLMFP architecture introduces two distinct VLMs:
- **SimVLM:** A vision–language model specialized for simulating environment dynamics, describing scenarios, and judging goal reachability based on current state and action sequences.
- **GenVLM:** A generative vision–language model (e.g., GPT-4o) tasked with producing and refining the PDDL files interactively, leveraging feedback arising from discrepancies between symbolic and simulated outcomes.

## 2. Architecture and Workflow

VLMFP employs a Dual-VLM workflow orchestrated through four key stages:

1. **Scenario Encoding:**  
   SimVLM generates a concise natural language description $n_p$ summarizing spatial relationships and object configurations derived from $n_d$ and $i_p$.

2. **Initial PDDL Generation:**  
   GenVLM uses $n_d$ and $n_p$ to synthesize initial candidate PDDL files $(f_d^{(0)},f_p^{(0)})$.

3. **Simulation Consistency Check and Feedback Collection:**  
   Random action sequences $q$ are sampled; SimVLM reports simulated transition outcomes while the planner executes those sequences on $(f_d^{(t)},f_p^{(t)})$. Discrepancies (i.e., $E_{\text{sim}}(q) \neq E_{f_d,f_p}(q)$) are collected as feedback for further refinement.

4. **Iterative Refinement:**  
   GenVLM receives mismatch feedback and updates the files. This loop continues until either perfect alignment is reached (measured by the EW score, see Section 4), a valid plan is found, or the maximum number of iterations is reached.

Pseudocode for this process, following [2510.03182], is as follows:
```python
Input: domain text n_d, image i_p
Output: PDDL domain file f_d and problem file f_p, plus plan π

1. n_p ← SimVLM.describe(n_d, i_p)
2. (f_d^0, f_p^0) ← GenVLM.generate_initial(n_d, n_p)
3. t ← 0
4. repeat
5.     if not syntax_valid(f_d^t) or not syntax_valid(f_p^t):
6.         (f_d^t, f_p^t) ← GenVLM.regenerate(n_d, n_p)
7.         continue
8.     sample Q = {q₁, ..., q_N}
9.     for q in Q:
10.        r_sim ← SimVLM.simulate(n_d, n_p, q)
11.        r_pddl ← Planner.execute(f_d^t, f_p^t, q)
12.        if r_sim ≠ r_pddl:
13.           collect mismatch feedback
14.    score ← EW_score(f_d^t, f_p^t)
15.    if score == 1.0 and Planner.solve(f_d^t, f_p^t) succeeds:
16.        return (f_d^t, f_p^t, π)
17.    (f_d^{t+1}, f_p^{t+1}) ← GenVLM.refine(n_d, n_p, f_d^t, f_p^t)
18.    t ← t + 1
19. until t ≥ T_max
```

Termination occurs when no mismatches are observed and the symbolic planner can solve the formalized problem, or after a preset iteration cap.

## 3. Output Representation: PDDL Domain and Problem Files

The PDDL domain file $f_d$ encodes type signatures, predicates, and action schemas, whereas the problem file $f_p$ introduces instance-specific objects, initial state predicates, and goal formulations.

An illustrative example for the "FrozenLake" environment is as follows:

- **Domain file ($f_d$):**
  ```lisp
  (define (domain frozenlake)
    (:requirements :strips)
    (:predicates
      (at ?x)
      (ice‐hole ?x)
      (up_direction ?from ?to)
      (down_direction ?from ?to)
      (left_direction ?from ?to)
      (right_direction ?from ?to)
    )
    (:action move-up
      :parameters (?from ?to)
      :precondition (and (at ?from) (up_direction ?from ?to) (not (ice-hole ?from)))
      :effect (and (not (at ?from)) (at ?to)))
      ;; move-down, move-left, move-right analogously
  )
  ```

- **Problem file ($f_p$):**
  ```lisp
  (define (problem FL-instance1)
    (:domain frozenlake)
    (:objects
      pos-1-1 pos-1-2 pos-1-3 pos-1-4
      pos-2-1 pos-2-2 pos-2-3 pos-2-4
      pos-3-1 pos-3-2 pos-3-3 pos-3-4
      pos-4-1 pos-4-2 pos-4-3 pos-4-4 - position
    )
    (:init
      (at pos-1-1)
      (ice-hole pos-1-3) (ice-hole pos-2-2) (ice-hole pos-3-3)
      (up_direction pos-2-1 pos-1-1) (down_direction pos-1-1 pos-2-1)
      ;; ... other direction predicates
    )
    (:goal (and (at pos-4-4)))
  )
  ```

The same domain file $f_d$ generalizes across all instances for a given problem class, while $f_p$ is adapted to the particular state configuration generated from the visual input.

## 4. Evaluation Metrics and Empirical Results

Multiple quantitative indicators are established for benchmarking VLMFP [2510.03182]:

- **SimVLM Metrics:**  
  - Task description accuracy
  - Execution reason accuracy
  - Execution result accuracy
  - Goal-reaching judgment accuracy  
  For seen and unseen appearances, SimVLM achieves high performance, with rates ranging from 82% to 95.5%.

- **Planning Validity:**  
  The success rate is defined as the proportion of tested instances where the planner, operating on generated $(f_d, f_p)$, reaches the designated goal.  
  VLMFP with GPT-4o yields a planning validity of 70.0% for seen appearances and 54.1% for unseen ones, markedly superior to CodePDDL baselines (32.3% for unseen cases).

- **EW (Exploration Walk) Score:**  
  EW quantifies the bidirectional agreement between SimVLM simulations and PDDL execution across sampled action sequences:
  $$
  \text{EW} = \frac{2}{(1/A + 1/B)}
  $$
  where $A$ and $B$ are averages of the expected valid sequence rates under SimVLM and PDDL models, respectively. A high EW score indicates close alignment between simulated and formalized domains.

| Metric                         | Seen (%) | Unseen (%) |
|-------------------------------|----------|------------|
| SimVLM TaskDesc               | 95.5     | 92.6       |
| SimVLM ExecResult             | 85.5     | 87.8       |
| SimVLM GoalReach              | 82.4     | 85.6       |
| VLMFP Planning Validity       | 70.0     | 54.1       |
| CodePDDL Baseline Validity    | 30.7     | 32.3       |

This demonstrates robust generalization both to novel visual environments and to previously unseen instance configurations.

## 5. Generalization and System Limitations

The generalization capabilities of Simulation-to-Rules (VLMFP) are evidenced at multiple levels:
- **Visual generalization:** SimVLM accuracy remains above 82% when evaluated on domains rendered in unseen visual styles.
- **Rule generalization:** Novel FrozenLake rule variants (e.g., teleportation, skip-action after hazard) show reasoning/execution rates for SimVLM between 59–99% in most cases. Notably, specific rules requiring multi-step state resets present systematic challenges—e.g., the skip-action rule shows correct textual reasoning (71%) but only 0% execution fidelity due to state-tracking errors.
- **Scalability:** The same $f_d$ supports all instances within a domain, with $f_p$ automatically adapted, illustrating broad intra-domain generalization.

Limitations observed include:
- Occasional omission of required predicates (e.g., directional constraints) in the generated problem file, leading to incomplete or unexecutable plans.
- In complex domains requiring many object types and intricate preconditions (e.g., Sokoban, Printer), initial generation may lack necessary constraints, with planning accuracy collapsing in the absence of iterative refinement.
- SimVLM’s state-tracking capacity limits plan fidelity in rules requiring non-local state resets or novel dynamics.

A plausible implication is that while VLMFP can formalize and generalize many classes of planning tasks, reliance on vision–language models for environment simulation imposes an upper bound on achievable logical fidelity when rules depart structurally from training data.

## 6. Related Research and Positioning

VLMFP is positioned directly in response to prior hybrid approaches in neuro-symbolic planning which leverage VLMs to convert visual problems into PDDL for downstream symbolic planning, yet require human-authored domain files or extensive interaction with the environment for verification. Unique to VLMFP is the end-to-end automation of both domain and problem file synthesis and the closure of the formalization loop by iterative simulation-based refinement mediated by a pair of specialized VLMs [2510.03182].

Simulation-to-Rules stands in contrast to classical model-free data-driven simulation in computational mechanics [2101.10730], where transition rules are constructed from structured (often physical) data and classification of behavioral regimes rather than extracted from visual-linguistic input. The unification of simulation and symbolic formalization within VLMFP reflects a general trend toward integrative, data-efficient neuro-symbolic planning frameworks.

## 7. Significance and Future Directions

Simulation-to-Rules (VLMFP) demonstrates viable automatic formalization of complex visual planning domains with generalization to new visual styles, object configurations, and varied rule sets. The approach moves beyond instance-specific reasoning, enabling symbolic execution at scale directly from perception data.

Open challenges and future potential include:
- Enhancing VLM reasoning for rule sets requiring non-local state dependencies and memory of complex transitions
- Robust extraction of implicit constraints in domains with high combinatorial complexity
- Scaling to more general classes of planning problems beyond grid-based environments

Continued development of dual-model simulation-to-rule pipelines promises to increase the autonomy, transferability, and transparency of visual-to-symbolic planning systems.

Source: https://www.emergentmind.com/topics/simulation-to-rules-vlmfp