---
title: 'PlanBench: Automated Planning Evaluation'
url: https://www.emergentmind.com/topics/planbench-dataset
type: topic
---

# PlanBench: Automated Planning Evaluation

PlanBench is an extensible benchmark designed for the rigorous evaluation of large language models (LLMs) and large reasoning models (LRMs) on classical automated planning, reasoning about change, and plan verification. Developed in the tradition of the International Planning Competition (IPC), PlanBench grounds all tasks in precise PDDL-style (Planning Domain Definition Language) domains, enabling mechanical validation of outputs and encompassing diverse task types, representation styles, and domain distributions. It systematically combines natural-language templates, symbolic encodings, and ground-truth solutions, providing a reproducible suite tailored for zero-shot, one-shot, and few-shot evaluation protocols [2206.10498].

## 1. Formal Structure and Problem Specification

PlanBench formulates each planning instance as a classical deterministic STRIPS-style planning problem:
\[
P = \langle S,\,A,\,T,\,s_0,\,g \rangle
\]
where:
- \(S\): Reachable state space, each state \(s \subseteq F\) is a set of ground fluents.
- \(A\): Finite set of actions, each equipped with \(\mathit{pre}(a)\)—precondition set, and effects \(\mathit{add}(a),\,\mathit{del}(a)\).
- \(T: S \times A \rightarrow S\): Deterministic transition function:
  \[
  T(s, a) = (s \setminus \mathit{del}(a)) \cup \mathit{add}(a) \quad \text{if } \mathit{pre}(a) \subseteq s, \text{ undefined otherwise.}
  \]
- \(s_0\): Initial state.
- \(g \subseteq F\): Goal condition (conjunctive fluents).

A plan \(\pi = [a_1, a_2, \dots, a_k]\) is valid if \(T(\dots T(T(s_0, a_1), a_2)\dots, a_k)\) is defined and achieves \(g\) [2409.13373].

## 2. Domains, Instance Types, and Representations

The dataset encompasses five principal domains:
- Blocksworld (600 instances),
- Mystery Blocksworld (600),
- Depots (500),
- Logistics (285),
- OD_Logistics (285).

Each instance is dual-encoded:
- **Natural-language template**: Controlled English listing actions, preconditions, effects, initial facts, and the goal. Designed to minimize ambiguity while testing linguistic generalization and reasoning.
- **PDDL encoding**: Canonical PDDL files for domains and problems; all actions are primitive operators without hierarchical constructs.

Additionally, "Obfuscated Mystery" tasks are included, where names are replaced with arbitrary tokens to evaluate robustness to symbol grounding [2206.10498, 2511.18165, 2409.13373]. Specific Blocksworld subsets introduce:
- Extended Blocksworld (110 hard instances, 6–20 blocks, plans up to 40 steps),
- Unsolvable Blocksworld (100 instances with contradictory goals).

## 3. Task Taxonomy and Curriculum

PlanBench's curriculum spans eight test types (each with automated prompt, completion, parse, and validation logic):
- Plan Generation (zero/one/few-shot, cost-unconstrained),
- Cost-Optimal Plan Generation,
- Plan Verification (validity of a proposed plan),
- Plan Execution Prediction (state after plan execution),
- Replanning (after exogenous change mid-plan),
- Plan Reuse (adapting/pruning plans for new goals),
- Plan Generalization (refactoring based on operator or object changes),
- Robustness to goal reformulation or domain obfuscation.

The auxiliary test sets include additional Blocksworld instances targeted at plan generalization, explicitly crafted to prevent memorization or surface-level pattern matching by LLMs [2206.10498].

## 4. Evaluation Metrics and Protocols

PlanBench mandates strict mechanical validation of all plans and responses. Key metrics include:
- **Success Rate (SR)** for plan generation:
  \[
  \mathrm{SR} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[\pi_i \text{ is valid}]
  \]
- **Plan Cost**:
  \[
  \mathrm{cost}(\pi) = \sum_{j=1}^{|\pi|} c(a_j)
  \]
- **Optimality Ratio (OR)**:
  \[
  \mathrm{OR}(\pi) = \frac{\mathrm{cost}(\pi)}{\mathrm{cost}(\pi^*)}
  \]
  where \(\pi^*\) is planner-computed optimal plan.

For unsolvable tasks: **True Negative Rate (TNR)**, **False Negative Rate (FNR)**.
For plan verification and reasoning: proportion (%) of exact matching answers to validator ground-truth.

Cost and efficiency metrics (average runtime, average monetary cost per 100 instances) are tracked in recent studies [2409.13373].

Experiments typically use zero-temperature sampling and direct translation of LLM outputs to formal plans, validated via VAL and Fast-Downward (or, for formal verification, NuSMV with LTL model checking) [2510.03469].

## 5. Annotation Formats, Tooling, and Extensibility

Annotations consist of natural language prompts, PDDL domain and problem files, and, optionally, reference (ground-truth) plans. Annotation schema is standardized for each domain:
- Initial state: set of fluents (e.g. “red block on table”).
- Goal: conjunctive set of fluents.
- Actions: complete lifted schema (name, parameters, precondition, effect).
- Instance metadata: domain, identifier, reference solution.
- For plan verification: binary valid/invalid label (possibly with error/failure type annotations).

Tooling is fully open-source; scripts auto-generate prompts, execute model queries, parse outputs, and validate plans. Researchers are encouraged to extend PlanBench via a modular directory structure, contributing new domains, templates, and validation scripts [2206.10498, 2409.13373].

## 6. Applications and Recent Benchmarking Findings

Recent studies have focused on several application modalities:
- **Structured Planning with Symbolic Models**: SymPlanner exploits PlanBench to ground LLM policies in symbolic state transition systems, leveraging iterative correction and contrastive ranking mechanisms to boost plan validity and diversity [2505.01479].
- **HTN Modeling**: L2HP applies LLMs to hierarchical planning over PlanBench, reporting parsing success around 36% in classical PDDL mode and notably low validity in hierarchical HDDL mode (~1%), highlighting benchmark inadequacy for structured HP tasks and the need for new hierarchical-annotated datasets [2511.18165].
- **Plan Verification via Formal Methods**: Natural language plans are automatically translated into Kripke structures and LTL formulas; GPT-5 achieves F1 = 96.3% on the plan verification subset, with almost perfect syntactic form but remaining semantic limits. Model checking analysis is central for bridging LLM output and formal guarantees [2510.03469].
- **Evaluating Large Reasoning Models**: OpenAI’s o1 (Strawberry) LRM demonstrates quantum improvements in PlanBench Blocksworld subsets, yet does not saturate benchmark accuracy; cost and runtime metrics are increasingly incorporated [2409.13373].

## 7. Challenges, Limitations, and Extensions

PlanBench enforces strict mechanical validation, but presents several open challenges:
- **Ambiguity in Natural Language**: Some instances risk underspecification without formal ground-truth anchoring; parallel PDDL encoding mitigates this, but not perfectly [2505.01479].
- **Scalability and Difficulty**: Increasing plan length (beyond 12–16 steps) or object cardinality leads to cascading error rates for LLMs and LRMs.
- **Hierarchical Planning**: PlanBench lacks HTN-specific ground-truth, so extension to hierarchical domains requires new annotation paradigms [2511.18165].
- **Efficient Evaluation**: Original PlanBench did not measure cost-efficiency; recent work has introduced cost-per-instance and runtime measurements for model selection [2409.13373].
- **Extensibility Protocols**: Researchers should supply both zero/one-shot templates and PDDL encodings per new domain, as well as hard instances for scaling analysis.

PlanBench remains the canonical benchmark for assessing the planning and reasoning capabilities of language- and reasoning-based AI models—providing a reproducible, extensible foundation for current and future research in structured plan generation, verification, and formal evaluation [2206.10498, 2409.13373, 2511.18165, 2505.01479, 2510.03469].

Source: https://www.emergentmind.com/topics/planbench-dataset