PlanBench-XL: LLM Planning Benchmark

• The paper demonstrates that LLM agents face significant adaptive planning challenges in retrieval-limited tool ecosystems, with performance dropping sharply under controlled blocking conditions.
• PlanBench-XL comprises 327 retail tasks involving 1,665 tools, requiring agents to iteratively retrieve tools, discover implicit sub-goals, and overcome partial observability challenges.
• Empirical results show strong correlations between datatype exploration and task accuracy, highlighting both the potential and limitations of current LLM planning strategies.

PlanBench-XL is an interactive benchmark for evaluating long-horizon planning by LLM tool-use agents in large-scale, retrieval-limited tool ecosystems. It was introduced to test whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls, infer implicit sub-goals, and adapt to dynamic environments over long horizons. The benchmark comprises 327 retail tasks over 1,665 tools and includes an optional blocking mechanism that simulates missing, failing, or distracting tool functions while preserving solvability. In experiments on ten leading LLMs, performance remained limited: GPT-5.4 reached 51.90% accuracy in block-free settings but fell to 11.36% under the most severe blocking condition (Liu et al., 21 Jun 2026).

1. Position in the PlanBench lineage

PlanBench-XL shares its name with an earlier line of benchmarks on planning, but its problem setting is substantially different. The original "PlanBench: An Extensible Benchmark for Evaluating LLMs on Planning and Reasoning about Change" focuses on IPC-style PDDL domains, automated validation, and tasks such as plan generation, plan verification, replanning, and reasoning about plan execution (Valmeekam et al., 2022). AutoPlanBench later automated PDDL-to-natural-language conversion, releasing code and an initial dataset converting 12 PDDL domains into NL and enabling broader, reproducible evaluation of LLM planning over classical planning tasks (Stein et al., 2023).

PlanBench-XL departs from that PDDL-centered setting. Rather than asking a model to synthesize or reason about plans over explicitly described action schemas, it places an agent inside a retrieval-mediated environment in which tools are only partially visible and intermediate sub-goals are implicit. The central difficulty is not merely selecting a valid next action under known symbolic dynamics, but discovering relevant tools, obtaining missing datatypes, and adapting when accessible tools are incomplete, deceptive, or broken (Liu et al., 21 Jun 2026).

This suggests that the benchmark marks a shift from classical planning evaluation toward agentic planning under partial observability and imperfect tool access. A related but distinct development is PlanningBench, which treats planning data generation as a controllable synthesis problem with verification checklists and broad task taxonomies rather than as a fixed benchmark collection (Zhao et al., 20 May 2026). PlanBench-XL instead fixes an interactive evaluation environment and studies process-level failures in long-horizon tool use.

2. Benchmark composition and task structure

PlanBench-XL is instantiated in the retail domain, covering fulfillment, orders, payments, returns and refunds, and accounts. Tasks begin from an initial datatype set $D_0$ and target a final datatype set $Y$; all intermediate sub-goals and tool paths are implicit and must be discovered by the agent during interaction (Liu et al., 21 Jun 2026).

Component	Value
Domain	retail
Datatypes	56
Tools	1,665
Evaluation queries	327
Shortest valid solution path length $L^\*$	$\{5,6,7,8,9\}$
Step budget $T_{\max}$	100
Per-retrieval return cap $\Lambda_{ret}^{cap}$	30

The task ontology is organized around interconnected retail flows. Fulfillment tasks involve chains such as warehouse request, shipment, and delivery attempts. Order-related tasks include draft order, quote or pricing snapshot, finalized order, and order items. Payment tasks traverse saved method, payment intent, authorization, and payment record. After-sales tasks include return request, return review or audit, and refund. Account-related tasks involve user or account identifiers and stored instrument references (Liu et al., 21 Jun 2026).

Each tool $\tau$ has typed input and output datatype sets, $D_{in}(\tau)$ and $D_{out}(\tau)$, together with a natural-language name and description and strict JSON-like argument schemas. Tool invocation adds $D_{out}(\tau)$ to the agent’s internal typed state and may unlock downstream tools. The library also contains noisy tools: semantically similar distractors that explicitly disclose unreliability in their descriptions, such as deprecated, stale, or non-authoritative variants. Backend data are structured and deterministic; a tool call returns the backend’s value for $Y$0, and answers cannot be inferred without calling tools (Liu et al., 21 Jun 2026).

3. Interaction protocol and formalization

At each step $Y$1, the agent emits exactly one action $Y$2. A retrieval action adds matched tools to the currently callable set. A tool-call executes a retrieved tool with structured arguments. An answer action terminates the episode and triggers scoring. Episodes also terminate when the step budget is exhausted; each model response counts as one step (Liu et al., 21 Jun 2026).

The environment-maintained latent state is formalized as

$Y$3

where $Y$4 is the query, $Y$5 is the set of retrieved callable tools, and $Y$6 is the set of datatypes obtained via successful tool calls. This state is not directly visible to the agent. For each query $Y$7, the benchmark defines an internal state graph

$Y$8

with nodes corresponding to datatype subsets and edges $Y$9 whenever $L^\*$0 and $L^\*$1. A trajectory $L^\*$2 reaches the target if $L^\*$3 (Liu et al., 21 Jun 2026).

The planning objective is to maximize task accuracy under a finite interaction budget:

$L^\*$4

with

$L^\*$5

where $L^\*$6 if the normalized predicted answer contains the normalized gold answer and the target datatypes have been reached, and $L^\*$7 otherwise. Final answers must therefore be grounded in tool-derived target datatypes $L^\*$8; guessing or copying without reaching $L^\*$9 is not credited (Liu et al., 21 Jun 2026).

The benchmark fixes several protocol parameters. Models were run with maximum token length 8192 and no persistent memory beyond the turn-by-turn transcript and the accumulating retrieved set $\{5,6,7,8,9\}$0. The unified inference prompt instructs agents to use retrieve_tools, tool_call, and final_answer actions, to enforce grounded answers, and to plan bi-directionally within retail flows (Liu et al., 21 Jun 2026).

4. Retrieval-limited visibility and blocking

The defining constraint in PlanBench-XL is retrieval-limited tool visibility. At step $\{5,6,7,8,9\}$1, the agent does not observe the full tool library $\{5,6,7,8,9\}$2; it only sees a retrieved subset $\{5,6,7,8,9\}$3, and $\{5,6,7,8,9\}$4 grows only when the agent issues a retrieval action. This creates a partial-observability regime in which an agent must plan under uncertainty about what tools exist, which intermediate datatypes are necessary, and how to recover if retrieved tools prove unhelpful or unavailable (Liu et al., 21 Jun 2026).

The retriever supports three modes. Input-conditioned retrieval performs forward anticipation from available inputs. Output-conditioned retrieval performs backward anticipation from desired outcomes. Input–output-conditioned retrieval constrains both ends simultaneously. Alias-aware mapping from natural-language queries to canonical datatypes uses a sparse hashing encoder over tokens and character $\{5,6,7,8,9\}$5-grams together with cosine-style similarity, after which exact typed matching over tool interfaces is performed (Liu et al., 21 Jun 2026).

A retrieval request returns between $\{5,6,7,8,9\}$6 and $\{5,6,7,8,9\}$7 executable tools, then fills the result list with noisy variants up to $\{5,6,7,8,9\}$8, distributing distractors evenly across matched tools. If no tool satisfies a one-hop typed constraint, the retriever returns “no direct one-step tool exists” with a suggestion to search via intermediate datatypes. In a single-step retrieval sanity check over 185 one-step tasks with Qwen3-8B, the reported values were: GT Tool Retrieved = 92.43%, Accuracy = 85.95%, Avg. Turns = 3.18, S/C = 0.95, ITCR = 0.00%, and EGT Prec. = 96.45%. The paper interprets this as evidence that multi-step failures in the full benchmark are primarily planning and selection issues rather than pure retrieval coverage failures (Liu et al., 21 Jun 2026).

The optional blocking mechanism modifies the environment at retrieval time. Three blocking types are defined. Explicit failure replaces a tool with one that has an identical surface schema but returns an explicit error. Implicit failure replaces a tool with one that returns a value violating its documented behavior or world consistency without an explicit error. Semantically misleading replaces a tool with one that has a related description but different functionality. Blocking is formalized by replacing retrieved tools $\{5,6,7,8,9\}$9 with alternatives $T_{\max}$0 while preserving solvability; candidate blocked sets are selected so that at least one feasible path remains, with constraints such as $T_{\max}$1 and $T_{\max}$2 for $T_{\max}$3, $T_{\max}$4, and $T_{\max}$5 (Liu et al., 21 Jun 2026).

5. Evaluation metrics and empirical results

PlanBench-XL reports metrics at the levels of task completion, exploration behavior, and execution quality. Accuracy is the fraction of tasks whose final answer is correct and datatype-grounded. EGT Precision measures overlap between executed datatypes and process-level ground-truth datatypes. Avg. Turns records total interaction length. Mean Explored Datatypes measures how many datatypes beyond the initial state are explored. Search-to-Call Ratio is the ratio of retrieval turns to tool-call turns. Invalid Tool Call Rate counts structurally invalid calls, and Untrusted Input Rejection Rate counts calls rejected because at least one argument comes from a noisy tool output (Liu et al., 21 Jun 2026).

In the default block-free setting, Gemini-3.1-Pro obtained the highest reported accuracy at 77.06%, with 91.47% EGT Precision, 19.55 turns on average, Mean EDT 27.41, S/C 1.59, ITCR 0.68%, and UIRR 0.30%. DeepSeek-V4-Flash ranked second by accuracy at 63.08%. GPT-5.4 achieved 51.90% accuracy with 72.92% EGT Precision, 22.92 turns, Mean EDT 20.65, S/C 2.70, ITCR 6.28%, and UIRR 1.91%. Gemini-3.5-Flash reached 52.19% accuracy but with 57.87 turns and a high S/C of 10.44. Llama-3.3-70B-Instruct scored 18.96%, while Qwen3-8B and Llama-3.1-8B-Instruct scored 0%; Qwen3-14B, Qwen3-32B, and GPT-5.4-Mini remained below 3.1% (Liu et al., 21 Jun 2026).

Several correlations reported in the paper diagnose what matters for success. Mean EDT and Accuracy have Pearson correlation $T_{\max}$6, indicating that broader datatype exploration tracks strongly with higher performance. More output-conditioned retrieval, corresponding to backward planning from desired outcomes, correlates with higher Accuracy with Pearson $T_{\max}$7. EGT Precision correlates with Accuracy at $T_{\max}$8, whereas ITCR correlates negatively at $T_{\max}$9. At the same time, frequent search alone is not sufficient: Gemini-3.5-Flash searched heavily yet trailed Gemini-3.1-Pro in both Accuracy and effective exploration (Liu et al., 21 Jun 2026).

Blocked settings reveal a sharper weakness. Increasing block ratio from 0.2 to 0.8 while preserving one feasible path degrades performance markedly; GPT-5.4 loses more than 20 points as viable-path count shrinks and falls to around 30% when only one feasible path remains. Silent failures are the most harmful: UIRR is highest under implicit failures at 11.99%, compared with 9.67% for explicit failures and 9.89% for semantically misleading tools. When only the longest feasible path is preserved, accuracy declines sharply across models; GPT-5.4 falls to 11.36% under the most severe blocking condition. Enforced exploration after incorrect final answers yields limited gains, typically below 5 points, and the decline with minimal path length $\Lambda_{ret}^{cap}$0 becomes steeper under blocking (Liu et al., 21 Jun 2026).

6. Failure modes, interpretation, and reproducibility

PlanBench-XL includes a detailed failure taxonomy: No Traction, Irrecoverable Drift, Weak Recovery, and Format Error. Irrecoverable Drift is dominant. For GPT-5.4 and Gemini-3.5-Flash, it accounts for 72.4% and 71.3% of failures, respectively, while Weak Recovery is rare at approximately 3% in aggregate. The benchmark therefore emphasizes not merely whether an agent fails, but whether it can recover once a trajectory deviates from a progress-making path (Liu et al., 21 Jun 2026).

The analysis indicates that many failures are not caused by inability to retrieve relevant tools. Before a non-progress call, a valid progress-making tool had already been retrieved in 78.0% of default-setting cases and 71.1% of blocked-setting cases. Non-progress calls disproportionately target recently retrieved tools: 74.1% in default and 63.6% in blocked settings. In 44.7% of default cases and 43.2% of blocked cases, the best next tool had been retrieved more than two retrieval windows earlier. Even when a progress-capable tool reappears later in the context, agents often fail to recover; this reappearance occurs in 42.5% of default cases and 53.4% of blocked cases (Liu et al., 21 Jun 2026).

Blocking exposes distinct error channels. Semantically misleading tools are rarely invoked, at or below 3% across models and zero for GPT-5.4 and Llama-3.3-70B. Explicit failures prevent value reuse but can still trigger wasted search along failed branches. Implicit failures produce value contamination: value-reuse rates after implicit failures are 55.9% for GPT-5.4, 75.5% for Llama-3.3-70B, and 42.2% on average, versus 0% for explicit failures. Model-specific termination fingerprints are stable across settings: GPT-5.4 is dominated by “Surrender,” DeepSeek-V4-Flash and Llama-3.3-70B often end with “wrong tool value,” and Gemini-3.5-Flash is dominated by “Search exhaustion” (Liu et al., 21 Jun 2026).

The paper’s interpretation is that planning over massive tool ecosystems remains difficult even for frontier models. Merely increasing the number of turns or enforcing more exploration does not close the gap. The analyses instead motivate diversity-aware retrieval, failure-aware verification, explicit backtracking and recovery, and blocker-aware training with trajectories containing explicit failures, implicit failures, and misleading tools (Liu et al., 21 Jun 2026).

PlanBench-XL is publicly released with code at https://github.com/JiayuJeff/PlanBench-XL under the MIT License, a dataset at Hugging Face under Creative Commons, and a project page at https://planbench-xl.github.io/. The evaluation harness uses JSON-tagged actions, deterministic decoding with temperature 0.0, per-turn step accounting, datatype-checked final answers, vLLM for open-source models, and vendor APIs for proprietary models. The paper also notes several limitations: a single retail domain, a controlled retriever that is cleaner than production retrieval, and simplified but representative blocking. This suggests that PlanBench-XL is best understood as a diagnostic testbed for adaptive planning under partial observability and unreliable tool access, rather than as an exhaustive model of deployed enterprise agent ecosystems (Liu et al., 21 Jun 2026).