PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems

Abstract: LLM agents increasingly operate in large tool ecosystems, where real-world tasks require discovering relevant tools, inferring implicit sub-goals, and adapting to dynamic environments over long horizons. However, existing benchmarks rarely evaluate planning under retrieval-limited tool visibility. To address this gap, we introduce PlanBench-XL, an interactive benchmark of 327 retail tasks over 1,665 tools that tests whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls toward the final goal. PlanBench-XL further features an optional blocking mechanism that simulates real-world unpredictability through missing, failing, or distracting tool functions, forcing agents to detect disrupted paths and adapt at runtime. Experiments on ten leading LLMs show that massive-tool planning remains challenging: while GPT-5.4 achieves 51.90% accuracy in block-free settings, it collapses to 11.36% under the most severe blocking condition. Further analysis shows that agents are especially vulnerable when failures lack explicit error signals or when recovery requires longer alternative tool-use paths. These results establish PlanBench-XL as a testbed for diagnosing agentic planning failures and highlight the need for robust adaptive planning in long-horizon tasks with large, imperfect tool environments.

• The paper introduces PlanBench-XL, a scalable benchmark evaluating LLM agents' long-horizon planning in large, dynamic tool ecosystems.
• It employs bi-directional exploration and retrieval-time blocking to reveal strong links between exploration metrics and task accuracy.
• The study highlights significant performance drops under blocker scenarios, stressing the need for improved adaptive planning and tool verification.

PlanBench-XL: Assessing Adaptive Long-Horizon Planning in Massive Tool Ecosystems

Motivation and Benchmark Design

PlanBench-XL addresses core limitations of prior agentic planning benchmarks by directly evaluating LLM agents in realistic, large-scale tool ecosystems where partial observability, unreliable tools, and retrieval-limited access fundamentally shape performance. It implements an interactive retail environment with 327 distinct queries, 1,665 tools, and complex, multi-turn workflows constructed over 56 typed domain datatypes. Unlike fixed or fully visible tool sets, PlanBench-XL simulates real-world tool-use by requiring iterative tool discovery via retriever-mediated search, bi-directional anticipation (forward from evidence and backward from goals), and adaptation to retrieval-time blocking events.

Figure 1: Overview of PlanBench-XL, illustrating data construction and runtime agent evaluation with bi-directional exploration and retrieval-time blockers.

Benchmarking in this environment is non-trivial: relevant tools must be discovered through high-level NL queries, intermediate sub-goals inferred, and sequential composition executed over long horizons (≈25 interaction turns per task). Tool reliability is systematically perturbed with explicit, implicit, and misleading blockers, testing recovery, failure detection, and robust reasoning. The design enables controlled ablation studies of agent behavior under varying block types, solution space reductions, and enforced exploration prompts.

Evaluation Metrics and Agent Analysis

PlanBench-XL reports holistic process-level metrics spanning task completion (final answer Accuracy, Executed Ground-Truth Datatype Precision), exploration (Mean Explored Datatypes, Search-to-Call Ratio), and execution quality (Invalid Tool Call Rate, Untrusted Input Rejection Rate). The main evaluation results demonstrate substantial variance across leading LLMs. Gemini-3.1-Pro achieves 77.06% accuracy in the default setting, while GPT-5.4 yields 51.90%. The majority of models—including Qwen3 and Llama3 variants—remain below 60% accuracy, and smaller models often fail entirely.

Success correlates strongly with exploration breadth (Mean EDT, $r=0.902$ with accuracy), but frequent retrieval does not guarantee effective discovery; high S/C ratios can indicate inefficient search loops rather than useful state expansion. Bi-directional anticipation is critical: backward exploration frequency ($r=0.800$) is predictive of success, but most models default to forward-only search, neglecting intermediate datatype requirements.

Execution precision (EGT Precision, $r=0.781$ with accuracy) highlights that accurate exploitation of discovered tools is as crucial as broad exploration. Invalid call rate (ITCR) inversely correlates ($r=-0.443$) with final answer correctness, emphasizing that protocol adherence and structural compatibility are foundational for success.

Robustness and Blocking Analysis

PlanBench-XL's retrieval-time blocking mechanism surfaces agentic brittleness under environmental disruptions. Block ratios (proportion of originally feasible path removals) drive sharp accuracy declines. When only one solution path remains, GPT-5.4 accuracy plunges from 51.90% to 11.36%. Implicit failures (silent, plausible but erroneous outputs) are notably more damaging than explicit errors or semantic distractors, as agents continue propagation, contaminating downstream planning.

Figure 2: Accuracy under retrieval-time blocking, emphasizing vulnerability to implicit failures and stronger block ratios.

Agents consistently struggle to re-plan via longer solution chains when direct paths become unavailable. In path-restricted settings preserving only the longest valid sequence, accuracy falls steeply (often to just over 10%), even for top models. This denotes a pronounced deficiency in adaptive planning depth.

Figure 3: Accuracy across blocked solution paths, showing performance degradation when only the longest path is preserved.

Enforced exploration—additional test-time prompts after wrong terminations—yields just marginal gains (<5 percentage points), reinforcing that the problem is not merely interaction budget, but insufficient contingency planning and correct branch abandonment.

Figure 4: Effect of enforced exploration under blocking, indicating quickly saturated and limited accuracy improvements.

Longer shortest-path groupings consistently show lower accuracy, and this trend intensifies under blocking scenarios, further quantifying the compounding difficulty of extended plan horizons.

Figure 5: Accuracy decline by shortest path length $L^*$, highlighting increased difficulty in longer-horizon planning and sharper degradation under blocking conditions.

Error Analysis: Failure Modes and Tool-Selection

PlanBench-XL's annotation and trajectory breakdown elucidate where agentic planning collapses. The dominant failure is Irrecoverable Drift: after initial progress, agents diverge from feasible paths and rarely recover, with weak recovery observed in only 3% of cases. Tool-selection, rather than retrieval failure, is frequently responsible—agents overlook already retrieved, progress-capable tools (78% of non-progress steps had at least one viable candidate in history), and display pronounced recency bias regardless of actual utility.

Figure 6: Distribution of trajectory failure modes under the default setting, confirming that most failures follow partial progress and drift.

Blocking further stress-tests these weaknesses. While semantic distractors are rarely trusted, explicit failures prevent value propagation yet do not necessarily inhibit search-based follow-up. Implicit failures, by contrast, induce direct value reuse in later tool calls (aggregated to 42.2% of implicit-failure follow-ups), deeply corrupting the solution trajectory.

Figure 7: Invoked block alternatives under mixed blocking, with explicit, implicit, and semantic misleading tool selections.

Figure 8: Follow-up agent behavior after invoking block alternatives, showing high rates of value reuse for implicit failures.

Distinct termination fingerprints emerge across models post-drift. GPT-5.4 surrenders, DeepSeek and Llama commit unsupported tool values, and Gemini-3.5-Flash exhausts search, consistently across default and block modes.

Implications and Future Directions

PlanBench-XL exposes critical gaps in LLM agent robustness, adaptive planning, and tool-use reliability at scale. Current models lack effective mechanisms for detecting subtle failures, retaining relevant tool candidates for re-planning, and efficiently combining forward/backward search. Long-horizon planning with unreliable tool access remains unsolved even among SOTA agents, as evidenced by steep accuracy drops in block settings and ineffective simple test-time scaling.

Practical implications extend to RL training with blocker-aware trajectories, diversity-aware exploration incentives, integrated tool verification phases, and explicit backtracking mechanisms—each essential for deployment in dynamic, imperfect tool environments. The benchmark signifies the need for progress in agentic meta-reasoning, contingency detection, and iterative recovery, particularly as real-world tool ecosystems grow in complexity and scale.

Theoretically, PlanBench-XL motivates new research in latent sub-goal inference, bi-directional NL retrieval, epistemic uncertainty calibration under noisy tool responses, and large-scale stochastic planning analysis. Its domain-general construction allows extension into broader domains, supporting coverage across enterprise, finance, healthcare, and software engineering.

Conclusion

PlanBench-XL provides a principled, scalable framework for dissecting LLM agent performance in massive, retrieval-mediated tool ecosystems. It demonstrates that adaptive planning, robust tool verification, and coherent exploitation are not guaranteed in current architectures, especially under adversarial blocking. The benchmark offers actionable diagnostics and training signals for the next generation of agents, setting a rigorous standard for dynamic tool-use competency and long-horizon reasoning (2606.22388).