TravelPlanner Benchmark
- TravelPlanner is a benchmark family for language-agent travel planning that transforms natural language trip requests into structured itineraries under multiple constraints.
- It features various interaction settings—sole-planning, two-stage, and multi-turn—enabling analysis of tool use, modular planning, and constraint adaptation.
- Empirical studies show that explicit modular architectures and structured prompts significantly boost performance metrics like Final Pass Rate in itinerary validation.
TravelPlanner is a benchmark family for real-world travel planning with language agents in which a model must transform a natural-language trip request into a structured, executable itinerary while satisfying intertwined budgetary, temporal, spatial, accommodation, dining, and commonsense constraints. In the literature it is simultaneously treated as a benchmark for static single-turn planning, a testbed for tool-using long-horizon agents, and a substrate for later multi-turn and priority-aware extensions, which has made it a central reference point for research on constraint-aware language-agent planning (Oh et al., 5 Jun 2025, Liu et al., 14 Feb 2026, Lee et al., 17 Jan 2025).
1. Origin, scope, and research context
TravelPlanner is repeatedly described as a benchmark for “real-world planning with language agents” and as a natural language planning benchmark centered on multi-day travel itinerary construction under user-defined constraints (Oh et al., 5 Jun 2025, Lee et al., 17 Jan 2025). In its widely cited form, the benchmark contains 45 training tasks, 180 validation tasks, and 1000 test tasks, with trip durations of 3, 5, or 7 days and difficulty levels labeled Easy, Medium, and Hard (Lee et al., 17 Jan 2025). The task family has therefore served both as an evaluation environment for prompt-based planning and as a controlled laboratory for studying long-horizon tool use, multi-agent coordination, and reinforcement learning.
TravelPlanner sits within a broader lineage of trip-planning research that predates LLM agents. Earlier work on extensible and personalizable multi-modal trip planning formulated route selection as shortest-path search over a transportation network augmented with Linear Temporal Logic constraints and a preferential cost function, with explicit support for walking, biking, driving, public transit, and taxi, as well as user-uploaded auxiliary geographic data such as crime rates (Liu et al., 2019). This suggests a continuity between classical constrained trip planning and TravelPlanner-era language-agent systems: both target realistic travel decisions, but TravelPlanner moves the burden of interpretation, decomposition, and itinerary synthesis into a language-based planning loop.
Subsequent papers have broadened the meaning of “TravelPlanner” from a single benchmark into a methodological arena. Some works treat it primarily as a sole-planning benchmark in which all relevant information is provided in the prompt, whereas others emphasize tool-mediated search, multi-agent execution, multi-turn adaptation, or live-information settings (Liu et al., 14 Feb 2026). As a result, the term now denotes both the original benchmark and a family of derived problem settings used to probe different failure modes of planning agents.
2. Task structure and itinerary representation
In its original single-shot form, TravelPlanner takes a user request such as “Plan a 3-day trip from city A to city B for people,” together with hard constraints such as budget, room type, room rules, and cuisine, plus a reference database of flights, accommodations, attractions, and restaurants. The required output is a structured itinerary, usually organized day by day, with entries for current city, transportation between cities, breakfast, lunch, dinner, attractions, and accommodation (Oh et al., 5 Jun 2025). Later work preserves this basic structure and often renders it as a JSON-like array of day objects with fields such as "days", "current_city", "transportation", "breakfast", "attraction", "lunch", "dinner", and "accommodation" (Oh et al., 5 Jun 2025).
Across papers, the natural-language query typically specifies origin and destination city or cities, dates or duration, number of travelers, budget, and optional preferences or prohibitions involving accommodation, transportation, restaurants, and attractions (Lee et al., 17 Jan 2025). In some formulations this remains a pure planning problem over a provided reference context; in others it becomes a reasoning-plus-acting problem in which the system must first gather facts through tools and only then synthesize an itinerary.
A commonly used tool-using formulation exposes six sandbox tools: SearchFlight(departure, destination, date), GoogleDistanceMatrix(departure, destination, mode), SearchCity(state), SearchRestaurant(city), SearchAttraction(city), and SearchAccommodation(city) (Wu et al., 23 Mar 2026). These tools define the operational boundary of the benchmark in its agentic form: valid plans must be grounded in entities returned by the sandbox, and successful behavior depends not only on itinerary construction but also on correct tool choice, argument construction, and interpretation of results.
3. Constraint taxonomy and evaluation protocol
TravelPlanner evaluations are commonly decomposed into Delivery Rate, commonsense constraint pass rate, hard constraint pass rate, and Final Pass Rate, with commonsense and hard constraint scores usually reported in both Micro and Macro forms (Liu et al., 14 Feb 2026). Delivery Rate measures whether a system returns a deliverable final plan; the pass-rate metrics assess constraint satisfaction; and Final Pass Rate is the strict success criterion requiring the plan to satisfy both commonsense and hard constraints under the benchmark’s evaluation logic.
In one widely used formalization, commonsense constraints include within sandbox, complete information, within current city, reasonable city route, diverse restaurants, diverse attractions, non-conflicting transportation, and minimum nights stay. Hard constraints include budget, room rules, room type, cuisine, and transportation preferences (Wu et al., 23 Mar 2026). This partition matters because hard constraints reflect explicit user requirements, while commonsense constraints encode the structural plausibility of the itinerary and the integrity of grounding in the benchmark environment.
Prompt-optimization work on sole-planning uses the same metric suite and reports that the GPT-4 Turbo baseline reaches a Final Pass Rate of 5.55, while a manually modified prompt reaches 12.78; the same study reports Commonsense Pass Rate, Hard Constraint Pass Rate, and Delivery Rate in the standard TravelPlanner format (Miin et al., 2024). The persistence of this metric structure across later papers has made TravelPlanner unusually amenable to cross-method analysis, even when the underlying interaction protocol differs.
4. Interaction settings and benchmark evolution
TravelPlanner now appears in several distinct interaction regimes. The core distinction is between a setting where all relevant information is already present and a setting where the agent must actively collect it.
| Setting | Core property | Representative description |
|---|---|---|
| Sole-Planning | No tool use | All necessary information is given in the prompt |
| Two-Stage | Tool-based planning | The agent uses six sandbox tools, then formulates a complete travel plan |
| Flex-TravelPlanner | Multi-turn planning | Constraints are introduced sequentially or with explicit priorities |
In the formulation emphasized by MASFly, TravelPlanner has two evaluation modes. In Two-Stage mode, the model must interact with a sandbox environment through six tools and then formulate a complete travel plan; in Sole-Planning mode, all relevant information is already present in the prompt, so the task reduces to planning and constraint satisfaction over provided information (Liu et al., 14 Feb 2026). This distinction is consequential because many methods that perform well when all information is given degrade sharply when they must also orchestrate retrieval.
Flex-TravelPlanner extends the original benchmark by converting it from a static, single-turn task into a multi-turn, constraint-evolving benchmark. It introduces two new settings: sequential constraint introduction across multiple turns and scenarios with explicitly prioritized competing constraints. The sequential setting is built from 120 validation queries and compares 1-turn, 2-turn, and 3-turn variants with identical final constraint sets but different introduction orders; the priority-aware setting contains 134 two-turn scenarios in which budget conflicts with later-added cuisine or rating preferences (Oh et al., 5 Jun 2025). This extension reframes TravelPlanner from a one-shot itinerary generator into a benchmark for plan revision and constraint hierarchy handling.
A further evolution places TravelPlanner in live-information and multi-turn feedback loops. ATLAS reports an 84% final pass rate in a realistic live-search setting with multi-turn feedback, compared with 59% for ReAct and 27% for a monolithic agent, thereby demonstrating that TravelPlanner-style reasoning can be transplanted from static sandbox conditions into interactive, dynamically updated search environments (Choi et al., 29 Sep 2025).
5. Methodological landscape and reported performance
TravelPlanner has become a proving ground for several distinct methodological traditions. One line treats the benchmark as a test of inference-time search. “Mind Evolution” frames TravelPlanner as a natural language planning task solved by evolutionary search over candidate itineraries, reporting 95.6% success on the validation set and 95.2% on the 1000-instance test set with Gemini 1.5 Flash, and 100% on validation plus 99.9% on test in a two-stage Flash-plus-Pro configuration, notably without a formal solver (Lee et al., 17 Jan 2025).
A second line develops dynamic or automatically generated multi-agent systems. MASFly constructs query-specific teams through retrieval-augmented SOP instantiation and supervises execution with a Watcher agent and a Personalized Experience Pool; it reports a 39.4% Final Pass Rate in the harder Two-Stage mode and 61.7% in Sole-Planning mode, using 20 TravelPlanner training instances to build its SOP repository and 180 test instances for evaluation (Liu et al., 14 Feb 2026). SwarmAgentic instead performs language-based swarm optimization over candidate agentic systems and reports a Final Pass Rate of 32.2 with GPT-4o on TravelPlanner, which it characterizes as a +261.8% relative improvement over ADAS in that setting (Zhang et al., 18 Jun 2025).
A third line emphasizes structural decomposition of the planning problem. Behavior Forest decomposes travel planning into parallel behavior trees for transportation, accommodation, dining, and attractions, coordinated by a global mechanism that enforces cross-task constraints. It reports a Final Pass Rate of 91.67 in the two-stage TravelPlanner setting and 94.44 in the sole-planning setting, together with a 6.67 percentage-point improvement over LLM-rwplanning in the two-stage configuration (Yuan et al., 23 Apr 2026). HiMAP-Travel adopts a different decomposition, splitting strategic coordination from day-level execution and adding a transactional monitor plus bargaining protocol; with Qwen3-8B it reports 52.78% validation and 52.65% test Final Pass Rate, as well as a +8.67 pp gain over a sequential DeepTravel baseline under a controlled same-model comparison (Bui et al., 5 Mar 2026).
A fourth line moves toward explicit program synthesis or solver construction. SCOPE separates query-specific reasoning from reusable code execution and reports 93.1% success on TravelPlanner with GPT-4o, described as a 61.6% gain over the best CoT baseline while cutting inference cost by 1.4x and time by approximately 4.67x (Deik et al., 14 Jan 2026). TriFlow combines retrieval, planning, and governance in a progressive multi-agent pipeline and reports 91.1% final pass on TravelPlanner together with over 10x runtime efficiency improvement compared to current SOTA (Chen et al., 12 Dec 2025).
A fifth line studies direct policy optimization. A comprehensive RL study using TravelPlanner as a long-horizon tool-using environment reports that a 7B Qwen2.5-Instruct model trained with the proposed recipe reaches 62.8% success, after examining reward shaping, model scaling, data composition, algorithm selection, and environmental stability (Wu et al., 23 Mar 2026). Reported numbers across these papers are setting-dependent—sole-planning, two-stage, static sandbox, live search, validation, and test protocols differ—so direct leaderboard-style comparison requires care. This suggests that TravelPlanner is less a single scorecard than a common substrate on which multiple agent-design questions can be isolated.
6. Cross-paper findings, limitations, and significance
A recurrent finding is that static one-shot success does not reliably predict adaptive planning ability. Flex-TravelPlanner shows that performance on single-turn tasks poorly predicts the ability to revise plans across multiple turns, that constraint introduction order significantly affects outcomes, and that models often violate budget to satisfy newly introduced lower-priority preferences: GPT-4o does so in 59% of such cases and LLaMA 3.1 70B in 61.9% (Oh et al., 5 Jun 2025). TravelPlanner therefore exposes not only whether a model can construct an itinerary, but whether it can preserve global commitments under changing requirements.
Another recurrent finding is that explicit structure improves reliability. Behavior Forest separates local and global constraints; SCOPE separates reasoning from execution; and TriFlow imposes a progressive retrieval–planning–governance pipeline with feasibility-first refinement (Yuan et al., 23 Apr 2026, Deik et al., 14 Jan 2026, Chen et al., 12 Dec 2025). A plausible implication is that TravelPlanner’s difficulty does not stem solely from lack of raw reasoning capacity, but from the mismatch between unconstrained next-token generation and the algebra of coupled itinerary constraints.
Empirical analyses also show that information management and supervision matter independently of model scale. In a TravelPlanner MAS study, adding a notebook reduces errors due to hallucinated details by 18%, and adding an orchestrator reduces errors by up to 13.5% within focused sub-areas; combining both mechanisms yields a 25% Final Pass Rate, a 17.5% absolute improvement over a single-agent baseline’s 7.5% (Ou et al., 18 Aug 2025). ATLAS similarly treats hallucination and hidden-constraint violation as products of inadequate search, missing constraint construction, and insufficient critique loops rather than of planning alone (Choi et al., 29 Sep 2025).
TravelPlanner has also revealed limits of prompt-only optimization. A semi-automated prompt-generation study reports that an initial GPT-4o-generated prompt attains a Final Pass Rate of 2.78, a human-refined one-step iteration reaches 6.67, and a fully manually modified prompt reaches 12.78 (Miin et al., 2024). This suggests that better prompts and more structured reference data help, but benchmark performance ultimately depends on deeper mechanisms for grounding, checking, and revising plans.
As a research object, TravelPlanner has consequently become more than a benchmark for “can an LLM write an itinerary.” It is now used to study dynamic constraint management, tool orchestration, long-horizon RL, multi-agent specialization, program synthesis, notebook-based information sharing, bargaining under infeasibility, and the relation between local decisions and global plan validity. Its continuing influence derives from that combination of applied realism and formal evaluability: the task is concrete enough to resemble genuine travel planning, yet structured enough to support reproducible analysis of why planning agents fail and how they can be improved.