TripTailor: Real-World Travel Planning Benchmark

• TripTailor is a personalized travel planning benchmark that converts natural language queries into detailed multi-day itineraries within a real-world sandbox.
• The dataset covers 40 Chinese cities with over 500,000 POIs and around 4,000 real itineraries, integrating attractions, hotels, restaurants, flights, and trains.
• Its evaluation framework rigorously measures feasibility, rationality, and personalization, exposing gaps between current LLM capabilities and human-level planning.

Searching arXiv for TripTailor and closely related travel-planning benchmark papers. TripTailor is a real-world benchmark and dataset for personalized multi-day travel planning that evaluates whether AI agents and LLMs can produce executable, rational, and preference-aligned itineraries under realistic constraints. It is centered on a closed sandbox of travel entities in China, including attractions, hotels, restaurants, trains, and flights, paired with nearly 4,000 real itineraries and corresponding user queries. Unlike earlier benchmarks that emphasize simulated settings or narrow hard-constraint checking, TripTailor is designed to test feasibility, rationality, spatial efficiency, and personalization against real human travel plans (Shen et al., 2 Aug 2025).

1. Definition and scope

TripTailor is defined as a benchmark for personalized travel planning in real-world scenarios, with a large sandbox environment and a multi-dimensional evaluation framework (Shen et al., 2 Aug 2025). Its problem setting is to map an open-ended natural-language request into a day-by-day itinerary that specifies transportation, accommodation, meals, attractions, time slots, and costs, while remaining within the benchmark’s sandbox and satisfying user constraints and preferences. In later work, it is explicitly characterized as a real-world benchmark for executable, personalized multi-day itinerary generation under temporal, spatial, budgetary, and preference constraints (Chen et al., 12 Dec 2025).

The benchmark’s role is diagnostic as much as evaluative. It asks not only whether a generated itinerary obeys explicit rules, but whether it approaches the quality of real itineraries made for real travelers. This focus distinguishes it from settings that optimize only a few formal constraints. A plausible implication is that TripTailor is intended as a benchmark for integrated planning competence rather than isolated reasoning skills.

2. Dataset and sandbox environment

TripTailor builds a closed planning environment over 40 popular tourist cities in China and contains more than 500,000 real POIs together with transportation schedules (Shen et al., 2 Aug 2025). The sandbox includes 5,622 attractions, 89,224 accommodations, 422,120 restaurants, 15,110 flights, and 28,832 trains. Attractions retain fields such as price, latitude, longitude, tag names, reference time, summary, and opening hours; hotels and restaurants include prices, categories, ratings, and coordinates; flights and trains include times, prices, and station or airport metadata.

The benchmark also includes approximately 4,000 real itineraries paired with user queries. The data split consists of 3,145 training itineraries and 703 test itineraries, with the test set evenly divided into Easy and Hard subsets. Easy corresponds to 2–3 day trips, while Hard corresponds to 4–7 day trips. These itineraries are derived from self-guided travel plans from online travel agencies, filtered for comprehensiveness and rating quality, then aligned to sandbox entities and lightly rewritten into coherent narratives without adding external POIs (Shen et al., 2 Aug 2025).

User queries are constructed from each itinerary by extracting duration, budget, hotel category, meal cost range, departure and return timing, and high-level interests such as attraction types, cuisine preferences, and itinerary intensity. This produces first-person requests that preserve the original trip’s structural parameters while abstracting away exact POI names. The benchmark therefore defines a planning problem in which a human-level solution is known to exist inside the provided candidate space.

Component	Scale	Notes
Cities	40	China tourist cities
Attractions	5,622	4A or higher, or mentioned in real itineraries
Hotels	89,224	With price, category, and rating
Restaurants	422,120	With price, category, rating, nearby attractions
Flights	15,110	One week of schedules
Trains	28,832	One week of schedules
Itineraries	~4,000	3,145 train / 703 test

The sandbox formulation is central to later systems. In TriFlow, the global data space is written as $$\mathcal{D} = \{\text{flights}, \text{accommodations}, \text{POIs}, \text{distances}, \text{prices}, \ldots\}$$, and generated itineraries are required to remain within a task-specific subset $\mathcal{D}' \subset \mathcal{D}$ to satisfy the benchmark’s Within Sandbox criterion (Chen et al., 12 Dec 2025).

3. Task formulation and itinerary structure

TripTailor requires a model to transform a query into a structured itinerary over multiple days. In reconstructed form, a user request may be written as

$$r = (\text{cities}, \text{dates}, \text{\#travellers}, \text{budget}, \text{preferences}, \ldots),$$

while the itinerary is

$\pi = \{\pi_1, \pi_2, \dots, \pi_T\},$

with each $\pi_t$ denoting a day-level schedule containing transportation, accommodation, meals, attractions, time slots, and costs (Chen et al., 12 Dec 2025).

The expected output is more than a POI sequence. It must include outbound transportation on the first day, return transportation on the last day, hotel coverage for all nights, and explicit activity scheduling. In related formulations, a feasible daily schedule must be sequential and non-overlapping, end before 22:30, include local transfers of at least 30 minutes between consecutive activities, and avoid idle gaps greater than 1 hour; each full day must contain lunch and dinner in prescribed time windows, and no activities may be scheduled 2 hours before flights or 1 hour before trains (Wang et al., 8 Jan 2026). These details indicate the level of procedural rigor increasingly associated with TripTailor-style planning, even though not all such temporal checks are part of the original benchmark’s rationality score.

The benchmark is intentionally sandboxed: during evaluation, models are typically provided with pre-searched subsets of transportation, attractions, restaurants, and hotels, and all POIs used by the reference plan are guaranteed to be present. This isolates planning quality from external search quality and makes failure modes more attributable to itinerary construction rather than retrieval (Shen et al., 2 Aug 2025).

4. Evaluation framework

TripTailor evaluates plans along three main dimensions: feasibility, rationality, and personalization (Shen et al., 2 Aug 2025). Feasibility measures whether the itinerary is executable within the benchmark’s sandbox and whether it contains complete information. Rationality measures internal consistency and travel quality, including restaurant diversity, attraction diversity, budget compliance, visit-duration appropriateness, and route efficiency. Personalization evaluates whether a generated plan matches user intent as well as, or better than, the paired real itinerary.

In later formalizations, Final Pass Rate is expressed as

$$\text{FPR} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\!\left(\bigwedge_j C_j(\pi_i) = 1\right),$$

where each $C_j(\pi_i)$ is a binary criterion for instance $i$ (Chen et al., 12 Dec 2025). This captures the benchmark’s “all constraints must pass” character. The original metric set includes Within Sandbox and Complete Information for feasibility, and Diverse Restaurants, Reasonable Meal Prices, Diverse Attractions, Appropriate Visit Duration, Defined Budget Limit, and Optimized Route for rationality (Shen et al., 2 Aug 2025).

Route efficiency is quantified by the average distance between consecutive POIs:

$$D_{\text{avg}} = \frac{\sum_{k=1}^{n_d} \left( \frac{\sum_{j=1}^{M_k-1} d_{j,j+1}^k}{M_k - 1} \right)}{n_d}.$$

Here, $n_d$ is the number of days, $\mathcal{D}' \subset \mathcal{D}$0 is the number of POIs on day $\mathcal{D}' \subset \mathcal{D}$1, and $\mathcal{D}' \subset \mathcal{D}$2 is the geographic distance between consecutive POIs on that day (Shen et al., 2 Aug 2025). This metric was introduced because hard-constraint satisfaction alone does not ensure that a route is geographically coherent.

Personalization is assessed with two evaluators. One is an LLM-as-a-judge procedure that compares the generated itinerary directly against the real itinerary on criteria such as experiences, itinerary intensity, cuisine fit, accommodation fit, transportation practicality, and budget versus value tradeoff. The other is a trained reward model $\mathcal{D}' \subset \mathcal{D}$3, optimized with pairwise logistic loss:

$\mathcal{D}' \subset \mathcal{D}$4

where $\mathcal{D}' \subset \mathcal{D}$5 is the preferred itinerary and $\mathcal{D}' \subset \mathcal{D}$6 is a less preferred alternative for query $\mathcal{D}' \subset \mathcal{D}$7 (Shen et al., 2 Aug 2025). The benchmark reports both Individual Preference surpassing rates and a combined Final Surpassing Rate.

5. Baselines, reference workflows, and subsequent systems

The original benchmark evaluates direct prompting, zero-shot chain-of-thought, ReAct, Reflexion, and a workflow-decomposition baseline that manually separates transportation selection, attraction ranking, itinerary drafting, and restaurant and hotel insertion (Shen et al., 2 Aug 2025). The workflow baseline is important because it shows that explicit decomposition can substantially outperform monolithic prompting on rationality and final pass metrics, even when using a smaller backbone.

TripTailor has since become a primary evaluation target for newer trip-planning frameworks. TriFlow introduces a three-stage progressive multi-agent pipeline of retrieval, planning, and governance and reports state-of-the-art performance on both TravelPlanner and TripTailor, with 97.7% Final Pass Rate on TripTailor and 99.5% micro feasibility, 99.1% macro feasibility, 99.4% micro rationality, and 97.7% macro rationality (Chen et al., 12 Dec 2025). Its design emphasizes retrieval-bounded search space reduction, monotonic feasibility during planning, and bounded governance loops capped at eight refinement iterations.

TourPlanner proposes a different architecture based on Personalized Recall and Spatial Optimization, Competitive consensus Chain-of-Thought, and constraint-gated reinforcement learning (Wang et al., 8 Jan 2026). It uses hard-versus-soft reward separation, where

$\mathcal{D}' \subset \mathcal{D}$8

so that preference and route-quality optimization are emphasized only after hard constraints are sufficiently satisfied. This design reflects a broader methodological lesson from TripTailor: travel planning is not well served by undifferentiated optimization over mixed constraints.

TripTailor also fits within a longer line of personalized and constraint-aware trip planning. Earlier work on an extensible and personalizable multi-modal trip planner used linear temporal logic for hard temporal constraints and a preferential cost function

$\mathcal{D}' \subset \mathcal{D}$9

to encode trade-offs among mode-specific travel time, fare, and auxiliary metrics such as crime or pollution (Liu et al., 2019). DeepAltTrip, by contrast, addressed top-$$r = (\text{cities}, \text{dates}, \text{\#travellers}, \text{budget}, \text{preferences}, \ldots),$$0 itinerary recommendation among POIs with explicit attention to diversity across alternatives, using a combined evaluation statistic

$$r = (\text{cities}, \text{dates}, \text{\#travellers}, \text{budget}, \text{preferences}, \ldots),$$1

for relevance–diversity analysis (Rashid et al., 2021). These works are not TripTailor benchmarks, but they illuminate adjacent design dimensions—formal constraints, preference scalarization, and diverse route generation—that TripTailor exposes in a more realistic, large-scale setting.

6. Empirical findings and significance

The central empirical finding of TripTailor is that strong LLMs remain far from human-level travel-planning performance in realistic settings (Shen et al., 2 Aug 2025). The paper reports that fewer than 10% of itineraries generated by the latest state-of-the-art LLMs achieve human-level performance. In the main benchmark table, the workflow baseline reaches a Final Pass Rate of 63.3% and a Final Surpassing Rate of 16.2%, while direct GPT-4o reaches a Final Pass Rate of 21.5% and a Final Surpassing Rate of 7.5%; DeepSeek-V3 reaches 14.4% and 7.8%, and o1-mini reaches 18.3% and 9.4% (Shen et al., 2 Aug 2025).

The benchmark reveals that feasibility is not the sole bottleneck. Many systems maintain high Complete Information scores, and repeated restaurants or attractions are comparatively easy to avoid, yet route efficiency, budget control, visit-duration validity, and meal-price consistency remain weak. For example, the workflow baseline reaches 98.6% on Reasonable Meal Prices, while GPT-4o and DeepSeek-V3 reach 44.0% and 36.3%, respectively (Shen et al., 2 Aug 2025). This suggests that decomposition and explicit structure help on operational details that generic autoregressive generation often mishandles.

Spatial inefficiency is especially prominent. Real plans have average attraction-to-attraction distances of 5.8 km, whereas direct LLM plans reach approximately 15.9–17.2 km, implying route-distance ratios around 5.6–6.1 depending on model (Shen et al., 2 Aug 2025). This is one reason later methods such as TourPlanner explicitly incorporate spatial clustering and route-efficiency terms, and why TriFlow’s planning stage enforces structural constraints such as Within Current City and duration consistency (Wang et al., 8 Jan 2026).

The benchmark therefore functions as evidence that high nominal reasoning ability does not automatically yield high-quality real-world planning. A plausible implication is that TripTailor has shifted research toward modular, verifier-backed, and retrieval-grounded architectures rather than direct one-shot prompting.

7. Limitations, controversies, and research directions

TripTailor is realistic relative to earlier travel-planning benchmarks, but it is not without limitations (Shen et al., 2 Aug 2025). Its geographic scope is restricted to China, so cross-cultural generalization is unresolved. Its user queries are LLM-generated rather than directly collected from real users, even though they are conditioned on real itineraries and filtered for coherence. Personalization evaluation depends on LLM judges and a learned reward model, both of which may introduce bias. The benchmark also focuses on single-turn planning rather than iterative trip design with clarification and revision.

A recurring misconception is that passing formal feasibility checks is equivalent to producing a good itinerary. TripTailor’s route-distance metrics and personalization comparisons were introduced precisely because this is false: a plan can satisfy budget and non-repetition constraints while still being geographically scattered, generic, or poorly aligned with stated interests (Shen et al., 2 Aug 2025). Another misconception is that the benchmark is fundamentally about retrieval. In the canonical evaluation setup, relevant POIs are pre-supplied so that the task isolates itinerary synthesis rather than search.

Future directions suggested across the surrounding literature converge on several themes. One is stronger feasibility-first planning, as in TriFlow’s retrieval–planning–governance pipeline (Chen et al., 12 Dec 2025). Another is multi-path reasoning and competitive consensus, as in TourPlanner’s persona-based deliberation and constraint-gated RL (Wang et al., 8 Jan 2026). A third is richer personalization, including better reward models and more faithful preference representations. More broadly, TripTailor appears to be pushing the field toward modular, verifiable, and preference-sensitive planning systems that combine structured validation with flexible language-based reasoning rather than relying on monolithic prompting alone.