Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning (2411.13904v1)

Abstract: How are LLM-based agents used in the future? While many of the existing work on agents has focused on improving the performance of a specific family of objective and challenging tasks, in this work, we take a different perspective by thinking about full delegation: agents take over humans' routine decision-making processes and are trusted by humans to find solutions that fit people's personalized needs and are adaptive to ever-changing context. In order to achieve such a goal, the behavior of the agents, i.e., agentic behaviors, should be evaluated not only on their achievements (i.e., outcome evaluation), but also how they achieved that (i.e., procedure evaluation). For this, we propose APEC Agent Constitution, a list of criteria that an agent should follow for good agentic behaviors, including Accuracy, Proactivity, Efficiency and Credibility. To verify whether APEC aligns with human preferences, we develop APEC-Travel, a travel planning agent that proactively extracts hidden personalized needs via multi-round dialog with travelers. APEC-Travel is constructed purely from synthetic data generated by Llama3.1-405B-Instruct with a diverse set of travelers' persona to simulate rich distribution of dialogs. Iteratively fine-tuned to follow APEC Agent Constitution, APEC-Travel surpasses baselines by 20.7% on rule-based metrics and 9.1% on LLM-as-a-Judge scores across the constitution axes.

• The paper presents TTG, a novel system that translates natural language travel queries into symbolic representations for optimal itinerary planning.
• It leverages a fine-tuned LLM with ~91–92% accuracy and a MILP-based solver that computes near-optimal solutions in about 5 seconds per request.
• User evaluations showed high satisfaction with NPS scores between 35 and 40, highlighting the system’s practical impact on automated travel planning.

Review of "To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning"

The paper "To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning" presents a novel system addressing the complexities inherent in travel planning through a hybrid model that combines LLMs with Mixed Integer Linear Programming (MILP) solvers. This approach allows for the translation of natural language travel requests into optimal travel itineraries, responding to user demands in near real-time (~5 seconds per request).

System Design and Architecture

TTG is structured around three main components:

1. Symbolic Travel Generator: This module generates user requests and corresponding travel information in symbolic form, leveraging a synthetic data pipeline. The pipeline uses real-world datasets to generate flight and hotel information without human annotation. This module plays a crucial role in bridging the gap between natural language inputs and formal constraints required by MILP solvers.
2. Instruction Translator: Using a fine-tuned LLM, this component translates natural language requests to symbolic form. A notable achievement is the exact match accuracy of ~91% on a backtranslation metric, highlighting the system's efficacy in maintaining fidelity between user requests and symbolic translations.
3. Travel Solver: At the heart of TTG is the MILP-based solver that ensures the calculated travel itinerary is optimal and feasible within the constraints provided. It uses a symbolic representation of the request to solve the underlying combinatorial optimization problem, ensuring that the response aligns with the user's instructions.

Strong Numerical Results

The paper reports impressive numerical results, demonstrating the efficiency and reliability of TTG. The Translator component achieves an exact match accuracy of 92.0% when constrained decoding is employed. Furthermore, the quality of solutions, even without exact matches, remains high with a mean score of 0.979, indicating near-optimal solutions compared to ground truth.

User Evaluation and Satisfaction

Human evaluation of TTG using online surveys and qualitative interviews indicates robust user satisfaction, with the system scoring consistently high Net Promoter Scores (NPS) between 35 and 40%. This suggests that users perceive the system's outputs as satisfying their travel requests effectively, offering good value and efficiency.

Implications and Future Directions

The theoretical implications of TTG extend to enhancing the practical applications of LLMs by embedding them into structured decision-making systems, bridging the gap between unstructured user inputs and structured outputs. Practically, TTG's results suggest promising developments for personal travel planning, potentially transforming how individuals approach itinerary management by incorporating automated, real-time recommendations.

Looking forward, future developments of TTG could extend to enhancing personalization features and improving model generalizability. The integration of multi-round dialogue systems and further refinement in response accuracy and personalization would enhance user experience and computational efficiency.

Conclusion

The paper successfully presents TTG as an innovative travel planning solution that optimally converges natural language processing and computational optimization technologies. By providing a framework that bridges natural language input with precise, algorithmically guaranteed outcomes, TTG sets a precedent for future research at the intersection of AI-based language systems and practical decision-making tools.