UserBench: Evaluating User-Centric LLM Agents

• UserBench is a user-centric benchmark framework that evaluates LLM-based agents in multi-turn, preference-driven travel scenarios with evolving user goals.
• It employs a modular Gymnasium environment to simulate realistic travel requests, revealing latent user preferences through iterative clarifying dialogue.
• Evaluation metrics such as final score, best exist rate, and preference elicitation highlight challenges in achieving effective user-alignment with LLM agents.

UserBench denotes a user-centric benchmark framework explicitly developed to evaluate language-model-based agents in interactive, multi-turn, preference-driven environments where user intent is initially underspecified, preferences are revealed incrementally, and communication is often indirect (Qian et al., 29 Jul 2025). The environment is designed to quantify not just task completion but an agent’s effectiveness at uncovering, clarifying, and acting on complex, evolving user goals. UserBench facilitates rigorous comparison across leading open- and closed-source LLMs in simulated travel-planning tasks, providing detailed multi-aspect evaluation to gauge both technical performance and user-alignment capabilities.

1. Motivation and Context

UserBench addresses the fundamental limitations in conventional agent evaluation where the focus is on task or tool execution, often under conditions that assume fully specified, unambiguous instructions. In realistic deployment settings, users frequently supply incomplete or evolving requests and reveal personal constraints gradually. This benchmark is constructed to expose these complexities, emphasizing three core phenomena:

• Underspecified initial user goals.
• Incremental and implicit preference revelation through multi-turn dialogue.
• Indirect or ambiguous user communication.

The overarching objective is to advance research in proactive, user-aligned agent development by operationalizing a reproducible, extensible testbed modeling these user interaction characteristics (Qian et al., 29 Jul 2025).

2. Environment Design and Implementation

UserBench utilizes the Gymnasium framework to implement a modular and repeatable environment for interactive evaluation. Each data point is a multi-aspect travel scenario composed of:

• A vague initial travel request (e.g., “I’d like to plan a short vacation”).
• A latent set of user preferences across multiple aspects (flight, hotel, car rental, restaurant, or apartment).
• A pre-generated database, in which candidate options for each aspect are exhaustively tagged (“best”, “correct”, “wrong”, or “noise”) based on hidden user constraints and scenario affordances.

UserBench includes three stratified difficulty tiers (Easy, Medium, Hard), driven by the number and entanglement of latent preferences per scenario. Simulated users (implemented via GPT-4o) commence with only basic trip information, incrementally disclosing additional preferences either in response to the agent’s clarifying queries or via a rule-based automatic mechanism after continued conversational stasis. The agent’s dialogue actions are categorized and influence whether explicit preference information is provided or withheld. This structure enables repeatable agent evaluation and supports both single-answer and multi-answer recommendation settings.

3. Evaluation Metrics and Scoring Scheme

UserBench implements a set of quantitative, aspect-sensitive evaluation metrics to capture both technical execution and user-alignment performance.

Primary Final Score

Given $n$ travel aspects in a scenario, the normalized final score is defined as:

$$\textrm{Final Score} = \frac{1}{n} \sum_{i=1}^n \textrm{score}(i)$$

where, in the single-choice setting:

• $\textrm{score}(i) = 1.0$ if the agent selects the "best" option,
• $\textrm{score}(i) = 0.8$ for any "correct" option (suboptimal but acceptable under preferences),
• $\textrm{score}(i) = 0.0$ otherwise.

For multi-choice settings, the maximum achieved score among all agent-proposed candidates in an aspect is retained:

$$\textrm{Final Score} = \frac{\max(1.0, 0.8) + \max(0.0,0.8,0.0)}{2} = 0.9$$

Secondary Micro-Averaged Metrics

• Best Exist Rate: Fraction of aspects with the “best” option present in agent’s choices.
• Correct Exist Rate: Fraction of aspects with any “correct” (acceptable) option present.
• Valid Search Attempt (%): Proportion of agent-generated queries that are syntactically valid and correctly structured for the intended tool/database.
• Valid Action Attempt (%): Frequency of targeted, clarifying agent queries categorized as Type “1” (explicit, preference-seeking).
• Preference Elicited (%): Proportion of total scenario preferences uncovered during interaction—partitioned into active elicitation (via agent queries) and passive elicitation (system-facilitated after repeated conversational irrelevance).

4. Empirical Findings and Model Performance

Experimental results indicate severe limitations in current LLM-based agents’ ability to align with nuanced user intent:

• The frequency with which agent outputs fully match all user preferences per scenario averages only 20%.
• Even the most advanced models, such as GPT-4o, actively uncover less than 30% of user preferences through clarifying dialogue.
• Valid search attempt rates are high (>80%), highlighting agent proficiency in forming correctly structured tool/database queries.
• There is a pronounced performance gap between single-answer and multi-answer modalities; forcing unique recommendation selection causes up to a 40% decrease in final scores compared to the less constrained setting.

A summary table:

Metric	State-of-the-art LLMs (avg)
Full Preference Alignment	~20%
Active Preference Elicitation	<30%
Valid Search Attempt	>80%
Performance Drop (Single→Multi-choice)	~40% reduction

Results further reveal that increased difficulty (more entangled or numerous preferences) leads to a marked decrease in alignment and recommendation quality. Efficiency-effectiveness trade-offs are evident: agents optimizing for speed often neglect thorough preference elicitation, resulting in suboptimal user-centric outcomes (Qian et al., 29 Jul 2025).

5. Challenges in User-Centric Agent Design

Key challenges illuminated by UserBench include:

• Preference Elicitation Brittleness: LLM-based agents “guess” at user needs prematurely, often defaulting to acceptable but non-optimal choices due to insufficient probing for latent constraints.
• Dialogue Planning: Merely increasing interaction turns does not guarantee improved alignment; without strategic dialogue planning, additional dialogue may compound redundancy or drift from goal-relevant querying.
• Trade-off Between Efficiency and Effectiveness: There is no automatic gain in user alignment by being more conversational; agents must balance efficiency (minimizing user burden) with the necessity of eliciting substantive, actionable details.
• Context State Tracking: Maintaining memory of elicited constraints and avoiding repetition remains an area for further research.

Potential improvements proposed include the application of reinforcement learning with reward shaping (e.g., turn-based reward decay for delayed clarifications) and more robust state tracking algorithms to support scalable, user-aligned agent planning (Qian et al., 29 Jul 2025).

6. Significance and Research Impact

UserBench introduces a robust, extensible paradigm for evaluating agent models on user-centric axes, focusing on substantive user alignment rather than naively maximizing task accuracy or tool invocation proficiency. The benchmark’s modular design, grounded in the Gymnasium environment, supports adaptation to further domains where user constraints and preferences are emergent, incremental, and only partially specified at the outset.

This framework highlights that, despite advances in tool use and task reasoning, meaningful user-centric collaboration remains a critical unresolved challenge. UserBench thus establishes a reproducible foundation for both quantifying current model limitations and driving future advances in interactive, alignment-sensitive agent architectures (Qian et al., 29 Jul 2025).