---
title: 'VitaBench: LLM Interactive Benchmark'
url: https://www.emergentmind.com/topics/vitabench
type: topic
---

# VitaBench: LLM Interactive Benchmark

VitaBench is a benchmark for evaluating LLM-based agents on versatile interactive tasks in real-world applications. Introduced around “life-serving” domains such as food delivery, in-store consumption, and online travel services, it was designed to capture the joint difficulty of extensive information processing, large and stateful tool spaces, and dynamic multi-turn user interaction [2509.26490]. In subsequent work, VitaBench became a recurring evaluation target for generalist tool-use agents, context-management methods, and long-horizon personalized assistants, while VitaBench 2.0 extended the original setting from task execution toward long-term personalization and proactive behavior in temporally ordered user interactions [2605.27141].

## 1. Benchmark scope and domain coverage

The original VitaBench defines a real-world interactive benchmark grounded in daily consumer applications. Its three main domains are Delivery, In-store Consumption, and Online Travel Agency (OTA), and it supports both single-scenario and cross-scenario tasks through flexible composition of domain tools and scenarios [2509.26490].

The benchmark contains 66 tools and 400 tasks in total: 100 cross-scenario tasks, which serve as the main benchmark, and 300 single-scenario tasks divided into 100 Delivery tasks, 100 In-store tasks, and 100 OTA tasks. The environment databases are large and heterogeneous. Reported statistics include 1,324 service providers, 6,946 products, and 447 transactions for the cross-scenario environment; 410 providers and 788 products for Delivery; 611 providers and 3,277 products for In-store; and 1,437 providers and 9,693 products for OTA [2509.26490].

The tool space is explicitly stateful and cross-domain composable. Reported tool counts are 20 for Delivery, 24 for In-store, and 38 for OTA, with 27 write tools, 33 read tools, and 6 general tools. Example tools include `address_to_longitude_latitude`, `longitude_latitude_to_distance`, `get_nearby`, `instore_book`, `create_delivery_order`, `pay_delivery_order`, `train_ticket_search`, `create_train_order`, and `weather`. This organization allows tasks that stay within a single operational domain as well as tasks that require domain switching and interleaved workflows across booking, ordering, routing, and payment [2509.26490].

A central design choice is the removal of domain-specific textual policies in favor of tool-graph structure. This makes the benchmark less about rote policy following and more about latent workflow discovery, tool dependency management, and user-facing decision making. Later papers often evaluate only selected subsets—such as Delivery, In-store or Store, OTA or Ota, and Cross or Cross-Scenarios—but these subsets are derived from the broader benchmark structure rather than redefining it [2509.26490].

## 2. Formal task model and complexity framework

VitaBench formalizes each environment as a POMDP. For an environment $e \in \mathcal{E}$, the formulation is
$$
(\mathcal{U},\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{T},r)_e .
$$
Here, $\mathcal{U}$ is the instruction space, $\mathcal{S}$ the state space, $\mathcal{A}$ the action space, $\mathcal{O}$ the observation space, $\mathcal{T}$ the state transition function, and $r$ the final success metric in $[0,1]$ [2509.26490].

The state and observation spaces factor into database and user components:
$$
\mathcal{S} = \mathcal{S}_{\text{db}} \otimes \mathcal{S}_{\text{user}},
\qquad
\mathcal{O} = \mathcal{O}_{\text{db}} \otimes \mathcal{O}_{\text{user}}.
$$
The action space contains both tool invocations and user dialogue actions. Database transitions are deterministic Python-function executions, whereas user transitions are stochastic because the user is simulated by an LLM. A trajectory under policy $\pi_\theta$ is written as
$$
\tau = (s_0, a_1, s_1, a_2, s_2, \ldots, a_T, s_T) \sim \pi_{\theta}(\tau \mid e, u).
$$
This structure makes VitaBench a benchmark for partially observable, interactive decision making rather than isolated function calling [2509.26490].

The benchmark’s explicit task-complexity framework decomposes complexity into three components:
$$
\mathcal{C}_{\text{task}} = \langle \mathcal{C}_{\text{reason}}, \mathcal{C}_{\text{tool}}, \mathcal{C}_{\text{interact}} \rangle .
$$
Reasoning complexity is tied to the entropy of the observation space and the degree of partial observability,
$$
\eta = 1 - \frac{|\mathcal{O}|}{|\mathcal{S}|}.
$$
Tool complexity is modeled by a directed dependency graph
$$
G = (V, E),
$$
with quantities such as edge density
$$
\rho = \frac{|E|}{|V|(|V|-1)}
$$
and the coverage ratio of task-relevant subgraphs. Interaction complexity is driven by persistent user profiles, behavior attributes, and evolving user state, including emotional style, interaction pattern, and dynamic engagement. Together these definitions operationalize why large search spaces alone do not explain task hardness: VitaBench treats uncertainty, dependencies, and user dynamics as co-equal sources of difficulty [2509.26490].

## 3. Task construction and evaluation protocol

Each task environment combines a user profile, a task instruction derived from multiple authentic user requests, structured environment information, and a set of rubrics used for evaluation. Environment information includes service providers, products or services with attributes, historical transactions, and distractors that only partially satisfy constraints. The resulting tasks require temporal reasoning, spatial reasoning, multi-step tool usage, proactive clarification, and long-horizon tracking of evolving intent [2509.26490].

Rubrics are the core evaluation primitive. For each task,
$$
\mathcal{R} = \{ r_1, r_2, \ldots, r_k \},
$$
where each $r_j$ is an atomic requirement such as a reservation time, delivery content, seat type, arrival deadline, or location constraint. Because many tasks admit multiple valid solution paths, VitaBench does not use exact-match trajectory scoring. Instead, it uses a rubric-based sliding window evaluator that splits long trajectories into overlapping windows, maintains a rubric state vector
$$
\mathbf{s} \in \{0, 1\}^k,
$$
and updates rubric satisfaction as the dialogue and tool-use trace unfolds [2509.26490].

The final task score is strict:
$$
\text{score} = \mathbbm{1}\Big[\sum_{j=1}^k s_j = k\Big].
$$
A task is successful only if all rubrics are satisfied at the end. This design supports diverse valid execution paths while still enforcing exact constraint satisfaction. The original paper reports that the baseline evaluator configuration—sliding windows plus rubrics—achieves task accuracy of 95.0%, rubric accuracy of 88.5%, and Cohen’s $\kappa = 0.828$ relative to human labels, whereas methods without rubrics have almost no agreement with human judgment [2509.26490].

The benchmark also standardizes repeated sampling. Each model–task pair is run four times with temperature $0.0$, and results are reported using Avg@4, Pass@4, and Pass$^4$. Avg@4 measures mean success across four runs, Pass@4 measures the probability that at least one of four runs succeeds, and Pass$^4$ measures the probability that all four runs succeed. This trio separates capability ceiling from stability and reveals whether a model solves tasks consistently or only intermittently [2509.26490].

## 4. Empirical difficulty and benchmarked capabilities

VitaBench was constructed to be hard even for frontier models. The original evaluation reports that even the most advanced models achieve only 30% success rate on cross-scenario tasks and less than 50% success rate on other tasks. For the main cross-scenario benchmark, the best reported thinking model is o3 (high) with Avg@4 = 30.0%, Pass@4 = 61.0%, and Pass$^4$ = 6.0%. For single-scenario tasks, the same model reaches Avg@4 = 53.5% on Delivery, 53.5% on In-store, and 37.8% on OTA [2509.26490].

Aggregate performance across all models reveals a clear domain gradient. Reported average performance is 42.1% for In-store, 38.0% for Delivery, 20.7% for OTA, and 16.2% for Cross-scenario. The accompanying environment statistics show that this gradient does not reduce to database size alone. In-store has the largest search space but substantially higher performance than OTA and Cross-scenario, whereas OTA and Cross-scenario combine more reasoning points with higher tool complexity and correspondingly lower scores [2509.26490].

The benchmark is explicitly intended to test six interlocking capabilities: handling extensive information, leveraging diverse tools and resources, temporal and spatial reasoning, proactive clarification of ambiguous instructions, tracking evolving user intent in long dialogues, and interaction robustness under user behavior variation. Example appendix trajectories span 50–100 turns and require reuse of reservation identifiers, synchronization of train arrival times with restaurant bookings and delivery windows, and later reminder generation based on prior decisions [2509.26490].

Error analysis in the original paper divides failures into reasoning, tool use, and interaction categories, with reported proportions of 61.8%, 21.1%, and 7.9%, respectively. This is consistent with the benchmark’s design emphasis: tool invocation matters, but the dominant bottleneck is reasoning over partially observed states, temporal and spatial constraints, and latent user goals. The gap between Pass@4 and Pass$^4$ further shows that models are brittle even when they sometimes discover a valid plan [2509.26490].

## 5. VitaBench in subsequent agent research

After its introduction, VitaBench quickly became a standard downstream benchmark for interactive tool-use agents. In “ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training,” VitaBench is used only for evaluation and never for training. The paper treats it as a strictly OOD benchmark, emphasizing that synthesized training domains are entirely disjoint from the evaluation domains and that the benchmark includes distinct data formats not encountered during training. ScaleEnv uses the cross-domain subset for reasoning generalization and the Delivery, In-store, and OTA domains for domain generalization. Reported zero-shot gains include Qwen3-SE-32B improving Cross from 5.3 to 10.8, Delivery from 27.0 to 31.3, In-store from 22.5 to 34.5, and OTA from 4.5 to 12.5, with the paper highlighting that performance on the most challenging cross-domain subset is doubled. A domain-scaling analysis further reports a monotonic increase in VitaBench Pass@4 as the number of synthetic training domains rises from $N \in \{0,2,4,8,16\}$, and performance has not yet plateaued at $N=16$ [2602.06820].

“EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL” uses VitaBench as one of its conversational benchmarks and reports scores on Deliver, Store, and Ota, together with their mean. Under this setup, Qwen3-4B improves from 7.67 to 11.33 after supervised fine-tuning and then to 16.00 after SFT+RL; Qwen3-8B reaches 18.67 after SFT+RL. The paper attributes the gains to verified stateful MCP environments, topology-aware dependency sampling, calibrated refinement for implicit user intent, and a composite RL reward combining trajectory similarity, state equivalence, and a length penalty [2605.18703].

“AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning” characterizes VitaBench as featuring standardized multi-turn conversations and standardized tool invocation formats, with scenarios such as food delivery and hospitality that require multiple rounds of reasoning and repeated interaction with simulated users. AutoForge-30B-A3B scores 17.5 on VitaBench, compared with 16.0 for its Qwen3-Thinking-30B-A3B backbone. The reported gain is smaller than on some other benchmarks, and the paper explicitly presents VitaBench as the hardest of its in-domain agentic benchmarks [2512.22857].

“U-Fold: Dynamic Intent-Aware Context Folding for User-Centric Agents” uses VitaBench as the primary stress test for long, noisy, user-centric dialogues. The paper argues that U-Fold’s advantages over ReAct and prior folding baselines are most pronounced on VitaBench because conversations are longer, tool outputs are substantially more verbose, and user intent is more complex. Reported results include, for Qwen3-Thinking-30B-A3B on the hard VitaBench variant, In-store performance of 27.0 for U-Fold versus 11.0 for ReAct, and Cross-Scenarios performance of 12.0 versus 4.0. The paper’s ablations further associate Delivery with the need for intent-aware conversation summarization and OTA with the need for dynamic extraction of relevant tool outputs from verbose histories [2601.18285].

## 6. VitaBench 2.0: long-term personalization and proactive behavior

VitaBench 2.0 extends the original benchmark from one-shot interactive task solving to long-term, user-centric interaction. It inherits the toolsets and environment scaffolding of the original benchmark but reorganizes tasks as temporally ordered sequences for individual users, with preferences embedded in fragmented and heterogeneous histories. The central question is no longer only whether an agent can execute explicit requests, but whether it can extract, use, and update implicit user preferences over time and behave proactively when necessary information is missing [2605.27141].

For each user $u$, the benchmark specifies a profile $P_u$, a preference set $\mathcal{N}_u = (n_1,\dots,n_L)$, and a task sequence $\mathcal{T}_u = (t_1,\dots,t_N)$. Interaction histories $\mathcal{H}_i$ are revealed between tasks, and the memory state evolves according to
$$
\mathcal{M}_i = Update(\mathcal{M}_{i-1}, \mathcal{H}_i),
$$
while action generation is conditioned on retrieved memory:
$$
a_t \sim \pi(a_t \mid o_t, \text{Retrieve}(\mathcal{M}_i, q_i)).
$$
The released benchmark includes 56 users and approximately 2.3k preferences, averaging about 40.8 preferences per user. Each user has 10–20 tasks, with an average of 14.6. Preferences can be added, deleted, or modified, and histories mix dialogue with platform behaviors such as search, order, review, and browsing, with significant noise and conflicting signals [2605.27141].

VitaBench 2.0 also introduces dedicated proactive tasks. These tasks are constructed so that conditional preferences cannot be applied correctly without first acquiring missing context. A canonical example is a coffee-ordering task where the user prefers high-caffeine coffee in the morning but low-caffeine coffee in the afternoon, while the initial query omits the meeting time. Correct behavior requires the agent to recognize the ambiguity, ask for the missing variable, and then act. Proactiveness is evaluated through rubric satisfaction rather than a separate metric, but the paper separately reports performance on proactive subsets and shows that proactive behavior is systematically weaker than general personalization [2605.27141].

The benchmark provides three memory settings: Full Context, Agentic Memory, and RAG Memory. Agentic Memory uses an LLM-maintained structured preference summary with a 4096-token cap; RAG Memory indexes chunked interaction logs and retrieves top-$k$ records by embedding similarity. Reported results show that state-of-the-art models remain far from reliable. Under Full Context, the best reported performance is Claude-Opus-4.6 with Avg@4 = 0.503, Pass@4 = 0.664, and Pass$^4$ = 0.337. The paper further reports that memory mechanisms often underperform Full Context, with RAG Memory typically degrading performance more than Agentic Memory, and that proactive-task performance is substantially lower than personalization-task performance across model families [2605.27141].

## 7. Limitations, misconceptions, and research significance

A common simplification is to treat VitaBench as merely another function-calling benchmark. The published specification is broader: the benchmark targets reasoning complexity, tool complexity, and interaction complexity simultaneously, uses stateful executable tools, and evaluates long trajectories with multiple valid solution paths rather than single-call correctness [2509.26490]. Another simplification is to identify the benchmark with the specific domain subsets reported in later papers. In practice, subsequent work frequently uses only parts of the benchmark—such as Delivery, Store/In-store, Ota/OTA, or Cross/Cross-Scenarios—because VitaBench serves both as a full benchmark suite and as a modular evaluation substrate [2602.06820].

The original VitaBench paper explicitly notes that both the user simulator and the evaluator are LLM-based, introducing stochasticity and potential bias, even though human studies report high fidelity and evaluator agreement. It also acknowledges that the benchmark is derived primarily from Chinese platforms and is restricted to life-serving domains rather than finance, healthcare, or enterprise workflows [2509.26490]. U-Fold adds a further limitation from the perspective of context-management research: existing benchmarks, including VitaBench, still fall short of the complexity of real-world multi-session, multi-user systems, even if VitaBench is substantially more realistic than earlier short-context benchmarks [2601.18285].

VitaBench 2.0 inherits these concerns and adds others. Its user profiles, preferences, and histories are synthetic or programmatically generated, and its abstract memory interface covers only a subset of possible memory architectures. The evaluation remains rubric-based and does not directly optimize for user satisfaction or trust. Yet these abstractions are deliberate: they provide controllability, reproducibility, and systematic comparison of memory mechanisms and proactive reasoning strategies [2605.27141].

The benchmark’s significance lies in how consistently it exposes the gap between current tool-use competence and real-world readiness. In the original release, top models remain near 30% on cross-scenario tasks and below 50% on other domains; in VitaBench 2.0, even the best Full Context systems remain around 0.5 Avg@4 with markedly lower performance on proactive tasks. Subsequent work has therefore used VitaBench not only as a leaderboard target, but as a diagnostic instrument for environment synthesis, RL reward design, memory architecture, intent tracking, and long-context agent behavior [2509.26490].

Source: https://www.emergentmind.com/topics/vitabench