VCWorld: Executable Benchmark for Cockpit Control
- VCWorld is a simulated, executable automotive cockpit environment that abstracts in-car systems into 30 device classes with real-time state management.
- It implements a state-based function call framework that leverages a shared global state to improve device selection and operational accuracy.
- The benchmark evaluates 1291 tasks across various interaction complexities to compare conventional function calling with state-based state transitions.
VCWorld, in the automotive agent literature, denotes VehicleWorld, a simulated but executable intelligent vehicle cockpit environment for studying how LLM agents control tightly coupled in-car systems. The paper does not distinguish VCWorld from VehicleWorld. It is presented as the first comprehensive environment for the automotive domain with 30 device/module classes, 250 APIs, 680 attributes/properties, real-time state information during agent execution, 302 initialization scenarios, and a 1291-task benchmark spanning single-turn, multi-turn, single-intent, and multi-intent interactions (Yang et al., 8 Sep 2025).
1. Scope and problem setting
VehicleWorld is motivated by the claim that intelligent vehicle cockpits are not well modeled by conventional function-calling environments. The environment targets a setting in which navigation, media, climate, lighting, doors, calls, touch control, and diagnostics are not isolated applications but tightly coupled subsystems with shared resources and continuously changing state. In that setting, stateless tool-use protocols require multiple exploratory calls to recover environmental context before they can act, which the paper identifies as a source of inefficiency and weak error recovery (Yang et al., 8 Sep 2025).
The environment abstracts cockpit functionality into four domains: Multimedia, Touch Control, Car Control, and Lighting. Examples named in the paper include Navigation, Video, AirConditioner, Door, Conversation, and a global Environment manager. Each device is represented as a module class with executable APIs and explicit internal properties. This design makes the benchmark less a conventional API-selection problem than a controlled study of world-state inference and state transition in a multi-device setting.
A terminological caution is necessary. In adjacent literature, the label VCWorld is also used for an unrelated biological world model for virtual cell simulation, where it denotes a cell-level white-box simulator for perturbation-response prediction rather than an automotive cockpit environment (Wei et al., 29 Nov 2025). In the vehicle-interaction literature, however, VCWorld refers to VehicleWorld.
2. Environment architecture and global state model
The architectural core of VehicleWorld is its treatment of cockpit subsystems as globally coupled rather than locally autonomous. The paper emphasizes that the environment is “highly integrated” because modules interact through shared global state and resource contention. The canonical example is the audio channel, which may be occupied by music, radio, or navigation prompts, such that only one device can use it at a time. Other shared properties include volume, temperature, sound channel, speaker, unit system, and time format (Yang et al., 8 Sep 2025).
To maintain consistency across subsystems, VehicleWorld defines a global static Environment class using a Singleton pattern. Devices access shared properties through this central interface rather than storing private copies. The Environment class is described as providing standardized access methods and concurrency control. This design choice is central: it makes failures of local reasoning visible at the world level, because an API call that is locally valid may still conflict with global state or another subsystem.
Each top-level device class may include nested inner classes for subcomponents, and every class exposes a to_dict() method that recursively serializes current state into structured JSON. The world state therefore has two layers. Device-level state includes fields such as whether navigation is active, current destination and midway route, whether a video is playing, playback quality, whether the air conditioner is on, and whether a door is locked or open. Global state includes variables such as volume, sound_channel, temperature, and speaker. The appendix further notes that each JSON attribute is annotated with value, type, and description to make the state more legible to the agent.
This architecture implies that VehicleWorld is not merely a collection of callable methods. It is an executable state machine over a shared cockpit world, with JSON serialization as the interface through which models observe that world.
3. Executability, scenarios, and benchmark construction
VehicleWorld is fully executable rather than symbolic. Every API is implemented as code that updates internal attributes, validates parameters, and returns structured outputs. The environment includes init() methods for all 30 device classes, and combinations of these methods are used to generate 302 initialization scenarios. The benchmark tasks themselves contain executable <inits> and <api_call> sections, and both are run locally to validate correctness (Yang et al., 8 Sep 2025).
Benchmark construction is semi-automatic. The authors first choose devices, APIs, and initialization methods based on real-world use cases, then use Claude 3.7 Sonnet to generate scenario templates with <scenario>, <inits>, <query>, and <api_call> tags. Automatic execution validates that the initialization code and reference action sequence are executable. After that, domain experts manually inspect each scenario for executability, semantic alignment between the query and the API call, and whether the query produces a meaningful state change.
The final benchmark contains 1291 tasks. These are grouped by domain and by interaction complexity using four labels: S-S, S-M, M-S, and M-M, where the first position indicates single-turn vs. multi-turn and the second indicates single-intent vs. multi-intent. The distribution reported in the paper is as follows: Multimedia has 59 S-S, 96 S-M, 251 M-S, and 53 M-M tasks; Touch Control has 34, 93, 107, and 15; Car Control has 135, 178, 205, and 173; and Light has 79, 75, 159, and 161. Car Control is the largest subset.
The benchmark statistics are meant to show that tasks are not degenerate. Average involved devices per task are approximately two in each domain: 2.03 for Touch Control, 2.02 for Light, 2.11 for Multimedia, and 2.06 for Car Control. Average unique APIs per task are 2.84, 3.54, 3.18, and 3.47, and average API calls are 3.07, 3.86, 3.34, and 3.73 respectively. The most complex tasks involve up to 5 devices, 13 API calls, and 12 unique APIs.
4. Function calling and the state-based reformulation
The paper’s central methodological claim is that conventional Function Calling (FC) is an ill-suited abstraction for tightly coupled cockpit control. FC is formalized as
where is the user query at step and is the set of available APIs (Yang et al., 8 Sep 2025).
In this formulation, the agent receives a query, searches available modules and APIs, and emits an API call with parameters. In the authors’ setup, this often begins with search_module and search_api. The paper argues that such behavior is fundamentally stateless and exploratory. Because the model lacks explicit world-state access at decision time, it often needs extra interaction rounds to infer which modules exist, what their current mode is, and which side effects matter globally.
The proposed alternative is State-based Function Call (SFC), formalized as
where is the current system state in JSON format and is the target state. This reframes control from “predict the next function call” to “predict the target world state, then generate transition code that realizes it.”
SFC is implemented as a two-stage process. In stage 1, the model receives the complete JSON state of all devices plus the user query and selects the relevant devices. In stage 2, it receives only the state JSON of the selected devices and predicts the target state together with the transition code. The paper explicitly notes that the two stages “operate independently without shared context.” Execution is then performed directly on the simulator through Python code that updates state variables or attributes, rather than through a full API-only plan.
This distinction is consequential. Pure SFC does not primarily search for APIs; it manipulates an explicit, serialized world state. The environment then returns execution logs, exceptions, and the updated state, which becomes the basis for subsequent turns. The authors therefore treat SFC as a more faithful abstraction for cockpit interaction because it externalizes persistent system state rather than leaving it implicit in the model’s hidden textual context.
5. Evaluation methodology and empirical findings
VehicleWorld is evaluated with state-based rather than action-sequence-based metrics. The three main metrics are F1 positive, F1 negative, and Accuracy (Acc), which respectively assess whether attributes that should change do change, whether attributes that should remain unchanged do remain unchanged, and whether final values are correct (Yang et al., 8 Sep 2025). For ambiguous numerical instructions such as “increase the volume,” the paper judges trend correctness—increase, decrease, or maintain—rather than exact value equality.
The paper also compares state-based evaluation with rule-based evaluation, where the latter checks exact API names, parameter keys, parameter values, and call counts. On 200 FC outputs from Qwen2.5 models, state-based evaluation is reported to have lower error rate relative to expert judgment, because it evaluates practical task completion rather than exact reproduction of a reference API trace.
The main empirical comparison is FC versus SFC under ReAct prompting across eight representative models: Claude-3.7-Sonnet, GPT-4o, DeepSeek-v3-250324, Qwen2.5-7B, Qwen2.5-14B, Qwen2.5-32B, Qwen2.5-72B, and Llama-3.1-8B. The headline result is that average accuracy rises from 51.1% under FC to 61.5% under SFC. The best FC model is GPT-4o at 70.2%, whereas the best pure SFC model is Claude-3.7-Sonnet at 73.4%. DeepSeek-v3-250324 is close at 70.0% under FC and 71.9% under SFC. Smaller open models improve substantially: Qwen2.5-7B rises from 33.0% to 49.0%, Qwen2.5-14B from 38.0% to 57.3%, and Llama-3.1-8B from 28.7% to 46.8%.
The domain breakdown indicates that gains are particularly large in more coupled control settings. Average Touch Control S-S improves from 46.8 to 65.5, Touch Control M-M from 37.1 to 49.0, Car Control S-M from 30.2 to 40.8, and Light S-S from 38.7 to 70.2. The paper also reports that SFC “significantly reduces both the number of interaction rounds and generated tokens across various models,” although exact token and turn counts are not printed in the text.
Ablation results refine this picture. With reflection, average FC accuracy rises from 53.1 to 63.4 for SFC across the evaluated subset, preserving SFC’s overall advantage. Without examples, both paradigms deteriorate, but FC degrades more: average no-example FC accuracy is 33.5, versus 50.3 for no-example SFC. The paper also reports that injecting irrelevant device states degrades all Qwen2.5 models, with smaller models degrading more severely; and that adding extra ReAct-style reasoning can hurt SFC, which the authors interpret as “overthinking.”
The most nuanced result is that pure SFC is not always dominant. When a task requires manipulating many device properties and there are strong high-level APIs that encapsulate those changes, FC can be advantageous. This motivates a hybrid FC+SFC pipeline in which SFC performs device selection and FC performs API execution within the selected subset. The paper reports that FC+SFC achieves the highest end-to-end accuracy overall.
6. Significance, limitations, and relation to adjacent work
VehicleWorld’s significance lies in making cockpit control a problem of explicit world-state transition rather than API retrieval. The environment shows that exposing full state can improve device selection, reduce exploratory interaction, and better align evaluation with what matters operationally: whether the intended state was achieved without disturbing irrelevant subsystems (Yang et al., 8 Sep 2025).
Several limitations are explicit. Environment construction remains heavily manual. The authors state that defining and verifying each device class requires substantial effort, and they do not yet have a clean automated way to integrate the global Environment class into entity-construction pipelines. The JSON state representation is also complex and long; although attribute-level value/type/description annotations help, they increase prompt size. The paper further stresses that VehicleWorld remains a controlled research simulation, not a production in-vehicle system. Safety, reliability, and user-interaction risks associated with real deployment are outside its scope.
In relation to nearby research, VehicleWorld occupies a distinct position. It is more integrated and executable than scene-understanding benchmarks that supervise video-based spatial cognition from indoor videos, such as ViCA, which focuses on spatial QA over real 3D scenes rather than direct control of coupled cockpit devices (Feng, 18 May 2025). It also differs from vehicle-in-virtual-environment methods for ADAS and CAD testing, which place a real vehicle inside a synchronized virtual traffic world rather than modeling an in-cabin multi-device software environment (Cao et al., 2024). A plausible implication is that these lines of work address different layers of a broader vehicle world model: VehicleWorld addresses in-cabin control and state management; video-based spatial cognition addresses scene understanding; and virtual-environment testing addresses closed-loop vehicle behavior in simulated traffic.
Within that landscape, VCWorld/VehicleWorld is best understood as an executable benchmark for stateful automotive agent control. Its main contribution is not merely the size of the API surface, but the claim that tightly coupled cockpit interaction is better formulated as an explicit state-transition problem over a shared global environment than as stateless function calling.