---
title: 'VCWorld: Executable Benchmark for Cockpit Control'
url: https://www.emergentmind.com/topics/vcworld
type: topic
---

# VCWorld: Executable Benchmark for Cockpit Control

VCWorld, in the automotive agent literature, denotes **VehicleWorld**, a simulated but executable intelligent vehicle cockpit environment for studying how large language model agents control tightly coupled in-car systems. The paper does not distinguish VCWorld from VehicleWorld. It is presented as the first comprehensive environment for the automotive domain with **30 device/module classes**, **250 APIs**, **680 attributes/properties**, real-time state information during agent execution, **302 initialization scenarios**, and a **1291-task** benchmark spanning single-turn, multi-turn, single-intent, and multi-intent interactions [2509.06736].

## 1. Scope and problem setting

VehicleWorld is motivated by the claim that intelligent vehicle cockpits are not well modeled by conventional function-calling environments. The environment targets a setting in which navigation, media, climate, lighting, doors, calls, touch control, and diagnostics are not isolated applications but tightly coupled subsystems with shared resources and continuously changing state. In that setting, stateless tool-use protocols require multiple exploratory calls to recover environmental context before they can act, which the paper identifies as a source of inefficiency and weak error recovery [2509.06736].

The environment abstracts cockpit functionality into four domains: **Multimedia**, **Touch Control**, **Car Control**, and **Lighting**. Examples named in the paper include **Navigation**, **Video**, **AirConditioner**, **Door**, **Conversation**, and a global **Environment** manager. Each device is represented as a module class with executable APIs and explicit internal properties. This design makes the benchmark less a conventional API-selection problem than a controlled study of world-state inference and state transition in a multi-device setting.

A terminological caution is necessary. In adjacent literature, the label **VCWorld** is also used for an unrelated biological world model for virtual cell simulation, where it denotes a cell-level white-box simulator for perturbation-response prediction rather than an automotive cockpit environment [2512.00306]. In the vehicle-interaction literature, however, VCWorld refers to VehicleWorld.

## 2. Environment architecture and global state model

The architectural core of VehicleWorld is its treatment of cockpit subsystems as globally coupled rather than locally autonomous. The paper emphasizes that the environment is “highly integrated” because modules interact through shared global state and resource contention. The canonical example is the **audio channel**, which may be occupied by music, radio, or navigation prompts, such that only one device can use it at a time. Other shared properties include **volume**, **temperature**, **sound channel**, **speaker**, **unit system**, and **time format** [2509.06736].

To maintain consistency across subsystems, VehicleWorld defines a global static **Environment** class using a **Singleton** pattern. Devices access shared properties through this central interface rather than storing private copies. The Environment class is described as providing standardized access methods and concurrency control. This design choice is central: it makes failures of local reasoning visible at the world level, because an API call that is locally valid may still conflict with global state or another subsystem.

Each top-level device class may include nested inner classes for subcomponents, and every class exposes a `to_dict()` method that recursively serializes current state into structured JSON. The world state therefore has two layers. **Device-level state** includes fields such as whether navigation is active, current destination and midway route, whether a video is playing, playback quality, whether the air conditioner is on, and whether a door is locked or open. **Global state** includes variables such as `volume`, `sound_channel`, `temperature`, and `speaker`. The appendix further notes that each JSON attribute is annotated with **value, type, and description** to make the state more legible to the agent.

This architecture implies that VehicleWorld is not merely a collection of callable methods. It is an executable state machine over a shared cockpit world, with JSON serialization as the interface through which models observe that world.

## 3. Executability, scenarios, and benchmark construction

VehicleWorld is fully executable rather than symbolic. Every API is implemented as code that updates internal attributes, validates parameters, and returns structured outputs. The environment includes `init()` methods for all **30** device classes, and combinations of these methods are used to generate **302 initialization scenarios**. The benchmark tasks themselves contain executable `<inits>` and `<api_call>` sections, and both are run locally to validate correctness [2509.06736].

Benchmark construction is semi-automatic. The authors first choose devices, APIs, and initialization methods based on real-world use cases, then use **Claude 3.7 Sonnet** to generate scenario templates with `<scenario>`, `<inits>`, `<query>`, and `<api_call>` tags. Automatic execution validates that the initialization code and reference action sequence are executable. After that, domain experts manually inspect each scenario for executability, semantic alignment between the query and the API call, and whether the query produces a meaningful state change.

The final benchmark contains **1291 tasks**. These are grouped by domain and by interaction complexity using four labels: **S-S**, **S-M**, **M-S**, and **M-M**, where the first position indicates **single-turn vs. multi-turn** and the second indicates **single-intent vs. multi-intent**. The distribution reported in the paper is as follows: Multimedia has **59 S-S**, **96 S-M**, **251 M-S**, and **53 M-M** tasks; Touch Control has **34**, **93**, **107**, and **15**; Car Control has **135**, **178**, **205**, and **173**; and Light has **79**, **75**, **159**, and **161**. Car Control is the largest subset.

The benchmark statistics are meant to show that tasks are not degenerate. Average involved devices per task are approximately two in each domain: **2.03** for Touch Control, **2.02** for Light, **2.11** for Multimedia, and **2.06** for Car Control. Average unique APIs per task are **2.84**, **3.54**, **3.18**, and **3.47**, and average API calls are **3.07**, **3.86**, **3.34**, and **3.73** respectively. The most complex tasks involve **up to 5 devices**, **13 API calls**, and **12 unique APIs**.

## 4. Function calling and the state-based reformulation

The paper’s central methodological claim is that conventional **Function Calling (FC)** is an ill-suited abstraction for tightly coupled cockpit control. FC is formalized as
$$
f_i = \mathcal{F}(q_i, A),
$$
where $q_i$ is the user query at step $i$ and $A$ is the set of available APIs [2509.06736].

In this formulation, the agent receives a query, searches available modules and APIs, and emits an API call with parameters. In the authors’ setup, this often begins with `search_module` and `search_api`. The paper argues that such behavior is fundamentally **stateless and exploratory**. Because the model lacks explicit world-state access at decision time, it often needs extra interaction rounds to infer which modules exist, what their current mode is, and which side effects matter globally.

The proposed alternative is **State-based Function Call (SFC)**, formalized as
$$
s_{i+1} = \mathcal{SF}(q_i, s_i),
$$
where $s_i$ is the current system state in JSON format and $s_{i+1}$ is the target state. This reframes control from “predict the next function call” to “predict the target world state, then generate transition code that realizes it.”

SFC is implemented as a **two-stage process**. In stage 1, the model receives the complete JSON state of all devices plus the user query and selects the **relevant devices**. In stage 2, it receives only the state JSON of the selected devices and predicts the target state together with the transition code. The paper explicitly notes that the two stages “operate independently without shared context.” Execution is then performed directly on the simulator through Python code that updates state variables or attributes, rather than through a full API-only plan.

This distinction is consequential. Pure SFC does not primarily search for APIs; it manipulates an explicit, serialized world state. The environment then returns execution logs, exceptions, and the updated state, which becomes the basis for subsequent turns. The authors therefore treat SFC as a more faithful abstraction for cockpit interaction because it externalizes persistent system state rather than leaving it implicit in the model’s hidden textual context.

## 5. Evaluation methodology and empirical findings

VehicleWorld is evaluated with **state-based** rather than action-sequence-based metrics. The three main metrics are **F1 positive**, **F1 negative**, and **Accuracy (Acc)**, which respectively assess whether attributes that should change do change, whether attributes that should remain unchanged do remain unchanged, and whether final values are correct [2509.06736]. For ambiguous numerical instructions such as “increase the volume,” the paper judges **trend correctness**—increase, decrease, or maintain—rather than exact value equality.

The paper also compares **state-based** evaluation with **rule-based** evaluation, where the latter checks exact API names, parameter keys, parameter values, and call counts. On **200 FC outputs** from Qwen2.5 models, state-based evaluation is reported to have lower error rate relative to expert judgment, because it evaluates practical task completion rather than exact reproduction of a reference API trace.

The main empirical comparison is FC versus SFC under **ReAct** prompting across eight representative models: **Claude-3.7-Sonnet, GPT-4o, DeepSeek-v3-250324, Qwen2.5-7B, Qwen2.5-14B, Qwen2.5-32B, Qwen2.5-72B, and Llama-3.1-8B**. The headline result is that **average accuracy rises from 51.1% under FC to 61.5% under SFC**. The best FC model is **GPT-4o at 70.2%**, whereas the best pure SFC model is **Claude-3.7-Sonnet at 73.4%**. **DeepSeek-v3-250324** is close at **70.0%** under FC and **71.9%** under SFC. Smaller open models improve substantially: **Qwen2.5-7B** rises from **33.0% to 49.0%**, **Qwen2.5-14B** from **38.0% to 57.3%**, and **Llama-3.1-8B** from **28.7% to 46.8%**.

The domain breakdown indicates that gains are particularly large in more coupled control settings. Average **Touch Control S-S** improves from **46.8** to **65.5**, **Touch Control M-M** from **37.1** to **49.0**, **Car Control S-M** from **30.2** to **40.8**, and **Light S-S** from **38.7** to **70.2**. The paper also reports that SFC “significantly reduces both the number of interaction rounds and generated tokens across various models,” although exact token and turn counts are not printed in the text.

Ablation results refine this picture. With **reflection**, average FC accuracy rises from **53.1** to **63.4** for SFC across the evaluated subset, preserving SFC’s overall advantage. Without examples, both paradigms deteriorate, but FC degrades more: average no-example FC accuracy is **33.5**, versus **50.3** for no-example SFC. The paper also reports that injecting irrelevant device states degrades all Qwen2.5 models, with smaller models degrading more severely; and that adding extra ReAct-style reasoning can hurt SFC, which the authors interpret as “overthinking.”

The most nuanced result is that pure SFC is not always dominant. When a task requires manipulating many device properties and there are strong high-level APIs that encapsulate those changes, FC can be advantageous. This motivates a hybrid **FC+SFC** pipeline in which SFC performs device selection and FC performs API execution within the selected subset. The paper reports that **FC+SFC achieves the highest end-to-end accuracy** overall.

## 6. Significance, limitations, and relation to adjacent work

VehicleWorld’s significance lies in making cockpit control a problem of **explicit world-state transition** rather than API retrieval. The environment shows that exposing full state can improve device selection, reduce exploratory interaction, and better align evaluation with what matters operationally: whether the intended state was achieved without disturbing irrelevant subsystems [2509.06736].

Several limitations are explicit. Environment construction remains heavily **manual**. The authors state that defining and verifying each device class requires substantial effort, and they do not yet have a clean automated way to integrate the global Environment class into entity-construction pipelines. The JSON state representation is also **complex** and long; although attribute-level value/type/description annotations help, they increase prompt size. The paper further stresses that VehicleWorld remains a **controlled research simulation**, not a production in-vehicle system. Safety, reliability, and user-interaction risks associated with real deployment are outside its scope.

In relation to nearby research, VehicleWorld occupies a distinct position. It is more integrated and executable than scene-understanding benchmarks that supervise video-based spatial cognition from indoor videos, such as ViCA, which focuses on spatial QA over real 3D scenes rather than direct control of coupled cockpit devices [2505.12312]. It also differs from vehicle-in-virtual-environment methods for ADAS and CAD testing, which place a real vehicle inside a synchronized virtual traffic world rather than modeling an in-cabin multi-device software environment [2410.04313]. A plausible implication is that these lines of work address different layers of a broader vehicle world model: VehicleWorld addresses in-cabin control and state management; video-based spatial cognition addresses scene understanding; and virtual-environment testing addresses closed-loop vehicle behavior in simulated traffic.

Within that landscape, VCWorld/VehicleWorld is best understood as an executable benchmark for **stateful automotive agent control**. Its main contribution is not merely the size of the API surface, but the claim that tightly coupled cockpit interaction is better formulated as an explicit state-transition problem over a shared global environment than as stateless function calling.

Source: https://www.emergentmind.com/topics/vcworld