ASURADA: Geo-Aware Automotive Agent

• ASURADA is a geo-adaptive multimodal agent designed for in-vehicle GUIs that integrates GPS input to provide real-time environmental and regulatory context for safe vehicle operation.
• It leverages Automotive‑ENV, a high-fidelity benchmark with OS-level APIs and dynamic task templates, to simulate various safety-critical automotive interactions.
• Empirical evaluations show ASURADA achieves notable improvements in safety tasks compared to baseline agents, highlighting its value in aligning actions with environmental conditions.

Searching arXiv for the specified paper to ground the article and citation. ASURADA is a geo‑adaptive multimodal agent designed specifically for in‑vehicle graphical user interfaces (GUIs), introduced together with Automotive‑ENV, a benchmark and interaction environment tailored to vehicle GUIs (Yan et al., 25 Sep 2025). It is motivated by the claim that automotive interaction differs fundamentally from desktop, web, or mobile GUI control because correct behavior depends on real‑time location, driving state, weather, and traffic conditions; driver attention is limited and noisy; many functions are location‑sensitive; and the action space is tightly coupled to vehicle subsystems with stateful constraints (Yan et al., 25 Sep 2025). Within that setting, ASURADA extends a baseline multimodal agent by integrating GPS as an input modality and by using a GPS reasoning module to transform raw coordinates into environmental context, road context, and regional regulations and cultural norms, so that behavior is not only functionally correct but safety‑aligned and culturally/regulatorily appropriate (Yan et al., 25 Sep 2025).

1. Problem Setting and Domain Specificity

ASURADA is formulated for a domain in which multimodal GUI interaction must be interpreted under highly dynamic, safety‑critical conditions. The motivating examples emphasize that an utterance such as “I can’t see through the windshield, it’s all fogged up” requires inference over visibility conditions and vehicle controls, specifically operation of the HVAC system by enabling the front defroster rather than modifying unrelated settings such as media or brightness (Yan et al., 25 Sep 2025). The system is therefore positioned not merely as a GUI controller but as an agent whose actions must remain consistent with environmental hazards and subsystem semantics.

The problem setting also assumes that driver commands are often brief, ambiguous, or incomplete. This establishes a distinction between literal instruction following and intent resolution under contextual uncertainty. In this framing, location is not an auxiliary feature: many functions are constrained or modified by regional laws, road types, and climate, including high‑beam use, speed limits, fog lights, and air‑conditioning behavior (Yan et al., 25 Sep 2025). The paper’s characterization of automotive GUIs as simplified for safety, yet tightly coupled to underlying vehicle subsystems, suggests that the core technical difficulty lies in contextually valid control rather than in generic GUI navigation.

A plausible implication is that conventional GUI benchmarks underrepresent the principal failure modes of automotive agents, because they typically rely on static screenshots and textual instructions rather than dynamic environmental state. In that sense, ASURADA is best understood as a domain‑specialized multimodal agent whose defining property is geo‑awareness rather than generic interface competence.

2. Automotive‑ENV as the Operating and Evaluation Substrate

ASURADA operates in Automotive‑ENV, described as the first high‑fidelity benchmark and interaction environment tailored for vehicle GUIs (Yan et al., 25 Sep 2025). The environment is built on a real Automotive OS commonly used in modern electric/premium vehicles, is simulated virtually on standard hardware without proprietary in‑car hardware, and mirrors production automotive software architecture, including event handling and OS‑level APIs for subsystems (Yan et al., 25 Sep 2025). The emphasis on OS‑level fidelity is central because the agent interacts through interfaces that resemble deployed vehicle software stacks rather than abstract benchmark wrappers.

Automotive‑ENV spans 8 functional modules—Maps, HVAC, Road, Phenomenon, Media, Apps, System, and Comms—with 185 parameterized task templates (Yan et al., 25 Sep 2025). Each template can be instantiated with random parameters, yielding effectively infinite task instances, denoted by “$\infty$” in the paper’s summary comparison (Yan et al., 25 Sep 2025). This parameterization matters because it converts the benchmark from a fixed test set into a family of task distributions over operational contexts.

The observation space is explicitly multimodal. It includes screenshots of the GUI, an accessibility tree containing structured UI elements, vehicle state information for subsystems such as HVAC and media, GPS location data with derived regional context, and network connectivity with optional web knowledge that is contextually gated by GPS, including weather and local regulations (Yan et al., 25 Sep 2025). The action space consists of touchscreen operations—taps, swipes, and text input—and automotive‑specific APIs such as safety alerts, safety popup messages, navigation start, and climate setpoints (Yan et al., 25 Sep 2025). ASURADA and other agents act through these OS‑level interfaces rather than low‑level CAN or OBD‑II buses, which the paper states simplifies integration and evaluation (Yan et al., 25 Sep 2025).

Reproducibility is enforced through state management procedures including system time reset, fixed app versions, offline operation, and explicit initialization and cleanup per task (Yan et al., 25 Sep 2025). Geographic parameterization permits instantiation either from the current GPS or from predefined region profiles defined by climate, rules, and cultural norms (Yan et al., 25 Sep 2025). Success is determined through direct OS‑level programmatic checks of subsystem states, such as check_fan_speed_max() and check_front_defroster_enable(), rather than by screen matching or textual heuristics (Yan et al., 25 Sep 2025). The paper defines success for a task with instruction $u$ and validation function $f$ as

$\text{Success}(u) = \begin{cases} 1 & \text{if } f(\text{OS state}) = \text{True},\[4pt] 0 & \text{otherwise}. \end{cases}$

and aggregate success rate over a task set $\mathcal{T}$ as

$$\text{SR} = \frac{1}{|\mathcal{T}|} \sum_{i \in \mathcal{T}} \text{Success}(u_i).$$

Safety metrics, including Driving Alignment and Environment Alerts, are evaluated similarly but conditioned on geo‑context (Yan et al., 25 Sep 2025). This evaluation design shifts the benchmark from appearance‑based correctness toward state‑based correctness.

3. Task Taxonomy and the Role of Safety Awareness

The benchmark partitions tasks into General and Safety‑Aware categories (Yan et al., 25 Sep 2025). General tasks comprise Explicit Control and Implicit Intent. Explicit Control includes direct commands such as “Set the temperature to 22 degrees” and “Turn the fan speed to Max,” which map relatively directly to GUI actions or APIs (Yan et al., 25 Sep 2025). Implicit Intent covers indirect expressions such as “My hands are freezing,” for which the agent must infer an appropriate intervention, such as increasing cabin temperature or turning on seat heating (Yan et al., 25 Sep 2025).

Safety‑Aware tasks comprise Driving Alignment and Environment Alerts (Yan et al., 25 Sep 2025). Driving Alignment requires compliance with regional rules and norms, including recognition that high beams are inappropriate in certain conditions or that current speed violates city limits (Yan et al., 25 Sep 2025). Environment Alerts requires detection of hazardous conditions such as fog, heat, or low visibility and corresponding proactive responses, including enabling fog lights, defrosters, or safety notifications (Yan et al., 25 Sep 2025).

This taxonomy is consequential because it separates basic operational control from contextually contingent control. The paper states that the taxonomy is central to evaluating whether geo‑aware reasoning yields tangible safety benefits (Yan et al., 25 Sep 2025). A plausible implication is that ASURADA’s primary contribution is not uniform performance improvement across all task types, but selective improvement in categories where environmental and regulatory context materially alters the admissible action set.

4. Architecture, State Representation, and Inference Workflow

ASURADA is described conceptually as an extension of a baseline multimodal agent, M3A, with two additions: GPS as an input modality alongside screen pixels, accessibility tree, and text instructions, and a GPS reasoning module that acts as a “virtual sensor” transforming raw coordinates into environmental context, road context, and regional regulations and cultural norms (Yan et al., 25 Sep 2025). The output of this module includes location descriptor, road type, legal speed limit, weather snapshot, and regional norms (Yan et al., 25 Sep 2025).

The agent state at interaction step $t$ is summarized as

$$s_t = \big( x_t^{\text{screen}}, x_t^{\text{a11y}}, x_t^{\text{text}}, x_t^{\text{GPS}}, x_t^{\text{veh}}, m_t \big),$$

where the terms denote screenshot pixels, accessibility tree, user instruction, GPS coordinates and derived attributes, vehicle subsystem states, and internal memory/history of previous actions and observations, respectively (Yan et al., 25 Sep 2025). Geo‑aware features are computed by a GPS reasoning module:

$$g(s_t) = \text{GeoContext}\big(x_t^{\text{GPS}}\big) \approx \big( \text{Location descriptor}, \text{Road type}, \text{Legal speed limit}, \text{Weather snapshot}, \text{Regional norms} \big).$$

An augmented state is then formed as

$s_t' = [\, s_t; g(s_t) \,],$

with concatenation interpreted at the representation level, including prompt augmentation for LLMs and feature concatenation for multimodal encoders (Yan et al., 25 Sep 2025). The policy is expressed as

$u$0

and is implemented via a VLM‑based agent architecture, with examples including GPT‑4o‑Mini and Gemini variants (Yan et al., 25 Sep 2025).

The agent produces structured JSON action plans containing GUI actions such as tap indices and text entries, API calls such as safety notification center or fog lights, reasoning traces using chain‑of‑thought and ReAct‑style thought/action interleaving, and reflection metadata for post‑action analysis (Yan et al., 25 Sep 2025). Its workflow consists of five iterative stages: perception, geo‑contextualization, reasoning and planning, execution, and reflection (Yan et al., 25 Sep 2025). Perception collects multimodal observations including vehicle state, GUI layout and accessibility tree, screenshot, GPS signals, and memory. Geo‑contextualization uses GPS signals to query external sources, if enabled, including weather APIs, road and traffic regulations, and region‑specific norms. Reasoning and planning generate a JSON plan with step‑by‑step reasoning in a ReAct + Reflexion style. Execution invokes touchscreen operations and OS‑level APIs such as HVACer, navigation, and safety alerts. Reflection compares pre‑ and post‑action OS states, assesses validation checks, and updates memory $u$1 with strategy successes and failures (Yan et al., 25 Sep 2025).

The paper states that ASURADA is primarily evaluated under prompted inference using proprietary LLMs and that no explicit RL training objective is specified (Yan et al., 25 Sep 2025). Agent behavior is driven by prompt engineering, ReAct‑style reasoning, and Reflexion‑based self‑critique, while performance is measured offline using Automotive‑ENV’s programmatic checks (Yan et al., 25 Sep 2025). The paper includes a hypothetical supervised imitation loss for future work,

$u$2

but explicitly notes that the current paper does not report such training (Yan et al., 25 Sep 2025).

5. Geo‑Aware Reasoning and Safety‑Conditioned Action Selection

ASURADA’s geo‑awareness is implemented through direct GPS input and GPS‑contextualized web queries (Yan et al., 25 Sep 2025). Direct GPS input includes raw coordinates, exemplified by $u$3, and derived meta‑information such as city, region, and urban versus highway classification (Yan et al., 25 Sep 2025). GPS‑contextualized web queries can retrieve local rules, weather conditions, and cultural norms, and these are textualized and injected into the prompt so that reasoning about safety and legality is grounded in location‑specific constraints (Yan et al., 25 Sep 2025).

The paper does not present explicit mathematical safety constraints, but it characterizes the behavior as corresponding to a constraint set $u$4 over actions:

$u$5

where $u$6 encodes violations of legal speed limits, visibility requirements, regional rules such as high‑beam usage and child lock behavior, and environmental safety conditions including heat stress and low cabin visibility (Yan et al., 25 Sep 2025). ASURADA uses geo‑context to decide whether to take proactive safety actions such as opening the safety notification center or enabling defrosters when such constraints are breached (Yan et al., 25 Sep 2025).

A concrete example is the urban speed‑violation scenario. Without geo‑context, the input “Current speed: 80 km/h” contains no information about location or speed limit, so 80 km/h is treated as not obviously unsafe; the model takes no safety action, the status remains infeasible, and the result is failure in Driving Alignment (Yan et al., 25 Sep 2025). With geo‑context, the input is extended to include GPS $u$7, the location “Paris city center,” and the local rule “urban roads limited to 50 km/h unless otherwise posted,” enabling the model to recognize that $u$8 and therefore unlawful and unsafe (Yan et al., 25 Sep 2025). The resulting action is: “Opens the safety notification center. (click index 22),” and the outcome is success (Yan et al., 25 Sep 2025). The example demonstrates that geo‑context changes the semantics of an otherwise neutral speed reading into a safety‑relevant violation.

The paper also notes geo‑dependent climate behavior. For the utterance “I feel hot,” ASURADA may suggest opening windows in cool regions, while in persistently hot climates it directly activates air conditioning (Yan et al., 25 Sep 2025). These examples are presented as qualitative demonstrations of how GPS‑derived climate context interacts with linguistic cues to guide HVAC strategy. This suggests that geo‑awareness functions both as a legality prior and as a contextual prior for implicit intent resolution.

6. Empirical Evaluation and Performance Characteristics

ASURADA is evaluated against two baselines: T3A, a text‑only agent using the accessibility tree and instruction without screen pixels or GPS, and M3A, a multimodal agent using the accessibility tree and screen screenshot with Set‑of‑Mark annotations and ReAct + Reflexion prompting, but without GPS (Yan et al., 25 Sep 2025). ASURADA uses accessibility tree, screen, and GPS, with optional network‑based context queries informed by GPS (Yan et al., 25 Sep 2025). All agents follow a four‑phase cycle of interpretation, planning, execution, and reflection, and the primary metric is Success Rate across Explicit Control, Implicit Intent, Driving Alignment, and Environment Alerts, with human annotators providing an upper bound (Yan et al., 25 Sep 2025).

Human performance is reported as 90.0% on Explicit Control, 82.0% on Implicit Intent, 100.0% on Driving Alignment, and 88.0% on Environment Alerts (Yan et al., 25 Sep 2025). The effect of visual grounding from T3A to M3A is illustrated by improvements such as GPT‑4o‑Mini Explicit Control from 43.1% to 52.1% and Implicit Intent from 5.2% to 13.6%, and Gemini 1.5 Flash Explicit Control from 43.3% to 46.6% and Implicit Intent from 28.5% to 33.3%; safety categories also improve modestly, for example Driving Alignment from 55.0% to 75.0% for Gemini 1.5 Flash (Yan et al., 25 Sep 2025). The effect of GPS geo‑context from M3A to ASURADA is reported as small for general tasks but substantial for safety‑aware tasks. For GPT‑4o‑Mini, Explicit Control changes from 52.1 to 52.3 and Implicit Intent from 13.6 to 14.2, while Driving Alignment rises from 50.0 to 60.0 and Environment Alerts from 55.0 to 65.0 (Yan et al., 25 Sep 2025). For Gemini 1.5 Pro, Driving Alignment increases from 55.0 to 90.0 and Environment Alerts from 80.0 to 90.0. For Gemini 1.5 Flash, Driving Alignment rises from 75.0 to 90.0 and Environment Alerts from 55.0 to 85.0. For Gemini 2.0 Flash, Implicit Intent rises from 38.0 to 45.7, Driving Alignment from 65.0 to 90.0, and Environment Alerts from 45.0 to 70.0. For Gemini 2.0 Flash‑Lite, Driving Alignment increases from 42.8 to 90.0 (Yan et al., 25 Sep 2025).

The paper’s synthesis is that General tasks benefit mainly from visual grounding, while GPS yields modest additional improvement, whereas Safety‑Aware tasks show dramatic gains with GPS, often 15–30 percentage points or more (Yan et al., 25 Sep 2025). ASURADA’s overall success on safety tasks, such as 90% on Driving Alignment for multiple Gemini variants, approaches human Driving Alignment performance of 100%, but still lags on Environment Alerts and Implicit Intent (Yan et al., 25 Sep 2025). This suggests that geo‑aware context is especially effective when the principal source of ambiguity lies outside the GUI itself.

Further analysis in the paper addresses token length and task‑wise gains. With geo‑context, reasoning trajectories are shorter and more focused, typically under 1500 tokens and rarely above 2000; without geo‑context, outputs often extend to 2000–3000 tokens, which the paper interprets as less efficient and more redundant reasoning (Yan et al., 25 Sep 2025). Large relative improvements of 15–30% are reported in Media, Maps, HVAC, and Road, while gains are modest at 2–5% in System and Communication tasks (Yan et al., 25 Sep 2025). The authors interpret this as evidence that geo‑context simplifies reasoning by constraining plausible actions and providing stronger priors (Yan et al., 25 Sep 2025).

7. Limitations, Assumptions, and Broader Significance

The paper explicitly identifies a performance gap to humans. Even with GPS, representative ASURADA configurations, such as Gemini, achieve around 65% success overall, described as better than web‑adapted baselines but far below human 100% on safety‑critical tasks (Yan et al., 25 Sep 2025). It also notes reliance on GPS signals as a limitation, since real vehicles experience GPS outages in tunnels, underground parking, and urban canyons; the current system assumes reliable GPS, and outages could degrade geo‑aware behavior (Yan et al., 25 Sep 2025). Qualitatively, the models continue to struggle with multi‑step reasoning over ambiguous instructions and with edge cases in safety judgments where rules are subtle or conflicting (Yan et al., 25 Sep 2025).

The stated assumptions include reliable sensors and OS APIs for querying subsystem state, accessible geographic and regulatory data via GPS‑informed queries, and an offline operation regime for many tasks, though optional web queries assume connectivity (Yan et al., 25 Sep 2025). These assumptions delineate the extent to which the benchmark abstracts away infrastructure uncertainty. A plausible implication is that transition from emulated Automotive OS to production vehicles will require explicit handling of missing or stale geo‑context and more heterogeneous subsystem interfaces.

Future directions proposed in the paper include robust geo‑reasoning under partial observation by using LLMs as “virtual sensors” to interpolate position when GPS is temporarily unavailable; richer multimodal context, including additional sensors such as vehicle speed, camera images, and perhaps driver state; integration into real vehicles involving real drivers and regulators; and learning‑based optimization via RL or imitation learning using Automotive‑ENV’s programmatic reward signals, potentially with safety‑specific losses (Yan et al., 25 Sep 2025). These directions preserve the benchmark’s central thesis that safety‑aware automotive assistance depends on fusing GUI understanding with geographic and environmental priors.

In broader terms, ASURADA and Automotive‑ENV are presented as extending multimodal GUI agents into a safety‑critical, geo‑dependent domain with structured tasks and rich OS‑level rewards (Yan et al., 25 Sep 2025). The significance assigned by the paper lies in three linked claims: automotive GUIs constitute a distinct domain; geo‑aware multimodal agent design materially improves Driving Alignment and Environment Alerts; and programmatic, state‑based evaluation avoids the brittleness of screen‑based assessment while enabling precise measurement and possible future RL training (Yan et al., 25 Sep 2025). The concluding interpretation is that geo‑context is a critical ingredient in making automotive GUI agents reliable and safe, and that Automotive‑ENV provides the infrastructure for systematic progress toward safety‑aware in‑vehicle assistants (Yan et al., 25 Sep 2025).