Scene-Aware Conversational ADAS

• SC-ADAS is a driver-assistance system that uses generative AI and multimodal sensing to adapt dialogue based on driving scenes and driver state.
• It integrates vision-to-text, large language models, and structured function calling to provide real-time, interpretable recommendations and control actions.
• The technology bridges dynamic scene interpretation with adaptive communication, enhancing safety through validated, context-aware vehicular commands.

Scene-Aware Conversational ADAS (SC-ADAS) denotes a class of driver-assistance systems in which natural-language interaction is grounded in the driving scene, the driver’s attentional state, and executable assistance functions rather than in fixed alerts or isolated voice commands. The term is used explicitly for a modular framework that integrates Generative AI components including LLMs, vision-to-text interpretation, and structured function calling to enable real-time, interpretable, and adaptive driver assistance; closely related work defines an “Adaptive Communication Module for Driver Assistive Systems” that adapts communication to the driver’s psycho-physiological attributes and the environment, and also proposes vehicle-to-vehicle dialogue with a remote onboard multimodal LLM to complement the ego vehicle’s field-of-view in occluded urban scenes (Han et al., 14 Jul 2025, Hasan et al., 2024, Dona et al., 2024).

1. Conceptual scope and problem formulation

SC-ADAS emerges from two recurring critiques of conventional assistance systems. First, current ADAS are described as “largely rule-based and reactive,” lacking the ability to engage drivers through natural language conversation, interpret evolving road scenes, or adapt behavior in real time based on situational context and driver intent. Second, much existing in-vehicle communication follows a “one mode for all scenarios paradigm,” even though the appropriateness of a warning or explanation depends on both scenario hazard and the driver’s current distraction state (Han et al., 14 Jul 2025, Hasan et al., 2024).

Within that problem formulation, conversational assistance is not restricted to a voice wrapper around fixed functions. In the 2025 SC-ADAS formulation, the system supports multi-turn dialogue grounded in visual and sensor context, produces natural-language recommendations, and executes vehicle-level actuation only after explicit driver confirmation. In the 2026 Autoware-based framework, the interaction space is taxonomized into five categories: Information, Mission Control, Configuration, Cooperation, and Intervention. These categories span status queries, global operating-state changes, long-term preference updates, local maneuver requests, and attempts to correct a perception or planning error such as “The traffic light is green” or “Emergency, stop now” (Han et al., 14 Jul 2025, Seegert et al., 9 Jan 2026).

Taken together, these studies suggest that SC-ADAS should be understood as an interaction layer over scene interpretation and control validation, not merely as speech recognition plus command execution. Its central research question is how to align scene understanding, driver state, dialogue history, and safe actuation in a single closed loop.

2. Lineage in audio-visual scene-aware dialogue

The technical lineage of SC-ADAS is closely tied to Audio Visual Scene-aware Dialog (AVSD), which extended end-to-end dialogue modeling from text-only and image-only settings to dynamic audio-visual scenes. AVSD frames the task as generating the next answer in a dialogue about a video, conditioned on the current question, the audio-visual scene representation, and the dialogue history. In the 2018 formulation, the full context vector for the $n$-th answer is

$g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$

where $g_n^{(q)}$ is the current question encoding, $g_n^{(h)}$ is the hierarchical-LSTM history encoding, and $g_n^{(av)}$ is the audio-visual video encoding. Rather than a fixed fusion rule, the model uses multimodal attention:

$$g_n^{(av)} = \tanh\left(\sum_{k=1}^K \beta_{k,n} d_{k,n}\right), \qquad \beta_{k,n}=\frac{\exp(v_{k,n})}{\sum_{\kappa=1}^{K}\exp(v_{\kappa,n})}.$$

This allows the decoder to emphasize different modalities depending on the question context. On AVSD response generation, the best reported configuration in the table is QA + I3D + VGGish with attention, achieving BLEU-1 0.256, BLEU-2 0.161, BLEU-3 0.109, BLEU-4 0.078, METEOR 0.113, ROUGE-L 0.277, CIDEr 0.727. The paper’s broader conclusion is that multimodal features help, and multimodal attention helps more than naive fusion (Hori et al., 2018).

The dataset design is equally important. AVSD was collected from the Charades video corpus as typed conversations between two AMT workers, one of whom saw the full video while the other saw only the first, middle, and last frames. Each dialogue contains exactly 10 question-answer pairs, followed by a summary description. One reported split contains 6,172 dialogs for training, 732 dialogs for validation, and 733 dialogs for test; the later DSTC7-based work uses the official training set (7,659 dialogs), prototype-validation set (732 dialogs), and prototype-test set (733 dialogs) (Hori et al., 2018, Kumar et al., 2019).

The 2019 extension deepens this lineage in three ways. It treats dialog topics as contextual state using Latent Dirichlet Allocation (LDA) and Guided LDA over questions, answers, QA pairs, captions, history, and history+captions; it explores attention over dialog history and multimodal features; and it incorporates AclNet, an end-to-end audio classification ConvNet trained on ESC-50 that takes raw, amplitude-normalized 44.1 kHz audio samples as input. In the topic experiments, the baseline scores are Bleu1 0.273, Bleu2 0.173, Bleu3 0.118, Bleu4 0.084, Meteor 0.117, Rouge 0.291, CIDEr 0.766. Guided LDA reaches CIDEr 0.812 with Bleu1 0.265, Bleu2 0.170, Bleu3 0.117, Bleu4 0.084, Meteor 0.118, Rouge 0.293, CIDEr 0.812, and the authors state that “GuidedLDA + GloVe is our best performing model.” The same paper reports that attending over all sentence-LSTM output states outperforms the baseline on BLEU3, BLEU4, METEOR, CIDEr and ROUGE, while B+AclNet performs the best both on the overall dataset and on audio-related dialogs (Kumar et al., 2019).

For SC-ADAS, the relevance of AVSD is foundational rather than automotive-specific. It establishes the core computational problem of grounding multi-turn dialogue in dynamic scene evidence, audio, and conversational memory.

3. Modular conversational-control architectures

The explicit SC-ADAS architecture presented in 2025 decomposes the system into four Generative AI-powered modules: Query Refinement, Context Retrieval, Response Generation, and Command Generation. The driver’s spoken query is converted to text, refined into intent plus metadata indicating whether sensing context or actuation is required, optionally augmented with real-time sensor data and vision-to-text scene descriptions, and then used to generate a natural-language response. If the driver confirms an action, structured function calling produces an executable ADAS command. The paper illustrates the actuation interface with a function-call output such as:

$g_n^{(av)}$1

This design separates recommendation from control: conversation may suggest an action, but execution is routed through structured output only after confirmation (Han et al., 14 Jul 2025).

The same separation appears in the 2026 Autoware-based framework, but with a stronger emphasis on deterministic validation. A spoken instruction $I$, AV status information, and a knowledge base are passed to an LLM, which translates the request into an application-centric DSL. The target policy space is described as

$P = \{A, C\},$

with $A$ finite actions and $C$ continuous parameters. The structured command is then parsed by a ROS2 Interface Node that validates the action, enforces parameter bounds, maps the DSL explicitly to allowed Autoware operations, and returns the definitive success or failure state to the LLM for second-stage feedback generation. The paper stresses that the system sits on top of Autoware’s existing safety architecture and that the LLM never directly drives the vehicle (Seegert et al., 9 Jan 2026).

This dual pattern—LLM-mediated interpretation followed by deterministic validation—recurs across the literature. In SC-ADAS, dialogue is used for intent understanding, recommendation, and explanation; validated machine-readable commands remain the boundary to actuation. A plausible implication is that conversational naturalness and control authority are deliberately decoupled to preserve transparency and safety.

4. Adaptive communication and driver state awareness

A distinct line of work treats the conversational or communicative problem less as command execution and more as adaptive alert selection. In an online user study with $g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$0 participants, 144 of whom were retained for analysis, participants viewed 30 simulated driving videos recorded in the CARLA Driving Simulator from the driver’s perspective, including the windshield view and the left side mirror, rear-view mirror, and right side mirror. The scenarios, ordered by decreasing risk, were Pedestrian collision, Car collision, Lane deviation, Blind spot obstruction, and Stagnant at a green light, each under clear day, rainy day, and night conditions. Augmented versions manipulated the driver’s apparent attention through Drowsiness, Hyper-focus on the road, and Salient external agents (Hasan et al., 2024).

Participants ranked their top three preferred communication modes from a set including non-flashing and flashing dashboard icons, flashing mirror icons, spoken language alerts, a single beep, multiple beeps, steering-wheel vibrations, seat vibrations, text alerts, and image alerts. Preferences varied systematically with scene hazard. For high-risk / emergent scenarios such as pedestrian collisions and car collisions, participants preferred repeating sound alerts and vibrations. For lower-risk scenarios such as stagnation at a green light, they preferred a single beep and spoken language alerts. The least preferred modes across scenarios were non-flashing dashboard icons, text alerts, and image alerts on the dashboard or console (Hasan et al., 2024).

The paper also quantifies how preferences change when the same scene is presented under altered attention states. After converting top-3 choices into full rankings, the authors report average correlations between normal and augmented versions of Kendall’s $g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$1, Spearman’s $g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$2, and RBO $g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$3, with $g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$4 for all metrics. The correlations are described as only moderately positive, indicating that hazard structure persists but does not fully determine communication preference under distraction. Statistical support is also reported for scenario dependence via one-way ANOVA ($g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$5, $g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$6) and chi-square goodness-of-fit ($g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$7, $g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$8) (Hasan et al., 2024).

The study’s conceptual contribution is the Adaptive Communication Module for Driver Assistive Systems, defined as technology that “learns and adapts the mode of communication of information to be relayed by Assistive Systems to the changes in the driver’s psycho-physiological attributes while taking into consideration the environment around the driver (and their vehicle) as well as the driver’s communication preferences.” Support for adaptivity is substantial but not unqualified: mean perceived safety with such a module is $g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],$9 as driver and $g_n^{(q)}$0 as passenger; over 80% reported feeling safer as a driver and over 77% as a passenger. At the same time, mean comfort with audio recordings is $g_n^{(q)}$1 and with video recordings $g_n^{(q)}$2, with more than 60% uncomfortable with audio recording and 67% uncomfortable with video recording. The trust hypothesis is reported as inconclusive (Hasan et al., 2024).

5. Situation awareness, symbolic reasoning, and audited risk interpretation

Another SC-ADAS strand emphasizes modeling what the driver has perceived and understood, especially in takeover situations. In semi-autonomous driving at SAE level 3/4, the challenge is that the driver may need to rebuild situation awareness within the commonly cited 7–10 seconds of takeover time. The SituSYS framework, developed in the SituWare project, addresses this through a ROS-based modular system with three layers: SituSENSORS, SituMODEL, and SituHMI. It senses the environment and driver gaze, estimates situation awareness, and directs attention through specialized HMIs (Suchan et al., 2023).

The formal core is an ASP-based representation of the driving domain,

$g_n^{(q)}$3

where $g_n^{(q)}$4 are domain objects, $g_n^{(q)}$5 spatial entities, $g_n^{(q)}$6 spatial relations, $g_n^{(q)}$7 time points, $g_n^{(q)}$8 fluents, and $g_n^{(q)}$9 events. Scene data include ego-vehicle state, automation state, other-vehicle properties, and fixation information; gaze is measured with a PupilLabs mobile eye tracker at up to 120 Hz with about 0.60° accuracy. The reasoning pipeline updates a mental belief state using fixation thresholds and a Kalman filter motion model, constructs an ASP problem, and solves it with Clingo to obtain an interpretation model $g_n^{(h)}$0 and projection model $g_n^{(h)}$1. The system is reported to run in real time at about 30 Hz and to handle scenes with up to 20 vehicles plus the ego vehicle (Suchan et al., 2023).

Where SituSYS models a specific driver’s awareness, the 2026 scenario-centric framework audits how LLMs interpret fixed driving situations. Raw multimodal logs are converted into a normalized semantic layer at 1 Hz, then materialized into deterministic scenario windows according to

$g_n^{(h)}$2

For the reported evaluation, each scenario uses a symmetric 3-second pre and 3-second post window around an anchor time, yielding seven-second scenario windows across a curated near-people scenario set of 16 cases. Models are constrained to use only provided structured data and images if available, and must return JSON-only outputs with numeric codes. The closed risk taxonomy defines overall_risk_level from 0 Not identified through 6 Critical, and high risk is explicitly defined as overall risk level $g_n^{(h)}$3 (Carvalho et al., 29 Mar 2026).

Under these constraints, GPT-4o-vision assigns higher average risk and escalates a larger fraction of windows to high risk than the text-only GPT-4o-mini and DeepSeek-Chat models. The text-only models cite more evidence signals on average, while all models agree that pedestrian-related risk dominates the near-people dataset but differ in secondary attribution: DeepSeek-Chat emphasizes rear-end conflict, GPT-4o-mini emphasizes lateral conflict, and GPT-4o-vision emphasizes traffic density. The paper interprets this as evidence of intrinsic semantic indeterminacy rather than isolated model failure, and introduces a composite uncertainty score normalized to $g_n^{(h)}$4 to characterize inter-model divergence (Carvalho et al., 29 Mar 2026).

For SC-ADAS, the combined lesson is that scene awareness involves not only perception and command selection, but also a model of what the driver knows and an auditable representation of semantic ambiguity.

6. Cooperative and distributed scene grounding

SC-ADAS has also been proposed as a cooperative vehicle-to-vehicle capability for occluded urban scenes. The central idea is that the ego vehicle need not receive raw sensor streams from a nearby vehicle; instead, it can query the remote vehicle’s onboard multimodal LLM about what that vehicle “sees.” This is presented as a communication-efficient alternative to sensor streaming. The paper gives the bandwidth intuition that transmitting a 218.6 KB image over a 1 Mbps link with 10% overhead would take about 1.98 seconds, whereas a detailed dialogue in text occupies far less data—roughly what would fill about 124 pages of text. The remote vehicle’s LLM acts as a communication interface, an abstraction layer over the remote sensors and internal ML stack, and a way to share only task-relevant scene descriptions rather than raw data (Dona et al., 2024).

The reported experiments use the Waymo Open Dataset, with a subset of 126 images with exactly one pedestrian on a crosswalk and 138 control images with no visible pedestrians on crosswalks, annotated using the VGG Image Annotator (VIA). Two models are compared: GPT-4V and GPT-4o. For binary pedestrian detection, GPT-4V achieves TP = 120, FP = 5, FN = 6, TN = 129, with Recall = 95.24%, Specificity = 96.27%, Precision = 96.00%, Accuracy = 95.77%, F1 = 95.62%, and MCC = 91.53%. GPT-4o achieves TP = 125, FP = 11, FN = 1, TN = 127, with Recall = 99.21%, Specificity = 92.03%, Precision = 91.91%, Accuracy = 95.45%, F1 = 95.42%, and MCC = 91.18% (Dona et al., 2024).

Localization is substantially weaker. Using normalized coordinates in the format $g_n^{(h)}$5, the overlap or union rate across three runs is 30.42% for GPT-4V and 42.18% for GPT-4o. When overlap exists, recall is 37.38% for GPT-4V and 49.76% for GPT-4o, with high standard deviations (28.81% and 35.77%, respectively). Prompt engineering produces only slight improvements that are reported as not statistically significant, and 52.94% of the missed cases occur under difficult lighting such as dusk, sunset, shade, or strong solar glare (Dona et al., 2024).

This line of work therefore positions LLM-based V2V dialogue as a semantic complement to perception rather than as a replacement for geometric localization. A plausible implication is that conversationally transmitted scene semantics may be useful for occlusion handling, but current models are not yet reliable for coordinate-level cooperative perception.

7. Empirical trade-offs, deployment constraints, and research directions

The 2025 SC-ADAS prototype makes the trade-offs of scene-aware conversation explicit. Implemented in CARLA v0.9.15 with local simulation and cloud-hosted GPT-4o on Azure OpenAI, it distinguishes Conversational-Only Service, Conversational ADAS Service, and Scene-Aware Conversational Service. In a $g_n^{(h)}$6 scene-understanding evaluation over highway (CARLA Town4) and downtown (Town3) settings under clear, rainy, and foggy conditions, the query “What speed do you recommend for this road?” produces the following recommendations: Highway: 65 MPH in clear weather, 40 MPH in rain, 30 MPH in fog; Downtown: 45 MPH in clear weather, 20 MPH in rain, 20 MPH in fog. However, the Scene-Aware Service has an average latency of approximately 11 seconds, about 8 seconds slower than the base Conversational-Only Service at 3 seconds. The Refiner and Responder typically contribute 2–5 seconds, the Actuator about 1 second, and Vision is the dominant and most variable cost, with end-to-end latencies ranging from 8 to 17 seconds in visually complex scenes. Input token counts grow over time as dialogue history accumulates, while output tokens remain relatively stable and generally below the 500-token maximum (Han et al., 14 Jul 2025).

The Autoware-based framework offers a different operating point. In translation evaluation over $g_n^{(h)}$7 spoken instructions, the baseline configuration with AV status, knowledge base, and in-context learning achieves 97.0% accuracy with mean translation time $g_n^{(h)}$8 and median 1.669 s. Ablations yield 96.0% for No AV Status, 73.0% for Zero-Shot, and 66.3% for KB-Only. In planning simulation, the full system executes a five-category scenario 30 times, succeeds in 29/30 runs, and reports Task Success Rate (TSR): 96.7%. Command Execution latency has mean $g_n^{(h)}$9 ms, median 0.461 ms, and standard deviation 8.491 ms, while Feedback Generation has mean $g_n^{(av)}$0 s, median 1.778 s, and standard deviation 0.355 s (Seegert et al., 9 Jan 2026).

Across the literature, deployment constraints remain prominent. The AVSD architectural studies report improvements mainly on offline text-generation metrics and do not address latency, safety, robustness, or open-world deployment constraints; they also identify richer multimodal fusion and better representations for objects, pose, and activity recognition as open directions (Kumar et al., 2019). The adaptive-communication study is careful that preferences for AdaCoM concern a hypothetical system, so support for adaptivity is not final evidence of real-world acceptance (Hasan et al., 2024). SituSYS demonstrates feasibility and real-time performance, but explicitly calls for a long-term empirical study of driving-performance impact and cognitive adequacy (Suchan et al., 2023). The cooperative V2V work calls for a standardized message interchange format between vehicles and further work on hallucination detection and mitigation (Dona et al., 2024). The scenario-centric risk-auditing framework argues that SC-ADAS must incorporate explicit ambiguity management because disagreement can arise from semantic underdetermination rather than component failure (Carvalho et al., 29 Mar 2026). The 2025 SC-ADAS prototype proposes future work on lightweight embedded models, Retrieval-Augmented Generation, and output validation for safety and reliability in real driving (Han et al., 14 Jul 2025).

Taken together, these results suggest that SC-ADAS is technically feasible as a modular synthesis of scene grounding, dialogue management, adaptive communication, structured control, and safety-aware validation. They also indicate that the field remains transitional: current systems can converse about scenes, recommend context-dependent actions, and mediate validated control, but latency, consistency, privacy, ambiguity handling, and safety-case construction remain unresolved research problems.