---
title: Scene-Aware Conversational ADAS
url: https://www.emergentmind.com/topics/scene-aware-conversational-adas-sc-adas
type: topic
---

# Scene-Aware Conversational ADAS

Scene-Aware Conversational ADAS (SC-ADAS) denotes a class of driver-assistance systems in which natural-language interaction is grounded in the driving scene, the driver’s attentional state, and executable assistance functions rather than in fixed alerts or isolated voice commands. The term is used explicitly for a modular framework that integrates Generative AI components including large language models, vision-to-text interpretation, and structured function calling to enable real-time, interpretable, and adaptive driver assistance; closely related work defines an “Adaptive Communication Module for Driver Assistive Systems” that adapts communication to the driver’s psycho-physiological attributes and the environment, and also proposes vehicle-to-vehicle dialogue with a remote onboard multimodal LLM to complement the ego vehicle’s field-of-view in occluded urban scenes [2507.10500, 2403.03312, 2408.10794].

## 1. Conceptual scope and problem formulation

SC-ADAS emerges from two recurring critiques of conventional assistance systems. First, current ADAS are described as “largely rule-based and reactive,” lacking the ability to engage drivers through natural language conversation, interpret evolving road scenes, or adapt behavior in real time based on situational context and driver intent. Second, much existing in-vehicle communication follows a “one mode for all scenarios paradigm,” even though the appropriateness of a warning or explanation depends on both scenario hazard and the driver’s current distraction state [2507.10500, 2403.03312].

Within that problem formulation, conversational assistance is not restricted to a voice wrapper around fixed functions. In the 2025 SC-ADAS formulation, the system supports multi-turn dialogue grounded in visual and sensor context, produces natural-language recommendations, and executes vehicle-level actuation only after explicit driver confirmation. In the 2026 Autoware-based framework, the interaction space is taxonomized into five categories: Information, Mission Control, Configuration, Cooperation, and Intervention. These categories span status queries, global operating-state changes, long-term preference updates, local maneuver requests, and attempts to correct a perception or planning error such as “The traffic light is green” or “Emergency, stop now” [2507.10500, 2601.05806].

Taken together, these studies suggest that SC-ADAS should be understood as an interaction layer over scene interpretation and control validation, not merely as speech recognition plus command execution. Its central research question is how to align scene understanding, driver state, dialogue history, and safe actuation in a single closed loop.

## 2. Lineage in audio-visual scene-aware dialogue

The technical lineage of SC-ADAS is closely tied to Audio Visual Scene-aware Dialog (AVSD), which extended end-to-end dialogue modeling from text-only and image-only settings to dynamic audio-visual scenes. AVSD frames the task as generating the next answer in a dialogue about a video, conditioned on the current question, the audio-visual scene representation, and the dialogue history. In the 2018 formulation, the full context vector for the \(n\)-th answer is

$$
g_n = [g_n^{(q)}, g_n^{(av)}, g_n^{(h)}],
$$

where \(g_n^{(q)}\) is the current question encoding, \(g_n^{(h)}\) is the hierarchical-LSTM history encoding, and \(g_n^{(av)}\) is the audio-visual video encoding. Rather than a fixed fusion rule, the model uses multimodal attention:

$$
g_n^{(av)} = \tanh\left(\sum_{k=1}^K \beta_{k,n} d_{k,n}\right), \qquad
\beta_{k,n}=\frac{\exp(v_{k,n})}{\sum_{\kappa=1}^{K}\exp(v_{\kappa,n})}.
$$

This allows the decoder to emphasize different modalities depending on the question context. On AVSD response generation, the best reported configuration in the table is **QA + I3D + VGGish** with attention, achieving **BLEU-1 0.256, BLEU-2 0.161, BLEU-3 0.109, BLEU-4 0.078, METEOR 0.113, ROUGE-L 0.277, CIDEr 0.727**. The paper’s broader conclusion is that multimodal features help, and multimodal attention helps more than naive fusion [1806.08409].

The dataset design is equally important. AVSD was collected from the Charades video corpus as typed conversations between two AMT workers, one of whom saw the full video while the other saw only the first, middle, and last frames. Each dialogue contains exactly **10 question-answer pairs**, followed by a summary description. One reported split contains **6,172 dialogs** for training, **732 dialogs** for validation, and **733 dialogs** for test; the later DSTC7-based work uses the official **training set (7,659 dialogs)**, **prototype-validation set (732 dialogs)**, and **prototype-test set (733 dialogs)** [1806.08409, 1912.10132].

The 2019 extension deepens this lineage in three ways. It treats dialog topics as contextual state using **Latent Dirichlet Allocation (LDA)** and **Guided LDA** over questions, answers, QA pairs, captions, history, and history+captions; it explores attention over dialog history and multimodal features; and it incorporates **AclNet**, an end-to-end audio classification ConvNet trained on **ESC-50** that takes raw, amplitude-normalized **44.1 kHz** audio samples as input. In the topic experiments, the baseline scores are **Bleu1 0.273, Bleu2 0.173, Bleu3 0.118, Bleu4 0.084, Meteor 0.117, Rouge 0.291, CIDEr 0.766**. Guided LDA reaches **CIDEr 0.812** with **Bleu1 0.265, Bleu2 0.170, Bleu3 0.117, Bleu4 0.084, Meteor 0.118, Rouge 0.293, CIDEr 0.812**, and the authors state that “GuidedLDA + GloVe is our best performing model.” The same paper reports that attending over all sentence-LSTM output states outperforms the baseline on **BLEU3, BLEU4, METEOR, CIDEr and ROUGE**, while **B+AclNet performs the best both on the overall dataset and on audio-related dialogs** [1912.10132].

For SC-ADAS, the relevance of AVSD is foundational rather than automotive-specific. It establishes the core computational problem of grounding multi-turn dialogue in dynamic scene evidence, audio, and conversational memory.

## 3. Modular conversational-control architectures

The explicit SC-ADAS architecture presented in 2025 decomposes the system into four Generative AI-powered modules: **Query Refinement**, **Context Retrieval**, **Response Generation**, and **Command Generation**. The driver’s spoken query is converted to text, refined into intent plus metadata indicating whether sensing context or actuation is required, optionally augmented with real-time sensor data and vision-to-text scene descriptions, and then used to generate a natural-language response. If the driver confirms an action, structured function calling produces an executable ADAS command. The paper illustrates the actuation interface with a function-call output such as:

```json
{ "name": "set_speed", "arguments": { "speed": 50 } }
```

This design separates recommendation from control: conversation may suggest an action, but execution is routed through structured output only after confirmation [2507.10500].

The same separation appears in the 2026 Autoware-based framework, but with a stronger emphasis on deterministic validation. A spoken instruction \(I\), AV status information, and a knowledge base are passed to an LLM, which translates the request into an application-centric DSL. The target policy space is described as

$$
P = \{A, C\},
$$

with \(A\) finite actions and \(C\) continuous parameters. The structured command is then parsed by a **ROS2 Interface Node** that validates the action, enforces parameter bounds, maps the DSL explicitly to allowed Autoware operations, and returns the definitive success or failure state to the LLM for second-stage feedback generation. The paper stresses that the system sits on top of Autoware’s existing safety architecture and that the LLM never directly drives the vehicle [2601.05806].

This dual pattern—LLM-mediated interpretation followed by deterministic validation—recurs across the literature. In SC-ADAS, dialogue is used for intent understanding, recommendation, and explanation; validated machine-readable commands remain the boundary to actuation. A plausible implication is that conversational naturalness and control authority are deliberately decoupled to preserve transparency and safety.

## 4. Adaptive communication and driver state awareness

A distinct line of work treats the conversational or communicative problem less as command execution and more as adaptive alert selection. In an online user study with **\(N=147\)** participants, **144** of whom were retained for analysis, participants viewed **30** simulated driving videos recorded in the **CARLA Driving Simulator** from the driver’s perspective, including the windshield view and the left side mirror, rear-view mirror, and right side mirror. The scenarios, ordered by decreasing risk, were **Pedestrian collision**, **Car collision**, **Lane deviation**, **Blind spot obstruction**, and **Stagnant at a green light**, each under **clear day**, **rainy day**, and **night** conditions. Augmented versions manipulated the driver’s apparent attention through **Drowsiness**, **Hyper-focus on the road**, and **Salient external agents** [2403.03312].

Participants ranked their top three preferred communication modes from a set including non-flashing and flashing dashboard icons, flashing mirror icons, spoken language alerts, a single beep, multiple beeps, steering-wheel vibrations, seat vibrations, text alerts, and image alerts. Preferences varied systematically with scene hazard. For **high-risk / emergent scenarios** such as **pedestrian collisions** and **car collisions**, participants preferred **repeating sound alerts** and **vibrations**. For **lower-risk scenarios** such as **stagnation at a green light**, they preferred a **single beep** and **spoken language alerts**. The least preferred modes across scenarios were **non-flashing dashboard icons**, **text alerts**, and **image alerts** on the dashboard or console [2403.03312].

The paper also quantifies how preferences change when the same scene is presented under altered attention states. After converting top-3 choices into full rankings, the authors report average correlations between normal and augmented versions of **Kendall’s \(\tau = 0.569 \pm 0.372\)**, **Spearman’s \(\rho = 0.603 \pm 0.383\)**, and **RBO \(= 0.619 \pm 0.309\)**, with **\(p < 0.0004\)** for all metrics. The correlations are described as only moderately positive, indicating that hazard structure persists but does not fully determine communication preference under distraction. Statistical support is also reported for scenario dependence via **one-way ANOVA** (\(F = 28.19\), \(p < 0.05\)) and **chi-square goodness-of-fit** (\(\chi^2(3,5) = 158.85\), \(p < 0.001\)) [2403.03312].

The study’s conceptual contribution is the **Adaptive Communication Module for Driver Assistive Systems**, defined as technology that “learns and adapts the mode of communication of information to be relayed by Assistive Systems to the changes in the driver’s psycho-physiological attributes while taking into consideration the environment around the driver (and their vehicle) as well as the driver’s communication preferences.” Support for adaptivity is substantial but not unqualified: mean perceived safety with such a module is **\(3.93 \pm 0.91\)** as driver and **\(3.95 \pm 1.03\)** as passenger; **over 80%** reported feeling safer as a driver and **over 77%** as a passenger. At the same time, mean comfort with **audio recordings** is **\(2.32 \pm 1.20\)** and with **video recordings** **\(2.14 \pm 1.19\)**, with more than **60%** uncomfortable with audio recording and **67%** uncomfortable with video recording. The trust hypothesis is reported as **inconclusive** [2403.03312].

## 5. Situation awareness, symbolic reasoning, and audited risk interpretation

Another SC-ADAS strand emphasizes modeling what the driver has perceived and understood, especially in takeover situations. In semi-autonomous driving at **SAE level 3/4**, the challenge is that the driver may need to rebuild situation awareness within the commonly cited **7–10 seconds** of takeover time. The **SituSYS** framework, developed in the **SituWare** project, addresses this through a **ROS-based modular system** with three layers: **SituSENSORS**, **SituMODEL**, and **SituHMI**. It senses the environment and driver gaze, estimates situation awareness, and directs attention through specialized HMIs [2308.15895].

The formal core is an **ASP-based** representation of the driving domain,

$$
\Sigma = \langle \mathcal{O}, \mathcal{E}, \mathcal{R}, \mathcal{T}, \Phi, \Theta \rangle,
$$

where \(\mathcal{O}\) are domain objects, \(\mathcal{E}\) spatial entities, \(\mathcal{R}\) spatial relations, \(\mathcal{T}\) time points, \(\Phi\) fluents, and \(\Theta\) events. Scene data include ego-vehicle state, automation state, other-vehicle properties, and fixation information; gaze is measured with a **PupilLabs mobile eye tracker** at up to **120 Hz** with about **0.60° accuracy**. The reasoning pipeline updates a mental belief state using fixation thresholds and a **Kalman filter motion model**, constructs an ASP problem, and solves it with **Clingo** to obtain an interpretation model \(\mathcal{IM}\) and projection model \(\mathcal{PM}\). The system is reported to run in real time at about **30 Hz** and to handle scenes with **up to 20 vehicles plus the ego vehicle** [2308.15895].

Where SituSYS models a specific driver’s awareness, the 2026 scenario-centric framework audits how LLMs interpret fixed driving situations. Raw multimodal logs are converted into a normalized semantic layer at **1 Hz**, then materialized into deterministic scenario windows according to

$$
\mathcal{S} \;=\; \mathcal{B}\!\left(\mathcal{U}(a,t),\, \theta\right).
$$

For the reported evaluation, each scenario uses a symmetric **3-second pre** and **3-second post** window around an anchor time, yielding **seven-second scenario windows** across a curated **near-people scenario set** of **16** cases. Models are constrained to use only provided structured data and images if available, and must return **JSON-only** outputs with numeric codes. The closed risk taxonomy defines `overall_risk_level` from **0 Not identified** through **6 Critical**, and **high risk** is explicitly defined as **overall risk level \(\ge 4\)** [2603.27536].

Under these constraints, **GPT-4o-vision** assigns higher average risk and escalates a larger fraction of windows to high risk than the text-only **GPT-4o-mini** and **DeepSeek-Chat** models. The text-only models cite more evidence signals on average, while all models agree that pedestrian-related risk dominates the near-people dataset but differ in secondary attribution: **DeepSeek-Chat** emphasizes **rear-end conflict**, **GPT-4o-mini** emphasizes **lateral conflict**, and **GPT-4o-vision** emphasizes **traffic density**. The paper interprets this as evidence of **intrinsic semantic indeterminacy** rather than isolated model failure, and introduces a composite uncertainty score normalized to \([0,1]\) to characterize inter-model divergence [2603.27536].

For SC-ADAS, the combined lesson is that scene awareness involves not only perception and command selection, but also a model of what the driver knows and an auditable representation of semantic ambiguity.

## 6. Cooperative and distributed scene grounding

SC-ADAS has also been proposed as a cooperative vehicle-to-vehicle capability for occluded urban scenes. The central idea is that the ego vehicle need not receive raw sensor streams from a nearby vehicle; instead, it can query the remote vehicle’s onboard multimodal LLM about what that vehicle “sees.” This is presented as a communication-efficient alternative to sensor streaming. The paper gives the bandwidth intuition that transmitting a **218.6 KB image** over a **1 Mbps link with 10% overhead** would take about **1.98 seconds**, whereas a detailed dialogue in text occupies far less data—roughly what would fill about **124 pages of text**. The remote vehicle’s LLM acts as a communication interface, an abstraction layer over the remote sensors and internal ML stack, and a way to share only task-relevant scene descriptions rather than raw data [2408.10794].

The reported experiments use the **Waymo Open Dataset**, with a subset of **126 images with exactly one pedestrian on a crosswalk** and **138 control images with no visible pedestrians on crosswalks**, annotated using the **VGG Image Annotator (VIA)**. Two models are compared: **GPT-4V** and **GPT-4o**. For binary pedestrian detection, **GPT-4V** achieves **TP = 120, FP = 5, FN = 6, TN = 129**, with **Recall = 95.24%**, **Specificity = 96.27%**, **Precision = 96.00%**, **Accuracy = 95.77%**, **F1 = 95.62%**, and **MCC = 91.53%**. **GPT-4o** achieves **TP = 125, FP = 11, FN = 1, TN = 127**, with **Recall = 99.21%**, **Specificity = 92.03%**, **Precision = 91.91%**, **Accuracy = 95.45%**, **F1 = 95.42%**, and **MCC = 91.18%** [2408.10794].

Localization is substantially weaker. Using normalized coordinates in the format \((X,Y), (X',Y')\), the overlap or union rate across three runs is **30.42%** for **GPT-4V** and **42.18%** for **GPT-4o**. When overlap exists, recall is **37.38%** for **GPT-4V** and **49.76%** for **GPT-4o**, with high standard deviations (**28.81%** and **35.77%**, respectively). Prompt engineering produces only slight improvements that are reported as **not statistically significant**, and **52.94% of the missed cases** occur under difficult lighting such as dusk, sunset, shade, or strong solar glare [2408.10794].

This line of work therefore positions LLM-based V2V dialogue as a semantic complement to perception rather than as a replacement for geometric localization. A plausible implication is that conversationally transmitted scene semantics may be useful for occlusion handling, but current models are not yet reliable for coordinate-level cooperative perception.

## 7. Empirical trade-offs, deployment constraints, and research directions

The 2025 SC-ADAS prototype makes the trade-offs of scene-aware conversation explicit. Implemented in **CARLA v0.9.15** with local simulation and cloud-hosted **GPT-4o** on Azure OpenAI, it distinguishes **Conversational-Only Service**, **Conversational ADAS Service**, and **Scene-Aware Conversational Service**. In a \(2 \times 3\) scene-understanding evaluation over highway (**CARLA Town4**) and downtown (**Town3**) settings under **clear**, **rainy**, and **foggy** conditions, the query “What speed do you recommend for this road?” produces the following recommendations: **Highway**: **65 MPH** in clear weather, **40 MPH** in rain, **30 MPH** in fog; **Downtown**: **45 MPH** in clear weather, **20 MPH** in rain, **20 MPH** in fog. However, the **Scene-Aware Service** has an average latency of approximately **11 seconds**, about **8 seconds** slower than the base **Conversational-Only Service** at **3 seconds**. The **Refiner** and **Responder** typically contribute **2–5 seconds**, the **Actuator** about **1 second**, and **Vision** is the dominant and most variable cost, with end-to-end latencies ranging from **8 to 17 seconds** in visually complex scenes. Input token counts grow over time as dialogue history accumulates, while output tokens remain relatively stable and generally below the **500-token** maximum [2507.10500].

The Autoware-based framework offers a different operating point. In translation evaluation over **\(N = 200\)** spoken instructions, the baseline configuration with AV status, knowledge base, and in-context learning achieves **97.0% accuracy** with mean translation time **\(t_r = 1.723\,\text{s}\)** and median **1.669 s**. Ablations yield **96.0%** for **No AV Status**, **73.0%** for **Zero-Shot**, and **66.3%** for **KB-Only**. In planning simulation, the full system executes a five-category scenario **30 times**, succeeds in **29/30** runs, and reports **Task Success Rate (TSR): 96.7%**. **Command Execution** latency has mean **\(t = 5.737\) ms**, median **0.461 ms**, and standard deviation **8.491 ms**, while **Feedback Generation** has mean **\(t = 1.846\) s**, median **1.778 s**, and standard deviation **0.355 s** [2601.05806].

Across the literature, deployment constraints remain prominent. The AVSD architectural studies report improvements mainly on offline text-generation metrics and do not address latency, safety, robustness, or open-world deployment constraints; they also identify richer multimodal fusion and better representations for **objects, pose, and activity recognition** as open directions [1912.10132]. The adaptive-communication study is careful that preferences for AdaCoM concern a **hypothetical system**, so support for adaptivity is not final evidence of real-world acceptance [2403.03312]. SituSYS demonstrates feasibility and real-time performance, but explicitly calls for a **long-term empirical study** of driving-performance impact and cognitive adequacy [2308.15895]. The cooperative V2V work calls for a **standardized message interchange format** between vehicles and further work on hallucination detection and mitigation [2408.10794]. The scenario-centric risk-auditing framework argues that SC-ADAS must incorporate **explicit ambiguity management** because disagreement can arise from semantic underdetermination rather than component failure [2603.27536]. The 2025 SC-ADAS prototype proposes future work on lightweight embedded models, Retrieval-Augmented Generation, and output validation for safety and reliability in real driving [2507.10500].

Taken together, these results suggest that SC-ADAS is technically feasible as a modular synthesis of scene grounding, dialogue management, adaptive communication, structured control, and safety-aware validation. They also indicate that the field remains transitional: current systems can converse about scenes, recommend context-dependent actions, and mediate validated control, but latency, consistency, privacy, ambiguity handling, and safety-case construction remain unresolved research problems.

Source: https://www.emergentmind.com/topics/scene-aware-conversational-adas-sc-adas