---
title: 'DriveAgent: LLM-Driven Multimodal Driving Analysis'
url: https://www.emergentmind.com/topics/driveagent
type: topic
---

# DriveAgent: LLM-Driven Multimodal Driving Analysis

DriveAgent is a **multi-agent, LLM-driven reasoning framework for autonomous driving** that fuses **multimodal sensor data**—camera, LiDAR, GPS, and IMU—with structured language-model reasoning. It is explicitly **not a low-level controller or an end-to-end driving policy**; instead, it sits on top of the perception stack and focuses on selecting and summarizing **critical moments** in a drive, performing **vehicle-level diagnostics** and **environment-level reasoning**, and producing **urgency-aware, interpretable decisions** or recommendations. Its central claim is that a hierarchy of specialized agents, coordinated by an LLM or VLM, can transform raw multimodal data into **structured, explainable understanding** and **prioritized action suggestions** for complex autonomous driving scenarios [2505.02123].

## 1. Conceptual scope and claimed novelty

DriveAgent decomposes driving analysis into four modules: descriptive analysis, vehicle reasoning, environmental reasoning, and response generation. The stated emphasis is not actuator control but **sensor-aware, interpretable diagnostics** and **causal explanation**. The framework therefore occupies a layer between conventional perception outputs and downstream planning, where it can explain what changed, what is inconsistent across sensors, which objects require caution, and which issue should be prioritized in the response [2505.02123].

The paper identifies four main contributions. First, it introduces a **multi-modal agent system** in which each stage of driving analysis is handled by dedicated agents. Second, it fine-tunes **LLaMA-3.2-Vision (11B)** with instruction-style supervision and structured prompts for object detection and traffic scene interpretation. Third, it defines **self-reasoning benchmarks** for vehicle reasoning and environmental reasoning, and compares prompting strategies with the proposed multi-agent pipeline. Fourth, it releases a **three-tier real-world dataset** with synchronized camera, LiDAR, and GNSS/IMU across three routes, designed to cover standard, typical, and more challenging scenarios [2505.02123].

What distinguishes DriveAgent from earlier LLM-for-driving systems, according to the paper, is the combination of **multi-agent, multi-stage LLM reasoning** with **multiple sensors simultaneously**, the explicit targeting of **sensor-aware, interpretable diagnostics** such as camera misalignment and LiDAR misplacement, the inclusion of **urgency-aware decision generation**, and the tight coupling of LLM reasoning to **explicit symbolic quantities** such as positions, motions, and discrepancies computed from sensor data. The paper characterizes this as moving beyond a setting in which an LLM merely reads images and answers a question, toward a **structured, pipeline-driven reasoning system over time-series, multimodal data** [2505.02123].

A recurrent misconception is that DriveAgent is a closed-loop autonomous driving policy. The paper rejects that interpretation directly: the system operates **open-loop** on logged sensor data, proposes maneuvers, and does not describe command transmission to a low-level controller. A plausible implication is that it is better understood as a **high-level reasoning and monitoring module** than as a replacement for the full planning-and-control stack.

## 2. Multi-agent architecture and information flow

The architecture is organized into four modules, denoted M1–M4. **Module 1 – Descriptive Analysis (T1)** selects *critical timestamps* along a route based on motion patterns derived from GPS/IMU. **Module 2 – Vehicle Reasoning (T2)** contains a **Vision agent** \( \mathcal{M}_V \), a **LiDAR agent** \( \mathcal{M}_L \), and an **Analyzer / diagnostic agent** \( \mathcal{M}_D \) that compares the two modalities and diagnoses vehicle or sensor abnormalities. **Module 3 – Environmental Reasoning (T3)** uses an environmental change agent and a **Causal analysis agent** \( \mathcal{M}_C \) to explain what changed between timestamps and why. **Module 4 – Response Generation (T4)** uses an **urgency-aware decision-generation agent** \( \mathcal{M}_R \) to rank issues and propose the best action [2505.02123].

The information flow is explicitly structured. Module 1 outputs critical times and triggers,
\[
\{(T_i, F_i)\}_{i=0}^n.
\]
Module 2 augments these with sensor-specific descriptions, producing LiDAR tuples \(\{(T_i, F_i, L_i)\}\), vision tuples \(\{(T_i, F_i, V_i)\}\), and diagnoses \(D_i\). Module 3 computes environmental changes \(E_{i+1}\) and caution flags \(C_i\). Module 4 merges \(D_i\) and \(C_i\) into final responses \(\mathcal{R}_i\) containing priority and recommended maneuvers. The resulting pipeline is summarized as: Sensors \(\rightarrow\) timestamp selection \(\rightarrow\) vehicle reasoning \(\rightarrow\) environment and causal reasoning \(\rightarrow\) urgency-aware decisions [2505.02123].

Module 1 formalizes route categorization using speed \(S\) and urban complexity \(U\):
\[
\mathcal{R}(S, U) \in \{\,r_1 :: \theta_1,\; r_2 :: \theta_2,\; r_3 :: \theta_3\}
\]
with \(r_1\) corresponding to high-speed, low-complexity routes, \(r_2\) to medium-speed, medium-complexity routes, and \(r_3\) to variable-speed, high-complexity routes. Thresholds are selected through
\[
\theta_i = G(S, U, r_i),
\]
where \(G\) adapts standard kinematic baselines such as angular velocity \(10^\circ/s\), linear acceleration \(8\,m/s^2\), and yaw rate \(10^\circ/s\). Threshold crossings define critical timestamps \(T_i\) and trigger factors \(F_i\) [2505.02123].

In vehicle-level fusion, the agents compute per-object motion as
\[
\Delta p_i = p_i(t+1) - p_i(t),
\]
restrict analysis to objects within LiDAR range \(R\) using
\[
\Omega = \{\, i \mid \|\mathbf{L}_i(t)\| \le R\,\},
\]
and measure cross-sensor consistency by
\[
\Delta_i(t) = \|\mathbf{L}_i(t) - \mathbf{C}_i(t)\|.
\]
Large \(\Delta_i(t)\) across many objects is treated as evidence of sensor misalignment or noise. In environmental reasoning, the framework defines visual changes \(\Delta v_i(t)\), LiDAR changes \(\Delta \ell_j(t)\), cross-sensor distances
\[
\Delta_{i,j}(t) = \|v_i(t) - \ell_j(t)\|,
\]
and long-interval object changes
\[
\Delta \mathbf{O}_i(t) = \mathbf{O}_i(t) - \mathbf{O}_i(t-\Delta t).
\]
These quantities are presented to the LLM through structured prompts, so the fusion is **mid-level / feature-level**, not raw-data fusion [2505.02123].

Response generation is formalized around a set of insights
\[
\mathcal{A} = \{(a_i, c_i)\}_{i=1}^{N},
\]
an urgency function \(\Psi(a_i, c_i)\), the most urgent issue
\[
\hat{a} = \arg\max_{(a_i, c_i) \in \mathcal{A}} \Psi(a_i, c_i),
\]
and an action set \(\Phi(\hat{a}) = \{\phi_1,\dots,\phi_M\}\), from which the selected action is
\[
\phi^* = \arg\max_{\phi_j \in \Phi(\hat{a})} \mathrm{Score}(\phi_j).
\]
The final response is
\[
\mathcal{R} = \bigl(\hat{a}, \phi^*, \mathcal{A}^-\bigr),
\]
with \(\mathcal{A}^- = \mathcal{A} \setminus \{\hat{a}\}\). The paper notes that \(\Psi\) and \(\mathrm{Score}\) are not explicitly given. It also states that the “agents” are largely **roles of one or more LLM instances plus auxiliary code**, coordinated in a **centralized, sequential** manner; it further states that this is conceptually similar to a **blackboard architecture** [2505.02123].

## 3. Sensor fusion, perception stack, and model implementation

DriveAgent uses a time-synchronized sensor suite consisting of **Robosense Ruby-128** LiDAR with 128 beams, 10 Hz, up to 230 m, and 360° horizontal field of view; **Basler acA2440-75uc** cameras at 10 Hz and \(2448 \times 2048\) resolution; and **Xsens MTI-680G** GNSS/INS, with GNSS at 4 Hz and IMU/RTK at 400 Hz and cm-level positioning. Synchronization is central because the framework depends on comparing camera- and LiDAR-derived object states at aligned timestamps [2505.02123].

The LiDAR branch uses **PointPillars** plus a clustering strategy for real-time 3D object detection and tracking, producing object labels together with 3D positions and sizes over time. The vision branch relies on a fine-tuned VLM to detect and classify **four-wheel vehicles, non-four-wheel vehicles, pedestrians, signs, fixed installations, plants, and monitors**, while also producing per-frame object positions \(p_i(t)\). GNSS/IMU signals are pre-processed into speed, acceleration, angular velocity, yaw rate, and related motion indicators for timestamp selection [2505.02123].

The VLM foundation is **LLaMA-3.2-Vision (11B)**, composed of a pre-trained vision tower, the **LLaMA-3.2 11B** language model, and a learnable linear projection layer mapping visual features into the LLM token space. The adaptation strategy fine-tunes both the vision encoder and the LLM using **LoRA**, trains the projection layer from scratch, and uses **AdamW** with learning rate \(2 \times 10^{-4}\), batch size 2, cosine learning-rate schedule with 0.03 warm-up ratio, and 10 epochs on instruction-style supervision, all on an **NVIDIA H100 GPU**. Prompts are formatted as instruction–response pairs with special `<Image>` tokens and **JSON-structured** fields containing sensor-derived features, natural-language task instructions, and role constraints for the active agent [2505.02123].

The paper stresses that structured outputs are enforced by **annotation guidelines** and role-specific prompt formats. This is important because later modules consume earlier outputs as structured facts rather than free-form prose. A descriptive analysis agent, for example, receives critical timestamps and triggers; the vehicle reasoning agent receives explicit discrepancies \(\Delta_i(t)\); and the response-generation stage receives categorized insights for urgency ranking. This suggests that the LLM is not being used as a raw perception engine, but as a reasoning layer tightly conditioned on numerically grounded intermediate representations.

## 4. Dataset, benchmark tasks, and empirical performance

The dataset contains three real-world routes. **R1 (1277.76 m)** is a controlled environment with small environmental dynamics, forward camera only, and higher average speed of approximately \(7.3\,\text{m/s}\). **R2 (969.19 m)** loops around an urban square, has large environmental dynamics and more traffic activity, uses forward plus left and right cameras, and has lower average speed of approximately \(4.17\,\text{m/s}\). **R3 (1125.91 m)** has medium dynamics and roadside obstructions, also uses forward plus left and right cameras, and has average speed approximately \(4.29\,\text{m/s}\). All routes have synchronized LiDAR, cameras, and GNSS/IMU [2505.02123].

The evaluation is organized into three tasks. **Task 1 – Object and Category Detection** is trained on R2 and R3 and evaluated on R1 using Precision, Recall, and F1. **Task 2 – Vehicle Reasoning** tests LiDAR misplacement diagnosis and misaligned camera-view detection on R2 and R3, including left and right variants. **Task 3 – Environmental Reasoning** asks the system to identify independently moving objects versus static ones by comparing two timestamps on R2 and R3. Baselines include **LLaMA-3.2-Vision-Instruct**, **GPT-4o-mini**, **Pixtra-large**, **GPT-4o**, and **Claude-3.7-Sonnet** for Task 1, and **Zero-Shot**, **CoT**, and **CoT + Self-Refine** for Tasks 2 and 3 [2505.02123].

The main quantitative results are summarized below.

| Task | Setting | Result |
|---|---|---|
| Task 1 | DriveAgent (fine-tuned VLM) | \(P=\mathbf{89.96}\%\), \(R=63.52\%\), \(F1=\mathbf{71.62}\) |
| Task 2 | Vision reasoning, R2 | \(96.84\%\) |
| Task 2 | Vision reasoning, R3 | \(87.62\%\) |
| Task 3 | Environmental reasoning, R2 | \(58.95\%\) |
| Task 3 | Environmental reasoning, R3 | \(65.71\%\) |

For **Task 1**, the fine-tuned DriveAgent VLM achieves the best precision and best F1 among the reported models, surpassing **Claude-3.7-Sonnet** on F1 and recall-weighted performance while retaining markedly higher precision. The paper further states that DriveAgent is the **only model that consistently detects “monitors”**, described as overhead displays or cameras, which other models often miss. That observation is used to argue for the effect of structured annotations and domain-specific training [2505.02123].

For **Task 2**, the strongest gains appear in **vision reasoning** for camera misalignment. Zero-Shot and CoT prompting perform very poorly on left/right distractor views, sometimes as low as \(0\%\)–\(3\%\), whereas DriveAgent obtains \(96.84\%\) on R2, \(58.25\%\) on R2-left, \(71.30\%\) on R2-right, \(87.62\%\) on R3, \(68.75\%\) on R3-left, and \(63.16\%\) on R3-right. In **LiDAR reasoning**, the paper characterizes DriveAgent as competitive and more stable, with best performance on some variants such as \(69.90\%\) on R2-left and \(51.25\%\) on R3-left, while avoiding the catastrophic drops seen in some prompting baselines [2505.02123].

For **Task 3**, DriveAgent yields the best reported environmental-reasoning accuracy on both routes: \(58.95\%\) on R2 versus \(37.89\%\) for Zero-Shot, \(56.84\%\) for CoT, and \(43.16\%\) for CoT+Self-Refine; and \(65.71\%\) on R3 versus \(36.19\%\), \(62.86\%\), and \(56.19\%\), respectively. Although the paper does not present classical ablations, the comparison with Zero-Shot, CoT, and CoT+Self-Refine suggests that plain prompting is insufficient for nuanced multimodal reasoning and that the explicit multi-agent pipeline yields more robust behavior [2505.02123].

## 5. Interpretability, robustness, and limitations

Interpretability is one of DriveAgent’s central design goals. Each module produces **human-readable diagnostic statements** about sensor disagreement, environmental change, and causal explanations, while the final response explicitly lists the most pressing issue \(\hat{a}\), the selected action \(\phi^*\), and the remaining issues \(\mathcal{A}^-\) [2505.02123].

The paper’s examples illustrate how this structure supports diagnosis. In a **misaligned camera views** case, persistent large \(\Delta_i(t)\) across many objects leads the diagnostic agent to conclude “camera misalignment” rather than “environment changed”, with a medium urgency and a recommendation such as “recalibrate camera / rely more on LiDAR.” In an **independently moving pedestrian** case, the environmental reasoning stage uses \(\Delta \mathbf{O}_i(t)\) to separate pedestrian motion from static background and recommends “slow down / prepare to yield” with high urgency. The paper’s broader point is that the decisions are explainable because they are grounded in named objects, explicit motion quantities, and causal reasoning steps [2505.02123].

Robustness is evaluated indirectly through two kinds of difficulty. First, **misaligned camera views** act as noisy, conflicting inputs; most baselines fail to recognize the misalignment, while DriveAgent diagnoses it with substantially higher accuracy. Second, the three routes expose the system to small, medium, and large environmental dynamics. The paper argues that the use of cross-sensor consistency measures such as \(\Delta_i(t)\) and \(\Delta_{i,j}(t)\) makes the system more robust to single-sensor failures because disagreement can itself become an object of reasoning [2505.02123].

The limitations are equally explicit. The framework is **not real-time closed-loop**; the paper provides no latency numbers or real-time guarantees. It remains dependent on **LLM correctness**, so hallucinated or numerically misinterpreted reasoning remains possible even with structured prompts. Dataset coverage is limited to **three routes in one city**, without night, rain, snow, or multi-lane highway settings. There are **no formal safety guarantees**, and the action-selection functions \(\Psi\) and \(\mathrm{Score}\) are presented abstractly rather than instantiated in closed form [2505.02123].

A second misconception is that “multi-agent” here necessarily means multiple independently trained foundation models. The paper instead states that there is **no separate LLM per agent**; the same LLaMA-3.2-Vision base is reused with different prompts and tools per module. This matters because DriveAgent’s novelty lies less in model plurality than in **prompted role specialization plus structured inter-agent dataflow**.

## 6. Relation to adjacent autonomous-driving agent paradigms

Within the broader LLM-for-driving literature, DriveAgent occupies a distinctive niche. **Agent-Driver**, introduced as “A Language Agent for Autonomous Driving,” also places an LLM above a conventional perception stack and adds a tool library, cognitive memory, chain-of-thought reasoning, task planning, motion planning, and self-reflection on the **nuScenes** benchmark [2311.10813]. By contrast, DriveAgent emphasizes **multimodal sensor fusion across camera, LiDAR, GPS, and IMU**, **sensor-aware diagnostics**, and **urgency-aware recommendation** rather than direct language-generated trajectory planning [2505.02123].

Relative to end-to-end systems such as **DriveTransformer**, which unifies detection, prediction, mapping, and planning in a single transformer with **Task Parallelism**, **Sparse Representation**, and **Streaming Processing**, DriveAgent remains explicitly modular and open-loop [2503.07656]. This difference is methodological: DriveTransformer targets scalable end-to-end policy learning, whereas DriveAgent targets explainable reasoning over structured multimodal features and time-indexed events.

Subsequent work further diversifies the design space. **DriveAgent-R1** introduces **active perception** and **hybrid thinking** for VLM-based high-level planning, allowing the agent to switch between text-only reasoning and tool-augmented visual reasoning based on scene complexity [2507.20879]. **DriveJudge** moves in another direction, using a VLM to decide which evaluation rules are useful in a given context while retaining deterministic physical metrics for scoring [2606.17362]. These systems indicate that “driving agents” now span perception, planning, diagnosis, evaluation, and scheduling, but DriveAgent’s contribution remains its tight coupling of LLM reasoning to symbolic, cross-sensor, time-series evidence [2505.02123].

The paper itself suggests that DriveAgent can sit alongside a traditional planner as a **high-level reasoning and monitoring module**. A plausible implication is that its longer-term significance lies not in replacing existing autonomous-driving stacks, but in furnishing those stacks with a layer that can inspect multimodal evidence, diagnose sensor inconsistencies, explain environmental change, and prioritize responses in human-readable form.

Source: https://www.emergentmind.com/topics/driveagent