DriveAgent: LLM-Driven Multimodal Driving Analysis

• DriveAgent is a multimodal LLM reasoning framework that integrates camera, LiDAR, GPS, and IMU data to generate structured, explainable diagnostic insights.
• Its architecture decomposes driving analysis into four modules—descriptive analysis, vehicle reasoning, environmental reasoning, and response generation—to detect sensor discrepancies and prioritize actionable recommendations.
• The system leverages fine-tuned LLaMA-3.2-Vision, self-reasoning benchmarks, and real-world datasets to achieve high precision and interpretability in diagnosing complex autonomous driving scenarios.

DriveAgent is a multi-agent, LLM-driven reasoning framework for autonomous driving that fuses multimodal sensor data—camera, LiDAR, GPS, and IMU—with structured language-model reasoning. It is explicitly not a low-level controller or an end-to-end driving policy; instead, it sits on top of the perception stack and focuses on selecting and summarizing critical moments in a drive, performing vehicle-level diagnostics and environment-level reasoning, and producing urgency-aware, interpretable decisions or recommendations. Its central claim is that a hierarchy of specialized agents, coordinated by an LLM or VLM, can transform raw multimodal data into structured, explainable understanding and prioritized action suggestions for complex autonomous driving scenarios (Hou et al., 4 May 2025).

1. Conceptual scope and claimed novelty

DriveAgent decomposes driving analysis into four modules: descriptive analysis, vehicle reasoning, environmental reasoning, and response generation. The stated emphasis is not actuator control but sensor-aware, interpretable diagnostics and causal explanation. The framework therefore occupies a layer between conventional perception outputs and downstream planning, where it can explain what changed, what is inconsistent across sensors, which objects require caution, and which issue should be prioritized in the response (Hou et al., 4 May 2025).

The paper identifies four main contributions. First, it introduces a multi-modal agent system in which each stage of driving analysis is handled by dedicated agents. Second, it fine-tunes LLaMA-3.2-Vision (11B) with instruction-style supervision and structured prompts for object detection and traffic scene interpretation. Third, it defines self-reasoning benchmarks for vehicle reasoning and environmental reasoning, and compares prompting strategies with the proposed multi-agent pipeline. Fourth, it releases a three-tier real-world dataset with synchronized camera, LiDAR, and GNSS/IMU across three routes, designed to cover standard, typical, and more challenging scenarios (Hou et al., 4 May 2025).

What distinguishes DriveAgent from earlier LLM-for-driving systems, according to the paper, is the combination of multi-agent, multi-stage LLM reasoning with multiple sensors simultaneously, the explicit targeting of sensor-aware, interpretable diagnostics such as camera misalignment and LiDAR misplacement, the inclusion of urgency-aware decision generation, and the tight coupling of LLM reasoning to explicit symbolic quantities such as positions, motions, and discrepancies computed from sensor data. The paper characterizes this as moving beyond a setting in which an LLM merely reads images and answers a question, toward a structured, pipeline-driven reasoning system over time-series, multimodal data (Hou et al., 4 May 2025).

A recurrent misconception is that DriveAgent is a closed-loop autonomous driving policy. The paper rejects that interpretation directly: the system operates open-loop on logged sensor data, proposes maneuvers, and does not describe command transmission to a low-level controller. A plausible implication is that it is better understood as a high-level reasoning and monitoring module than as a replacement for the full planning-and-control stack.

2. Multi-agent architecture and information flow

The architecture is organized into four modules, denoted M1–M4. Module 1 – Descriptive Analysis (T1) selects critical timestamps along a route based on motion patterns derived from GPS/IMU. Module 2 – Vehicle Reasoning (T2) contains a Vision agent $\mathcal{M}_V$, a LiDAR agent $\mathcal{M}_L$, and an Analyzer / diagnostic agent $\mathcal{M}_D$ that compares the two modalities and diagnoses vehicle or sensor abnormalities. Module 3 – Environmental Reasoning (T3) uses an environmental change agent and a Causal analysis agent $\mathcal{M}_C$ to explain what changed between timestamps and why. Module 4 – Response Generation (T4) uses an urgency-aware decision-generation agent $\mathcal{M}_R$ to rank issues and propose the best action (Hou et al., 4 May 2025).

The information flow is explicitly structured. Module 1 outputs critical times and triggers,

$\{(T_i, F_i)\}_{i=0}^n.$

Module 2 augments these with sensor-specific descriptions, producing LiDAR tuples $\{(T_i, F_i, L_i)\}$, vision tuples $\{(T_i, F_i, V_i)\}$, and diagnoses $D_i$. Module 3 computes environmental changes $E_{i+1}$ and caution flags $\mathcal{M}_L$0. Module 4 merges $\mathcal{M}_L$1 and $\mathcal{M}_L$2 into final responses $\mathcal{M}_L$3 containing priority and recommended maneuvers. The resulting pipeline is summarized as: Sensors $\mathcal{M}_L$4 timestamp selection $\mathcal{M}_L$5 vehicle reasoning $\mathcal{M}_L$6 environment and causal reasoning $\mathcal{M}_L$7 urgency-aware decisions (Hou et al., 4 May 2025).

Module 1 formalizes route categorization using speed $\mathcal{M}_L$8 and urban complexity $\mathcal{M}_L$9: $\mathcal{M}_D$0 with $\mathcal{M}_D$1 corresponding to high-speed, low-complexity routes, $\mathcal{M}_D$2 to medium-speed, medium-complexity routes, and $\mathcal{M}_D$3 to variable-speed, high-complexity routes. Thresholds are selected through

$\mathcal{M}_D$4

where $\mathcal{M}_D$5 adapts standard kinematic baselines such as angular velocity $\mathcal{M}_D$6, linear acceleration $\mathcal{M}_D$7, and yaw rate $\mathcal{M}_D$8. Threshold crossings define critical timestamps $\mathcal{M}_D$9 and trigger factors $\mathcal{M}_C$0 (Hou et al., 4 May 2025).

In vehicle-level fusion, the agents compute per-object motion as

$\mathcal{M}_C$1

restrict analysis to objects within LiDAR range $\mathcal{M}_C$2 using

$\mathcal{M}_C$3

and measure cross-sensor consistency by

$\mathcal{M}_C$4

Large $\mathcal{M}_C$5 across many objects is treated as evidence of sensor misalignment or noise. In environmental reasoning, the framework defines visual changes $\mathcal{M}_C$6, LiDAR changes $\mathcal{M}_C$7, cross-sensor distances

$\mathcal{M}_C$8

and long-interval object changes

$\mathcal{M}_C$9

These quantities are presented to the LLM through structured prompts, so the fusion is mid-level / feature-level, not raw-data fusion (Hou et al., 4 May 2025).

Response generation is formalized around a set of insights

$\mathcal{M}_R$0

an urgency function $\mathcal{M}_R$1, the most urgent issue

$\mathcal{M}_R$2

and an action set $\mathcal{M}_R$3, from which the selected action is

$\mathcal{M}_R$4

The final response is

$\mathcal{M}_R$5

with $\mathcal{M}_R$6. The paper notes that $\mathcal{M}_R$7 and $\mathcal{M}_R$8 are not explicitly given. It also states that the “agents” are largely roles of one or more LLM instances plus auxiliary code, coordinated in a centralized, sequential manner; it further states that this is conceptually similar to a blackboard architecture (Hou et al., 4 May 2025).

3. Sensor fusion, perception stack, and model implementation

DriveAgent uses a time-synchronized sensor suite consisting of Robosense Ruby-128 LiDAR with 128 beams, 10 Hz, up to 230 m, and 360° horizontal field of view; Basler acA2440-75uc cameras at 10 Hz and $\mathcal{M}_R$9 resolution; and Xsens MTI-680G GNSS/INS, with GNSS at 4 Hz and IMU/RTK at 400 Hz and cm-level positioning. Synchronization is central because the framework depends on comparing camera- and LiDAR-derived object states at aligned timestamps (Hou et al., 4 May 2025).

The LiDAR branch uses PointPillars plus a clustering strategy for real-time 3D object detection and tracking, producing object labels together with 3D positions and sizes over time. The vision branch relies on a fine-tuned VLM to detect and classify four-wheel vehicles, non-four-wheel vehicles, pedestrians, signs, fixed installations, plants, and monitors, while also producing per-frame object positions $\{(T_i, F_i)\}_{i=0}^n.$0. GNSS/IMU signals are pre-processed into speed, acceleration, angular velocity, yaw rate, and related motion indicators for timestamp selection (Hou et al., 4 May 2025).

The VLM foundation is LLaMA-3.2-Vision (11B), composed of a pre-trained vision tower, the LLaMA-3.2 11B LLM, and a learnable linear projection layer mapping visual features into the LLM token space. The adaptation strategy fine-tunes both the vision encoder and the LLM using LoRA, trains the projection layer from scratch, and uses AdamW with learning rate $\{(T_i, F_i)\}_{i=0}^n.$1, batch size 2, cosine learning-rate schedule with 0.03 warm-up ratio, and 10 epochs on instruction-style supervision, all on an NVIDIA H100 GPU. Prompts are formatted as instruction–response pairs with special <Image> tokens and JSON-structured fields containing sensor-derived features, natural-language task instructions, and role constraints for the active agent (Hou et al., 4 May 2025).

The paper stresses that structured outputs are enforced by annotation guidelines and role-specific prompt formats. This is important because later modules consume earlier outputs as structured facts rather than free-form prose. A descriptive analysis agent, for example, receives critical timestamps and triggers; the vehicle reasoning agent receives explicit discrepancies $\{(T_i, F_i)\}_{i=0}^n.$2; and the response-generation stage receives categorized insights for urgency ranking. This suggests that the LLM is not being used as a raw perception engine, but as a reasoning layer tightly conditioned on numerically grounded intermediate representations.

4. Dataset, benchmark tasks, and empirical performance

The dataset contains three real-world routes. R1 (1277.76 m) is a controlled environment with small environmental dynamics, forward camera only, and higher average speed of approximately $\{(T_i, F_i)\}_{i=0}^n.$3. R2 (969.19 m) loops around an urban square, has large environmental dynamics and more traffic activity, uses forward plus left and right cameras, and has lower average speed of approximately $\{(T_i, F_i)\}_{i=0}^n.$4. R3 (1125.91 m) has medium dynamics and roadside obstructions, also uses forward plus left and right cameras, and has average speed approximately $\{(T_i, F_i)\}_{i=0}^n.$5. All routes have synchronized LiDAR, cameras, and GNSS/IMU (Hou et al., 4 May 2025).

The evaluation is organized into three tasks. Task 1 – Object and Category Detection is trained on R2 and R3 and evaluated on R1 using Precision, Recall, and F1. Task 2 – Vehicle Reasoning tests LiDAR misplacement diagnosis and misaligned camera-view detection on R2 and R3, including left and right variants. Task 3 – Environmental Reasoning asks the system to identify independently moving objects versus static ones by comparing two timestamps on R2 and R3. Baselines include LLaMA-3.2-Vision-Instruct, GPT-4o-mini, Pixtra-large, GPT-4o, and Claude-3.7-Sonnet for Task 1, and Zero-Shot, CoT, and CoT + Self-Refine for Tasks 2 and 3 (Hou et al., 4 May 2025).

The main quantitative results are summarized below.

Task	Setting	Result
Task 1	DriveAgent (fine-tuned VLM)	$\{(T_i, F_i)\}_{i=0}^n.$6, $\{(T_i, F_i)\}_{i=0}^n.$7, $\{(T_i, F_i)\}_{i=0}^n.$8
Task 2	Vision reasoning, R2	$\{(T_i, F_i)\}_{i=0}^n.$9
Task 2	Vision reasoning, R3	$\{(T_i, F_i, L_i)\}$0
Task 3	Environmental reasoning, R2	$\{(T_i, F_i, L_i)\}$1
Task 3	Environmental reasoning, R3	$\{(T_i, F_i, L_i)\}$2

For Task 1, the fine-tuned DriveAgent VLM achieves the best precision and best F1 among the reported models, surpassing Claude-3.7-Sonnet on F1 and recall-weighted performance while retaining markedly higher precision. The paper further states that DriveAgent is the only model that consistently detects “monitors”, described as overhead displays or cameras, which other models often miss. That observation is used to argue for the effect of structured annotations and domain-specific training (Hou et al., 4 May 2025).

For Task 2, the strongest gains appear in vision reasoning for camera misalignment. Zero-Shot and CoT prompting perform very poorly on left/right distractor views, sometimes as low as $\{(T_i, F_i, L_i)\}$3–$\{(T_i, F_i, L_i)\}$4, whereas DriveAgent obtains $\{(T_i, F_i, L_i)\}$5 on R2, $\{(T_i, F_i, L_i)\}$6 on R2-left, $\{(T_i, F_i, L_i)\}$7 on R2-right, $\{(T_i, F_i, L_i)\}$8 on R3, $\{(T_i, F_i, L_i)\}$9 on R3-left, and $\{(T_i, F_i, V_i)\}$0 on R3-right. In LiDAR reasoning, the paper characterizes DriveAgent as competitive and more stable, with best performance on some variants such as $\{(T_i, F_i, V_i)\}$1 on R2-left and $\{(T_i, F_i, V_i)\}$2 on R3-left, while avoiding the catastrophic drops seen in some prompting baselines (Hou et al., 4 May 2025).

For Task 3, DriveAgent yields the best reported environmental-reasoning accuracy on both routes: $\{(T_i, F_i, V_i)\}$3 on R2 versus $\{(T_i, F_i, V_i)\}$4 for Zero-Shot, $\{(T_i, F_i, V_i)\}$5 for CoT, and $\{(T_i, F_i, V_i)\}$6 for CoT+Self-Refine; and $\{(T_i, F_i, V_i)\}$7 on R3 versus $\{(T_i, F_i, V_i)\}$8, $\{(T_i, F_i, V_i)\}$9, and $D_i$0, respectively. Although the paper does not present classical ablations, the comparison with Zero-Shot, CoT, and CoT+Self-Refine suggests that plain prompting is insufficient for nuanced multimodal reasoning and that the explicit multi-agent pipeline yields more robust behavior (Hou et al., 4 May 2025).

5. Interpretability, robustness, and limitations

Interpretability is one of DriveAgent’s central design goals. Each module produces human-readable diagnostic statements about sensor disagreement, environmental change, and causal explanations, while the final response explicitly lists the most pressing issue $D_i$1, the selected action $D_i$2, and the remaining issues $D_i$3 (Hou et al., 4 May 2025).

The paper’s examples illustrate how this structure supports diagnosis. In a misaligned camera views case, persistent large $D_i$4 across many objects leads the diagnostic agent to conclude “camera misalignment” rather than “environment changed”, with a medium urgency and a recommendation such as “recalibrate camera / rely more on LiDAR.” In an independently moving pedestrian case, the environmental reasoning stage uses $D_i$5 to separate pedestrian motion from static background and recommends “slow down / prepare to yield” with high urgency. The paper’s broader point is that the decisions are explainable because they are grounded in named objects, explicit motion quantities, and causal reasoning steps (Hou et al., 4 May 2025).

Robustness is evaluated indirectly through two kinds of difficulty. First, misaligned camera views act as noisy, conflicting inputs; most baselines fail to recognize the misalignment, while DriveAgent diagnoses it with substantially higher accuracy. Second, the three routes expose the system to small, medium, and large environmental dynamics. The paper argues that the use of cross-sensor consistency measures such as $D_i$6 and $D_i$7 makes the system more robust to single-sensor failures because disagreement can itself become an object of reasoning (Hou et al., 4 May 2025).

The limitations are equally explicit. The framework is not real-time closed-loop; the paper provides no latency numbers or real-time guarantees. It remains dependent on LLM correctness, so hallucinated or numerically misinterpreted reasoning remains possible even with structured prompts. Dataset coverage is limited to three routes in one city, without night, rain, snow, or multi-lane highway settings. There are no formal safety guarantees, and the action-selection functions $D_i$8 and $D_i$9 are presented abstractly rather than instantiated in closed form (Hou et al., 4 May 2025).

A second misconception is that “multi-agent” here necessarily means multiple independently trained foundation models. The paper instead states that there is no separate LLM per agent; the same LLaMA-3.2-Vision base is reused with different prompts and tools per module. This matters because DriveAgent’s novelty lies less in model plurality than in prompted role specialization plus structured inter-agent dataflow.

6. Relation to adjacent autonomous-driving agent paradigms

Within the broader LLM-for-driving literature, DriveAgent occupies a distinctive niche. Agent-Driver, introduced as “A Language Agent for Autonomous Driving,” also places an LLM above a conventional perception stack and adds a tool library, cognitive memory, chain-of-thought reasoning, task planning, motion planning, and self-reflection on the nuScenes benchmark (Mao et al., 2023). By contrast, DriveAgent emphasizes multimodal sensor fusion across camera, LiDAR, GPS, and IMU, sensor-aware diagnostics, and urgency-aware recommendation rather than direct language-generated trajectory planning (Hou et al., 4 May 2025).

Relative to end-to-end systems such as DriveTransformer, which unifies detection, prediction, mapping, and planning in a single transformer with Task Parallelism, Sparse Representation, and Streaming Processing, DriveAgent remains explicitly modular and open-loop (Jia et al., 7 Mar 2025). This difference is methodological: DriveTransformer targets scalable end-to-end policy learning, whereas DriveAgent targets explainable reasoning over structured multimodal features and time-indexed events.

Subsequent work further diversifies the design space. DriveAgent-R1 introduces active perception and hybrid thinking for VLM-based high-level planning, allowing the agent to switch between text-only reasoning and tool-augmented visual reasoning based on scene complexity (Zheng et al., 28 Jul 2025). DriveJudge moves in another direction, using a VLM to decide which evaluation rules are useful in a given context while retaining deterministic physical metrics for scoring (Sun et al., 15 Jun 2026). These systems indicate that “driving agents” now span perception, planning, diagnosis, evaluation, and scheduling, but DriveAgent’s contribution remains its tight coupling of LLM reasoning to symbolic, cross-sensor, time-series evidence (Hou et al., 4 May 2025).

The paper itself suggests that DriveAgent can sit alongside a traditional planner as a high-level reasoning and monitoring module. A plausible implication is that its longer-term significance lies not in replacing existing autonomous-driving stacks, but in furnishing those stacks with a layer that can inspect multimodal evidence, diagnose sensor inconsistencies, explain environmental change, and prioritize responses in human-readable form.