BEVDriver: BEV-Centric Autonomous Driving

• BEVDriver is a BEV-centric autonomous driving architecture that integrates multi-view RGB and LiDAR data into a unified top-down representation.
• Its design couples a specialized BEV encoder, a Q-Former for language alignment, and an LLM planner to predict future trajectories in closed-loop settings.
• Benchmark results show significant performance gains over existing methods while revealing challenges in pedestrian forecasting and safety-critical decision making.

Searching arXiv for BEVDriver and closely related work. BEVDriver is a bird’s-eye-view-centered autonomous-driving architecture in which multi-sensor perception is compressed into a latent BEV representation and coupled to a learned planner operating in closed loop. In its most specific published form, introduced as an LLM-based model for end-to-end closed-loop driving in CARLA, BEVDriver fuses multi-view RGB images and a 3D LiDAR point cloud into latent BEV features, aligns those features with natural-language navigation instructions through a Q-Former, predicts future waypoints with a LoRA-adapted Llama backbone, and converts the predicted trajectory into vehicle controls through PID tracking (Winter et al., 5 Mar 2025).

1. Definition and scope

In the literature represented here, BEVDriver most specifically denotes the 2025 CARLA model of Winter et al., designed for robust closed-loop driving from latent BEV features and language instructions (Winter et al., 5 Mar 2025). Its core premise is that BEV serves as the spatially grounded interface between heterogeneous sensors and high-capacity sequence models: cameras and LiDAR are first fused into a common top-down state, and only then passed into an LLM-based planner.

The term also appears more broadly in adjacent work as a convenient label for a BEV-centric end-to-end driving pattern rather than a single frozen architecture. A hierarchical DRL paper presents an in-depth description of a “BEVDriver framework” in which BEV perception, temporal modeling, and PPO-based control are tightly integrated (Lu et al., 2024). A later BEVLM paper gives a step-by-step “BEVDriver” design guide built around semantic distillation from a frozen LLM teacher into BEV representations (Monninger et al., 6 Mar 2026). This suggests that BEVDriver functions both as a named system and as a shorthand for end-to-end agents whose intermediate state is a geometrically coherent BEV tensor rather than raw perspective imagery alone.

A broader antecedent is the 2023 intersection-driving system built from BEV-V2X perception, IMM-UKF-based fusion prediction, and DRL-based decision making and planning in complex traffic intersections (Li et al., 2023). That work is not explicitly named BEVDriver in the supplied material, but it exemplifies the same general integration of BEV perception, state prediction, decision making, and planning.

2. Core architecture

The 2025 BEVDriver model consists of three principal modules: a BEV encoder, a Q-Former, and an LLM planner (Winter et al., 5 Mar 2025). The encoder fuses four RGB views—front, left, right, and rear—with a 3D LiDAR point cloud into a latent BEV feature map. The Q-Former then projects BEV tokens from a 256-dimensional visual space into the 768-dimensional embedding space expected by the LLM. The planner is an instruction-tuned Llama backbone, in 7B or 8B-Instruct form, adapted with LoRA and coupled to GRU and MLP output heads.

The published data flow is explicit. Sensor streams are first transformed by the BEV encoder into a 256-D BEV feature tensor of shape $(H, W, C)$. Those BEV features, together with historical frames, are processed by the Q-Former into 32 visual tokens of dimension 768. The visual tokens are concatenated with tokenized navigation instructions and passed to the LoRA-adapted LLM, whose hidden state is decoded by a GRU and an MLP into future waypoints and a completion flag; the resulting waypoints are then executed by PID controllers (Winter et al., 5 Mar 2025).

The perception stack is multimodal and asymmetric. The image branch uses ResNet-50 to produce multi-scale feature maps, while the LiDAR branch uses ResNet-18 to encode voxelized point clouds. A transformer-based encoder projects both modalities into a unified 256-D BEV feature space. During pre-training, decoder heads are attached for object detection with L1 loss, traffic-light classification with cross-entropy, front-view semantic segmentation with focal loss, and temporal contrastive alignment (Winter et al., 5 Mar 2025). A common misconception is to treat BEVDriver primarily as an LLM policy; the published architecture instead makes the LLM only one stage in a perception-heavy, geometrically grounded stack.

3. BEV construction and language alignment

BEVDriver’s geometric core is the projection of camera and LiDAR observations into a shared BEV lattice. For camera features, if $X_{\text{cam}} \in \mathbb{R}^{3\times N}$ denotes points in the sensor frame, with intrinsic $K$ and extrinsic $[R|t]$, the transformation is specified as

$$x_{\text{world}} = R X_{\text{cam}} + t,\qquad (u,v)=K[R|t]X_{\text{cam}}.$$

The world coordinates $x_{\text{world}}=(x,y,z)$ are then discretized onto an $H\times W$ grid with cell size $\Delta$ via

$$i=\left\lfloor \frac{x-x_{\min}}{\Delta}\right\rfloor,\qquad j=\left\lfloor \frac{y-y_{\min}}{\Delta}\right\rfloor.$$

LiDAR points $P\in\mathbb{R}^{3\times M}$ are transformed to the ego-vehicle frame and binned analogously:

$X_{\text{cam}} \in \mathbb{R}^{3\times N}$0

Features that land in the same BEV cell are aggregated by attention or average pooling to yield a latent tensor $X_{\text{cam}} \in \mathbb{R}^{3\times N}$1 (Winter et al., 5 Mar 2025).

Alignment with language is handled by a Q-Former adapted from BLIP-2. It uses 32 learnable query tokens $X_{\text{cam}} \in \mathbb{R}^{3\times N}$2 and performs multi-head cross-attention over flattened BEV features $X_{\text{cam}} \in \mathbb{R}^{3\times N}$3 projected into 768 dimensions. The attention computation is given as

$X_{\text{cam}} \in \mathbb{R}^{3\times N}$4

$X_{\text{cam}} \in \mathbb{R}^{3\times N}$5

The result is a set of 32 visual tokens already aligned to the LLM embedding space (Winter et al., 5 Mar 2025).

Instruction conditioning is serialized rather than fused by a bespoke planner head. If the navigation instruction is tokenized as $X_{\text{cam}} \in \mathbb{R}^{3\times N}$6 and the aligned visual tokens are $X_{\text{cam}} \in \mathbb{R}^{3\times N}$7, the input sequence is

$X_{\text{cam}} \in \mathbb{R}^{3\times N}$8

This ordering makes the planner explicitly multimodal, but with BEV tokens as the sole visual interface (Winter et al., 5 Mar 2025). The design choice matters: it replaces raw-image token streams with a compact, topologically consistent representation whose geometry is already normalized to the ego frame.

4. Trajectory prediction, closed-loop execution, and optimization

BEVDriver does not directly output low-level controls from the LLM. Instead, the planner predicts a short-horizon waypoint set and a binary completion indicator. A GRU adapter takes the final hidden state $X_{\text{cam}} \in \mathbb{R}^{3\times N}$9 of the LLM and predicts $K$0 future waypoints,

$K$1

while an MLP adapter outputs a completion flag $K$2 (Winter et al., 5 Mar 2025).

Training uses a waypoint regression term and a completion-flag classification term:

$K$3

$K$4

$K$5

The optimization target is therefore a grounded trajectory-decoding objective rather than next-token imitation in natural language alone (Winter et al., 5 Mar 2025).

At runtime in CARLA, the system is fully closed loop. At each time step, new sensor data are acquired, BEV features are updated, the Q-Former and LLM are re-run, and a fresh waypoint set $K$6 is produced. Longitudinal PID control uses distance error to regulate throttle and brake, while lateral PID control uses cross-track error to regulate steering. The system also performs frame-wise scenario assessment by monitoring infractions such as collisions and red-light events. If an unsafe state is detected or the current instruction is complete, planning is interrupted and re-executed with updated context. A further safety constraint is specified: if a predicted waypoint falls inside an obstacle region in the BEV map, it is overridden by an evasive sub-trajectory (Winter et al., 5 Mar 2025).

This control decomposition is technically significant because it preserves the expressive, instruction-conditioned planning capacity of the LLM while delegating actuator stabilization to conventional feedback control. In that sense, BEVDriver is end-to-end at the level of trajectory generation, but not a pure direct-control language policy.

5. Benchmark results and ablation behavior

On the LangAuto benchmark, BEVDriver is evaluated with Route Completion (RC), Infraction Score (IS), and Driving Score (DS), where

$K$7

The reported closed-loop results for the Llama-7B configuration are as follows (Winter et al., 5 Mar 2025).

LangAuto split	DS	RC
Long	48.9%	59.7%
Short	66.7%	77.8%
Tiny	70.2%	81.3%

These results correspond to gains over state-of-the-art methods of +11.1% DS and +12.2% RC on LangAuto-Long, +18.9% DS and +14.5% RC on LangAuto-Short, and +5.5% DS and +3.4% RC on LangAuto-Tiny (Winter et al., 5 Mar 2025). Relative to LMDrive LLaVA-7B, the Driving Score margins are reported as +35.1% on Long, +31.2% on Short, and +5.5% on Tiny.

The ablation results sharpen the interpretation of where the gains come from. In an unseen Town-5 evaluation, BEVDriver achieves DS 28.4% and RC 38.9% on Town-5 Long, reported as the best among models trained with and without Town-5 (Winter et al., 5 Mar 2025). End-to-end unfreezing of the encoder produces a mixed effect: RC increases by 5% on LangAuto-Tiny, but IS deteriorates from 0.87 to 0.64. This indicates that additional perceptual plasticity can improve route progress while simultaneously degrading driving safety. Another ablation removes misleading commands; unexpectedly, this lowers DS from 70.2% to 60.6%, suggesting that periodic instructions improve attention and planning.

The limitations reported in the same study are equally revealing. Distance-based commands such as “in 50 m turn left” are executed prematurely, indicating inadequate temporal grounding. The semantic segmentation branch misses fine road markings such as crosswalks. Proposed future directions include spatiotemporal attention in the BEV encoder, autoregressive trajectory generation, full end-to-end fine-tuning including traffic-light loss, and scene-description token generation for enhanced explainability (Winter et al., 5 Mar 2025).

6. Safety-critical evaluation under interactive pedestrian behavior

The most pointed external evaluation of BEVDriver comes from HABIT, a CARLA benchmark designed to expose weaknesses that remain hidden in scripted simulations (Ramesh et al., 24 Nov 2025). In that benchmark, BEVDriver is connected through the standard Leaderboard API, receives camera and LiDAR streams at 20 Hz, and returns a trajectory command at 10 Hz. Route waypoints are converted into language with the same route-to-language encoder used in LangAuto and supplied to the LLM planner as fixed context at the start of each route.

HABIT scenarios are generated from 110 CARLA routes under 12 weather conditions. Each scenario contains one ego vehicle, 30 scripted vehicles, 20 interactive pedestrians, and 10 ambient pedestrians. The interactive pedestrians are drawn from 4,730 semantically filtered motions and transition from idle to attempting-to-cross to crossing based on ego distance and orientation triggers (Ramesh et al., 24 Nov 2025). This setup specifically stresses the planner’s response to diverse, non-scripted pedestrian intent.

The benchmark reports “lower is better” metrics. Collision rate is defined as number of collisions divided by distance driven in kilometers; AIS 3+ injury risk is the percentage of events with AIS at least 3; FPBR is the number of false-positive braking events divided by total braking events (Ramesh et al., 24 Nov 2025). Under this evaluation, BEVDriver records 7.19 collisions/km and 12.35% pMAIS 3+. FPBR is marked as not applicable because BEVDriver does not explicitly brake on predicted pedestrian trajectories, and ADE is not reported because it has no explicit pedestrian forecasting module.

The qualitative diagnosis is consistent with those numbers. HABIT reports that BEVDriver treats all pedestrians as static obstacles in the BEV map and does not predict their future trajectories. Consequently, jaywalking, sudden diagonal crossings, and hesitant stop-and-go motions lead to late reactions and collisions; pedestrians emerging from behind occluders or approaching from unusual directions are often missed until braking or steering is too late (Ramesh et al., 24 Nov 2025). The same study argues that, unlike InterFuser, BEVDriver does not exhibit an overcautious-braking trade-off; it instead follows its nominal LLM-predicted trajectory, which yields a collision rate comparable to TransFuser but without false-positive braking statistics to offset the safety interpretation. Because speed is not modulated around pedestrians, impacts tend to occur at higher relative velocities, pushing pMAIS 3+ into the double-digit range.

Three remedies are highlighted. The first is to integrate an explicit pedestrian-forecasting head into the BEV encoder so that the planner can attend to predicted trajectories rather than generic obstacles. The second is to augment training with HABIT’s diverse motion library, especially rare behaviors such as jaywalking and hesitant crossings. The third is to incorporate severity-aware loss functions, for example by weighting predicted collisions by estimated AIS risk (Ramesh et al., 24 Nov 2025). These proposals directly target the mismatch between geometrically grounded global planning and fine-grained human-intent anticipation.

7. Position within the BEV-centric driving literature

BEVDriver sits within a wider transition from raw-perspective tokenization to unified BEV abstraction as the interface between perception and high-level reasoning. BEVLM argues that existing LLM-based driving systems often process multi-view and multi-frame images independently, producing redundant computation and weak cross-view spatial consistency, whereas semantically distilled BEV representations provide a geometrically coherent alternative (Monninger et al., 6 Mar 2026). In that framework, BEV features are downsampled, projected by an MLP, wrapped with special $K$8 tokens, and fed to a frozen LLM teacher; the paper reports 46% improvement in cross-view reasoning accuracy and 29% improvement in safety-critical closed-loop performance after semantic distillation. Its implementation guide explicitly presents projector variants such as Q-Former modules, making it closely adjacent to the published BEVDriver architecture.

A parallel line is BEV-VLM, which replaces latent-token alignment through a Q-Former with an explicit BEV-HD Map image consumed by Qwen2.5-VL (Chen et al., 27 Sep 2025). That system uses BEVFusion to extract $K$9 from six cameras and LiDAR, applies PCA for 3-channel visualization, overlays a local HD-map crop, and predicts a discretized 3-second trajectory with six waypoint tokens. On nuScenes, the Qwen-7B version reports 0.13 / 0.16 / 0.16 for $[R|t]$0, 0.15 average displacement, and 0.00 collision at 3 seconds. The contrast with BEVDriver is architectural rather than conceptual: both rely on BEV as the dominant visual abstraction, but differ in whether that abstraction remains latent or is rendered into a VLM-consumable image space.

Alternative BEV-centric controllers omit LLMs entirely. The hierarchical DRL framework of 2024 couples lift-and-splat BEV perception to MLP encoders for road, vehicle, and navigation vectors, then uses a GRU plus PPO actor-critic to produce throttle/brake and steering (Lu et al., 2024). Across all maps and both congestion regimes, it reports an average collision-rate reduction of roughly 20%, with low-congestion collision rate improving from 0.34 for DRL to 0.12 for the six-camera configuration. ME$[R|t]$1-BEV pushes this direction further by combining BEV perception with a Mamba state-space temporal module and PPO, reporting collision-rate reduction from 0.81 to 0.26 and Driving Score increase from 39.7 to 71.2 in low-density CARLA experiments (Lu et al., 8 Aug 2025). These systems clarify that the decisive factor is often the BEV state representation itself; the planner may be an LLM, a GRU-PPO policy, or a Mamba-augmented RL stack.

Taken together, the literature positions BEVDriver as a representative of a broader architectural thesis: top-down, ego-aligned spatial abstraction is an effective interface for decision-making modules that must reason across sensors, time, and instructions. The evidence is correspondingly mixed. BEVDriver achieves strong closed-loop LangAuto results and benefits from multimodal compression into BEV, yet HABIT shows that BEV grounding alone does not resolve intent forecasting, temporal grounding, or safety under realistic pedestrian interaction (Winter et al., 5 Mar 2025, Ramesh et al., 24 Nov 2025). The central research question has therefore shifted from whether BEV should be used to how much semantic, predictive, and interaction-aware structure must be injected into BEV before a high-capacity planner can act robustly in the open world.