BEV-LLM: Bird’s-Eye-View Language Models
- Bird’s-Eye-View Large Language Models are multimodal systems that ground language processing on top-down scene representations, improving spatial coherence and 3D reasoning.
- They process diverse inputs such as multi-camera images, LiDAR, and GPS data to support tasks like scene captioning, navigation, and decision-making in autonomous systems.
- Empirical results indicate that BEV integration boosts spatial accuracy and planning efficiency, overcoming limitations of perspective-only models.
Searching arXiv for papers explicitly using BEV with LLMs and closely related systems. Bird’s-Eye-View LLM (BEV-LLM) denotes a class of multimodal systems in which a LLM or multimodal LLM is grounded not primarily in perspective-view image tokens, but in a bird’s-eye-view (BEV) representation of a scene. In this formulation, BEV serves as the spatial interface between perception and language: multi-camera images, LiDAR, GPS, remote-sensing imagery, or collaborative multi-agent sensing are transformed into an ego-centric or map-centric top-down representation, and that representation is then injected into a language-capable model for question answering, captioning, planning, driving, communication-state prediction, or other reasoning tasks. The term is used explicitly in some papers and is also a useful umbrella label for adjacent systems such as BEV-InMLLM, BEVDriver, BEVLM, BeLLA, Talk2BEV, and task-specific BEV-augmented MLLM baselines for driving and remote sensing (Brandstaetter et al., 25 Jul 2025, Ding et al., 2024, Winter et al., 5 Mar 2025, Monninger et al., 6 Mar 2026, Mohan et al., 5 Dec 2025, Choudhary et al., 2023, Park et al., 17 Mar 2025, Wang et al., 8 May 2026).
1. Conceptual scope and problem formulation
BEV-LLM research is motivated by a recurring deficiency in generic VLMs and MLLMs: perspective-image tokenization does not naturally yield a geometrically coherent 3D world model. Across autonomous driving, vehicle-to-infrastructure reasoning, embodied navigation, and built-environment interpretation, authors repeatedly identify the same failure mode: LLMs can process visual tokens, but they are weak at absolute spatial understanding, cross-view consistency, ego-centric directionality, and compositional geometric reasoning unless an explicit spatial substrate is supplied (Winter et al., 5 Mar 2025, Ehsani et al., 4 Sep 2025, Ding et al., 2024).
In this literature, BEV is that substrate. It places scene elements into a common top-down coordinate frame in which relative position, road layout, occupancy, actor interactions, and motion are easier to encode than in disconnected front-, side-, or rear-view images. For driving, this enables queries about left/right/front/rear relations, ego-motion planning, traffic flow, and risk. For communication-oriented systems, it supports reasoning about blockage, line-of-sight, and spatial occlusion. For remote sensing, it supports quantitative estimation of built-environment ratios from overhead imagery, although that line of work is closer to top-down multimodal reasoning than to fully geospatially grounded language agents (Ehsani et al., 4 Sep 2025, Park et al., 17 Mar 2025, Wang et al., 8 May 2026).
The phrase “BEV-LLM” therefore does not identify a single architecture. It spans at least four distinct formulations. First, some systems use BEV as a latent multimodal scene representation injected directly into an LLM or MLLM, as in BEVDriver, BeLLA, and the scene-captioning model titled BEV-LLM (Winter et al., 5 Mar 2025, Mohan et al., 5 Dec 2025, Brandstaetter et al., 25 Jul 2025). Second, some use BEV as a plug-in adapter for a frozen backbone, as in BEV-InMLLM and the V2I collaborative perception framework (Ding et al., 2024, Ehsani et al., 4 Sep 2025). Third, some use BEV as the input interface for language reasoning and then distill the resulting semantics back into a BEV encoder, as in BEVLM (Monninger et al., 6 Mar 2026). Fourth, some use BEV as a symbolic or structured map queried by a general-purpose LLM, as in Talk2BEV (Choudhary et al., 2023).
This diversity has an important consequence: a BEV-LLM is not necessarily an end-to-end generative driving agent. In several cases the LLM is frozen, the BEV encoder is external, and the trainable component is a lightweight projector, Q-Former, or LoRA adapter. A plausible implication is that the field is better understood as an interface-design problem—how to connect spatially grounded BEV representations to pretrained LLMs—than as a settled model family.
2. Architectural patterns and BEV–language interfaces
A concise taxonomy of representative systems is useful before examining their mechanics.
| System | BEV–language interface | Primary task |
|---|---|---|
| Talk2BEV | Structured language-enhanced BEV map | QA and spatial reasoning |
| BEV-InMLLM | Instruction-aware BEV Q-Former plus cross-attention injection | Holistic driving QA |
| BEVDriver | Latent BEV encoder plus Q-Former into LLM | Closed-loop driving |
| BEVLM | Serialized BEV tokens plus projector; later semantic distillation | VQA and driving improvement |
| BeLLA | Single-token BEV embedding inserted into LLM prompt | Driving QA |
| BEV-LLM | Positional Q-Former over BEVFusion features | Scene captioning |
One major design axis is whether BEV is represented as dense tokens, compressed latent queries, or structured symbolic records. Talk2BEV textualizes BEV into an object-centric language-enhanced map containing position, area, and LVLM-derived object/background descriptions, and relies on an LLM plus spatial operators for inference (Choudhary et al., 2023). At the other extreme, BEVLM down-samples a BEV grid to , yielding 2,500 BEV tokens bracketed by special tokens and projected by an MLP into the LLM’s embedding space (Monninger et al., 6 Mar 2026).
A second axis is whether BEV is passed to the LLM directly or first distilled by a bottleneck module. BEV-InMLLM uses an instruction-aware BEV Q-Former,
and injects the resulting tokens into the multi-view visual token stream before the frozen MLLM (Ding et al., 2024). The V2I collaborative perception paper uses an instruction-aware Q-Former to distill task-relevant information from an aggregated multi-agent BEV map, followed by cross-attention back into ego features and LLM decoding (Ehsani et al., 4 Sep 2025). BEVDriver likewise uses a modified BLIP-2 Q-Former with 32 learnable $768$-dimensional queries to align latent BEV features with navigation instructions before passing them to a LoRA-adapted LLM (Winter et al., 5 Mar 2025). The captioning model titled BEV-LLM uses a Positional Q-Former initialized from BERT, with 512 queries for the 1B model and 256 for the 8B model, to compress BEVFusion features before projection into Llama-3 token space (Brandstaetter et al., 25 Jul 2025).
A third axis concerns how explicitly geometry is encoded before language interaction. Some systems rely on a pretrained BEV encoder such as BEVFormer, UniAD/BEVFormer, InterFuser-derived encoders, BEVFusion, or LSS (Mohan et al., 5 Dec 2025, Monninger et al., 6 Mar 2026, Winter et al., 5 Mar 2025, Ding et al., 2024). Others add dedicated geometric machinery before language injection. The V2I framework performs BEVFusion on each agent, temporal self-attention over neighboring frames, GPS-based warping into the ego frame,
and then concatenation plus a convolution to form the collaborative BEV (Ehsani et al., 4 Sep 2025). NuPlanQA’s BEV-LLM uses BEVFormer-style temporal self-attention,
followed by spatio-temporal fusion attention,
to fuse BEV back into the multi-view visual stream (Park et al., 17 Mar 2025).
There is also substantial variation in compression strategy. BeLLA compresses the entire BEV feature map into a single token,
0
using a deeper convolutional projector and then replaces a reserved placeholder position in the LLM token stream with that embedding (Mohan et al., 5 Dec 2025). This is far more aggressive than BEVLM’s 2,500-token interface or BEVDriver’s 32-query interface. The contrast illustrates a core unresolved design question: whether BEV-to-language alignment benefits more from preserving spatial granularity or from imposing an extreme bottleneck that forces compact scene abstraction.
3. Data construction, supervision, and training regimes
BEV-LLM work is tightly coupled to dataset design. Several papers contribute not only models but also specialized language-grounded datasets whose supervision structure determines what kind of BEV reasoning is learned.
NuInstruct is an early large driving benchmark of this type. Built on nuScenes, it contains 91,355 instruction-response pairs over 11,850 keyframes sampled from 850 videos, using six camera views and three uniformly sampled frames per example. It organizes supervision into four task families—Perception, Prediction, Risk, and Planning with Reasoning—further divided into 17 subtasks (Ding et al., 2024). Its SQL-based generation pipeline operationalizes a hierarchical driving logic: 1 NuPlanQA extends this dataset-centric line with a much larger resource: NuPlanQA-1M contains about 1 million QA pairs from 90K frames, while NuPlanQA-Eval provides about 8K multiple-choice items from 4K frames across nine subtasks grouped into Road Environment Perception, Spatial Relations Recognition, and Ego-Centric Reasoning (Park et al., 17 Mar 2025).
Captioning-oriented work follows a different supervision route. The scene-captioning BEV-LLM paper trains on nuCaption and introduces nuView and GroundView. nuCaption has 240K samples total with 169K/70K train/val; nuView has 205K total with 169K/36K train/val and focuses on environmental conditions and viewpoints; GroundView has 7.4K total with 6K/1.4K train/val and emphasizes object-centric grounding-style summaries (Brandstaetter et al., 25 Jul 2025). BeLLA uses NuScenes-QA and DriveLM-NuScenes, with over 460K QA pairs over 34K timesteps for NuScenes-QA and over 377K QA pairs over 4072 training frames for DriveLM (Mohan et al., 5 Dec 2025).
Remote-sensing work reveals a different supervision pattern. The built-environment paper constructs a benchmark of 14,523 images and 101,661 visual question-answer pairs using Mapbox imagery and OpenStreetMap-derived labels, with 9,523 images / 66,661 QA pairs for training and 5,000 images / 35,000 QA pairs for testing. The task targets quantitative metrics such as building footprint ratio, green area percentage, urban blue area percentage, impervious space ratio, and several pairwise ratios (Wang et al., 8 May 2026). This is narrower than open-ended planning, but it shows how BEV-style top-down reasoning can be grounded in GIS-derived scalar targets.
Training protocols are predominantly adapter-based and parameter-efficient. In the built-environment benchmark, LoRA with 2 and 3 is applied to attention query and value projections, with AdamW, learning rate 4, weight decay 0.01, batch size 32, gradient accumulation 8, 2 epochs, 5% warmup, and FP16 (Wang et al., 8 May 2026). BEVDriver freezes the pretrained BEV encoder in the primary setup and adapts the LLM with LoRA of rank 16, scaling factor 32, and 5% dropout, while training waypoint and completion heads (Winter et al., 5 Mar 2025). BeLLA uses LoRA with rank 64 and 5, keeping the BEVFormer encoder frozen and training only the projector and the LLM adapters during QA fine-tuning (Mohan et al., 5 Dec 2025). The V2I framework freezes both the multimodal encoder and Llama-3.2-11B-Vision backbone, updating only the BEV-fusion connector with AdamW, weight decay 6, cosine schedule from 7, 5% warmup, batch size 16, and 15 epochs (Ehsani et al., 4 Sep 2025).
A recurrent pattern is two-stage or curriculum-style training. BeLLA first performs BEV-text pretraining with frame-level descriptions synthesized from NuScenes metadata and CAN bus signals, optimizing
8
before QA fine-tuning (Mohan et al., 5 Dec 2025). NuPlanQA’s BEV-LLM first adapts LLaMA-3.2-Vision-11B to multi-timestep front-view inputs by training only the multimodal projector for one epoch, and only then activates the BEV-Fusion module (Park et al., 17 Mar 2025). LiDAR-LLM, a close precursor to BEV-centric language grounding, uses a three-stage progression: scene-level captioning, object-level grounding, and finally high-level instruction tuning (Yang et al., 2023). A plausible implication is that BEV-language alignment is difficult to learn directly from downstream QA alone; most successful systems first establish a modality bridge.
4. Application domains
The dominant application domain is autonomous driving, but the BEV-LLM paradigm has diversified into communication-state reasoning, embodied navigation, scene captioning, and urban remote sensing.
In driving QA and scene understanding, BEV-InMLLM and BeLLA target holistic reasoning over multi-view scenes rather than only perception or captioning. NuInstruct’s tasks cover distance estimation, speed estimation, closest-object identification, future status, crossing, oncoming, overtaking, braking, lane changing, and planning with reasoning (Ding et al., 2024). BeLLA focuses on NuScenes-QA and DriveLM, with reported strength on Status, Prediction, Planning, and Behavior questions relative to appearance-heavy perception questions (Mohan et al., 5 Dec 2025). NuPlanQA pushes toward multi-view, short-horizon ego-centric reasoning over 1.5 seconds of context and explicit vehicle signals such as velocity and steering angle (Park et al., 17 Mar 2025).
Closed-loop driving is represented most directly by BEVDriver. It consumes four synchronized RGB camera views, a 3D LiDAR point cloud, and a natural-language navigation instruction, predicts five future waypoints and a completion flag, and uses PID controllers for steering, throttle, and brake. Its closed-loop evaluation on LangAuto makes it one of the clearest examples in which a LLM conditioned on latent BEV features contributes directly to driving behavior rather than only explanation or QA (Winter et al., 5 Mar 2025).
Vehicle-to-infrastructure collaborative perception extends the concept beyond driving-policy generation. The V2I paper constructs BEV from ego and neighboring helper vehicles using RGB, LiDAR, and GPS, then asks an MLLM to predict LoS/NLoS, link availability, and blockage risk. The downstream tasks are communication-specific, but the architecture is explicitly framed as a BEV injection method for spatially aware language reasoning over 3D collaborative scenes (Ehsani et al., 4 Sep 2025).
Scene captioning and explainability form another cluster. The paper explicitly titled BEV-LLM uses BEVFusion over six surround-view RGB images and LiDAR, a Positional Q-Former, and Llama-3-family backbones to generate captions and view-specific descriptions (Brandstaetter et al., 25 Jul 2025). LiDAR-LLM, though named differently, is also BEV-centered at the interface level because it flattens voxel features along the 9-axis into a BEV feature map and connects that map to LLaMA through a View-Aware Transformer (Yang et al., 2023). These systems position captioning as an interpretability tool: the vehicle can describe what it perceives rather than only act on it.
Embodied navigation instruction generation offers a related but distinct use case. BEVInstructor fuses perspective and BEV representations derived from egocentric route observations, uses them as prompt-like visual context for LLaMA-7B, and adds instance-guided iterative refinement. Its inputs are trajectory observations and selected actions; its outputs are route descriptions rather than control signals or answers (Fan et al., 2024). This work shows that BEV-grounded language modeling is not confined to road-scene QA.
Built-environment reasoning from remote sensing broadens the paradigm further. The remote-sensing paper investigates whether VLMs can infer built-form composition and environmental balance from bird’s-eye-view urban tiles, using overhead Mapbox imagery at zoom level 17 and OSM-derived labels over 335 U.S. metropolitan areas (Wang et al., 8 May 2026). It does not propose a dedicated BEV-LLM architecture, but it demonstrates that top-down multimodal reasoning over quantitative urban metrics is a natural extension of the BEV-language interface.
5. Empirical findings
A central empirical theme is that BEV integration materially improves spatial reasoning relative to perspective-only or ego-only baselines, but the magnitude and locus of the gain depend on the task.
On NuPlanQA-Eval, BEV-LLM reaches 78.7% total accuracy with multi-view, multi-frame input and 76.2% with single-frame input, making it the best overall open-source model reported there. The ablation without BEV-Fusion yields only 65.5 total accuracy under multi-view, multi-frame training, so adding BEV-Fusion gives a +13.2 point improvement overall, with especially large gains in Spatial Relation Recognition (+13.3) and Ego-Centric Reasoning (+14.9) (Park et al., 17 Mar 2025). This is one of the clearest demonstrations that BEV contributes most strongly where ego-centric geometry and actor relations matter most.
In closed-loop driving, BEVDriver reports 48.9 DS / 59.7 RC / 0.82 IS on LangAuto long, 66.7 / 77.8 / 0.87 on LangAuto short, and 70.2 / 81.3 / 0.87 on LangAuto tiny with a Llama-7B backbone. The paper states up to 18.9% higher Driving Score relative to prior state of the art and much larger gains relative to LMDrive’s LLaVA-based variants (Winter et al., 5 Mar 2025). It also reports open-loop ADE 0.04 and FDE 0.07, suggesting that latent BEV grounding can support precise waypoint prediction rather than only high-level maneuver selection.
In holistic driving QA, BEV-InMLLM shows that adding multi-view and temporal modeling already improves base MLLMs, but BEV injection adds further gains that are strongest on explicitly spatial subtasks. For MiniGPT-4, removing BEV degrades spatial MAE from 13.9 to 18.0 and spatial accuracy from 32.9 to 30.0 in the ablation grouped by Temporal, Multi-view, Spatial, and Holistic categories (Ding et al., 2024). This supports the common interpretation that BEV’s distinct contribution is not generic semantic enrichment, but stronger spatial disambiguation.
BEVLM provides two linked empirical results. In representation alignment, replacing image tokens with BEV tokens improves cross-view reasoning on Ego3D from 42.02 to 61.34 average MCQ accuracy and reduces L1 distance error from 9.01 m to 7.05 m with InternVL3-1B, which the paper summarizes as a 46.0% improvement in MCQ accuracy and a 27.8% reduction in L1 error (Monninger et al., 6 Mar 2026). In downstream driving, distilling semantics from a frozen 8B teacher into the BEV encoder increases NeuroNCAP score from 2.10 to 2.71 and reduces collision rate from 0.62 to 0.55, which the paper describes as a 29.0% improvement in NeuroNCAP score and an 11.3% decrease in collision rate (Monninger et al., 6 Mar 2026). This establishes that language supervision can improve BEV representations even when the LLM is absent at driving-time inference.
BeLLA’s results are more mixed, which is instructive. On NuScenes-QA it reaches 59.6% overall accuracy with LLaMA 3.2 3B and is especially strong on Status at 69.9, but it is not universally best against all baselines. On DriveLM, Qwen-based BeLLA reports BLEU-4 45.62, METEOR 33.90, ROUGE-L 72.12, CIDEr 3.17, with especially strong Behavior performance but weaker Perception performance (Mohan et al., 5 Dec 2025). The paper attributes this asymmetry to BEV’s strength on dynamics, layout, and motion status rather than appearance-heavy cues.
Scene-captioning results likewise emphasize that BEV can be competitive with far smaller language backbones when the representation bridge is well designed. On nuCaption, BEV-LLM-1B achieves BLEU-4 20.28, slightly above LiDAR-LLM’s 19.26, while using 1.35B parameters compared with 6.96B for LiDAR-LLM (Brandstaetter et al., 25 Jul 2025). The same paper reports on nuView BLEU-4 20.96 for the 1B model and 21.04 for the 8B model, with the 1B model slightly better on BLEU-1/2/3 and BERT-F1 (Brandstaetter et al., 25 Jul 2025). This suggests that BEV representation quality and positional conditioning can offset substantial differences in raw LLM scale.
Communication-oriented BEV injection produces similarly strong relative gains. In the V2I paper, macro-average accuracy rises from 73.3% for the ego-only LLM baseline to 87.2% for the BEV injection model, an absolute gain of 13.9 percentage points, and robustness gains in rain and night increase by up to 32.7% (Ehsani et al., 4 Sep 2025). This extends the BEV-LLM argument beyond classical visual QA: explicit BEV context improves any downstream task whose latent structure is fundamentally geometric.
Remote-sensing results reveal a different empirical regime. Zero-shot performance of general-purpose and remote-sensing-specific VLMs is only modestly above random guessing on four-way multiple choice, but LoRA fine-tuning improves InternVL-3-8B from 36.21 to 62.57 on Building Footprint Ratio and from 45.72 to 72.15 on Green Area Percentage, with the paper summarizing this as roughly 90% relative improvement (Wang et al., 8 May 2026). The implication is that overhead-scene quantitative reasoning is learnable but not reliably emergent in zero-shot models.
6. Limitations, controversies, and open directions
The field remains methodologically heterogeneous and several limitations recur across papers.
A first limitation is that many so-called BEV-LLM systems are not fully end-to-end language-grounded world models. In BeLLA, the BEV encoder is frozen and compressed into a single token (Mohan et al., 5 Dec 2025). In the V2I framework, the LLM and multimodal encoder are frozen and only the connector is trained (Ehsani et al., 4 Sep 2025). In BEVLM, the most consequential downstream gains come from distilling semantics into a BEV encoder used later by a conventional UniAD driving stack rather than by online LLM control (Monninger et al., 6 Mar 2026). This suggests that current successes often come from using LLMs as adapters or teachers, not from making LLMs the primary online decision engine.
A second limitation is incompleteness of spatial grounding. The built-environment study has no coordinates, no projection-aware reasoning, no vector-map fusion, no temporal aggregation, and no explicit geolocation conditioning (Wang et al., 8 May 2026). Talk2BEV relies on object-centric JSON-like descriptions and explicit spatial operators rather than a native BEV token interface, which is effective but reveals that pure LLM spatial reasoning over text descriptions is inadequate for precise metric inference (Choudhary et al., 2023). NuPlanQA’s BEV-LLM improves short-horizon driving understanding, but the paper leaves many implementation details unspecified, including optimizer settings, image preprocessing, and exact freezing strategy (Park et al., 17 Mar 2025).
A third limitation concerns appearance and semantics lost in BEV compression. BeLLA explicitly notes difficulty with fine-grained appearance-centric questions because BEV does not preserve color, texture, or traffic-light appearance well (Mohan et al., 5 Dec 2025). Remote-sensing quantitative reasoning works better for Green Area Percentage than for relational metrics such as Blue–Green Ratio because the former aligns with stronger visual semantics while the latter requires multi-step reasoning (Wang et al., 8 May 2026). BEVLM itself acknowledges that projecting semantics into BEV can create a bottleneck that discards perspective cues (Monninger et al., 6 Mar 2026).
A fourth limitation is dataset and supervision quality. Talk2BEV depends on caption quality from BLIP-2, InstructBLIP-2, or MiniGPT-4, and performance changes only modestly when replacing predicted BEV with ground-truth BEV, implying that semantics, not BEV geometry, may be the dominant bottleneck in that pipeline (Choudhary et al., 2023). The built-environment benchmark attributes city-level variability partly to inconsistent OSM annotation quality (Wang et al., 8 May 2026). NuPlanQA and other large driving QA datasets rely heavily on LLM-generated labels with human review, which introduces residual prompt bias and possible annotation noise (Park et al., 17 Mar 2025).
A fifth limitation is efficiency and deployment realism. BroadBEV and MamBEV, though not BEV-LLMs themselves, show that perception-side BEV construction still entails nontrivial computational tradeoffs. BroadBEV reports 158 ms latency for the full model versus 83 ms baseline on a single A100, with ColFusion attention dominating cost (Kim et al., 2023). MamBEV improves scaling relative to dense attention, but its temporal component grows sharply with frame count and its implementation does not yet realize the full theoretical efficiency of the SSM formulation (Ke et al., 18 Mar 2025). Since most BEV-LLM pipelines layer language reasoning on top of BEV perception, these upstream costs remain central.
The most plausible near-term direction is not a monolithic “driving GPT,” but increasingly structured BEV-language systems with stronger geometry, better compression, and more explicit world-state semantics. The literature already suggests several trajectories. One is richer BEV tokenization: from single-token compression toward object- or patch-level token sets with continuous positional grounding (Mohan et al., 5 Dec 2025, Monninger et al., 6 Mar 2026). Another is stronger temporal and multi-agent memory, as seen in the V2I connector and MamBEV’s efficient BEV representation learning (Ehsani et al., 4 Sep 2025, Ke et al., 18 Mar 2025). A third is semantics distilled into BEV itself rather than only decoded through an LLM, as in BEVLM and open-vocabulary BEV segmentation with geometry-aware constraints (Monninger et al., 6 Mar 2026, Choi et al., 23 Jun 2026). A fourth is broader domain transfer, from driving to embodied navigation, smart-city remote sensing, and communication-state reasoning (Fan et al., 2024, Wang et al., 8 May 2026, Ehsani et al., 4 Sep 2025).
Taken together, the literature indicates that BEV-LLM is best understood not as a single model archetype but as a research program: make language-capable models reason over a spatially coherent top-down world representation, and use that representation to close the gap between generic multimodal pattern recognition and the geometric requirements of embodied, driving, and map-centric reasoning.