BEVFormer: 3D Perception from Multi-Camera Views

• BEVFormer Architecture is a unified multi-camera 3D perception framework that constructs bird’s-eye-view representations using deformable spatiotemporal transformers without relying on LiDAR.
• It employs a two-stage pipeline where a perspective head extracts dense 2D/3D features and a BEV head lifts these into a canonical BEV grid for efficient object detection.
• The framework demonstrates competitive performance on benchmarks such as nuScenes, with BEVFormer v2 achieving improved convergence and compatibility with modern image backbones via perspective-space supervision.

BEVFormer is a unified multi-camera 3D perception framework that constructs Bird’s-Eye-View (BEV) scene representations from raw, synchronized images. By leveraging deformable spatiotemporal transformers, BEVFormer and its successor BEVFormer v2 introduce a BEV-centric pipeline for 3D object detection and map segmentation, providing competitive performance on benchmarks such as nuScenes without relying on LiDAR or explicit depth pretraining. BEVFormer v2 further augments the approach with perspective-space supervision and a two-stage architecture for improved convergence and compatibility with modern image backbones.

1. Architectural Overview: Two-Stage BEVFormer Pipeline

The BEVFormer v2 framework processes multi-camera perception in two principal stages (Yang et al., 2022). The architecture comprises:

• Stage 1: Perspective Head
  • Multi-scale image features are extracted by a standard 2D backbone (e.g., ResNet, InternImage) with an FPN.
  • The Perspective Head directly predicts 2D/3D bounding box parameters (class scores, box offsets, depth, orientation, size, and 3D confidence) per pixel location in the perspective view in a dense FCOS3D style.
  • This head provides strong, dense 3D supervision, enforcing fine-grained representation learning within the backbone.
• Stage 2: BEV Head
  • Image features are spatially transformed (“lifted”) to a canonical BEV grid via a Spatial Encoder utilizing multi-scale deformable cross-attention.
  • Optionally, a Temporal Encoder warps and fuses prior (and, for offline, subsequent) BEV features based on ego-motion.
  • A deformable-DETR-style decoder integrates “hybrid” object queries: learned per-dataset embeddings and the Stage 1 proposal centers projected into BEV space.
  • Final 3D boxes are predicted using these hybrid queries.

The total loss for end-to-end training is:

$$\mathcal{L}_{\rm total} = \lambda_{\rm bev}\,\mathcal{L}_{\rm bev} + \lambda_{\rm pers}\,\mathcal{L}_{\rm pers}$$

with default weights $\lambda_{\rm bev} = \lambda_{\rm pers} = 1$.

BEVFormer v1 (the original variant) consists of a spatiotemporal transformer encoder, where predefined BEV queries organize evidence gathering (spatial cross-attention from camera views, temporal self-attention over BEV histories), followed by a detection or segmentation head (Li et al., 2022).

2. Core Components and Attention Schemes

Backbone and FPN

• BEVFormer supports any standard 2D backbone without LiDAR or depth-specific pretraining. An FPN outputs five scales with fixed channel width ($C=256$).

Perspective Head (Stage 1 in v2)

• Utilizes FCOS3D/MonoDIS logic: dense per-pixel regression of box parameters, leveraging focal and IoU losses for 2D and L1 losses for 3D parameters.
• Non-Maximum Suppression (NMS) is applied first in perspective space per camera, then in BEV coordinates to select top proposals as input for Stage 2.

Spatial Encoder (BEV Lifting)

• Deformable cross-attention “lifts” image features to BEV. For each BEV grid cell, a set of vertical reference points is projected through camera matrices, then multi-scale deformable attention samples features from all image views and aggregated to form the BEV representation.

Temporal Encoder (Memory Fusion)

• Ego-motion compensated BEV feature maps from past (and optionally future) frames are warped and concatenated, then reduced back to $C$ channels via a residual block. The original BEVFormer v1 used a recurrent transformer, but BEVFormer v2 achieves equivalent or better performance with explicit warping and channel concatenation.

BEV Decoder (Stage 2 in v2)

• Deformable DETR layers operate on the fused BEV representation. Queries comprise both (i) learned positional embeddings and (ii) per-image reference points from Stage 1’s perspective proposals.
• Each query samples BEV features around its learned or designated BEV coordinate using predicted offsets, through cross-attention.

The table below contrasts the Stage 1 and Stage 2 queries in BEVFormer v2:

Query Type	Source	Function
Learned queries	Per-dataset positional embeddings	General object location priors in BEV
Proposal queries	Projected Stage 1 (perspective) centers	Anchor object-specific attention in BEV

3. Mathematical Formulation and Losses

BEVFormer formalizes camera-to-BEV transformation by:

• Mapping BEV coordinates $\mathbf{X}_{\rm bev}$ to image space through camera intrinsics/extrinsics:

$\mathbf{u} = K [R|t]\,\mathbf{X}_{\rm bev}$

• Backprojecting pixel positions (with predicted depth) to camera and BEV coordinates:

$$\mathbf{X}_{\rm cam} = d\,K^{-1}[u,v,1]^\top, \; \mathbf{X}_{\rm bev} = R_{cw}\mathbf{X}_{\rm cam}+t_{cw}$$

Losses combine dense perspective-space objectives ($\mathcal{L}_{\rm pers}$: 2D/3D box regression, confidence, derived from mono-depth heads) and BEV-space detection losses (classification, L1 regression in BEV, assignment via Hungarian matching).

4. Spatiotemporal Transformer and Encoder Stack

Original BEVFormer implements each encoder layer as:

• Temporal Self-Attention: Fuses current BEV queries with warped BEV from the prior time step using deformable self-attention, accommodating both motion and occlusion cues.
• Spatial Cross-Attention: For each BEV grid cell, projects fixed 3D “pillar” points through each camera to sample features with multi-head deformable attention from all views.
• Feed-Forward Network: Per-grid feed-forward update.

After $L$ encoder layers, the final BEV grid encodes context across space, time, and cameras (Li et al., 2022).

5. Training Protocols and Implementation

• Optimizer: AdamW, with backbone-specific learning rates and layer-wise decay (0.96–0.94 for InternImage).
• Augmentation: Random horizontal flip, color jitter, image-level augmentations.
• Temporal sampling: 4-second intervals for historical frames.
• Batch sizes of 16–32 and up to 48 epochs (v2 converges rapidly: 24–48 epochs versus 72 for BEV-only models).
• Both heads are trained jointly with equal loss weights; no depth pretraining is required.

Ablation studies show Stage 1’s dense perspective head yields a consistent +3NDS/+2mAP improvement over BEV-only models and supports a broad spectrum of backbone architectures (Yang et al., 2022).

6. Empirical Results, Limitations, and Extensions

• nuScenes test performance (no depth pretraining):
  • BEVFormer v2 (InternImage-B): NDS=0.620, mAP=0.540
  • BEVFormer v2 (InternImage-XL): NDS=0.634, mAP=0.556
• BEVFormer v1 achieves NDS=0.569 with VoVNet backbone and yields state-of-the-art results in vehicle detection and map segmentation (Li et al., 2022, Yang et al., 2022).
• BEVFormer’s deformable attention only sparsely samples image features and cannot benefit from direct LiDAR-based depth supervision. This motivates alternatives such as VoxelFormer, which incorporates dual-view attention, achieving up to +4.9% NDS improvement over the original BEVFormer by more exhaustively aggregating image information and enabling direct lidar supervision (Li et al., 2023).

7. Context, Related Approaches, and Future Directions

Prior to BEVFormer, multi-camera 3D detection often relied on depth-pretrained backbones or sparse projections, limiting interoperability with strong 2D vision models. BEVFormer’s framework detaches BEV representation learning from depth-specific pretraining and is compatible with modern segmentation/detection heads.

Subsequent developments investigate alternative strategies for BEV feature construction, such as dual-view attention (Li et al., 2023), which overcomes BEVFormer’s limitations by exhaustively encoding image features and enabling stronger supervision. The field continues to pursue tighter coupling of camera and BEV representations, improved temporal recurrence, and deeper integration with multi-task settings (detection, segmentation, tracking) in BEV space.

BEVFormer and its variants remain central architectures for image-based 3D scene understanding within autonomous driving, enabling state-of-the-art performance without dependence on 3D sensors or handcrafted geometric pipelines.