OnlineBEV: Online 3D BEV Perception

• OnlineBEV is a multi-camera BEV perception system that integrates a recurrent transformer fusion mechanism for online, memory-efficient temporal feature aggregation.
• It employs motion-guided deformable cross-attention to align and fuse current and historical BEV features, effectively compensating for independent object motion.
• The design achieves state-of-the-art 3D detection on nuScenes while maintaining constant memory usage and robust performance in dynamic scenes.

OnlineBEV is an architecture for multi-camera 3D perception that advances temporal feature fusion in Bird's Eye View (BEV) representations. It uniquely integrates a recurrent transformer-based structure with motion-guided deformable cross-attention to enable online, memory-efficient aggregation of sequential BEV features, effectively resolving misalignment due to independent object motion. This system achieves state-of-the-art results for image-based 3D detection on the nuScenes benchmark, surpassing prior dense BEV architectures.

1. Architectural Overview and Data Flow

OnlineBEV processes sequences of surround-view camera images, extracting BEV features for each timestep and aggregating historical context via a recurrent fusion mechanism. Input images $\{I_{t}^c | c=1...6\}$ are encoded with a shared backbone (e.g. ResNet or VoVNet) to generate per-camera feature maps. These are projected from perspective view to BEV space using the Lift–Splat–Shoot (LSS) transform, producing the current frame's BEV feature $F_t \in \mathbb{R}^{C \times H \times W}$. The historical BEV feature $H_{t-1}$ is stored in a single memory slot. OnlineBEV applies its Motion-Guided BEV Fusion Network (MBFNet), a compact $L$-layer transformer, to fuse $F_t$ with $H_{t-1}$, yielding the updated historical BEV feature $H_t$. The aggregated BEV is then consumed by a detection head, typically CenterPoint, to generate 3D bounding boxes, class labels, and velocities. The memory slot is recurrently updated, supporting unbounded temporal fusion with constant memory consumption.

2. Recurrent Temporal Fusion Mechanism

The core temporal fusion mechanism operates over $L$ transformer layers, aligning and aggregating BEV queries by explicit modeling of feature dynamics. At each layer $l$:

• The current and historical BEV are initialized as $q_t^{(0)} \leftarrow F_t$ and $q_{t-1}^{(0)} \leftarrow H_{t-1}$.
• Motion Feature Extraction: Compute $$M_t^{(l)} = \mathrm{CWA}(\mathrm{FC}(q_t^{(l)} - q_{t-1}^{(l)}))$$, where channel-wise attention (CWA) highlights salient motion-induced changes.
• Deformable Alignment: The historical query is aligned to current via deformable cross-attention:

$$Z_{t-1}^{(l)}(p) = \mathrm{DeformAttn}(Q=M_{t}^{(l)}(p),\ \text{ref}=p, V=q_{t-1}^{(l)})$$

leading to

$$\hat q_{t-1}^{(l)} = \mathrm{LN}\left(q_{t-1}^{(l)} + \mathrm{Dropout}(Z_{t-1}^{(l)})\right), \quad q_{t-1}^{(l+1)} = \mathrm{FFN}_1(\hat q_{t-1}^{(l)})$$

• Fusion: Updated queries

$$q_t^{(l+1)} = \mathrm{FFN}_2([q_t^{(l)}\,\|\,\hat q_{t-1}^{(l)}])$$

are concatenated and fused. Final outputs are $H_t = q_t^{(L)}$ and an additional aligned historical counterpart $\hat H_t = q_{t-1}^{(L)}$.

This design affords adaptive spatial warping at every cell of the BEV grid, compensating not just for ego-motion but also for independently moving scene elements.

3. Motion-Guided BEV Fusion Network (MBFNet) Structure

MBFNet, used recurrently in OnlineBEV, comprises two interlocked modules within each transformer layer:

• Motion Feature Extractor (MFE): Difference tensor $\Delta = q_t^{(l)} - q_{t-1}^{(l)}$ is linearly projected and then modulated by CWA, compressing channel dimension ($C \to C'$, with $C=256$, $C'=64$) to form motion excitation features $M_t^{(l)}$. The parameter budget for MFE is approximately 0.1 M.
• Motion-Guided BEV Warping Attention (MGWA): Deformable cross-attention as in Deformable DETR, guided by the cellwise $M_t^{(l)}$. Offsets and weights per grid cell $p$ are predicted from $M_t^{(l)}(p)$, enabling dynamic warping.
• Layer Configuration: Three MBFNet layers ($L=3$) are stacked, each including dropout (0.1) and LayerNorm before each FFN.

This composition allows OnlineBEV to robustly align historical BEV features, even under substantial object motion.

4. Temporal Consistency Loss and Training Objective

Explicit spatial alignment is enforced via Temporal Consistency Learning Loss as follows:

• Two detection heads (sharing weights) predict heatmaps $Q_t$ and $\hat Q_t$ from $H_t$ and $\hat H_t$ respectively.
• The Heatmap-based Temporal Consistency (HTC) loss:

$L_{\mathrm{cons}} = \| Q_t - \hat Q_t \|_2^2$

is minimized, with stop-gradient on $Q_t$ so only $\hat H_t$ is optimized toward $H_t$.

• The full objective is:

$$L = \omega_{\mathrm{cls}} L_{\mathrm{cls}} + \omega_{\mathrm{reg}} L_{\mathrm{reg}} + \omega_{\mathrm{cons}} L_{\mathrm{cons}}$$

Standard weights are $\omega_{\mathrm{cls}}=1$, $\omega_{\mathrm{reg}}=0.25$, $\omega_{\mathrm{cons}}=2$. This formulation anchors the warped historical BEV, stabilizing long-term temporal alignment and promoting more accurate spatiotemporal fusion.

5. Ego-Motion Compensation and Sequence Dynamics

To ensure meaningful temporal integration, historical BEV features are ego-motion compensated into the current coordinate frame using precise vehicle poses before fusion. Memory is reset at detected scene boundaries, preventing contamination between unrelated trajectories. The architecture maintains $\sim3.4$ GB constant memory usage, independent of sequence length. This enables “infinite” context exploitation without resource blow-up. Dynamic objects, which cannot be reconciled by ego-motion alone, are handled via the MBFNet’s cellwise warping, addressing smearing artifacts prevalent in non-motion-aware fusion approaches.

6. Empirical Results on Benchmark Datasets

OnlineBEV's testing on nuScenes demonstrates notable improvements over established methods:

nuScenes Validation Set (ResNet50, 256×704):

Method	#Frames	mAP ↑	NDS ↑
SOLOFusion	17	42.7%	53.4%
StreamPETR	rnt	43.2%	54.0%
SparseBEV	8	43.2%	54.5%
OnlineBEV	rnt	44.4%	54.5%

nuScenes Test Set (V2-99, 640×1600):

Method	#Frames	mAP ↑	NDS ↑
SOLOFusion	17	54.0%	61.9%
StreamPETR	rnt	55.0%	63.6%
SparseBEV	8	55.6%	63.6%
OnlineBEV	rnt	55.8%	63.9%

In ablation studies, recurrent fusion achieves nearly the same performance as parallel fusion of 17 frames but with only a single memory slot. Addition of MBFNet alignment and HTC-loss yield measurable improvements in NDS, supporting the design’s effectiveness.

7. Computational Complexity, Latency, and Design Implications

Comparison of resource use demonstrates OnlineBEV’s efficiency:

Model	GFLOPs	Latency	Memory	Params
SOLOFusion	198.7	72.8ms	3.9GB	65 M
OnlineBEV	205.7	79.3ms	3.4GB	65 M

Query-based methods (SparseBEV, StreamPETR) offer greater speed but lack dense BEV outputs needed for segmentation and occupancy estimation. OnlineBEV trades minimal extra compute (+3.5% GFLOPs) for dense BEV support. The design avoids storing $K$ frames, permitting continual context accumulation without increased storage.

Channel-wise attention in MFE provides measurable gain in alignment and object velocity (mAVE), outperforming naïve difference-based warping.

8. Significance and Context within Perception Research

OnlineBEV embodies a shift in multi-camera BEV perception: it offers online, memory-efficient, and motion-aware recurrent fusion that matches or exceeds parallel fusion strategies in detection accuracy. The motion-guided deformable attention and temporal consistency supervision address limitations of ego-motion-only approaches, particularly for dynamic scene elements. This architecture establishes a new level of dense BEV fusion performance for camera-only 3D object detection in autonomous driving, influencing the design of future BEV-based temporal perception systems (Koh et al., 11 Jul 2025).