CTFusion: Camera-Radar Temporal Fusion

• CTFusion is a camera–radar temporal fusion framework that fuses multi-view images, radar data, and past BEV frames using explicit motion compensation to align moving objects.
• It employs modules such as Multi-View Fusion (with RCA), Motion Feature Estimation, and Motion Guided Temporal Fusion to effectively merge data from dynamic scenes.
• Experimental results on nuScenes show significant improvements in 3D detection metrics, with notable gains in mAP and NDS compared to camera-only approaches.

CTFusion is an informal shorthand for CRT-Fusion, a Camera–Radar–Temporal Fusion framework for 3D object detection that fuses multi-view camera images, automotive radar, and temporal information in a bird’s-eye view (BEV) representation. The method is designed for settings such as autonomous vehicles and robotics, where dynamic objects invalidate naïve temporal aggregation: if BEV features from multiple frames are concatenated without motion compensation, moving objects are blurred or smeared in feature space. CRT-Fusion addresses this by combining multi-modal BEV fusion with explicit motion estimation and motion-guided temporal alignment, and predicts 3D bounding boxes and velocities with a CenterPoint-style BEV head (Kim et al., 2024).

1. Task formulation and motivation

The framework takes as input 6 RGB cameras, 5 radars, radar point clouds accumulated over several past sweeps such as 6, and a temporal window of BEV frames such as 6 past frames for the full model. Its output is a set of 3D boxes with location, size, orientation, and velocity. Evaluation follows the standard nuScenes protocol with mAP and NDS, where NDS aggregates mAP with the error terms mATE, mASE, mAOE, mAVE, and mAAE.

The motivation is rooted in a limitation of prior radar-camera BEV detectors such as CRN and RCBEVDet. These methods typically fuse radar and camera information per frame and then perform temporal fusion by naïve concatenation or simple aggregation of BEV features over time. That treatment ignores object motion: a moving vehicle does not remain stationary in BEV across timestamps, so temporal concatenation mixes misaligned evidence. CRT-Fusion replaces this with explicit per-pixel motion and occupancy estimation in BEV and uses those estimates to warp earlier BEV features before aggregation. In effect, temporal fusion becomes a motion-aware alignment process rather than a purely convolutional stacking process.

2. Architectural organization and temporal modeling

At each timestamp $t-k$, the framework produces a fused BEV feature map $B_{t-k}$. Over a temporal window of $N$ frames, the model handles the sequence

$\{B_t, B_{t-1}, \dots, B_{t-N}\}.$

The pipeline consists of four computational stages before detection. First, the camera branch uses a backbone such as ResNet or ConvNeXt, while the radar branch uses PointPillars. Second, the Multi-View Fusion (MVF) module fuses radar and image features in both the camera view and BEV, producing a unified BEV feature $B_{t-k}$ for each frame. Third, the Motion Feature Estimator (MFE) predicts a per-pixel velocity map and a BEV occupancy map from each fused BEV feature. Fourth, the Motion Guided Temporal Fusion (MGTF) module warps and fuses BEV features recurrently across time using those motion and occupancy estimates. A CenterPoint-style BEV detection head then consumes the final temporally fused BEV representation.

The temporal recurrence is operational rather than recurrent-neural in the parameter-sharing sense. MGTF starts from the earliest feature map, aligns it to the next timestamp, fuses the result, and repeats until the current frame is reached: $$\tilde{B}_{t-N} \rightarrow B'_{t-N+1} \rightarrow \dots \rightarrow B_t^{\text{final}}.$$ The implementation uses a memory bank, so already fused BEV features are retained and earlier frames do not need to be recomputed. This is the mechanism the paper identifies as essential for keeping temporal cost manageable as the number of historical frames increases.

3. Multi-View Fusion: perspective-view and BEV fusion

MVF produces the per-frame fused BEV representation and comprises two stages: perspective-view fusion via Radar–Camera Azimuth Attention (RCA) and BEV fusion via gated fusion.

In the perspective-view stage, camera features are denoted

$F_c \in \mathbb{R}^{N \times C \times H \times W},$

and radar BEV features are denoted

$F_r \in \mathbb{R}^{C \times X \times Y}.$

For each camera view $i$, a single-image feature $F_i \in \mathbb{R}^{C \times H \times W}$ is compressed along width and height to obtain

$B_{t-k}$0

The width-compressed feature $B_{t-k}$1 corresponds to a camera azimuth angle $B_{t-k}$2, while each radar BEV cell $B_{t-k}$3 has azimuth $B_{t-k}$4. RCA groups radar cells by azimuth: $B_{t-k}$5 For each grouped radar feature, an MLP-based attention mechanism computes weights and forms an enhanced width descriptor: $B_{t-k}$6

$B_{t-k}$7

$B_{t-k}$8

The enhanced descriptor is combined with the height-compressed feature by element-wise multiplication,

$B_{t-k}$9

and then concatenated with the original feature and convolved to obtain the fused perspective-view representation. This mechanism uses radar to guide the camera feature along azimuth rather than injecting raw radar points into image depth directly.

The second stage transforms the perspective features into BEV. For each view, the network predicts a depth distribution over discrete bins plus a foreground channel,

$N$0

Foreground scores are thresholded with $N$1 to select likely object pixels, which are projected to BEV using camera intrinsics, extrinsics, and depth bins, yielding camera-derived BEV features $N$2. In parallel, PointPillars produces radar BEV features $N$3. These are fused by a gated fusion network: $N$4

$N$5

The result is the unified BEV feature $N$6.

The module is motivated by a complementary sensing regime: radar provides accurate but sparse and noisy depth, while cameras provide dense appearance with ambiguous depth. The reported ablations attribute a large portion of the detector’s gains to MVF. BEV radar fusion alone improves mAP by +8.0% over the baseline BEVDepth configuration, and RCA adds a further +1.1% mAP and +1.2% NDS beyond BEV fusion (Kim et al., 2024).

4. Motion Feature Estimator and Motion Guided Temporal Fusion

The Motion Feature Estimator takes each fused BEV feature $N$7 and predicts two dense maps: $N$8 Here $N$9 is a per-pixel velocity in the BEV plane, and $\{B_t, B_{t-1}, \dots, B_{t-N}\}.$0 approximates the probability that an object occupies that BEV cell. Both heads are small CNNs with $\{B_t, B_{t-1}, \dots, B_{t-N}\}.$1 and $\{B_t, B_{t-1}, \dots, B_{t-N}\}.$2 convolutions.

Supervision is defined by the BEV overlap between each grid cell and the projection of ground-truth 3D boxes. If $\{B_t, B_{t-1}, \dots, B_{t-N}\}.$3 denotes the physical box associated with a BEV cell and $\{B_t, B_{t-1}, \dots, B_{t-N}\}.$4 denotes the BEV projection of the set of ground-truth boxes $\{B_t, B_{t-1}, \dots, B_{t-N}\}.$5, the occupancy ratio is

$\{B_t, B_{t-1}, \dots, B_{t-N}\}.$6

With threshold $\{B_t, B_{t-1}, \dots, B_{t-N}\}.$7, the ground-truth velocity and occupancy maps are

$\{B_t, B_{t-1}, \dots, B_{t-N}\}.$8

and

$\{B_t, B_{t-1}, \dots, B_{t-N}\}.$9

These motion and occupancy predictions are not used directly as detector outputs; they are auxiliary geometric signals for temporal alignment.

The Motion Guided Temporal Fusion module then aligns BEV features across time. For each location $B_{t-k}$0, with velocity $B_{t-k}$1 and frame interval $B_{t-k}$2, the displacement is

$B_{t-k}$3

Only cells whose velocity magnitude exceeds threshold $B_{t-k}$4 — approximately 1 m/s — are treated as dynamic. If

$B_{t-k}$5

then the shifted feature map is

$B_{t-k}$6

This forward shift approximates moving each BEV cell along its estimated velocity vector. The aligned feature is then fused with the next-frame feature by occupancy-gated concatenation: $B_{t-k}$7 Occupancy gating suppresses background regions unlikely to contain objects and reduces the impact of spurious shifted features.

The paper’s ablations make a strong modality-specific claim about this motion branch: adding MFE and MGTF to camera-only BEVDepth reduces performance, with NDS 46.9 versus 47.4, indicating that motion estimation is not accurate enough without radar. With radar present, however, MFE and MGTF add +1.1% NDS and +1.1% mAP over the preceding fusion stage, which the paper attributes to radar-supported velocity prediction (Kim et al., 2024).

5. Optimization, configurations, and empirical performance

Training uses the total loss

$B_{t-k}$8

Here $B_{t-k}$9 is the CenterPoint 3D detection loss; $$\tilde{B}_{t-N} \rightarrow B'_{t-N+1} \rightarrow \dots \rightarrow B_t^{\text{final}}.$$0 is a binary cross-entropy loss over depth bins plus foreground; $$\tilde{B}_{t-N} \rightarrow B'_{t-N+1} \rightarrow \dots \rightarrow B_t^{\text{final}}.$$1 is a BCE foreground segmentation loss in perspective view; $$\tilde{B}_{t-N} \rightarrow B'_{t-N+1} \rightarrow \dots \rightarrow B_t^{\text{final}}.$$2 is an MSE velocity regression loss; and $$\tilde{B}_{t-N} \rightarrow B'_{t-N+1} \rightarrow \dots \rightarrow B_t^{\text{final}}.$$3 is a binary focal loss for BEV occupancy. The reported loss weights are

$$\tilde{B}_{t-N} \rightarrow B'_{t-N+1} \rightarrow \dots \rightarrow B_t^{\text{final}}.$$4

The model is trained on nuScenes with 700 train, 150 val, and 150 test scenes for 24 epochs. The first 6 epochs train the per-frame modules without MGTF, and the remaining 18 epochs train the full model with temporal fusion. The full CRT-Fusion uses 6 past BEV frames; CRT-Fusion-Light uses 3 past frames. Camera backbones include ResNet-50, ResNet-101, and ConvNeXt-B. The radar branch uses PointPillars with 6 sweeps, and CRT-Fusion-Light removes the heavy 2D CNN on radar BEV. The reported BEV grids are 128×128 for the ResNet-50 setup and 256×256 for ResNet-101, with image sizes 256×704 and 512×1408, respectively.

On the nuScenes validation set, the method exhibits a monotonic gain across its main components:

Configuration	NDS	mAP
BEVDepth (camera-only, reported)	47.5	35.1
Reproduced baseline with temporal fusion like SOLOFusion	47.4	37.8
+ BEV radar fusion	55.4	47.8
+ RCA	56.1	48.9
+ MFE + MGTF (full CRT-Fusion)	57.2	50.0

In the ResNet-50, no-CBGS configuration, CRT-Fusion improves over CRN by +1.2% NDS and +1.0% mAP. With CBGS and ResNet-50, it improves over RCBEVDet by +2.9% NDS and +5.5% mAP. With ResNet-101 and 512×1408 input resolution, the method reaches NDS 62.1 and mAP 55.4, compared with NDS 60.7 and mAP 54.5 for CRN. On the nuScenes test set with ConvNeXt-B, CRT-Fusion reports NDS 64.9, mAP 58.3 without TTA and NDS 65.6, mAP 58.9 with TTA; the abstract summarizes this as +1.7% NDS and +1.4% mAP over the previous best radar-camera method (Kim et al., 2024).

6. Interpretation, naming, robustness, and limitations

The central intuition is that motion-aware temporal fusion reduces the burden on the detector to learn motion compensation implicitly. For a moving vehicle, naïve feature stacking presents the object at different BEV positions across frames. CRT-Fusion instead estimates per-cell velocity, shifts historical features accordingly, and accumulates them at approximately consistent locations. For static objects and background, predicted velocity remains near zero, so spatial support is preserved. The paper reports that improvements appear across all object speed ranges and are especially strong at medium velocities, where naïve concatenation struggles most.

The framework also couples motion handling with multi-level multi-modal fusion. Radar informs image-space processing through RCA, which improves depth prediction quality before view transformation, and BEV gated fusion balances the contributions of camera and radar in the final per-frame representation. This multi-stage design is associated with better robustness under difficult conditions. The paper reports more than 15% mAP improvement over camera-only under all weather and lighting settings, and specifically notes improved night performance relative to CRN, with mAP 33.0 versus 30.4.

The term CTFusion in this context refers to the same concept as CRT-Fusion: a camera-radar BEV detector that performs temporal fusion guided by motion. The method’s three named modules — MVF, MFE, and MGTF — are the core of that designation. It is therefore a naming shorthand rather than a distinct algorithmic variant.

The reported limitations are also explicit. Computational cost still grows with the temporal horizon, even though the memory-bank design is more efficient than some baselines. MGTF depends on the quality of MFE’s velocity predictions, so very sparse radar or very high-speed regimes may degrade alignment quality. The framework is designed for camera+radar and does not directly incorporate LiDAR. Deployment further assumes synchronized multi-camera and multi-radar streams and non-trivial compute, so real-time embedded use remains challenging, although CRT-Fusion-Light is presented as a lighter alternative. The authors identify longer-horizon recurrent fusion, additional sensing modalities such as LiDAR, and improved motion estimation as natural future directions (Kim et al., 2024).