---
title: 'CTFusion: Camera-Radar Temporal Fusion'
url: https://www.emergentmind.com/topics/ctfusion
type: topic
---

# CTFusion: Camera-Radar Temporal Fusion

CTFusion is an informal shorthand for **CRT-Fusion**, a **Camera–Radar–Temporal Fusion** framework for 3D object detection that fuses multi-view camera images, automotive radar, and temporal information in a **bird’s-eye view (BEV)** representation. The method is designed for settings such as autonomous vehicles and robotics, where dynamic objects invalidate naïve temporal aggregation: if BEV features from multiple frames are concatenated without motion compensation, moving objects are blurred or smeared in feature space. CRT-Fusion addresses this by combining multi-modal BEV fusion with explicit motion estimation and motion-guided temporal alignment, and predicts 3D bounding boxes and velocities with a CenterPoint-style BEV head [2411.03013].

## 1. Task formulation and motivation

The framework takes as input **6 RGB cameras**, **5 radars**, radar point clouds accumulated over several past sweeps such as **6**, and a temporal window of BEV frames such as **6 past frames** for the full model. Its output is a set of 3D boxes with **location, size, orientation, and velocity**. Evaluation follows the standard **nuScenes** protocol with **mAP** and **NDS**, where NDS aggregates mAP with the error terms **mATE, mASE, mAOE, mAVE,** and **mAAE**.

The motivation is rooted in a limitation of prior radar-camera BEV detectors such as CRN and RCBEVDet. These methods typically fuse radar and camera information per frame and then perform temporal fusion by naïve concatenation or simple aggregation of BEV features over time. That treatment ignores object motion: a moving vehicle does not remain stationary in BEV across timestamps, so temporal concatenation mixes misaligned evidence. CRT-Fusion replaces this with explicit **per-pixel motion and occupancy estimation in BEV** and uses those estimates to warp earlier BEV features before aggregation. In effect, temporal fusion becomes a motion-aware alignment process rather than a purely convolutional stacking process.

## 2. Architectural organization and temporal modeling

At each timestamp \(t-k\), the framework produces a fused BEV feature map \(B_{t-k}\). Over a temporal window of \(N\) frames, the model handles the sequence
\[
\{B_t, B_{t-1}, \dots, B_{t-N}\}.
\]
The pipeline consists of four computational stages before detection. First, the camera branch uses a backbone such as **ResNet** or **ConvNeXt**, while the radar branch uses **PointPillars**. Second, the **Multi-View Fusion (MVF)** module fuses radar and image features in both the camera view and BEV, producing a unified BEV feature \(B_{t-k}\) for each frame. Third, the **Motion Feature Estimator (MFE)** predicts a per-pixel velocity map and a BEV occupancy map from each fused BEV feature. Fourth, the **Motion Guided Temporal Fusion (MGTF)** module warps and fuses BEV features recurrently across time using those motion and occupancy estimates. A **CenterPoint-style** BEV detection head then consumes the final temporally fused BEV representation.

The temporal recurrence is operational rather than recurrent-neural in the parameter-sharing sense. MGTF starts from the earliest feature map, aligns it to the next timestamp, fuses the result, and repeats until the current frame is reached:
\[
\tilde{B}_{t-N} \rightarrow B'_{t-N+1} \rightarrow \dots \rightarrow B_t^{\text{final}}.
\]
The implementation uses a **memory bank**, so already fused BEV features are retained and earlier frames do not need to be recomputed. This is the mechanism the paper identifies as essential for keeping temporal cost manageable as the number of historical frames increases.

## 3. Multi-View Fusion: perspective-view and BEV fusion

MVF produces the per-frame fused BEV representation and comprises two stages: **perspective-view fusion via Radar–Camera Azimuth Attention (RCA)** and **BEV fusion via gated fusion**.

In the perspective-view stage, camera features are denoted
\[
F_c \in \mathbb{R}^{N \times C \times H \times W},
\]
and radar BEV features are denoted
\[
F_r \in \mathbb{R}^{C \times X \times Y}.
\]
For each camera view \(i\), a single-image feature \(F_i \in \mathbb{R}^{C \times H \times W}\) is compressed along width and height to obtain
\[
W_i \in \mathbb{R}^{C \times 1 \times W}, \qquad
H_i \in \mathbb{R}^{C \times H \times 1}.
\]
The width-compressed feature \(W_i(j)\) corresponds to a camera azimuth angle \(\theta_c(j)\), while each radar BEV cell \(F_r(x,y)\) has azimuth \(\theta_r(x,y)\). RCA groups radar cells by azimuth:
\[
\mathcal{R}_j =
\left\{F_r(x,y)\mid (x,y)\in \operatorname{argmin}_{(x,y)}^{M}
\left|\theta_c(j)-\theta_r(x,y)\right|\right\}.
\]
For each grouped radar feature, an MLP-based attention mechanism computes weights and forms an enhanced width descriptor:
\[
\mathcal{M}_i(m)=\mathrm{MLP}_2\!\left(\mathrm{MLP}_1(\mathrm{concat}(W_i(j),\mathcal{R}_j(m)))\right),
\]
\[
\alpha_m=\mathrm{softmax}\big(\mathrm{MLP}_3(\mathcal{M}_i(m))\big),
\]
\[
W'_i(j)=\sum_{m=1}^{M}\alpha_m \mathcal{M}_i(m).
\]
The enhanced descriptor is combined with the height-compressed feature by element-wise multiplication,
\[
F'_i = W'_i \odot H_i,
\]
and then concatenated with the original feature and convolved to obtain the fused perspective-view representation. This mechanism uses radar to guide the camera feature along azimuth rather than injecting raw radar points into image depth directly.

The second stage transforms the perspective features into BEV. For each view, the network predicts a depth distribution over discrete bins plus a foreground channel,
\[
D_i \in \mathbb{R}^{(b+1)\times H \times W}.
\]
Foreground scores are thresholded with \(T_p\) to select likely object pixels, which are projected to BEV using camera intrinsics, extrinsics, and depth bins, yielding camera-derived BEV features \(B_c\). In parallel, PointPillars produces radar BEV features \(B_r\). These are fused by a gated fusion network:
\[
\alpha = \sigma(\mathrm{Conv}(\mathrm{concat}(B_c,B_r))),
\]
\[
B = \alpha \odot B_c + (1-\alpha)\odot B_r.
\]
The result is the unified BEV feature \(B_{t-k}\).

The module is motivated by a complementary sensing regime: radar provides accurate but sparse and noisy depth, while cameras provide dense appearance with ambiguous depth. The reported ablations attribute a large portion of the detector’s gains to MVF. BEV radar fusion alone improves mAP by **+8.0%** over the baseline BEVDepth configuration, and RCA adds a further **+1.1% mAP** and **+1.2% NDS** beyond BEV fusion [2411.03013].

## 4. Motion Feature Estimator and Motion Guided Temporal Fusion

The **Motion Feature Estimator** takes each fused BEV feature \(B_{t-k}\) and predicts two dense maps:
\[
M_{t-k}\in \mathbb{R}^{2\times X \times Y}, \qquad
O_{t-k}\in \mathbb{R}^{1\times X \times Y}.
\]
Here \(M_{t-k}(x,y)=(v_x,v_y)\) is a per-pixel velocity in the BEV plane, and \(O_{t-k}(x,y)\) approximates the probability that an object occupies that BEV cell. Both heads are small CNNs with **\(3\times 3\)** and **\(1\times 1\)** convolutions.

Supervision is defined by the BEV overlap between each grid cell and the projection of ground-truth 3D boxes. If \(H(x,y)\) denotes the physical box associated with a BEV cell and \(P(G)\) denotes the BEV projection of the set of ground-truth boxes \(G\), the occupancy ratio is
\[
r(x,y)=\frac{|H(x,y)\cap P(G)|}{|H(x,y)|}.
\]
With threshold \(T_{\text{IoU}}=0.5\), the ground-truth velocity and occupancy maps are
\[
M^{\text{GT}}_{t-k}(x,y)=
\begin{cases}
(v_x^{\text{gt}},v_y^{\text{gt}}), & r(x,y)\ge T_{\text{IoU}}\\
(0,0), & \text{otherwise}
\end{cases}
\]
and
\[
O^{\text{GT}}_{t-k}(x,y)=
\begin{cases}
1, & r(x,y)\ge T_{\text{IoU}}\\
0, & \text{otherwise.}
\end{cases}
\]
These motion and occupancy predictions are not used directly as detector outputs; they are auxiliary geometric signals for temporal alignment.

The **Motion Guided Temporal Fusion** module then aligns BEV features across time. For each location \((i,j)\), with velocity \(M_{t-k}(i,j)=(v_x,v_y)\) and frame interval \(t_s\), the displacement is
\[
\Delta x = v_x \cdot t_s,\qquad \Delta y = v_y \cdot t_s.
\]
Only cells whose velocity magnitude exceeds threshold \(T_v\) — approximately **1 m/s** — are treated as dynamic. If
\[
S(x,y)=\{(i,j)\mid x=i+[\Delta x],\ y=j+[\Delta y],\ |M_{t-k}(i,j)|>T_v\},
\]
then the shifted feature map is
\[
\tilde{B}_{t-k}(x,y)=
\begin{cases}
\frac{1}{|S(x,y)|}\sum_{(i,j)\in S(x,y)} B_{t-k}(i,j), & |S(x,y)|>0\\
B_{t-k}(x,y), & \text{otherwise.}
\end{cases}
\]
This forward shift approximates moving each BEV cell along its estimated velocity vector. The aligned feature is then fused with the next-frame feature by occupancy-gated concatenation:
\[
B'_{t-k+1} = \mathrm{concat}(\tilde{B}_{t-k}, B_{t-k+1}) \odot O_{t-k+1}.
\]
Occupancy gating suppresses background regions unlikely to contain objects and reduces the impact of spurious shifted features.

The paper’s ablations make a strong modality-specific claim about this motion branch: adding MFE and MGTF to **camera-only BEVDepth** reduces performance, with **NDS 46.9** versus **47.4**, indicating that motion estimation is not accurate enough without radar. With radar present, however, MFE and MGTF add **+1.1% NDS** and **+1.1% mAP** over the preceding fusion stage, which the paper attributes to radar-supported velocity prediction [2411.03013].

## 5. Optimization, configurations, and empirical performance

Training uses the total loss
\[
\mathcal{L}_{\text{total}} =
\mathcal{L}_{\text{det}}
+ \lambda_{\text{depth}}\mathcal{L}_{\text{depth}}
+ \lambda_{\text{seg}}\mathcal{L}_{\text{seg}}
+ \lambda_{\text{vel}}\mathcal{L}_{\text{vel}}
+ \lambda_{\text{occ}}\mathcal{L}_{\text{occ}}.
\]
Here \(\mathcal{L}_{\text{det}}\) is the CenterPoint 3D detection loss; \(\mathcal{L}_{\text{depth}}\) is a binary cross-entropy loss over depth bins plus foreground; \(\mathcal{L}_{\text{seg}}\) is a BCE foreground segmentation loss in perspective view; \(\mathcal{L}_{\text{vel}}\) is an MSE velocity regression loss; and \(\mathcal{L}_{\text{occ}}\) is a binary focal loss for BEV occupancy. The reported loss weights are
\[
\lambda_{\text{depth}}=3.0,\quad
\lambda_{\text{seg}}=25,\quad
\lambda_{\text{vel}}=1.0,\quad
\lambda_{\text{occ}}=30.
\]

The model is trained on **nuScenes** with **700 train**, **150 val**, and **150 test scenes** for **24 epochs**. The first **6 epochs** train the per-frame modules without MGTF, and the remaining **18 epochs** train the full model with temporal fusion. The full CRT-Fusion uses **6 past BEV frames**; **CRT-Fusion-Light** uses **3 past frames**. Camera backbones include **ResNet-50**, **ResNet-101**, and **ConvNeXt-B**. The radar branch uses **PointPillars with 6 sweeps**, and CRT-Fusion-Light removes the heavy 2D CNN on radar BEV. The reported BEV grids are **128×128** for the ResNet-50 setup and **256×256** for ResNet-101, with image sizes **256×704** and **512×1408**, respectively.

On the **nuScenes validation** set, the method exhibits a monotonic gain across its main components:

| Configuration | NDS | mAP |
|---|---:|---:|
| BEVDepth (camera-only, reported) | 47.5 | 35.1 |
| Reproduced baseline with temporal fusion like SOLOFusion | 47.4 | 37.8 |
| + BEV radar fusion | 55.4 | 47.8 |
| + RCA | 56.1 | 48.9 |
| + MFE + MGTF (full CRT-Fusion) | 57.2 | 50.0 |

In the **ResNet-50, no-CBGS** configuration, CRT-Fusion improves over CRN by **+1.2% NDS** and **+1.0% mAP**. With **CBGS and ResNet-50**, it improves over RCBEVDet by **+2.9% NDS** and **+5.5% mAP**. With **ResNet-101** and **512×1408** input resolution, the method reaches **NDS 62.1** and **mAP 55.4**, compared with **NDS 60.7** and **mAP 54.5** for CRN. On the **nuScenes test** set with **ConvNeXt-B**, CRT-Fusion reports **NDS 64.9, mAP 58.3** without TTA and **NDS 65.6, mAP 58.9** with TTA; the abstract summarizes this as **+1.7% NDS** and **+1.4% mAP** over the previous best radar-camera method [2411.03013].

## 6. Interpretation, naming, robustness, and limitations

The central intuition is that motion-aware temporal fusion reduces the burden on the detector to learn motion compensation implicitly. For a moving vehicle, naïve feature stacking presents the object at different BEV positions across frames. CRT-Fusion instead estimates per-cell velocity, shifts historical features accordingly, and accumulates them at approximately consistent locations. For static objects and background, predicted velocity remains near zero, so spatial support is preserved. The paper reports that improvements appear across all object speed ranges and are especially strong at medium velocities, where naïve concatenation struggles most.

The framework also couples motion handling with multi-level multi-modal fusion. Radar informs image-space processing through RCA, which improves depth prediction quality before view transformation, and BEV gated fusion balances the contributions of camera and radar in the final per-frame representation. This multi-stage design is associated with better robustness under difficult conditions. The paper reports **more than 15% mAP improvement over camera-only** under all weather and lighting settings, and specifically notes improved night performance relative to CRN, with **mAP 33.0 versus 30.4**.

The term **CTFusion** in this context refers to the same concept as **CRT-Fusion**: a camera-radar BEV detector that performs temporal fusion guided by motion. The method’s three named modules — **MVF**, **MFE**, and **MGTF** — are the core of that designation. It is therefore a naming shorthand rather than a distinct algorithmic variant.

The reported limitations are also explicit. Computational cost still grows with the temporal horizon, even though the memory-bank design is more efficient than some baselines. MGTF depends on the quality of MFE’s velocity predictions, so very sparse radar or very high-speed regimes may degrade alignment quality. The framework is designed for **camera+radar** and does not directly incorporate **LiDAR**. Deployment further assumes synchronized multi-camera and multi-radar streams and non-trivial compute, so real-time embedded use remains challenging, although CRT-Fusion-Light is presented as a lighter alternative. The authors identify longer-horizon recurrent fusion, additional sensing modalities such as LiDAR, and improved motion estimation as natural future directions [2411.03013].

Source: https://www.emergentmind.com/topics/ctfusion