SpaRC-AD: Sparse Radar–Camera Fusion

• The paper introduces a novel end-to-end radar–camera fusion framework using a sparse-query decoder to jointly optimize detection, tracking, mapping, and planning.
• It employs range-adaptive radar aggregation and Doppler-based velocity refinement to address challenges such as adverse weather, occlusions, and imprecise motion estimates.
• Empirical results on datasets like nuScenes and Bench2Drive demonstrate significant improvements in key metrics over state-of-the-art vision-only baselines.

Searching arXiv for the relevant paper and closely related work on SpaRC-AD. SpaRC-AD is a query-based, end-to-end camera–radar fusion framework for planning-oriented autonomous driving. It is designed to address limitations of vision-only pipelines in adverse weather, partial occlusions, precise velocity estimation, and long-horizon motion understanding by combining six-view surround cameras with multi-sweep radar point clouds in a unified sparse-query decoder. Its central claim is that sparse 3D feature alignment, range-adaptive radar aggregation, sparse frustum fusion, and Doppler-based velocity refinement can improve not only 3D detection, but also tracking, online mapping, motion prediction, and trajectory planning within a single model (Wolters et al., 14 Aug 2025).

1. Problem formulation and operational scope

SpaRC-AD targets the standard end-to-end autonomous driving objective in which perception, motion forecasting, and planning are optimized jointly rather than as disjoint modules. The motivating failure modes of camera-only systems are stated explicitly: vulnerability in rain, fog, and night conditions; degradation under partial occlusions; and imprecise velocity estimation because monocular depth and motion are inferred implicitly from images and temporal smoothing rather than from direct geometric constraints. Radar is introduced as the complementary modality because it is robust in adverse conditions, provides long-range sensing, and directly measures relative velocity via the Doppler effect (Wolters et al., 14 Aug 2025).

The framework operates on 360-degree surround-view cameras and multi-sweep radar point clouds. On nuScenes, the sensor configuration is six cameras and five radar units at $2\ \mathrm{Hz}$; Bench2Drive uses a similar sensor setup. The design goal is not dense scene rasterization in bird’s-eye view, but a sparse, planning-oriented scene representation in which agents, maps, and ego trajectories are each represented by learned query sets. This places SpaRC-AD within the query-centric end-to-end lineage, while specifically adapting it to radar–camera fusion (Wolters et al., 14 Aug 2025).

The intended output space is correspondingly broad. Agent queries support 3D detection and tracking, map queries support online vectorized mapping, and a dedicated ego query produces multi-modal trajectories conditioned on discrete driving commands. This coupling is operational rather than merely architectural: radar-derived motion cues are used to refine agent anchors, map geometry, and the ego planning state, so the same fused representation participates in all major downstream tasks (Wolters et al., 14 Aug 2025).

2. Sparse-query architecture and scene representation

SpaRC-AD avoids dense BEV grids and instead represents the driving scene through sparse query sets. The camera backbone is a ResNet-50 with an FPN neck that extracts multi-view, multi-scale features. The radar stream encodes multi-sweep radar points by point-wise encoding and serialization with a Point Transformer, producing embedded 3D radar features rather than gridded radar tensors (Wolters et al., 14 Aug 2025).

Three query families define the latent state. Agent queries use anchors $B_d \in \mathbb{R}^{N_d \times 11}$ with parameters

$$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$

paired with instance features $F_d \in \mathbb{R}^{N_d \times C}$. Map queries represent static road elements as polylines

$L_m \in \mathbb{R}^{N_m \times N_p \times 2},$

with up to 20 waypoints per polyline and paired instance features $F_m \in \mathbb{R}^{N_m \times C}$. Planning is handled by an ego query that predicts multi-modal trajectories conditioned on the commands turn left, turn right, and go straight. The trajectories are then rescored using proximity to other agents’ predicted paths in order to encourage socially compliant planning (Wolters et al., 14 Aug 2025).

The decoder alternates deformable perspective aggregation, radar cross-attention, self-attention among queries, and feed-forward networks. Query initialization and refinement are task-specific. Agent queries are initialized from learned instance prototypes and iteratively refined in position, yaw, dimensions, and velocity. Map queries initialize polylines and iteratively refine waypoint positions. The ego planning query pools fused features, including sparse frustum fusion outputs, to initialize a trajectory decoding state (Wolters et al., 14 Aug 2025).

A key structural point is that fusion happens inside the unified decoder rather than in a separate modality-specific preprocessor. Radar therefore enters the pipeline as a first-class contributor to query refinement. This suggests that SpaRC-AD is organized around sparse object- and polyline-level state updates rather than around sensor-level rasterization.

3. Radar–camera alignment, proximity-aware fusion, and Doppler integration

SpaRC-AD’s fusion mechanics are defined by sparse 3D feature alignment. Radar points are projected into camera frustums for sparse frustum fusion, enabling cross-attention between radar features and local image features in perspective space. Standard calibration relations are used: a camera pixel $x_{\mathrm{cam}} = [u,v,1]^T$ with depth $Z_{\mathrm{cam}}$ is back-projected as

$$X_{\mathrm{cam}} = Z_{\mathrm{cam}} K^{-1} x_{\mathrm{cam}}, \qquad X_{\mathrm{ego}} = T_{\mathrm{cam}\rightarrow \mathrm{ego}} X_{\mathrm{cam}},$$

while radar points are transformed by

$$X_{\mathrm{ego}} = T_{\mathrm{radar}\rightarrow \mathrm{ego}} X_{\mathrm{radar}}.$$

These operations align sparse radar points with image features and query anchors (Wolters et al., 14 Aug 2025).

Feature aggregation is explicitly range-adaptive. For query features $B_d \in \mathbb{R}^{N_d \times 11}$0, radar keys and values $B_d \in \mathbb{R}^{N_d \times 11}$1, and 3D positions $B_d \in \mathbb{R}^{N_d \times 11}$2, attention is defined as

$B_d \in \mathbb{R}^{N_d \times 11}$3

The penalty term downweights spatially distant radar points and enforces localized fusion. The same locality principle is applied to map refinement through a point-to-polyline distance defined as the minimum distance from a radar point $B_d \in \mathbb{R}^{N_d \times 11}$4 to a segment between consecutive waypoints $B_d \in \mathbb{R}^{N_d \times 11}$5 and $B_d \in \mathbb{R}^{N_d \times 11}$6 (Wolters et al., 14 Aug 2025).

Doppler-based velocity estimation is a second core component. Radar measures radial velocity

$B_d \in \mathbb{R}^{N_d \times 11}$7

where $B_d \in \mathbb{R}^{N_d \times 11}$8 is wavelength, $B_d \in \mathbb{R}^{N_d \times 11}$9 is Doppler shift, $$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$0 is the speed of light, and $$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$1 is the carrier frequency. Given the line-of-sight unit vector $$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$2, SpaRC-AD maps the radial measurement into ego-frame components by

$$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$3

and uses these components to refine $$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$4 in the agent anchors. Multi-sweep aggregation and proximity-aware attention are reported to stabilize these estimates, which in turn improves $$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$5, tracking, and downstream motion prediction (Wolters et al., 14 Aug 2025).

The geometric consequence is that SpaRC-AD does not treat radar as an auxiliary confidence prior. Radar is used as a metric source for range and motion, and the fusion operators are explicitly designed to preserve that physical structure during query updates.

4. Multi-task learning, decoding heads, and optimization

SpaRC-AD jointly optimizes detection, tracking, mapping, motion prediction, and planning. For 3D detection, the model outputs classification scores and box regressions over $$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$6. Hungarian matching assigns predictions to ground truth. Classification uses focal loss; regression uses $$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$7 or smooth $$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$8 on centers, sizes, orientation, and velocity; and a depth head in perspective view is supervised with $$\{x, y, z, w, h, l, \sin\theta, \cos\theta, v_x, v_y, v_z\},$$9 to stabilize geometry (Wolters et al., 14 Aug 2025).

Tracking is implemented through a streaming decoder with an instance memory queue of three key frames. Cross-attention over history supports learned association without a separate data-association stage. Online mapping predicts polylines such as lane boundaries, dividers, and crosswalks, again with Hungarian matching and $F_d \in \mathbb{R}^{N_d \times C}$0 waypoint losses. Motion prediction is multi-modal for both agents and ego; modes are scored by a classifier with focal loss and supervised by trajectory regression with $F_d \in \mathbb{R}^{N_d \times C}$1. Planning is evaluated by displacement, collision, and consistency metrics, including $F_d \in \mathbb{R}^{N_d \times C}$2, $F_d \in \mathbb{R}^{N_d \times C}$3, Collision Rate, and Trajectory Prediction Consistency (TPC) (Wolters et al., 14 Aug 2025).

The standard minimum average displacement error is given as

$F_d \in \mathbb{R}^{N_d \times C}$4

Planning queries produce multi-modal trajectories over a $F_d \in \mathbb{R}^{N_d \times C}$5 horizon, while motion forecasting spans $F_d \in \mathbb{R}^{N_d \times C}$6. The training procedure is two-stage: first, the multi-modal sparse encoder and detection head are trained for 100 epochs with AdamW and cosine annealing; second, the entire system is jointly fine-tuned for 10 epochs. Reported configuration values include 900 agent anchors, 100 map polylines, 6 decoder layers, a perception range of $F_d \in \mathbb{R}^{N_d \times C}$7, a three-frame instance memory queue, input sizes of $F_d \in \mathbb{R}^{N_d \times C}$8 for nuScenes and $F_d \in \mathbb{R}^{N_d \times C}$9 for Bench2Drive, and batch size 48. Ego status information is deactivated to avoid leakage (Wolters et al., 14 Aug 2025).

The source code for all experiments is reported as available at https://phi-wol.github.io/sparcad/. From a systems perspective, the use of sparse queries and sparse radar points is intended to reduce the memory and computation overhead associated with dense BEV fusion while retaining physically grounded alignment.

5. Empirical performance across perception, prediction, and planning

On nuScenes validation, SpaRC-AD reports gains over a state-of-the-art vision-only baseline across all major tasks. The improvements are especially pronounced in velocity-sensitive metrics, tracking, and long-horizon planning, consistent with the framework’s direct use of radar Doppler and range cues (Wolters et al., 14 Aug 2025).

Task	SpaRC-AD	Vision-only baseline
3D detection	mAP 46.6; NDS 57.0	mAP 41.8; NDS 52.5
Multi-object tracking	AMOTA 0.469; AMOTP 1.129	AMOTA 0.386; AMOTP 1.254
Online mapping	mAP 56.9	mAP 55.1
Motion prediction	mADE 0.58; mFDE 0.93; MR 0.121; EPA 0.53	mADE 0.62; mFDE 0.99; MR 0.136; EPA 0.482
Open-loop planning	Avg. L2 0.50 m; Collision Rate 0.09%; TPC 0.48	Avg. L2 0.61 m; TPC 0.57

The detection improvement includes a marked reduction in velocity error, with $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$0 versus $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$1. This is consistent with the explicit Doppler-based anchor refinement. On Turning-nuScenes, a benchmark constructed from harder turning scenarios, SpaRC-AD reports larger gains: average $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$2 drops from $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$3 to $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$4, average Collision Rate from $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$5 to $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$6, and average TPC from $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$7 to $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$8. At $L_m \in \mathbb{R}^{N_m \times N_p \times 2},$9 horizons, the model also improves long-range consistency on both nuScenes and T-nuScenes, with lower $F_m \in \mathbb{R}^{N_m \times C}$0, lower Collision Rate, and lower TPC than the vision-only baseline (Wolters et al., 14 Aug 2025).

Closed-loop results on Bench2Drive are also reported. SpaRC-AD achieves open-loop average $F_m \in \mathbb{R}^{N_m \times C}$1 versus $F_m \in \mathbb{R}^{N_m \times C}$2, driving score $F_m \in \mathbb{R}^{N_m \times C}$3 versus $F_m \in \mathbb{R}^{N_m \times C}$4, and success rate $F_m \in \mathbb{R}^{N_m \times C}$5 versus $F_m \in \mathbb{R}^{N_m \times C}$6. Qualitative analyses described in the evaluation note improved detection of fast oncoming or camouflaged vehicles at night and in rain, and more stable trajectories under occlusion. The documented ablations attribute these gains to sparse alignment, distance-penalized radar attention, Doppler integration, and history-aware unified queries (Wolters et al., 14 Aug 2025).

6. Relation to adjacent frameworks, limitations, and naming ambiguity

SpaRC-AD is not an isolated design; it sits at the intersection of sparse-query end-to-end driving and sparse radar–camera fusion. Its use of sparse frustum fusion and range-adaptive radar aggregation is directly aligned with SpaRC, a sparse radar–camera fusion transformer for 3D object detection that introduced sparse frustum fusion, range-adaptive radar aggregation, and local self-attention in a query-based perception setting (Wolters et al., 2024). Its end-to-end sparse-query formulation is also closely adjacent to SparseAD, which represents the driving scenario across perception, motion prediction, and planning without dense BEV features, but uses the name “SparseAD” rather than “SpaRC-AD” (Zhang et al., 2024).

The framework’s reported limitations are concrete. Radar point clouds in nuScenes and Bench2Drive are sparse and the training setup uses only a $F_m \in \mathbb{R}^{N_m \times C}$7 perception range, which underutilizes radar’s stated $F_m \in \mathbb{R}^{N_m \times C}$8 potential. nuScenes radar lacks full 4D height information, limiting vertical localization. CARLA-based closed-loop evaluation is constrained by simplified radar phenomenology and suboptimal extrinsics. Temporal and spatial calibration are explicitly identified as critical: miscalibration can degrade alignment and fusion effectiveness (Wolters et al., 14 Aug 2025).

Several future directions are stated. These include comparison with dense BEV fusion baselines, use of raw radar tensors such as range–Doppler or range–angle–time representations, extension to $F_m \in \mathbb{R}^{N_m \times C}$9 perception range, curation of planning-oriented camera–radar datasets, and V2X radar–camera fusion. A plausible implication is that SpaRC-AD should be understood less as a single detector upgraded with radar and more as a planning-oriented sparse-query recipe for multimodal autonomy.

The name itself is potentially ambiguous across literatures. “SpaRC-AD” in this context denotes the autonomous-driving framework described above (Wolters et al., 14 Aug 2025). Similar strings appear in unrelated domains, including Alzheimer’s disease biomarker prediction using sparse coding and correntropy (Wu et al., 2021), FPGA-based adaptive-optics real-time control under the name SPARC (Surendran et al., 2018), and the earlier radar–camera detector SpaRC (Wolters et al., 2024). This naming overlap suggests that domain qualification is necessary when citing the term in technical writing.