Online Radar-Camera Calibration Methods

• Online radar-camera calibration is the process of estimating and updating the extrinsic alignment between radar and camera sensors without relying on dedicated calibration targets.
• It leverages deep learning techniques and cross-modal correspondence, using methods like cost-volume alignment and geometric refinement to overcome radar sparsity versus image density.
• State-of-the-art approaches integrate temporal filtering and iterative regression to deliver full 6-DoF or rotation-only calibration with real-time performance in diverse conditions.

Searching arXiv for papers on online radar-camera calibration and closely related targetless extrinsic calibration. Online radar-camera calibration denotes the estimation or update of the extrinsic relationship between a radar coordinate frame and a camera coordinate frame during system operation, typically without dedicated calibration targets. In the literature, the problem has been instantiated as rotational auto-calibration for sparse traffic radar, full $SE(3)$ extrinsic recovery from learned cross-modal correspondences, planar homography under a common ground-plane assumption, and direct end-to-end regression inside multi-sensor networks. Across these formulations, the recurring technical obstacle is the mismatch between sparse or uncertain radar observations and dense camera imagery, which has led to two recurrent design patterns: learned common-feature or cost-volume alignment, and geometric refinement or temporal filtering layered on top of learned estimates (Cheng et al., 2023, Schöller et al., 2019, Luu et al., 28 May 2025).

1. Development of the problem

The 2019 work on targetless rotational auto-calibration formulated the online ITS problem as recovery of the $3\times 3$ rotation $R$ aligning sparse radar detections with the same vehicles in the camera image, while explicitly treating translation as secondary because small translational errors in the centimeter range have negligible projective impact at tens of meters distance. It used a coarse and a fine convolutional neural network trained in cascade, with the second network learning the residual decalibration left by the first (Schöller et al., 2019).

A later targetless method shifted from rotation-only correction to full extrinsic estimation by extracting a learned common feature from raw radar Range-Doppler-Angle data and RGB images, classifying whether radar and image detections represent the same physical object, and then estimating the extrinsic transformation matrix through RANSAC followed by Levenberg-Marquardt nonlinear optimization (Cheng et al., 2023). This established a hybrid template in which deep correspondence learning is coupled to classical projective geometry.

Subsequent work expanded both the geometric scope and the systems context. A radar-camera fused multi-object tracking framework treated online calibration as a homography-estimation problem between the radar’s Range-Azimuth plane and the image plane, under the assumption that tracked objects lie approximately on the common ground plane (Cheng et al., 23 Oct 2025). RC-AutoCalib presented itself as the first online automatic geometric calibration method for radar and camera systems and addressed full 6-DoF calibration end-to-end through a Dual-Perspective representation, Selective Fusion Mechanism, Multi-Modal Cross-Attention Mechanism, Explicit Matching Supervision, and a Noise-Resistant Matcher (Luu et al., 28 May 2025). CLRNet extended the setting to camera-lidar-radar calibration, while retaining a pairwise camera-radar branch, and RLCNet introduced simultaneous online calibration of LiDAR, RADAR, and camera with a weighted moving average and outlier rejection for dynamic adjustment of calibration parameters (Kegl et al., 16 Mar 2026, Cholakkal et al., 9 Dec 2025).

Taken together, these works suggest a progression from partial, application-specific correction toward full-pose, targetless, real-time calibration frameworks that can operate either per frame or under short causal temporal windows.

2. Geometric formulations

The classical full extrinsic formulation seeks a rigid-body transform

$$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$

with $R\in SO(3)$ and $t\in\mathbb{R}^3$, such that a radar point $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$ maps into the camera frame as $p_{\text{cam}}=T\cdot p_{\text{radar}}$, and projection into the image satisfies $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$. In the notation of Eq. (1) of the targetless common-feature method, $sP_p=K\,Q\,P_r$ with $3\times 3$0 (Cheng et al., 2023).

The rotational auto-calibration formulation keeps the same projective relationship but restricts the unknown to $3\times 3$1. The full extrinsic is written as $3\times 3$2, but the network predicts a unit quaternion $3\times 3$3, while tilt, pan, and roll are reported through a canonical Euler-angle decomposition. This choice reflects the stated observation that tilt, pan, and roll errors dominate misalignment in modern traffic-radar settings (Schöller et al., 2019).

A distinct formulation arises when the scene is assumed to be approximately planar. In the MOT-oriented framework, all tracked objects are assumed to lie approximately on the common ground plane $3\times 3$4, and the calibration is reduced to a $3\times 3$5 homography $3\times 3$6 between image coordinates $3\times 3$7 and radar Range-Azimuth coordinates $3\times 3$8: $3\times 3$9 Cartesian ground-plane coordinates are then reconstructed as $R$0 and $R$1 (Cheng et al., 23 Oct 2025).

End-to-end regression methods retain the full pose but alter the estimation procedure. RC-AutoCalib predicts an incremental transform $R$2 and updates the current estimate iteratively according to

$R$3

Its output is encoded as a small rotation vector and translation (Luu et al., 28 May 2025). CLRNet and RLCNet likewise represent extrinsics as translation plus unit quaternion, but integrate pairwise estimates into multi-sensor consistency objectives. CLRNet imposes loop closure through $R$4, whereas RLCNet uses a message-passing refinement enforcing consistency relations among predicted transforms (Kegl et al., 16 Mar 2026, Cholakkal et al., 9 Dec 2025).

These formulations are not interchangeable. A plausible implication is that the choice among rotation-only, planar homography, and full $R$5 estimation is driven less by abstract preference than by radar sparsity, scene geometry, deployment range, and the downstream role of the calibration.

3. Cross-modal representation learning and correspondence formation

The targetless common-feature pipeline is explicitly correspondence-centric. A YOLO-based radar detector localizes peaks in the RDA tensor and produces 3D bounding boxes in radar space; a standard YOLO detector on the RGB image produces 2D bounding boxes; a small 3D-CNN extracts a radar feature vector $R$6 from each cropped RDA volume; the image branch produces a corresponding image feature vector $R$7; and a fully connected common feature discriminator classifies whether $R$8 correspond to the same physical object. The discriminator output is $R$9, and a correspondence is declared when $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$0 (Cheng et al., 2023).

The 2019 rotational method does not form explicit object-to-object correspondences. Instead, it constructs a two-stream regression network: an RGB stream based on MobileNet v1 truncated after “conv_dw_3_relu,” followed by two MlpConv layers and a 50-unit embedding, and a radar stream that takes a sparse $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$1 image of projected detections storing $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$2, performs a single $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$3 max-pool to preserve sparsity, and maps it to another 50-unit embedding. Concatenation and fully connected regression then yield a quaternion estimate, trained with Euclidean loss in quaternion space $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$4 (Schöller et al., 2019).

The MOT calibration framework returns to explicit matching, but with a different representation. Its common-feature discriminator uses a radar branch based on a 3D-CNN over dense radar RAD tensors, with separate real and imaginary parts, to produce a 128-D embedding $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$5, and a YOLOv4 backbone to produce a 128-D image embedding $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$6. The classifier $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$7 outputs a match probability, and candidate correspondences are accepted when the match score exceeds $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$8 (Cheng et al., 23 Oct 2025).

RC-AutoCalib makes correspondence learning more explicit and more structured. It projects the data into a frontal view and a bird’s-eye view, extracts feature tensors $$T=\begin{bmatrix}R & t\ 0 & 1\end{bmatrix}\in SE(3),$$9, applies Multi-Modal Cross-Attention in each view, computes an assignment matrix $R\in SO(3)$0 together with per-token matchability scores, and supervises this matching through a dedicated matching loss $R\in SO(3)$1. During training, the Noise-Resistant Matcher excludes unreliable radar points from the positive match set $R\in SO(3)$2 by checking whether an axis-aligned box around a radar point contains at least $R\in SO(3)$3 LiDAR points (Luu et al., 28 May 2025).

CLRNet and RLCNet shift the emphasis from object-level matching to dense feature matching. CLRNet converts the radar point cloud into an equirectangular multi-channel depth image with channels for range, radar cross section, compensated Doppler velocity, and echo time stamp, extracts radar and camera features with modified ResNet-18 backbones, builds a PWC-Net-style cost volume by normalized cross-correlation, and compresses it into a 512-dim global descriptor $R\in SO(3)$4 for translation and rotation regression (Kegl et al., 16 Mar 2026). RLCNet similarly combines an RGB image, an inverse-depth radar projection, and a radar BEV rasterization, computes two 4D cost volumes with displacement $R\in SO(3)$5, then fuses the resulting features either by direct concatenation or by learned soft-mask fusion before quaternion-and-translation prediction (Cholakkal et al., 9 Dec 2025).

Taken together, the cited works suggest two broad representational strategies: object-centric common-feature learning and dense cost-volume or feature-correlation learning. The first aligns naturally with geometric estimation from matched entities; the second aligns naturally with direct pose regression.

4. Estimation procedures and online operation

In correspondence-based calibration, robust geometry remains central. The common-feature method records tentative correspondences between the 3D radar box center $R\in SO(3)$6 and the 2D image center $R\in SO(3)$7, then applies RANSAC by randomly sampling minimal subsets of four correspondences, solving a PnP for $R\in SO(3)$8, computing reprojection error

$R\in SO(3)$9

counting inliers with $t\in\mathbb{R}^3$0, and keeping the hypothesis with the maximum inlier count. The selected inliers are then refined by Levenberg-Marquardt, minimizing

$t\in\mathbb{R}^3$1

The paper describes $t\in\mathbb{R}^3$2 as on the order of 10–20 pixels and the number of RANSAC iterations $t\in\mathbb{R}^3$3 as on the order of 100–1000 (Cheng et al., 2023).

The planar-homography MOT pipeline also relies on robust estimation, but its online procedure is temporal and spatially stratified. Over $t\in\mathbb{R}^3$4 consecutive frames, it accumulates matched pairs, applies block-based sampling with block size $t\in\mathbb{R}^3$5 such as $t\in\mathbb{R}^3$6 pixels to ensure spatially uniform sampling, splits correspondences into upper and lower image regions, and estimates $t\in\mathbb{R}^3$7 and $t\in\mathbb{R}^3$8 separately using DLT plus RANSAC. The default tolerance is 3 m in RA space, and the current Python implementation yields approximately 19 Hz, with bottlenecks identified as large RAD tensor I/O and common-feature-discriminator inference over $t\in\mathbb{R}^3$9 object pairs (Cheng et al., 23 Oct 2025).

Direct-regression methods eliminate online iterative geometric solvers but often reintroduce iteration at the network level. The rotational auto-calibration approach uses a two-stage cascade $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$0, each pass requiring on the order of 5–10 ms, for overall latency below 20 ms per frame and $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$1 Hz on a modern GPU such as an NVIDIA Titan X or Jetson TX2. It optionally averages the predicted quaternion over a short temporal window of at least five frames to suppress per-frame noise (Schöller et al., 2019). RC-AutoCalib performs iterative online updates by repeatedly warping radar to frontal-view and BEV maps with the current $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$2, extracting features, regressing $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$3 through a 3-step LSTM autoregressive decoder, and updating $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$4; its reported inference is in the tens of milliseconds on a GPU (Luu et al., 28 May 2025).

CLRNet addresses large miscalibrations through a cascade of four separately trained models spanning $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$5. In rigid-platform mode it can aggregate four consecutive frames and take the median of per-frame predictions, while in non-rigid mode each frame is calibrated independently. On a single NVIDIA-1080 Ti, projection plus forward pass is reported as approximately 20 ms per frame, or 50 fps (Kegl et al., 16 Mar 2026).

RLCNet adds an explicit online calibration module around its per-frame predictions. Translation is smoothed by a weighted moving average with window length $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$6 and decay factor $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$7, while rotation is smoothed by iterative SLERP. Outlier rejection checks $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$8 with $p_{\text{radar}}=[x_r,y_r,z_r,1]^T$9 and $p_{\text{cam}}=T\cdot p_{\text{radar}}$0 with $p_{\text{cam}}=T\cdot p_{\text{radar}}$1 cm before admitting estimates to the moving window. With $p_{\text{cam}}=T\cdot p_{\text{radar}}$2 and $p_{\text{cam}}=T\cdot p_{\text{radar}}$3, the system is reported to react to a 1 cm shift or $p_{\text{cam}}=T\cdot p_{\text{radar}}$4 drift within two frames. The full five-stage iterative cascade runs in approximately 54 ms, i.e. about 18 Hz, on an NVIDIA RTX 3080 laptop GPU (Cholakkal et al., 9 Dec 2025).

5. Reported empirical performance

Reported metrics are not directly uniform across the literature: some papers optimize angular error only, others report 2D reprojection error, others report homography residuals in pixels or RA space, and recent end-to-end methods report translation and rotation pose error. This suggests that numerical comparison is meaningful only within a shared formulation and dataset.

Paper	Calibration setting	Reported result
"Targetless Rotational Auto-Calibration of Radar and Camera for Intelligent Transportation Systems" (Schöller et al., 2019)	Rotation only	On $p_{\text{cam}}=T\cdot p_{\text{radar}}$5, after $p_{\text{cam}}=T\cdot p_{\text{radar}}$6: tilt $p_{\text{cam}}=T\cdot p_{\text{radar}}$7, pan $p_{\text{cam}}=T\cdot p_{\text{radar}}$8, roll $p_{\text{cam}}=T\cdot p_{\text{radar}}$9, total $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$0; on unseen-gantry $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$1: $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$2 tilt, $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$3 pan, $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$4 roll
"Online Targetless Radar-Camera Extrinsic Calibration Based on the Common Features of Radar and Camera" (Cheng et al., 2023)	Full extrinsic via correspondences, RANSAC, LM	24 total correspondences, 21 RANSAC inliers; MARE over all 24 points $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$5 px, RMSRE $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$6 px; MARE over inliers $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$7 px, RMSRE $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$8 px
"Radar-Camera Fused Multi-Object Tracking: Online Calibration and Common Feature" (Cheng et al., 23 Oct 2025)	Homography under ground-plane assumption	Without up-down split: Rad→Cam reprojection error $s\,p_{\text{pixel}}=K\,[R\mid t]\,p_{\text{radar}}$9 px, Cam→Rad $sP_p=K\,Q\,P_r$0 px; with split: Rad→Cam $sP_p=K\,Q\,P_r$1–$sP_p=K\,Q\,P_r$2 px, Cam→Rad $sP_p=K\,Q\,P_r$3–$sP_p=K\,Q\,P_r$4 px; fusion MOTA $sP_p=K\,Q\,P_r$5 in controlled scenes and $sP_p=K\,Q\,P_r$6 in real-world scenarios
"RC-AutoCalib: An End-to-End Radar-Camera Automatic Calibration Network" (Luu et al., 28 May 2025)	Full 6-DoF end-to-end on nuScenes	For $sP_p=K\,Q\,P_r$7: rotation mean $sP_p=K\,Q\,P_r$8, translation mean $sP_p=K\,Q\,P_r$9 cm; for $3\times 3$00: rotation mean $3\times 3$01, translation mean $3\times 3$02 cm
"CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning" (Kegl et al., 16 Mar 2026)	Camera-radar branch on VoD	CLRNet median errors: $3\times 3$03 cm and $3\times 3$04 in scenario A, $3\times 3$05 cm and $3\times 3$06 in scenario B, $3\times 3$07 cm and $3\times 3$08 in scenario C; CLRNet+4 reaches $3\times 3$09 cm and $3\times 3$10 in rigid+4
"RLCNet: An end-to-end deep learning framework for simultaneous online calibration of LiDAR, RADAR, and Camera" (Cholakkal et al., 9 Dec 2025)	Simultaneous online calibration, radar-camera subset	Final mean errors for the radar-camera pair: $3\times 3$11, $3\times 3$12 cm; closed-loop error $3\times 3$13 cm

The experimental setups also differ materially. The common-feature targetless method used a TI AWR1843BOOST mmWave transceiver with DCA1000 capture board, a USB8MP02G 8 MP USB camera, synchronized capture at 30 Hz, and 1800 frames over the first minute in a real-world parking-lot scenario with a car and a moving pedestrian; qualitatively, projected radar points aligned tightly with tracked image-feature trajectories beyond the calibration window (Cheng et al., 2023). The rotational ITS method recorded a highway scene on Autobahn A9 and generated synthetic decalibrations over real samples, yielding 37,929 total samples for one gantry with held-out tests on both the same and a physically distinct second gantry (Schöller et al., 2019). RC-AutoCalib evaluated on a nuScenes subset of 12.6k train, 1.6k validation, and 1.6k test samples, while CLRNet used View-of-Delft and Dual-Radar and explicitly reported cross-domain degradation during VoD↔Dual-Radar transfer (Luu et al., 28 May 2025, Kegl et al., 16 Mar 2026).

A practical reading of these results is that online radar-camera calibration has bifurcated into two empirical regimes. One regime accepts stronger geometric assumptions or narrower correction targets and often reports low angular or reprojection residuals; the other pursues full 6-DoF correction and increasingly reports centimeter-scale translation and sub-degree rotation on public datasets, particularly when multi-frame aggregation or multi-sensor consistency is available.

6. Assumptions, limitations, and open issues

A common misconception is that online radar-camera calibration is a single problem with a single metric. The literature instead spans rotation-only correction, planar homography, full $3\times 3$14 regression, and simultaneous trilateral calibration; correspondingly, evaluation is reported in angular error, pixel reprojection error, RA-space residuals, translation error, rotation error, and downstream MOT metrics. This heterogeneity is explicit in the surveyed methods and complicates direct benchmarking.

Several limitations recur. Radar sparsity remains foundational: the 2019 ITS method restricted itself to rotation because traffic radars return a sparse set of object centroids and because roll recovery is intrinsically harder to learn, especially at long distances (Schöller et al., 2019). Height ambiguity and false radar returns remain serious enough that RC-AutoCalib introduced a Dual-Perspective representation and a Noise-Resistant Matcher, while also acknowledging vulnerability to the pretrained dense depth predictor in extreme weather and moderately high translation error in large mis-calibration $3\times 3$15 (Luu et al., 28 May 2025). The MOT-oriented homography approach is bounded by the planar ground assumption, explicitly notes that non-flat terrain or sensor pitch or roll drift may violate the model, and identifies common-feature interpretability as open because the learned embeddings $3\times 3$16 remain a black box (Cheng et al., 23 Oct 2025).

Runtime and deployment constraints are also uneven. Some pipelines already report embedded or desktop-GPU real-time behavior, such as $3\times 3$17 Hz for the cascaded rotational method and 50 fps for CRNet/CLRNet, while others remain below typical 30 Hz automotive requirements in their current implementation, such as approximately 19 Hz for the Python homography pipeline and approximately 18 Hz for the five-stage RLCNet cascade (Schöller et al., 2019, Kegl et al., 16 Mar 2026, Cheng et al., 23 Oct 2025, Cholakkal et al., 9 Dec 2025). A plausible implication is that “online” in this literature denotes causal operation during system use, but not a uniform throughput target.

Generalization is another unresolved axis. The 2019 rotational work reported degradation on a second, physically distinct gantry, especially in pan and roll, and CLRNet’s VoD↔Dual-Radar experiments showed that cross-domain errors rise even when the method retains a 20–30% advantage over the prior state of the art (Schöller et al., 2019, Kegl et al., 16 Mar 2026). This suggests that hardware configuration, viewing geometry, and dataset-specific radar statistics remain significant domain variables.

Open directions are stated explicitly across the papers: more diverse multi-site and multi-angle training data, short temporal sequences or filtering for stability, adaptive calibration with continuous homography refinement, 3D-aware fitting that incorporates elevation variations, interpretable common-feature visualization, temporal consistency, self-supervised updates on the fly, and fully 3D transformer models on raw point clouds and images (Schöller et al., 2019, Cheng et al., 23 Oct 2025, Luu et al., 28 May 2025). The aggregate trajectory of the field is therefore not merely toward higher accuracy, but toward calibration models that remain causal, targetless, robust to drift and sparsity, and portable across sensors, scenes, and operating conditions.