Coarse-to-Fine Online Calibration

• Coarse-to-fine online calibration is a multi-stage method that begins with a robust, approximate alignment and refines sensor parameters iteratively in real-time.
• It leverages techniques such as RANSAC, sliding-window optimization, and attention-based neural refinement to minimize systematic errors.
• This approach has been successfully applied to LiDAR-camera, multi-camera, and IMU sensor fusion, offering improved accuracy over traditional methods.

Coarse-to-fine online calibration refers to a class of iterative, real-time methods for estimating and refining sensor extrinsic parameters or alignment transformations in robotics, computer vision, and autonomous systems. These techniques operate by first providing a robust, approximate (coarse) solution, typically using minimal assumptions or data, and then progressively refining this estimate (fine stage) using more sophisticated, data-driven, or optimization-based strategies. This multi-stage structure enables online calibration in dynamic, unstructured, or targetless environments, where traditional batch or target-based calibration is either infeasible or insufficient.

1. Conceptual Foundations and Motivations

Coarse-to-fine online calibration emerged to address the critical need for real-time, automatic sensor alignment in multi-modal perception systems where manual procedures or static target-based approaches are impractical. This paradigm is driven by several factors:

• Environmental non-stationarity (e.g., changing vehicle loads, ground deformations, sensor drift).
• The necessity of robust initialization for iterative solvers to avoid local minima.
• The requirement to operate with partial, ambiguous, or noisy data without specialized targets or controlled motion.

Methods in this family typically segment the calibration process into two or more stages:

1. Coarse (Bootstrap) Initialization: Rapid, robust, and often targetless estimation of sensor geometry, typically exploiting correspondences from data itself, such as object detections, odometry, or human pose keypoints.
2. Fine (Iterative/Optimization-Based) Refinement: Exploitation of accumulating data, temporal consistency, and advanced optimization (e.g., sliding-window factor graphs, attention modules, pose graph optimization) to reduce systematic errors and handle complex, non-rigid transformations.

2. Representative Methodologies

Several canonical frameworks exemplify the coarse-to-fine online calibration principle:

LiDAR–Camera Targetless Calibration (CalibRefine)

• Coarse Stage: Establishes a planar homography between LiDAR and camera by matching object-level features via a deep common feature discriminator. Reliable cross-modal correspondences are robustly selected with RANSAC and spatial filtering.
• Fine Stage: Iterative online refinement accumulates correspondence sets over time, optimizing the homography based on reducing reprojection error as more data becomes available. A subsequent attention-based post-refinement uses a Vision Transformer and cross-attention between modalities to non-linearly correct spatial distortions, improving alignment beyond planar assumptions (Cheng et al., 24 Feb 2025).

Human Keypoint–Based Multi-Camera Calibration

• Coarse Priming: Uses known camera intrinsics and rough initial pose guesses to obtain a preliminary multi-view geometry from 2D person keypoints detected by edge devices.
• Fine Iteration: After sufficient multi-view observations, pose parameters are repeatedly optimized in a factor-graph, refining camera poses via bundle-adjustment–like updates conditioned on detected 3D person hypotheses (Pätzold et al., 2022).

LiDAR–VGGT Metric Mapping

• Pre-fusion (Coarse Alignment): Applies a Sim(3) transformation (scale, rotation, translation) to bring monocular VGGT camera trajectories into coarse metric alignment with LiDAR/IMU world coordinates, using Umeyama’s method. Robust session-wise scale estimation is performed with RANSAC.
• Post-fusion (Fine Alignment): Refines cross-modal Sim(3) alignment using bounding-box regularization to preserve scale consistency, followed by global pose graph optimization (PGO) to yield globally consistent metric maps (Wang et al., 3 Nov 2025).

Camera–to–Ground Calibration in Autonomous Driving

• Coarse Feature Initialization: Employs wheel odometry to estimate initial camera motions and predict region-of-interest for ground feature tracking.
• Fine Sliding-Window Factor-Graph: Ingests tracked features into a nonlinear factor graph, optimizing for camera–ground transformation and instantaneous ground geometry to adapt to non-rigid deformations (Li et al., 2023).

IMU–LiDAR–Camera Co-calibration

• Stage 1: Pairwise, Filter-Based Coarse Initialization: Performs independent IMU–camera and IMU–LiDAR extrinsics estimation using hand-eye calibration and online VIO/LIO.
• Stage 2: Joint Fine Optimization: Constructs cross-sensor constraints (e.g., chessboard features) to jointly refine all extrinsic parameters using a unified sliding-window nonlinear least-squares optimizer (lanhua, 2022).

3. Optimization and Update Algorithms

The fine stages of these frameworks universally employ iterative optimization on temporal windows of data. The methodologies include:

• Factor Graph Optimization: States (e.g., sensor poses, ground plane parameters) are connected through factor graphs representing priors, sensor models, and reprojection or residuals, solved via Gauss–Newton or Levenberg–Marquardt algorithms. Marginalization and robust cost functions like Huber norms are used to control computational complexity and manage outliers (Li et al., 2023, Pätzold et al., 2022, lanhua, 2022).
• Attention-Based Neural Refinement: Deep learning modules, particularly self-attention and cross-attention (e.g., ViT), can model global and inter-sensor spatial correlations for non-rigid corrections beyond parametric models (Cheng et al., 24 Feb 2025).
• Sim(3) Registration with Regularization: Alternating minimization for similarity transform estimation incorporates additional scale constraints to prevent divergence in the presence of incomplete FOV overlap (Wang et al., 3 Nov 2025).
• Procedural Matching and Data Association: Greedy closest-point or bipartite matching, RANSAC for outlier rejection, PCA to diagnose degenerate motion, and block-based filtering to ensure spatial diversity in matches (Cheng et al., 24 Feb 2025, Wang et al., 3 Nov 2025).

4. Evaluation Metrics and Empirical Performance

Relevant works standardize on several quantitative criteria for assessing calibration quality and convergence:

Metric	Definition/Goal	Typical Values in Recent Methods
Reprojection Error	2D error of projected correspondences	AED and RMSE < 30 px post-refinement (CalibRefine)
Pose RMSE/Accuracy	$\ell_2$ distance or angular deviation from reference	0.052 m / 0.44° (multi-camera) (Pätzold et al., 2022)
Scale Error	$|s-1|$ in Sim(3) alignment	<1% after full refinement (Wang et al., 3 Nov 2025)
Chamfer/ICP/Color Score	Map alignment or color consistency	3–5× reduction vs. baselines (Wang et al., 3 Nov 2025)

Performance is reported as substantial improvements over both static extrinsics and single-step online alternatives. For example, (Cheng et al., 24 Feb 2025) shows Coarse→Fine stage reductions in AED by 35 px in urban scenes; sliding-window bundle methods reduce translation/rotation errors by 30–50% and 5–10%, respectively, relative to two-step methods (lanhua, 2022).

5. Distinctions, Constraints, and Extensions

Coarse-to-fine online calibration differs from conventional offline, target-based routines in multiple respects:

• Targetless/Markerless Operation: Many pipelines are designed to avoid explicit calibration targets, functioning in-the-wild with object, human, or scene features only.
• Online/Streaming Capability: Sliding-window and streaming optimization accommodate new data as it arrives, maintaining calibration in non-stationary environments.
• Robustness to Degenerate Motion: PCA and motion diagnosis are used to avoid ill-conditioning, especially during phase transitions (e.g., stationarity or highly linear trajectories) (Wang et al., 3 Nov 2025).
• Non-Rigid and Temporally Varying Transformations: Factorization into planar, piecewise, or attention-modeled transformations allows adaptation to dynamic, non-rigid scenarios (e.g., road undulation, chassis deformation) (Li et al., 2023).
• Multi-Modal, Multi-Sensor Fusion: Unified calibration across IMU, LiDAR, and camera is tractable via shared optimization structures, handling geometric and appearance-based constraints (lanhua, 2022).

6. Practical Implementation and Real-Time Considerations

All surveyed methods detail real-time or near-real-time implementations, typically leveraging:

• Sliding-Window Size Control: Fixed lag or blockwise optimization to constrain computational burden.
• Partial Updates/Triggering: Factoring in only sensors or portions of the state affected by new data to localize computation (IMU at high frequency, LiDAR/camera at lower rates).
• Marginalization: Schur complement or equivalent to maintain bounded problem size (lanhua, 2022, Li et al., 2023).
• Optimization Libraries: GTSAM and Ceres are frequently used, given their support for factor graphs and automatic differentiation.

In practice, full bundles converge in several tens of milliseconds per keyframe block, yielding real-time capability (10–20 Hz calibration), even with multi-modal input and high-dimensional state vectors (lanhua, 2022, Wang et al., 3 Nov 2025).

7. Impact, Limitations, and Future Directions

The coarse-to-fine online calibration paradigm has demonstrated significant improvements in accuracy, robustness, and deployability across autonomous driving, robotics, and surveillance applications. Its flexibility allows adaptation to markerless, non-static, and multi-sensor environments, often outperforming both classical target-based and single-stage online approaches in head-to-head evaluations.

Nevertheless, challenges remain:

• Outlier Management: Early coarse stages may propagate erroneous correspondences, necessitating robust RANSAC/enhanced filtering.
• Degenerate Motion/Observability: Linear motion or stationary sensors impair refinement and must be explicitly recognized.
• Non-Rigidity/Complex Deformations: While approaches leveraging attention mechanisms mitigate some issues, non-parametric motion remains challenging.
• Computation/Scalability: High-rate sensor streams and large state spaces demand efficient parallelization and effective state marginalization.

Future advances are anticipated in self-supervised refinement, learning-based outlier rejection, localization under minimal visual cues, and truly continuous, life-long online calibration across diverse sensor suites.