Rolling Shutter SfM

• Rolling Shutter SfM is a 3D reconstruction method that accounts for per-row temporal distortions in rolling shutter cameras.
• It extends traditional multi-view geometry by replacing the fixed essential matrix with a row-dependent variant, enhancing accuracy under variable camera poses.
• Recent approaches integrate minimal solvers, RS-aware bundle adjustment, and learning methods to correct distortions and resolve critical motion ambiguities.

Rolling Shutter Structure-from-Motion (SfM) encompasses the algorithms, geometric models, and system-level methodologies for recovering three-dimensional (3D) scene structure and camera motion from sequences of images captured by cameras employing a rolling shutter (RS) mechanism. RS image formation interleaves spatial and temporal acquisition, causing each image row (or column) to be captured at a distinct time during exposure. These effects fundamentally alter classic multi-view geometry and optimization techniques that underpin SfM, requiring model generalizations, new solvers, and careful handling of critical motion sequences, degeneracy, and ambiguity.

1. Rolling Shutter Camera Geometry and Essential Matrix

The classical SfM assumes global shutter (GS) cameras, where all pixels are exposed simultaneously and the inter-frame geometry is encapsulated by a single essential matrix $\mathbf{E}$. For RS cameras, the pose of the camera continuously changes during image exposure, so each image row corresponds to a distinct effective camera pose. This temporal variation invalidates the fixed essential matrix model.

To address this, Dai, Li, and Kneip introduced the rolling shutter essential matrix, a generalized, row-dependent matrix $\mathbf{E}_{rs}(\lambda,\lambda')$ parameterized by scanline timing indices (1605.00475). The generalized epipolar constraint, for corresponding normalized points $\mathbf{x}$ and $\mathbf{x}'$, reads: $$\mathbf{x}'^\top \mathbf{E}_{rs}(\lambda, \lambda') \mathbf{x} = 0$$ where $\lambda$, $\lambda'$ encode the readout time (row) in each image. For the important case of constant velocity (linear RS model),

$$\mathcal{E}_{rs}(\lambda, \lambda') = \mathbf{R}_{\lambda'}^{T} [\mathbf{t}_{\lambda'} - \mathbf{R}_{\lambda'} \mathbf{R}_{\lambda}^T \mathbf{t}_{\lambda}]_\times \mathbf{R}_{\lambda}$$

with $\mathbf{R}_\lambda$, $\mathbf{t}_\lambda$ the pose at row $\lambda$.

This generalization leads to epipolar geometry not as a set of corresponding lines in two images, but as a family of higher-order curves parameterized by exposure times. Robust geometric estimation (e.g., Sampson distance for RS) requires extending classic notions to account for this higher-order, row-coupled geometry.

2. Hierarchy of Multi-Perspective Camera Models

Rolling shutter cameras fall within the larger hierarchy of multi-perspective camera models. Perspective (pinhole) cameras have a single optical center and all rays intersect at that center. Linear push-broom cameras, frequently used in line-scan systems, vary the center of projection along a single trajectory (with no rotation). RS cameras generalize both:

• Each image row (or column) is exposed from a different, time-varying camera center and orientation.
• The RS model is a superset, allowing simultaneous translation and rotation during acquisition.

The paper (1605.00475) formalizes this hierarchy:

Perspective Camera <--- Push-Broom Camera <--- Rolling Shutter Camera <--- General Multi-Perspective Camera (trans only) (trans & rot)

Push-broom and RS models are linked; the push-broom camera emerges as a special case of the RS model with restricted motion.

3. Self-Calibration and Critical Motion Sequences

Self-calibration of RS cameras, particularly under pure rotational motion, can be reframed as the self-calibration of an “imaginary camera” with unknown, variable skew and aspect ratio, plus a 1D nonlinear distortion analogous to lens distortion (1611.05476). Under a linearized pure rotation model,

$$\begin{bmatrix} c \ r \ 1 \end{bmatrix} \propto (\mathbf{I} + r[\vec{\phi}]_\times)\mathbf{R}\{ \vec{X} - (\vec{p} + r\vec{v}) \}$$

where $r$ indexes the image row and $\vec{\phi}$ is the angular velocity.

The RS effect can be approximately disentangled into:

• A nonlinear transformation (similar to lens distortion).
• A linear mapping interpretable as variable intrinsic skew and aspect ratio.

This reformulation allows treating RS SfM as a self-calibration problem, where unknown, variable skew and aspect ratio are estimated for each image.

A central issue is the existence of critical motion sequences (CMSs): camera trajectories for which the self-calibration (and thus RS SfM) problem becomes degenerate. The general representation of CMSs—derived from the self-calibration constraints—includes all cases where the rolling shutter direction is parallel across frames. In these cases, a 1D gauge freedom permits unconstrained scaling along the RS direction, yielding ambiguous or unstable solutions unless additional constraints (such as setting a reference parameter to zero in a distortion-free image) are imposed.

4. Bundle Adjustment and Optimization under Rolling Shutter

Bundle adjustment (BA) serves as the core of high-quality SfM, jointly optimizing camera poses, 3D structure, and, for RS, camera velocity or scanline-wise pose variables.

In RS-BA, the projection for each feature is governed by its corresponding instantaneous pose. This leads to a significantly higher-dimensional, nonlinear optimization problem:

• Each keyframe has associated pose and velocity (or more general temporal models, e.g., B-splines for continuous-time calibration (2108.07200)).
• Each feature observation is assigned its exact exposure time, often proportional to its row index.

Normalization of measurement points to camera coordinates, followed by explicit modeling of the visual residual covariance (to standardize error contributions), has been shown to make RS-BA robust to planar degeneracy and less sensitive to filming configuration (2209.08503). The covariance-weighted cost is minimized: $$\min_\theta \sum_{j, i} \mathbf{e}_{i}^{j,T} \Sigma_{i}^{j,-1} \mathbf{e}_i^j$$ where the covariance $\Sigma_i^j$ is analytically derived from the RS projection model.

Numerical optimization is accelerated by leveraging the block-sparse structure of the Jacobian and applying customized two-stage Schur complements, enabling efficient updates even for large-scale problems.

5. Differential, Minimal, and Direct Solvers

Several classes of RS SfM solvers have been proposed for core pose estimation and correction:

• Minimal solvers: e.g., under Ackermann motion for automotive scenes, 4-line RANSAC minimal solvers directly estimate angular and translational velocity (1712.03159). These enable real-time frame-wise rolling shutter compensation and SfM initialization in cars.
• Differential methods: For small inter-frame motions, modified differential SfM algorithms exploit per-scanline timing, linearly scaling optical flow to compensate for RS effects under constant velocity or employing minimal 9-point algorithms for constant acceleration. Dense depth maps and RS-aware image rectification can then be performed (1903.03943).
• Direct optimization: In direct sparse odometry, keyframe velocities are estimated explicitly, enabling joint photometric and geometric optimization for rolling-shutter input (1808.00558).

Insoluble degeneracies in BA can be remedied by constraining certain parameters (e.g., fixing one image’s skew/ratio distortion variable to zero in a near-distortion-free frame (1611.05476)).

6. Learning-based and Hybrid Correction Approaches

Deep learning-based RS correction architectures, such as SUNet (2108.04775), context-aware frame synthesis (2205.12912), and bidirectional self-supervised training using reversed RS images (2305.19862), focus on restoring global shutter–equivalent imagery by modeling per-row or per-frame time-varying distortion. These systems estimate dense undistortion flows or correction fields—often symmetrically from multiple frames—and leverage cost volume processing, feature warping, and context aggregation for high-quality, temporally consistent outputs.

Accurate RS correction improves downstream SfM by restoring geometry suitable for standard pipelined approaches (e.g., COLMAP), with robust feature matching, triangulation, and bundle adjustment under corrected, distortion-free conditions.

Hybrid approaches may combine analytic correction fields (such as quadratic rolling shutter motion solvers (2303.18125)) with deep 3D video architectures (RSA²-Net) for correcting under nonlinear motion and extreme occlusion, further advancing practical applicability to real-world, high-performance RS SfM.

7. Dataset, Benchmarking, and Calibration Considerations

Advances are grounded in benchmarking on synthetic and real RS datasets, often with ground truth GS frames for evaluation. The RSLF dataset (2311.01292) provides synthetic benchmark scenes for RS light-field cameras, enabling evaluation of joint 3D shape and motion estimation from a single RS-LF image.

Spatiotemporal calibration—particularly in camera-IMU assemblies—requires continuous-time B-spline modeling of both pose and IMU bias trajectories, with explicit per-row timing. Accurate estimation and calibration of inter-line delay $d$ is demonstrably crucial: neglecting rolling shutter effects during calibration produces errors up to $1^\circ$ in orientation and $2\,\text{cm}$ in translation (2108.07200).

8. Modern System-Level Integrations and Future Directions

Recent frameworks for real-time, robust, and collaborative SfM employ innovations in data association and optimization but are amenable to rolling shutter–specific adaptations. Methods based on Hierarchical Navigable Small World (HNSW) graphs for scalable matching (2407.03939), self-adaptive weighting in local bundle adjustment, and collaborative multi-agent 3D reconstruction pipelines are compatible with rolling shutter scenarios, especially when underpinned by robust RS-aware geometric solvers, bundle adjustment, and pre-correction networks.

A foundational direction appears in model-free, minimal solvers for scanline-dependent pose estimation that use only scanline/line intersection geometry, requiring no motion model (2506.22069). This modular view allows RS SfM to flexibly tackle arbitrary motion and scene configurations, with solvers well-suited for RANSAC initialization and integration into larger pipelines with RS bundle adjustment.

---

Aspect	Rolling Shutter SfM Contributions	Impact
Geometric modeling	Row/time-dependent epipolar geometry, RS essential matrices	Accurate multi-view geometry
Optimization	RS-aware bundle adjustment, covariance modeling, B-splines	Robust, efficient large-scale SfM
Initialization	Minimal/differential solvers, self-calibration, scanline pose	Reliable, model-free RANSAC
Learning-based correction	Dense undistortion flows, self-supervised dual RS input	GS-quality input for classical SfM
System-level integration	Robust distributed matching, multi-agent, modular pipelines	Real-time/collaborative 3D
Calibration	Joint estimation of extrinsics, temporal offsets, line delays	High-precision spatial alignment
Benchmarking	Synthetic & real RS datasets, RS-LF datasets	Standardized evaluation

Rolling Shutter SfM continues to evolve through advances in geometric understanding, optimization methods, model-free algebraic solvers, learning-based correction, and scalable system integration, enabling accurate 3D vision across the broad spectrum of rolling shutter imaging systems.