Rolling Shutter Essential Matrix

• Rolling shutter essential matrix is a generalization of the classical epipolar matrix that accounts for row-dependent camera poses during image capture.
• It models scanline-dependent effects like instantaneous angular and linear velocities using first-order, uniform, and polynomial methods.
• Research advances have led to robust solvers, including Groebner basis and gyro-aided strategies, that enhance pose estimation in high-motion scenarios.

A rolling shutter essential matrix generalizes the standard essential matrix of epipolar geometry to the case where images are acquired with rolling shutter cameras, whose line-by-line readout induces row-dependent camera poses during frame exposure. Unlike the global-shutter case, the pose at which each row is imaged varies as a function of time, leading to fundamentally different two-view geometry. The rolling shutter essential matrix formalizes the algebraic epipolar constraint that relates feature correspondences across two frames while explicitly incorporating these scanline-dependent effects such as instantaneous angular and linear velocities.

1. Mathematical Formulation and Model Variants

In rolling shutter cameras, each image row $v$ is acquired at a time $t_v = t_0 + \lambda v$, where $\lambda$ is the row readout time. The pose of the camera (both rotation and translation) evolves smoothly or even abruptly as a function of the readout progression. For a 3D world point observed at row $v$ in frame $i$ and row $v'$ in frame $j$, the corresponding instantaneous camera poses affect its projection into each image.

The classical essential matrix $E = [t]_\times R$ encapsulates a unique, rigid body motion between two camera frames. For rolling shutter, the essential matrix must depend on the row indices (hence exposure times) in both frames and incorporates higher-order models for the motion across readout. Several parametric variants exist:

• First-order (RS₁) model: Pose evolves linearly with scanline parameter $v$ (Hahn et al., 2024). $R(v) = R_0 + v R_1$, $t(v) = t_0 + v t_1$.
• Uniform velocity model: Constant angular velocity $\omega$ and translation $v$ induce affine changes during readout (Dai et al., 2016).
• Gyroscope-aided model: Uses measured angular velocity to parameterize per-row small-angle rotations for each frame (Lee et al., 2019).
• General polynomial models: Allow for higher-degree scanline dependencies, forming a hierarchy of multi-perspective cameras (Dai et al., 2016).

The generalized rolling shutter epipolar constraint, for correspondences $$(\mathbf x_1, v_1) \leftrightarrow (\mathbf x_2, v_2)$$, takes the bilinear form:

$$\mathbf x_2^\top E_\mathrm{rs}(v_1, v_2) \mathbf x_1 = 0,$$

where $E_\mathrm{rs}(v_1, v_2)$ is a polynomial matrix in $v_1, v_2$ and encodes the underlying motion model (Hahn et al., 2024).

2. Algebraic Structure and Block Representations

The form and dimensionality of the rolling shutter essential matrix depend on the degree and type of readout-dependent motion being modeled:

• Polynomial Structure: For example, in the RS₁ model, $E_\mathrm{rs}(v_1, v_2)$ is typically of total degree two in scanline parameters,

$$E_\mathrm{rs}(v_1, v_2) = E_{00} + v_1 E_{10} + v_2 E_{01} + v_1^2 E_{20} + v_2^2 E_{02} + v_1 v_2 E_{11},$$

with $E_{ij}$ being $3 \times 3$ matrices built from underlying motion parameters (e.g., $R_0, R_1, t_0, t_1$) (Hahn et al., 2024).

• Block Matrix Structure: For the constant angular/linear velocity model, the epipolar constraint can be “flattened” into a $6\times 6$ matrix acting on extended feature vectors such as $f(i) = [x_i; i x_i]$:

$f'(j)^\top E_{RS} f(i) = 0,$

where

$$E_{RS} = \begin{bmatrix} E_0 & E_j \ E_i & E_{ij} \end{bmatrix},$$

with $E_0$ the classical $[t]_\times R$ term, and the others encoding scanline derivatives (Dai et al., 2016).

• Rank and Singularity Constraints: For each fixed $(v_1, v_2)$, $E_\mathrm{rs}(v_1, v_2)$ must have rank 2, generalizing the classical essential matrix’s algebraic properties. In block form, $E_{RS}$ has precisely 10 DOF, and its SVD has two pairs of repeated nonzero singular values and two zeros (Dai et al., 2016).

3. Minimal Solvers and Estimation Pipelines

The higher complexity of rolling shutter geometry leads to a substantial increase in the number of unknowns to estimate. Minimal problem sizes and solution strategies depend on the chosen model:

Model / Approach	Minimal Correspondences	DOF
Classical global-shutter essential	5	5
Uniform RS (block/small-angle model)	10 (rows)	10
RS$_1$ polynomial model	9 point matches (RS₁)	15
Gyro-aided (IMU) RS	5 (with IMU)	5 (post-reduction)
Linear/uniform RS (no IMU)	20 or 44	11-17 [model-specific]

Solvers typically use either:

• Gröbner basis elimination for closed-form minimal solvers, with offline elimination template computation (e.g., $205 \times 225$ action-matrix for gyro-aided model) (Lee et al., 2019).
• Homotopy continuation or root-finding for polynomial systems with higher solution counts (Hahn et al., 2024).
• Nonlinear least squares (e.g., Levenberg–Marquardt) using the Sampson error generalized to the rolling shutter case (Dai et al., 2016).
• RANSAC for robust model selection, especially when minimal samples are large (e.g., $N=20$–44) (Lee et al., 2017).

Gyroscope-equipped variants exploit per-frame measured angular velocities to reduce the problem to a true minimal case solvable in closed form with just five correspondences, with all other methods requiring at least 9–20 point or point+line matches (Lee et al., 2019, Hahn et al., 2024).

4. Epipolar Constraint, Residuals, and Robust Cost Functions

For each correspondence, the rolling shutter essential matrix provides a bilinear constraint incorporating the scanline indices:

$$\mathbf x_2^\top E_\mathrm{rs}(v_1, v_2) \mathbf x_1 = 0.$$

Residuals for data fitting or RANSAC in rolling shutter geometry generalize the Sampson distance:

$$d_{RS}^2 = \frac{(f'(j)^\top E_{RS} f(i))^2}{\sum_k \|\partial_k(E_{RS} f(i))\|^2 + \sum_k \|E_{RS}^\top f'(j)\|^2},$$

where $f(i)$ and $f'(j)$ are extended features, and $\partial_k$ denotes row/column derivatives (Dai et al., 2016). This forms the basis for robust M-estimation and bundle adjustment pipelines, which require the explicit inclusion of scanline-dependent model terms.

5. Experimental Performance and Application Benchmarks

Empirical studies across multiple works demonstrate substantial improvements in relative pose estimation and monocular VO accuracy when employing rolling shutter essential matrix models. Key quantitative findings include:

• Gyro-aided rolling shutter estimation (G-RSRP) achieves median rotation accuracy $\leq1^\circ$ and translation error $\leq5^\circ$ under challenging synthetic and real high-velocity scenarios, using only 5 point correspondences and gyroscope readouts (Lee et al., 2019).
• Polynomial or block-model (no IMU) minimal solvers require 9–20+ correspondences but offer robust performance, with rotation errors reduced from 2.09°→0.48° and translation errors from 0.28 m→0.08 m under severe distortion compared to global-shutter methods (Lee et al., 2017).
• In real smartphone video, rolling-shutter methods improve inlier ratios from 47.2% (GS-only) to 62.9% on RS data, dramatically reducing outlier rates at image boundaries and during fast camera motion (Lee et al., 2017).
• Linear/uniform and 20-/44-point solvers deteriorate rapidly at moderate rolling shutter intensities or fail in RANSAC due to inconsistent inlier support, while IMU-aided or model-driven pipelines maintain robustness (Lee et al., 2019, Lee et al., 2017).

6. Limitations, Assumptions, and Multi-Perspective Context

Rolling shutter essential matrix methods inherently face increased computational cost and a risk of non-uniqueness or degeneracy:

• Computational Complexity: High minimal sample size (20–44) or polynomial solution degree (28–140) raises solver cost (Hahn et al., 2024).
• Linearization Accuracy: Taylor approximations of pose hold only for sufficiently small angular velocity or short readout duration; models become invalid under large rotational motion or rapid acceleration (Lee et al., 2017).
• Feature Distribution: Sufficiently rich and spatially distributed features across scanlines are essential to avoid underconstrained cases; planar or sparse scenes degrade estimation (Lee et al., 2017, Dai et al., 2016).
• Readout Model Consistency: Assumptions of constant $\lambda$ and rectilinear scan require adaptation in nonstandard CMOS sensor layouts (Lee et al., 2017).
• Gauge and DOF: Each matrix block in $E_{RS}$ is constrained via underlying geometric motion, reducing DOF below a naive block count and embedding gauge ambiguity inherent to two-view geometry (Dai et al., 2016).

Rolling shutter essential matrices are placed in a multi-perspective camera hierarchy by the degree of scanline dependence:

Model Degree $d$	Essential Matrix Size	DOF	Limiting Cases
$d=0$ (GS)	$3\times3$	4	Perspective
$d=1$ (pushbroom, etc)	$4\times4$	6	Pure translation or rotation
$d=2$ (linear RS)	$5\times5$ or $6\times6$	8–10	Full constant angular and linear velocity models
$d=3$ (poly camera)	$6\times6$	10	General polynomial camera models

This structure reveals the classical essential matrix as a special case and clarifies the geometric significance and interrelationships of rolling shutter models (Dai et al., 2016).

7. Extensions, Solver Strategies, and Research Directions

Research on the rolling shutter essential matrix continues with the development of more general models and improved algebraic solvers:

• General algebraic solution strategies include Gröbner-basis solvers, polynomial continuation, and tailored nonlinear optimization over the extended essential parameter space (Hahn et al., 2024).
• Minimal problem classification across point/line correspondences refines the set of practical solver signatures (e.g., 9 points → 60 solutions, 7 points + 2 lines → 48 solutions) (Hahn et al., 2024).
• Integration with inertial sensors (IMU/gyroscope) allows parameter reduction and increased efficiency, forming gyro-aided solvers that are robust and closed form (Lee et al., 2019).
• Multi-view and non-rectilinear scan models: Future work generalizes these methods to handle full video sequences and nonstandard rolling shutter readout patterns.
• Benchmarking: Systematic empirical validation across synthetic and real-world datasets provides further insight into numerical conditioning and failure modes.

Rolling shutter essential matrices provide the mathematical infrastructure for precise geometric computer vision in ubiquitous low-cost consumer camera settings, especially mobile and robotics applications affected by rolling shutter artifacts (Lee et al., 2019, Lee et al., 2017, Hahn et al., 2024, Dai et al., 2016).