---
title: Visual-Inertial Bundle Adjustment
url: https://www.emergentmind.com/topics/visual-inertial-bundle-adjustment-viba
type: topic
---

# Visual-Inertial Bundle Adjustment

Visual-Inertial Bundle Adjustment (VIBA) is the class of nonlinear optimization frameworks that jointly estimate sensor trajectory and environment geometry by fusing visual and inertial measurements, typically over a sliding window of keyframes. Visual residuals enforce geometric consistency through feature reprojection or direct photometric error, while inertial residuals enforce dynamic consistency via IMU preintegration. Modern VIBA encompasses a rich set of factor graph, marginalization, and parameterization strategies, supporting dense or sparse mapping, hybrid optimization, multi-sensor integration, and real-time deployment.

## 1. Problem Formulation and State Representation

At its core, VIBA defines a minimum energy or maximum a posteriori estimate over the stacked state of the system by minimizing the sum of visual, inertial, and optionally other sensor residuals. The state vector typically includes, for each keyframe \(i\):

- Pose in a reference frame: \( \mathbf{T}_i \in \mathrm{SE}(3) \); sometimes represented as separate rotation \( \mathbf{R}_i \in \mathrm{SO}(3) \), position \( \mathbf{p}_i \), with velocities \( \mathbf{v}_i \in \mathbb{R}^3 \).
- IMU biases: \( \mathbf{b}_{a,i}, \mathbf{b}_{g,i} \in \mathbb{R}^3 \).
- 3D map points, e.g., \( \mathbf{p}_j^v \in \mathbb{R}^3 \), or dense depth maps in direct methods.

The joint cost function is built from:

- Visual terms, e.g.,
  \[
  E_{\text{vis}} = \sum_{(i,j)\in V} \rho \Big( \left\| u_{i,j} - \pi( \mathbf{T}_i^{-1} \mathbf{p}_j^v ) \right\|_{\Omega_{ij}}^2 \Big)
  \]
- Inertial (IMU preintegration) terms, e.g.,
  \[
  E_{\text{IMU}} = \sum_{(i,k)\in I} \rho \left( r_{\text{IMU}}^\top \Omega_{\text{IMU}} r_{\text{IMU}} \right)
  \]
  where the IMU residuals connect consecutive poses and biases using preintegrated measurements and first-order bias correction ([1904.06504], [2404.01150], [2109.02182]).
- Optional priors/multimodal terms: e.g., LiDAR map residuals ([1803.01104]), wheel odometry ([2403.13714]), roll–pitch factors ([1904.06504]), etc.

The complete VIBA energy, in the case of multi-modal integration, is:
\[
J(\mathcal{X}) = E_{\text{vis}} + E_{\text{IMU}} + E_{\text{prior}} + \ldots
\]

State parameterizations exploit appropriate Lie group structures (e.g., \(\mathrm{SE}(3)\), \(\mathrm{SO}(3)\)), quaternions, or time-continuous bases (e.g., Chebyshev expansions [2404.01150]).

## 2. Visual and Inertial Residuals

### Visual Residuals

Visual residuals can be:

- Reprojection error: The difference between observed and projected image locations. For sparse feature-based BA:
  \[
  r_{\text{proj}}^{ij} = u_{i,j} - \pi( \mathbf{R}_i \mathbf{p}_j + \mathbf{t}_i )
  \]
  where \(\pi(\cdot)\) is the camera projection ([1706.03648], [1803.01104]).
- Photometric error: The intensity difference at matched pixels, often used in direct methods with or without rolling-shutter modeling, and accounting for affine brightness ([1911.01015], [2201.04114]).
- Dense optical flow errors and their learned confidence weights in deep-learning-augmented systems ([2403.13714]).

### Inertial Residuals

Inertial factors connect adjacent (or selected) keyframes using IMU preintegration as in Forster et al.:
\[
r_{\text{IMU}} = \begin{bmatrix}
r_{\Delta R} \\
r_{\Delta v} \\
r_{\Delta p} \\
r_{b}
\end{bmatrix}
\]
with bias correction Jacobians propagated to keep the linearization robust to bias changes ([1706.03648], [1904.06504], [2109.02182]).

### Cross-modal/Map Priors

Some VIBA frameworks include priors such as LiDAR-based constraints:
\[
r_{\text{map}}^j = n_m^{l \top} (q^j - p_m^l)
\]
point-to-plane, with robust information, anchoring local VIO to a global map ([1803.01104]).

## 3. Optimization Strategy, Marginalization, and Real-Time Operation

### Sliding Window and Marginalization

VIBA typically employs a sliding window of \( K \) keyframes; as new data is added, old frames are marginalized to bound computational cost. Marginalization is implemented via:

- Schur complement: Old states or landmarks are eliminated, resulting in a prior on the active variables ([1706.03648], [1904.06504], [2201.04114]).
- Square-root marginalization: Residuals are stored in square-root form via (block-)QR, enhancing numerical stability especially in single precision ([2109.02182]).

Efficient sparse matrix linear algebra (block-sparse Cholesky, Schur elimination) is used to exploit problem structure.

### Non-Rigid, Rigid, and Hybrid BA

- Non-rigid BA: Jointly optimizes all unknowns (poses, map points, biases). Provides maximal accuracy but can be less robust to poor initialization.
- Rigid BA: Holds points fixed, optimizing only poses/velocities/biases—useful for coarse registrations or fast convergence.
- Hybrid/alternating: Alternates rigid and non-rigid steps for practical trade-off ([1803.01104]).

### Structureless and Direct Formulations

- Structureless VIBA eliminates 3D landmarks analytically by formulating visual epipolar constraints between pairs of frames seen as pairwise residuals, favoring faster solve times and smaller state vectors at some accuracy cost ([2502.16598]).
- Direct methods operate on intensity residuals and model the full photometric error, supporting dense geometry and rolling shutter ([1911.01015], [2201.04114]).

### Adaptive, Learned, and Multi-modal Extensions

- Deep networks can augment the BA system by predicting correspondences or IMU biases, with all optimization steps fully differentiable for self-supervised continual adaptation ([2405.16754]).
- Multimodal priors (LiDAR, GNSS, wheel odometry) can be incorporated in a pose-graph or factor-graph setting for global consistency ([2403.13714], [1803.01104]).

## 4. Factor-Graph Structure, Observability, and Loop Closure

### Factor Graph Model

VIBA can be formalized as a factor graph with nodes (poses, velocities, biases, points) and factors (reprojection, IMU, priors). Sparse connectivity ensures scalability.

- Visual-inertial coupling enforces mixture observability, especially in pitch/roll degrees of freedom. Factor design, such as explicit roll–pitch constraints, eliminates underconstrained DOFs ([1904.06504]).
- Loop closure is managed by additional constraints (e.g., pose-graph relative transformation recovered by local/global BA), followed by loop-closure BA for drift correction ([1706.03648], [2306.01173]).

### Delayed Marginalization and Information Recovery

Delayed marginalization allows updating marginalization priors with new linearization points, providing more accurate scale/bias estimation and efficient tightly coupled IMU initialization through pose-graph bundle adjustment ([2201.04114]).

### Observability

Without inertial cues or priors, vision-only BA maintains a 4-DOF gauge freedom (3D translation + yaw). Inertial constraints (especially roll–pitch) render additional DOFs observable, yielding globally consistent estimation ([1904.06504], [1706.03648]).

## 5. Implementation and Experimental Practice

### Algorithmic Pipeline

Typical real-time VIBA pipeline:

1. Ingest image and IMU data.
2. Preintegrate IMU measurements over each interval.
3. Add keyframe when parallax/disparity exceeds threshold.
4. Form visual/inertial (and optionally cross-modal) residuals.
5. Linearize and assemble the normal equations (Gauss–Newton or Levenberg–Marquardt).
6. Marginalize oldest states via Schur or QR.
7. Solve system using block-sparse solvers, update state.
8. For learned systems, back-propagate BA errors to update network weights ([2405.16754]).

Typical window sizes are 8–12 keyframes ([1803.01104], [2502.16598]), with solve times below 50ms per frame, enabling real-time deployment on commodity CPUs/GPUs.

### Specialized Implementations

- Chebyshev polynomial optimization enables time-continuous VIBA, modeling poses as truncated polynomial expansions for direct, low-latency processing ([2404.01150]).
- Multi-fisheye and deep dense systems leverage GPU-based dense residual/Jacobian evaluation with sliding-window optimization and semi-pose-graph BA for large-scale global consistency ([2306.01173], [2403.13714]).

### Handling Dynamic Scenes and Non-Idealities

Dynamic-landmark suppression via inertial-motion prior and probabilistic gating improves robustness in non-static environments, separating static from dynamic feature tracks ([2503.23429]). Rolling-shutter models are critical for non-global-shutter cameras ([1911.01015]).

## 6. Applications, Empirical Performance, and Directions

VIBA enables high-precision, robust, and scale-consistent visual-inertial odometry and SLAM across robotics, automotive, AR/VR, and mobile domains.

Reported performance in representative datasets:

- EuRoC MAV: translation ATE reduction to 0.05m, rotation error 0.11°, sub-40ms solve times ([2502.16598]).
- Robustness to dynamic scenes with 40% ATE reduction vs. VINS-Fusion; negligible computational overhead ([2503.23429]).
- Dense mapping, real-time, and global driftless operation through GNSS/Map constraints and deep dense modules ([2403.13714]).
- Continual online learning yields up to 50% ATE improvement vs. non-adaptive baselines ([2405.16754]).

Limitations remain in degeneracy handling (e.g., low-parallax motion), sensitivity to calibration and parameter tuning, and real-time demands for dense/global optimization at scale ([1904.06504], [2404.01150]).

## 7. Future Developments and Research Frontiers

Open research directions include:

- Tight coupling of additional modalities (barometer, wheel odometry, multiple-cameras) and in-situ online extrinsic calibration ([1904.06504], [2403.13714]).
- Further reducing computational complexity via structureless methods, efficient marginalization, or continuous-time formulations ([2502.16598], [2404.01150]).
- Improved handling of dynamic objects and nonrigid environments ([2503.23429]).
- Fully differentiable, end-to-end visual-inertial learning systems with closed feedback from BA ([2405.16754]).
- Fast reliable initialization under scale/bias uncertainties via delayed-marginalization and pose-graph BA ([2201.04114]).

VIBA continues to be foundational to high-performance, robust, real-time visual-inertial state estimation and mapping, evolving rapidly through integrated algorithmic, machine learning, and sensor fusion advances.

Source: https://www.emergentmind.com/topics/visual-inertial-bundle-adjustment-viba