---
title: Window-Based Bundle Adjustment
url: https://www.emergentmind.com/topics/window-based-bundle-adjustment
type: topic
---

# Window-Based Bundle Adjustment

Window-based bundle adjustment refers to the formulation and solution of joint pose and/or structure optimization problems over a finite, sliding window of recent states (scans or frames), rather than over all historic measurements. This scheme is central to achieving scalable, real-time performance in visual and LiDAR odometry and mapping, and is now prominent in both classical visual-inertial SLAM and modern dense LiDAR mapping systems. Window-based BA systematically balances local consistency and computational tractability, mitigating drift by repeatedly re-linearizing and optimizing a temporally contiguous subset of variables while marginalizing out the effects of older states.

## 1. Mathematical Foundations of Window-Based Bundle Adjustment

The window-based BA framework optimizes a sequence of recent state variables (e.g., camera or LiDAR poses, velocities, biases) and, where applicable, selected map features or landmarks. Let $x = [x_p; x_l]$ denote the concatenation of pose parameters and landmark (feature) variables. The canonical objective function is
\[
E(x) = \frac{1}{2} \| r(x) \|^2,
\]
where $r(x)$ stacks all relevant residuals (e.g., reprojection errors, geometric feature residuals, inertial preintegration errors, or localization priors). 

To make real-time inference possible as new data arrives, only the most recent $N$ frames are maintained within a sliding window. The system defines a marginalization prior $E_m(x)$ summarizing all information from outside the active window. Optimization within the window takes the form
\[
E_{sw}(x) = \frac{1}{2} \| r_a(x) \|^2 + E_m(x),
\]
where $r_a(x)$ includes only residuals involving active window variables [2109.02182].

The Schur complement or square-root prior is employed for efficient and stable marginalization of old states [2109.02182], and, in selected LiDAR SLAM variants, the feature variables can be analytically eliminated to further reduce computational cost [2010.08215].

## 2. Landmark and Feature Handling in Sliding Windows

The treatment of landmark or feature variables within the window context is a source of major algorithmic diversity:

- **Visual SLAM:** Traditionally, BA jointly optimizes both pose and 3D landmark coordinates, typically using a Schur complement reduction to eliminate landmarks and produce a camera-only system amenable to efficient optimization [2109.02182].

- **LiDAR SLAM and Feature-Reduced BA:** In systems such as BALM and LMBAO, features are parameterized as planar or edge landmarks defined over sets of 3D points [2010.08215, 2209.08810]. The key advance in BALM is the analytic elimination of plane or edge parameters from the cost function: for plane patches, the optimal residual reduces to the smallest eigenvalue of the local sample covariance matrix; for edge features, the sum of the two smallest eigenvalues,
\[
\min_{n,q} \frac{1}{N_f} \sum_i r_i(T, n, q) =
\begin{cases}
\lambda_3 (A_f) & \text{plane} \\
\lambda_2 (A_f) + \lambda_3 (A_f) & \text{edge}
\end{cases}
\]
where $\lambda_k(A_f)$ is the $k$-th ordered eigenvalue of the covariance $A_f$ of the feature’s associated points [2010.08215].

Landmark maps can be actively maintained with observation counting and stability criteria (e.g., deletion if under-observed, excessive drift, or insufficient support). LMBAO maintains a persistent landmark map decoupled from the window, with active grading metrics for retention or culling [2209.08810].

## 3. Marginalization and Prior Construction

Marginalization is a fundamental operation required to remove old states from the active optimization, retaining their information via a prior. Several techniques are outlined:

- **Schur Complement Prior:** The canonical approach forms the linearized Hessian, partitions according to active and marginalized variables, and computes the Schur complement to construct the marginalized prior [2109.02182, 2209.08810].

- **Square-Root Marginalization:** To address numerical instability and conditioning issues arising from Hessian squaring, square-root marginalization stores the prior in a factorized form (matrix square root of the Hessian) and performs marginalization via in-place, rank-revealing Householder QR decomposition. This approach is algebraically equivalent to the Schur complement (and its Moore–Penrose generalized form), but maintains superior conditioning even in the presence of rank-deficient Jacobians [2109.02182].

- **Covariance Pre-accumulation in LiDAR BA:** In LMBAO and BALM, marginalized scans’ contributions to landmark covariances are pre-accumulated and treated as constants in subsequent window optimizations, ensuring that informative priors over landmark geometry are preserved [2010.08215, 2209.08810].

## 4. Efficient Data Association and Feature Extraction

Associating features or landmarks across scans efficiently is critical in both visual and LiDAR BA:

- **Adaptive Voxelization (BALM):** Features are extracted by recursively subdividing the space into octree voxels, split until tight planarity/linearity conditions are satisfied. Each leaf voxel then defines a plane or edge feature, characterized by its centroid, orientation, and associated eigenvalues. The hash-indexed octree allows constant or logarithmic time point-to-feature lookup, replacing costly nearest-neighbor Kd-tree queries [2010.08215].

- **Active Landmark Reprojection (LMBAO):** New scan features are associated with persistent landmarks via reprojection and nearest-neighbor or spherical-image lookup, with the observation count and drift tests performed to maintain or cull features [2209.08810].

This fast association underpins the ability to optimize over large window sizes with dense feature measurements at real-time rates.

## 5. Optimization, Jacobians, and Solvers

Optimization proceeds via standard Gauss–Newton or Levenberg–Marquardt iterations in the reduced pose or state space:

- **Closed-Form Jacobians and Hessians:** For eigenvalue-based residuals (e.g., BALM), analytic first and second derivatives of the eigenvalues with respect to pose/point parameters are derived, allowing second-order Taylor expansion of each residual and exact assembly of the block-sparse Jacobian and Hessian [2010.08215].

- **Sparse Block Structure:** For window size $M$, the pose parameter vector has $6M$ dimensions, and efficient solvers exploit this sparsity. In BALM, a $6M \times 6M$ system (e.g., $M=20 \Rightarrow 120 \times 120$) is solved in each local BA iteration [2010.08215].

- **Implementation Practices:** Many systems leverage efficient linear algebra libraries (e.g., Ceres Solver, BLAS/LAPACK) and, where applicable, automatic differentiation for Jacobian assembly [2209.08810]. The marginalization routines avoid storage of large dense Hessians by operating directly on factorizations or covariance summaries.

## 6. Experimental Results and System Performance

Empirical evaluations in both visual and LiDAR odometry/mapping scenarios consistently validate the benefits of window-based BA:

- **BALM:** Local BA in a window of $M=20$ scans achieves drift as low as $0.038\%$ on campus-scale hand-held LiDAR mapping and outperforms previous systems (LOAM, LeGO-LOAM) in both drift and runtime, solving each full refinement in under 100 ms at 2 Hz, integrated alongside odometry at 10 Hz [2010.08215].

- **LMBAO:** Using a window of $n=4$ active scans with a persistent landmark map, LMBAO reduces mapping APE RMSE by 30–50% over previous methods (Lego-LOAM, VLOM) on indoor and outdoor benchmarks, while supporting real-time inference (odometry step 29–40 ms per frame) [2209.08810].

- **Square-Root Marginalization:** Provides a 36% runtime reduction (compared to conventional Schur-complement marginalization) and avoids numerical failures in single precision, which are prevalent with Hessian-based schemes. The square-root prior is positive semi-definite by construction and preserves the estimator nullspace properties [2109.02182].

These results demonstrate that window-based BA, with efficient marginalization and data association, enables accurate and robust real-time SLAM on a variety of sensor modalities.

## 7. Impact, Variants, and Ongoing Developments

Window-based bundle adjustment constitutes an essential architectural choice in modern SLAM and odometry systems, applicable to visual, visual-inertial, and LiDAR mapping frameworks. Key differentiators among current approaches include: analytic feature elimination (enabling pose-only BA), factorized-matrix prior maintenance for numerical robustness, and the use of actively maintained persistent landmark maps.

A plausible implication is that as sensor modalities proliferate and maps grow in complexity, further research will address adaptive window sizing, multitask joint-optimizations (e.g., combining semantics and geometry), and robustification strategies for aggressive outlier or dynamic scene content.

Window-based BA remains central to SLAM system design, offering a scalable tradeoff between local accuracy and global consistency, with ongoing research advancing mathematical, algorithmic, and system-level optimizations [2010.08215, 2109.02182, 2209.08810].

Source: https://www.emergentmind.com/topics/window-based-bundle-adjustment