Window-Based Bundle Adjustment

• Window-based bundle adjustment is a technique that optimizes a finite sequence of recent states to maintain local consistency in visual and LiDAR SLAM systems.
• It employs advanced marginalization techniques, such as the Schur complement and square-root methods, to efficiently incorporate past information while mitigating drift.
• Experimental implementations like BALM and LMBAO demonstrate significant improvements in drift reduction and runtime performance in real-time mapping.

Window-based bundle adjustment refers to the formulation and solution of joint pose and/or structure optimization problems over a finite, sliding window of recent states (scans or frames), rather than over all historic measurements. This scheme is central to achieving scalable, real-time performance in visual and LiDAR odometry and mapping, and is now prominent in both classical visual-inertial SLAM and modern dense LiDAR mapping systems. Window-based BA systematically balances local consistency and computational tractability, mitigating drift by repeatedly re-linearizing and optimizing a temporally contiguous subset of variables while marginalizing out the effects of older states.

1. Mathematical Foundations of Window-Based Bundle Adjustment

The window-based BA framework optimizes a sequence of recent state variables (e.g., camera or LiDAR poses, velocities, biases) and, where applicable, selected map features or landmarks. Let $x = [x_p; x_l]$ denote the concatenation of pose parameters and landmark (feature) variables. The canonical objective function is

$E(x) = \frac{1}{2} \| r(x) \|^2,$

where $r(x)$ stacks all relevant residuals (e.g., reprojection errors, geometric feature residuals, inertial preintegration errors, or localization priors).

To make real-time inference possible as new data arrives, only the most recent $N$ frames are maintained within a sliding window. The system defines a marginalization prior $E_m(x)$ summarizing all information from outside the active window. Optimization within the window takes the form

$E_{sw}(x) = \frac{1}{2} \| r_a(x) \|^2 + E_m(x),$

where $r_a(x)$ includes only residuals involving active window variables (Demmel et al., 2021).

The Schur complement or square-root prior is employed for efficient and stable marginalization of old states (Demmel et al., 2021), and, in selected LiDAR SLAM variants, the feature variables can be analytically eliminated to further reduce computational cost (Liu et al., 2020).

2. Landmark and Feature Handling in Sliding Windows

The treatment of landmark or feature variables within the window context is a source of major algorithmic diversity:

• Visual SLAM: Traditionally, BA jointly optimizes both pose and 3D landmark coordinates, typically using a Schur complement reduction to eliminate landmarks and produce a camera-only system amenable to efficient optimization (Demmel et al., 2021).
• LiDAR SLAM and Feature-Reduced BA: In systems such as BALM and LMBAO, features are parameterized as planar or edge landmarks defined over sets of 3D points (Liu et al., 2020, Zhang et al., 2022). The key advance in BALM is the analytic elimination of plane or edge parameters from the cost function: for plane patches, the optimal residual reduces to the smallest eigenvalue of the local sample covariance matrix; for edge features, the sum of the two smallest eigenvalues,

$$\min_{n,q} \frac{1}{N_f} \sum_i r_i(T, n, q) = \begin{cases} \lambda_3 (A_f) & \text{plane} \ \lambda_2 (A_f) + \lambda_3 (A_f) & \text{edge} \end{cases}$$

where $\lambda_k(A_f)$ is the $k$-th ordered eigenvalue of the covariance $A_f$ of the feature’s associated points (Liu et al., 2020).

Landmark maps can be actively maintained with observation counting and stability criteria (e.g., deletion if under-observed, excessive drift, or insufficient support). LMBAO maintains a persistent landmark map decoupled from the window, with active grading metrics for retention or culling (Zhang et al., 2022).

3. Marginalization and Prior Construction

Marginalization is a fundamental operation required to remove old states from the active optimization, retaining their information via a prior. Several techniques are outlined:

• Schur Complement Prior: The canonical approach forms the linearized Hessian, partitions according to active and marginalized variables, and computes the Schur complement to construct the marginalized prior (Demmel et al., 2021, Zhang et al., 2022).
• Square-Root Marginalization: To address numerical instability and conditioning issues arising from Hessian squaring, square-root marginalization stores the prior in a factorized form (matrix square root of the Hessian) and performs marginalization via in-place, rank-revealing Householder QR decomposition. This approach is algebraically equivalent to the Schur complement (and its Moore–Penrose generalized form), but maintains superior conditioning even in the presence of rank-deficient Jacobians (Demmel et al., 2021).
• Covariance Pre-accumulation in LiDAR BA: In LMBAO and BALM, marginalized scans’ contributions to landmark covariances are pre-accumulated and treated as constants in subsequent window optimizations, ensuring that informative priors over landmark geometry are preserved (Liu et al., 2020, Zhang et al., 2022).

4. Efficient Data Association and Feature Extraction

Associating features or landmarks across scans efficiently is critical in both visual and LiDAR BA:

• Adaptive Voxelization (BALM): Features are extracted by recursively subdividing the space into octree voxels, split until tight planarity/linearity conditions are satisfied. Each leaf voxel then defines a plane or edge feature, characterized by its centroid, orientation, and associated eigenvalues. The hash-indexed octree allows constant or logarithmic time point-to-feature lookup, replacing costly nearest-neighbor Kd-tree queries (Liu et al., 2020).
• Active Landmark Reprojection (LMBAO): New scan features are associated with persistent landmarks via reprojection and nearest-neighbor or spherical-image lookup, with the observation count and drift tests performed to maintain or cull features (Zhang et al., 2022).

This fast association underpins the ability to optimize over large window sizes with dense feature measurements at real-time rates.

5. Optimization, Jacobians, and Solvers

Optimization proceeds via standard Gauss–Newton or Levenberg–Marquardt iterations in the reduced pose or state space:

• Closed-Form Jacobians and Hessians: For eigenvalue-based residuals (e.g., BALM), analytic first and second derivatives of the eigenvalues with respect to pose/point parameters are derived, allowing second-order Taylor expansion of each residual and exact assembly of the block-sparse Jacobian and Hessian (Liu et al., 2020).
• Sparse Block Structure: For window size $M$, the pose parameter vector has $6M$ dimensions, and efficient solvers exploit this sparsity. In BALM, a $6M \times 6M$ system (e.g., $M=20 \Rightarrow 120 \times 120$) is solved in each local BA iteration (Liu et al., 2020).
• Implementation Practices: Many systems leverage efficient linear algebra libraries (e.g., Ceres Solver, BLAS/LAPACK) and, where applicable, automatic differentiation for Jacobian assembly (Zhang et al., 2022). The marginalization routines avoid storage of large dense Hessians by operating directly on factorizations or covariance summaries.

6. Experimental Results and System Performance

Empirical evaluations in both visual and LiDAR odometry/mapping scenarios consistently validate the benefits of window-based BA:

• BALM: Local BA in a window of $M=20$ scans achieves drift as low as $0.038\%$ on campus-scale hand-held LiDAR mapping and outperforms previous systems (LOAM, LeGO-LOAM) in both drift and runtime, solving each full refinement in under 100 ms at 2 Hz, integrated alongside odometry at 10 Hz (Liu et al., 2020).
• LMBAO: Using a window of $n=4$ active scans with a persistent landmark map, LMBAO reduces mapping APE RMSE by 30–50% over previous methods (Lego-LOAM, VLOM) on indoor and outdoor benchmarks, while supporting real-time inference (odometry step 29–40 ms per frame) (Zhang et al., 2022).
• Square-Root Marginalization: Provides a 36% runtime reduction (compared to conventional Schur-complement marginalization) and avoids numerical failures in single precision, which are prevalent with Hessian-based schemes. The square-root prior is positive semi-definite by construction and preserves the estimator nullspace properties (Demmel et al., 2021).

These results demonstrate that window-based BA, with efficient marginalization and data association, enables accurate and robust real-time SLAM on a variety of sensor modalities.

7. Impact, Variants, and Ongoing Developments

Window-based bundle adjustment constitutes an essential architectural choice in modern SLAM and odometry systems, applicable to visual, visual-inertial, and LiDAR mapping frameworks. Key differentiators among current approaches include: analytic feature elimination (enabling pose-only BA), factorized-matrix prior maintenance for numerical robustness, and the use of actively maintained persistent landmark maps.

A plausible implication is that as sensor modalities proliferate and maps grow in complexity, further research will address adaptive window sizing, multitask joint-optimizations (e.g., combining semantics and geometry), and robustification strategies for aggressive outlier or dynamic scene content.

Window-based BA remains central to SLAM system design, offering a scalable tradeoff between local accuracy and global consistency, with ongoing research advancing mathematical, algorithmic, and system-level optimizations (Liu et al., 2020, Demmel et al., 2021, Zhang et al., 2022).