Geometric Patch-Based Online Bundle Adjustment

• The paper presents a geometric patch-based online bundle adjustment framework that enables real-time, high-fidelity 3D mapping from UAV imagery.
• It employs grid partitioning of images, DSM and GNSS/IMU priors, and sliding cluster optimization to dramatically reduce computational overhead while preserving accuracy.
• Empirical results on 50MP datasets show sub-second per-image processing and lower reprojection errors compared to conventional global bundle adjustment methods.

Geometric patch-based online bundle adjustment (BA) is a framework for real-time, high-fidelity pose and structure estimation from full-resolution unmanned aerial vehicle (UAV) imagery. This approach addresses the prohibitive computational complexity of conventional global BA methods applied to ultra-high-resolution datasets, especially in time-critical applications (e.g., disaster response). The method achieves real-time global mapping accuracy by partitioning images into grid-based patches, guiding their tracking with navigation and digital surface model (DSM) priors, and restricting optimization to localized sliding clusters of overlapping images. The system is designed for lightweight, onboard computation, enabling direct operation on UAV platforms equipped with modular camera systems, such as the DLR MACS (Iz et al., 8 Nov 2025).

1. Problem Formulation and Global Bundle Adjustment Framework

Let $I_i$ denote the $i$-th image in a temporal sequence ($n$ total), with known intrinsics $K$ and unknown exterior orientation $\{R_i\in SO(3),\, t_i\in\mathbb{R}^3\}$. The system seeks to estimate these poses and the positions of 3D scene points $\{X_j\in\mathbb{R}^3\}$. The classical geometric BA objective is

$$E_{BA}(\{R_i,t_i\},\{X_j\}) = \sum_{i=1}^n \sum_{j\in\Omega_i} \rho\!\Bigl(\bigl\|\,u_{ij} - \pi(K\,[R_i\,|\,t_i]\,X_j)\bigr\|^2\Bigr),$$

where $\pi$ is the pinhole projection model, $u_{ij}$ the measured image location of $X_j$ in image $I_i$, $\Omega_i$ the set of visible points, and $\rho(\cdot)$ a robust loss function (e.g., Huber) to mitigate outliers. The residual is

$$r_{ij} = u_{ij} - \pi(K(R_i X_j + t_i)) \in \mathbb{R}^2,$$

with iterative reweighted least squares weights

$$w_{ij} = \frac{\psi(\|r_{ij}\|)}{\|r_{ij}\|}, \qquad \psi = \rho'.$$

The geometric BA framework thus centers on minimizing $E_{BA}$ over camera orientations and 3D point locations subject to robustified reprojection errors.

2. Patch Partitioning and Propagation

Each full-resolution input image is divided into an $N\times N$ grid of square patches (default: $N=4$ or $5$, $150\times150$ px per patch), yielding $P = N^2$ indexed patches per image. Overlap is not explicit, but is handled by the re-initialization of patches drifting beyond image boundaries. Patches are tracked temporally using GNSS/IMU pose data and DSM-derived footprints. Specifically:

• Image registration guidance leverages GNSS/IMU coarse poses $(R_k, t_k)$.
• DSM footprints: External DSM (e.g., TanDEM-X or SRTM) allows projecting image corners into 3D, extracting georeferenced footprints.

Patch center propagation adopts one of two models:

• Rigid-body GNSS/IMU transform:

$$P_{ij}^{(k)} = K\,[R_k \mid t_k]\, (K^{-1} P_{ij}^{(k-1)}, d)$$

with nominal depth $d$.

• Footprint-based planar shift:

$$P_{ij}^{(k)} = R(\theta_k)(p-c) + c + (g_k - g_{k-1}),$$

where $g_k$ is the geocenter of $I_k$, $c$ is the patch-grid center, and $R(\theta_k)$ is a 2D rotation by the heading $\theta_k$.

The footprint-based model is selected at runtime due to higher accuracy and lower computational cost. Feature extraction (e.g., FAST+BRIEF) and matching occur within patch windows, with strict matching only across corresponding patch IDs in consecutive frames, effectively constraining the correspondence search space.

3. Online Overlap Detection and Sliding Cluster Assembly

Local bundle adjustment is performed on dynamically assembled sliding clusters:

• Overlap determination: Two images overlap if their DSM-projected ground plane footprints intersect, rapidly determined via UAV ECEF/ENU navigation data.
• Sliding cluster mechanism: With cluster size $M$ (e.g., 12 images), each new frame forms a cluster $C = \{I_{k-M+1}, ..., I_k\}$. When full, local BA is invoked on $C$.
• Continuity enforcement: The last 25% of images in a prior cluster are included in the next, ensuring result continuity. Within the overlapping segment, poses are either fixed (first half) or merged via weighted averaging:

$$T_\text{updated} = \frac{w_\text{old} T_\text{old} + w_\text{new} T_\text{new}}{w_\text{old} + w_\text{new}}$$

with weights set according to the number of matched features in each contributing cluster.

4. Cluster-Based Bundle Adjustment and Numerical Optimization

Each cluster-local BA instance is optimized via a Levenberg–Marquardt damped least-squares scheme:

• All observation residuals $r\in\mathbb{R}^{2N_o}$ (with $N_o$ correspondences) are stacked.
• The Jacobian $J = \partial r/\partial x$, for $x = \{R_i, t_i, X_j\}$, is built.
• The approximate Hessian is formed as $H \approx J^T W J$, where $W = \text{diag}(\{w_{ij}\} \otimes I_2)$.
• The update step is

$$\Delta x = - (H + \lambda D)^{-1} g, \qquad D = \text{diag}(H)$$

where $g = J^T W r$ is the preconditioned gradient.

• Camera rotations are updated via the exponential map to remain on $SO(3)$.

The architecture exploits dramatically reduced problem size (few hundred 3D points and $M$ poses) and efficient, patch-constrained feature matching, enabling practical real-time onboard execution.

5. Computational Performance and System Integration

The system is evaluated on MACS datasets with 50MP images. Full pipeline runtime is 66.45 s for 60 images (mean: 1.11 s/image pair), well within the real-time threshold of 2 s even on a 13th-Gen i7 laptop running MATLAB without GPU acceleration. Empirical results include:

• Feature extraction: 48.86 s
• Matching: 0.62 s for ~60 images (enabled by constrained matching windows)
• Local BA steps: ≈16 s total
• Mean reprojection error: 0.710 px ($\sigma=1.076$ px), outperforming incremental BA (0.727 px, $\sigma=1.756$ px)

The method reduces the size of $J$ (the Jacobian) and the number of constraints by two orders of magnitude vis-à-vis full-image BA. The pipeline is fully embedded within the DLR MACS system, processing images immediately post-acquisition for continuous, geo-referenced output suited to downstream real-time mapping and 3D modeling.

6. Applications, Limitations, and Extensions

This patch-based online BA framework is designed for scenarios requiring real-time georeferenced image products from ultra-high-resolution UAV imagery, including disaster response, infrastructure monitoring, and coastal protection. Integration with DSMs and navigation sensors underpins rapid, reliable spatial overlap determination and patch propagation. By forgoing any downsampling and limiting search and optimization spaces, the approach supports accurate 3D mapping at scale on resource-constrained onboard platforms.

A plausible implication is that further performance gains or broader applicability may depend on enhancing feature extraction parallelism, patch-overlap redundancy management, or hybridizing with learning-based feature encoders, but such developments fall outside the immediate remit of the described system.

7. Comparative Significance and Research Outlook

Geometric patch-based online bundle adjustment as implemented by the DLR group (Iz et al., 8 Nov 2025) offers a paradigm in real-time, large-scale photogrammetry for high-value, time-sensitive UAV missions. The approach preserves global bundle adjustment accuracy while strictly capping per-iteration complexity and latency via patch partitioning, navigation-informed search constraints, and cluster-wise local adjustment. Compared to traditional global or incremental BA pipelines, this yields lower reprojection error and sub-second per-frame performance without sacrificing ultrahigh spatial fidelity. This architecture demonstrates the viability of direct, full-resolution BA for on-board systems, opening directions for future research in scalable, robust, and adaptive aerial mapping workflows.