Differentiable Uncertainty-Aware Bundle Adjustment

• The paper introduces a differentiable framework incorporating learned measurement uncertainty into bundle adjustment, significantly enhancing robustness and convergence in noisy environments.
• It leverages Gauss–Newton and Levenberg–Marquardt solvers integrated with deep networks to enable end-to-end differentiation and uncertainty calibration.
• Applied across SLAM, SfM, and LiDAR domains, the method achieves faster convergence and improved accuracy while mitigating the impact of outlier measurements.

Differentiable uncertainty-aware bundle adjustment (UBA) refers to a class of optimization methods that integrate per-measurement or per-feature uncertainty models directly within the classic bundle adjustment pipeline, while maintaining end-to-end differentiability for integration with deep learning frameworks. Such methods address the key challenge that real-world measurement data, especially from vision or LiDAR sensors, is unavoidably noisy, and that the noise characteristics can often be predicted or estimated from upstream neural modules. UBA frameworks reinterpret all residuals as weighted by data-driven or measurement-driven uncertainty estimates, propagate such weights through nonlinear least squares (e.g., via Gauss–Newton or Levenberg–Marquardt solvers), and differentiate through the solver itself to enable joint learning or uncertainty calibration. This approach yields improved robustness, efficiency, and state-of-the-art accuracy across robotics, SLAM, and structure-from-motion problems in highly variable real-world conditions.

1. Mathematical Formulation and Differentiable BA as an Implicit Layer

The generic weighted bundle adjustment objective considers a parameter vector $\xi$ encoding camera poses, depths, or structure parameters, together with a set of $N$ correspondences $\{X_i \leftrightarrow u_i\}$, where $X_i\in\mathbb{R}^3$ and $u_i\in\mathbb{R}^2$. Uncertainty-aware BA solves

$$\min_{\xi} \sum_{i=1}^N w_i \cdot \|\pi(X_i; \xi) - u_i\|^2,$$

where $\pi(\cdot;\xi)$ is the projection function and the scalar weights $w_i>0$ encode reliability, uncertainty, or outlier confidence. In modern UBA pipelines, a deep network $f_\theta$ predicts not only the flow correction $\delta_i$ but also the per-correspondence uncertainty $N$0, forming

$N$1

Bundle adjustment is treated as an implicit differentiable layer: each forward pass solves for $N$2 via iterative (damped) Gauss–Newton or LM updates

$N$3

where $N$4 is the Jacobian and $N$5.

The backward pass employs the Implicit Function Theorem, computing for scalar loss $N$6 the gradients with respect to upstream network parameters: $N$7 where $N$8 is the normal equation matrix and $N$9 the residual. This structure ensures full backpropagation of uncertainty modeling into the network.

2. Sources of Instability and Gradient Variance in End-to-End Learning

Differentiable UBA layers, while conceptually principled, often suffer from slow convergence and unstable training unless carefully regularized and coordinated with the learning objective. Three key sources of gradient variance have been identified (Gurumurthy et al., 2024):

• Flow-loss interference: Outlier correspondences with large residuals inject high-variance terms into the flow loss gradients, overwhelming contributions from inlier points.
• Linearization errors: The Jacobian $\{X_i \leftrightarrow u_i\}$0 used for linearizing the BA objective is sensitive to noise in the current parameter estimates, propagating instability into the KKT differentiation chain.
• Weight-gradient residual dependence: The IFT shows that the weight gradients for a given point are amplified by the corresponding residual magnitude. Outliers can thus dominate weight updates, potentially leading to "weight collapse."

Empirical analyses demonstrate that even a small fraction of outliers can increase the variance and bias of BA gradients, substantially degrading the stability and efficiency of end-to-end optimization.

3. Uncertainty Modeling and Weighting Schemes

UBA methods integrate learned, predicted, or analytic uncertainty models into the residual weighting. Several representative weighting schemes include:

• Learned confidence prediction: Networks output $\{X_i \leftrightarrow u_i\}$1 or $\{X_i \leftrightarrow u_i\}$2 given local image, flow, or feature context. For example, in DTV-SfM (Chen et al., 2023), per-pixel confidence $\{X_i \leftrightarrow u_i\}$3 derives from an uncertainty-aware optical flow network, further combined with a RANSAC inlier mask $\{X_i \leftrightarrow u_i\}$4, yielding $\{X_i \leftrightarrow u_i\}$5.
• Analytic sensor models: In LiDAR-UBA (Ćwian et al., 7 Jan 2025), measurement standard deviation $\{X_i \leftrightarrow u_i\}$6 is computed via local sub-beam simulation, entering the residual normalization and robust loss.
• Differentiable regularization: ProBA (Chui et al., 27 May 2025) represents each landmark as a 3D Gaussian with isotropic or full SPD covariance $\{X_i \leftrightarrow u_i\}$7, propagating uncertainty through the nonlinear projection Jacobian to define a negative-loglikelihood reprojection loss.

In all cases, the core idea is to attenuate the influence of ambiguous, noisy, or inconsistent measurements, and to propagate that attenuation both during optimization and in the computation of uncertainty-aware loss gradients.

4. Variance Reduction Techniques and Training Objectives

To mitigate the core sources of gradient variance and instability, recent works introduce joint reweighting strategies. A central approach in (Gurumurthy et al., 2024) is to re-use the network-predicted weights $\{X_i \leftrightarrow u_i\}$8 not only in the inner BA loop but also to reweight the outer supervision loss: $\{X_i \leftrightarrow u_i\}$9 with the stop-gradient wrt $X_i\in\mathbb{R}^3$0 (i.e., $X_i\in\mathbb{R}^3$1) applied so as to decouple the learning of $X_i\in\mathbb{R}^3$2 from direct drive to zero. The full loss is

$X_i\in\mathbb{R}^3$3

where $X_i\in\mathbb{R}^3$4 is adaptively balanced. Empirically, this strategy yields substantially reduced gradient variance, more stable and faster convergence ($X_i\in\mathbb{R}^3$5–$X_i\in\mathbb{R}^3$6 training speedup), and improved or matched final pose accuracy across DPVO and DROID-SLAM testbeds.

Key aspects of successful variants include:

• Explicit gradient clipping for weight gradients to avoid collapse.
• Adaptive balancing of pose and flow losses by aligning gradient norms.
• In ablation, abrogation of the weighted flow loss or unmasking of pose loss recovers the unstable baseline or increases gradient variance.

5. Real-World Systems and Application Domains

UBA methodologies have been adopted across visual and LiDAR SLAM, dense two-view SfM, and visual odometry. Key instantiations include:

• DPVO and DROID-SLAM pipelines: integrating UBA layers achieving $X_i\in\mathbb{R}^3$7–$X_i\in\mathbb{R}^3$8 training speedup and $X_i\in\mathbb{R}^3$9 ATE improvement on TartanAir, and parity or improvement on EuRoC/TUM-RGBD (Gurumurthy et al., 2024).
• Dynamic monocular SLAM: DROID-W achieves ≈$u_i\in\mathbb{R}^2$0 FPS real-time operation with robust tracking in dynamic, cluttered scenes, using a learned per-pixel uncertainty field derived from DINOv2 feature inconsistency, and down-weights dynamic regions online (Li et al., 19 Mar 2026).
• Dense two-view SfM: DTV-SfM leverages per-pixel flow uncertainty, RANSAC inlier masks, and bidirectional weighting in a fully differentiable Gauss–Newton solver, achieving SOTA pose accuracy (Chen et al., 2023).
• LiDAR-based BA: MAD-BA co-optimizes pose and 3D surfel structure with per-measurement covariance estimation, achieving $u_i\in\mathbb{R}^2$1 pose RMSE reduction and >$u_i\in\mathbb{R}^2$2 map accuracy improvement against baselines (Ćwian et al., 7 Jan 2025).

Typical empirical findings demonstrate faster convergence, increased robustness to outliers and dynamics, and greater reliability in downstream geometric and semantic tasks.

6. Extensions, Limitations, and Generalizations

UBA frameworks generalize naturally to multi-agent, multi-modal, and general bilevel learning settings. Examples include:

• Multi-agent systems: (Muthukkumar, 18 Oct 2025) develops a differentiable rendering Cramér–Rao bound BA, fusing Fisher information across multiple agents (cameras), and establishing a rigorous uncertainty lower-bound for pose estimates consistent with classical vision theory.
• Pose-graph BA and visual-inertial odometry: Edge-uncertainties can be predicted and propagated for robust large-scale mapping (Gurumurthy et al., 2024).
• Dense SLAM and PnP: Reweighting photometric, geometric, or keypoint losses by learned confidences to handle ambiguities.
• Probabilistic BA: ProBA's framework jointly optimizes over 3D Gaussians and propagates their covariance through analytic Jacobians, removing the need for strong initialization or prior knowledge—demonstrating reliability with as few as 2–3 frames and robustness to unknown intrinsics (Chui et al., 27 May 2025).

Current limitations include dependence on upstream uncertainty estimation accuracy, sliding-window rather than global treatment of dynamic uncertainty (as in DROID-W), and limited expressiveness of simple uncertainty networks in complex, highly non-stationary settings (Li et al., 19 Mar 2026). Extending UBA to dynamic-object explicit tracking or richer uncertainty fields via spatial transformers or CNNs constitutes ongoing research.

7. Impact and Future Directions

Differentiable uncertainty-aware bundle adjustment represents a convergence of statistical modeling, deep learning, and nonlinear optimization, yielding a unified framework for robust, real-time, and learnable geometric estimation. By enabling uncertainty estimates to flow through all levels of the visual/robotic pipeline, these methods improve interpretability, resilience to outliers and dynamic scenes, and integrability with SLAM, NeRF, and multi-sensor fusion systems.

Emerging directions include global optimization of dynamic uncertainties, explicit representation of time-varying scene structure, integration with differentiable rendering pipelines for photorealistic reconstruction, and application to multi-agent and distributed perception networks (Muthukkumar, 18 Oct 2025). The fundamental principle—propagation and joint learning of uncertainty with geometric variables—continues to drive improvements in both state estimation and practical system robustness.