---
title: Omnidirectional Differentiable Bundle Adjustment
url: https://www.emergentmind.com/topics/omnidirectional-differentiable-bundle-adjustment-odba
type: topic
---

# Omnidirectional Differentiable Bundle Adjustment

Omnidirectional Differentiable Bundle Adjustment (ODBA) is a geometric optimization module designed for accurate pose and depth refinement in monocular omnidirectional visual odometry (OVO) systems leveraging 360-degree cameras. Unlike classical photometric or feature-based approaches, ODBA integrates a distortion-aware spherical feature interface and employs spherical trigonometric camera models, enabling joint optimization of camera poses and sparse landmark inverse depths via a differentiable Gauss–Newton solver with hand-derived Jacobians. This formulation supports end-to-end training, geometric supervision, and robust pose estimation under challenging conditions such as aggressive motion and severe image distortion [2601.02309].

## 1. Mathematical Foundations

ODBA models the geometry of a 360-degree equirectangular image as a mapping between image pixels and spherical coordinates. Each pixel $(u,v)$ in an image of size $H \times W$ represents spherical angles $(\theta,\phi)$, computed as $\theta=\arctan(x/z)$ and $\phi=\arcsin(y/\|X\|)$, with intrinsics
$$K = \begin{bmatrix} W/(2\pi) & 0 & W/2 \\ 0 & -H/\pi & H/2 \\ 0 & 0 & 1 \end{bmatrix}.$$
The forward projection $\Pi(X)$ and its inverse $\Pi^{-1}(p,d)$ follow:
- $\Pi(X) = K[\theta;\phi;1]$, $\theta = \arctan2(x,z)$, $\phi = \arcsin(y/\|X\|)$
- $\Pi^{-1}(p, d) = X = \frac{1}{d}\,[\cos\phi\,\sin\theta;\,\cos\phi\,\cos\theta;\,\sin\phi]$

For each patch $k$ with image coordinate $p_k^i$ and inverse-depth $d_k$, the 3D point corresponding to $p_k^i$ is reprojected from frame $i$ to frame $j$ via $T_{j i} = T_j T_i^{-1}$:
$$
p_{k j}' = \Pi\left(T_{j i} \cdot \Pi^{-1}(p_k^i, d_k)\right)
$$
Simultaneously, the learned flow head predicts an offset for patch matching:
$$
p_{k j}^* = p_k^i + F_{rnn}\left(\langle g_k, h_j \rangle\right)
$$
The objective minimized by ODBA is the weighted sum of squared reprojection errors:
$$
E(\{T\}, d) = \sum_{(k,j)\in E} \| p_{k j}^* - p_{k j}' \|^2_{\Sigma_{k j}}
$$
where $\Sigma_{k j}$ is a confidence-weighted covariance produced by the RNN updater.

## 2. Parameterization and Feature Interfaces

ODBA jointly refines the following:
- Camera poses $T_n\in SE(3)$, parameterized as 6D Lie algebra increments $\xi_n$
- Inverse depths $d_k$ associated with $N$ sparse “patch” landmarks

Rather than dense depth maps, ODBA associates each sparse patch $k$ with a single inverse-depth scalar $d_k$, initialized from the keyframe’s mean patch depth. The distortion-aware spherical feature extractor (DAS-Feat) computes matching features $f_i \in \mathbb{R}^{H \times W \times 128}$ and context features $h_i \in \mathbb{R}^{H \times W \times 384}$ via SphereResNet. A patchification module selects high saliency centers $p_k$ by evaluating the gradient magnitude of $f_i$ and crops $3 \times 3$ neighborhoods $g_k \in \mathbb{R}^{3 \times 3 \times 128}$, inheriting the center’s depth $d_k$. Edges $E$ are created to link each patch from a keyframe $i$ to neighbors $j$ in a temporal window.

This approach is distinct from pinhole bundle adjustment in two respects:
(a) Projections $\Pi, \Pi^{-1}$ use spherical trigonometry rather than perspective projection.
(b) Depth is parameterized per sparse patch, not per pixel or mesh vertex.

## 3. Optimization and Differentiability

Optimization utilizes a Gauss–Newton solver (without damping), typically running 3–5 iterations over Lie algebra pose increments and scalar depth updates. For each pair $(k,j)$, residuals and Jacobians are computed:
- $e_{k j} = p_{k j}^* - p_{k j}'$
- $J_i = \frac{\partial p_{k j}'}{\partial \xi_i} \in \mathbb{R}^{2 \times 6}$
- $J_j = \frac{\partial p_{k j}'}{\partial \xi_j} \in \mathbb{R}^{2 \times 6}$
- $J_d = \frac{\partial p_{k j}'}{\partial d_k} \in \mathbb{R}^{2 \times 1}$

The full block-Hessian (Eq. (9)) encompasses the system:
```
| wJ_iᵀJ_i   wJ_iᵀJ_j   wJ_iᵀJ_d | |Δξ_i|   | wJ_iᵀe |
| wJ_jᵀJ_i   wJ_jᵀJ_j   wJ_jᵀJ_d | |Δξ_j| = | wJ_jᵀe |
| wJ_dᵀJ_i   wJ_dᵀJ_j   wJ_dᵀJ_d | |Δd  |   | wJ_dᵀe |
```
where $w = \sqrt{\Sigma_{k j}^{-1}}$.

Schur complement eliminates $\Delta d$ for pose updates, followed by scalar depth updates:
- $T_n \leftarrow \exp(\Delta \xi_n)\,T_n$
- $d_k \leftarrow d_k + \Delta d_k$

Jacobian derivations follow:
- $J_j = \frac{\partial \Pi(X')}{\partial X'} \cdot \frac{\partial X'}{\partial \xi_j}$
- $J_i = -J_j\cdot \text{Adj}(T_{ij})$
- $J_d = \frac{\partial \Pi(X')}{\partial X'} \cdot T_{ij} \cdot \frac{\partial \Pi^{-1}(p,d)}{\partial d}$

All linear solvers and pose updates are implemented in PyTorch with autograd compatibility, enabling backpropagation of gradients through the residuals, Jacobians, and network modules.

## 4. Data Flow and Algorithmic Pipeline

Figure 2 in [2601.02309] depicts the workflow, summarized in the following pseudocode block:

```
Inputs: {I_i,…,I_j}, {T_i,…,T_j}, {d_1,…,d_N}
f_i, h_i ← SphereResNet(I_i) for all frames
{(p_k, d_k, g_k)} ← Patchify(f_i)
E ← build_edges(i, r)
for iter = 1 ... M do
  for each (k, j) ∈ E do
    gpatch ← g_k
    hpatch ← crop(h_j, Π(T_{ji} ⋅ Π⁻¹(p_k, d_k)))
    corr ← ⟨gpatch, hpatch⟩
    p_{k j}* ← p_k + F_rnn(corr)
    p_{k j}' ← Π(T_{ji} ⋅ Π⁻¹(p_k, d_k))
    e_{k j} ← p_{k j}* − p_{k j}'
    Compute J_i, J_j, J_d
  end
  Assemble H, b via ∑ JᵀJ, Jᵀe
  Solve [H][Δξ;Δd]=b (Schur complement)
  Update poses and depths
end
Output: refined {T_n}, {d_k}
```

At each iteration, the network computes patch-to-frame correlations and flow updates, applies spherical reprojection, evaluates residuals, computes Jacobians, constructs the Hessian system, and updates variables.

## 5. Experimental Validation

Ablation studies conducted on the real-world 360DVO benchmark and public synthetic datasets (TartanAir V2 and 360VO) demonstrate ODBA's efficacy:
- DPVO (pinhole + DBA) on 360DVO: ATE = 7.83 m (Easy), 6.92 m (Hard)
- Using ODBA but keeping a standard ResNet (no sphere): catastrophic failure (9.99 m)
- Pure SphereNet features + ODBA: divergence, indicating instability without residual connections
- Full 360DVO (SphereResNet + ODBA): ATE = 3.31 m (Easy), 4.35 m (Hard), 56% reduction over pinhole DPVO

On TartanAirV2 and synthetic 360VO datasets, 360DVO with ODBA reduces ATE by approximately 37.5% and 10%, respectively, over leading baselines including 360VO and OpenVSLAM.

End-to-end training with ODBA is essential; it enables accurate spherical geometry constraints and allows geometric supervision to backpropagate into DAS-Feat and the flow updater, critical for robust optimization in non-perspective geometries.

## 6. Distinctions and Implications

ODBA extends classical bundle adjustment to omnidirectional imagery by:
- Employing spherical trigonometry for pixel-to-ray conversion via custom projection heads
- Optimizing sparse inverse-depths per patch instead of dense pixelwise depth
- Utilizing end-to-end differentiability through all optimization blocks, facilitating joint learning of features and geometry

A plausible implication is that the interplay of feature learning (via DAS-Feat), robust spherical reprojection, and differentiable optimization constitutes a framework that achieves substantial improvements in robustness and accuracy over non-differentiable, perspective-only BA modules. The necessity of residual connections and spherical-aware networks is evidenced by failed or divergent runs in ablation studies, reinforcing the criticality of architectural choices for ODBA convergence.

In summary, ODBA comprises an omnidirectional, end-to-end differentiable bundle adjustment pipeline that tightly couples learned distortion-resistant features and spherical reprojection geometry via a Lie-algebra-based Gauss–Newton solver, yielding substantial gains in pose and depth estimation accuracy for monocular 360-degree visual odometry [2601.02309].

Source: https://www.emergentmind.com/topics/omnidirectional-differentiable-bundle-adjustment-odba