BA-T: An Iterative Transformer for Two-View Bundle Adjustment

Abstract: Feed-forward models for 3D reconstruction have achieved strong performance using deep cross-view attention to exchange information across images. However, these approaches often depend on heavy decoder stacks and lack a structured mechanism for geometry refinement, resulting in poor multi-view consistency. We address this by drawing inspiration from classical bundle adjustment (BA), which can be viewed as an iterative information propagation process between poses and local geometry. Inspired by BA, we propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space. Instead of relying on deep attention stacks, BA-T refines predictions based on latent residual by a single lightweight layer. Experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations, achieves stronger cross-view consistency than conventional decoders, and matches or surpasses substantially larger models while using only 16% of their decoder parameters. BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture. The code will be made publicly at https://github.com/zhangganlin/BA-T.

• The paper presents BA-T, which integrates classical bundle adjustment concepts into a reusable iterative transformer layer for robust 3D reconstruction.
• It refines latent camera and geometry tokens via adaptive cross-attention, reducing reprojection errors in just 3-4 iterations.
• Empirical results show BA-T achieves state-of-the-art two-view pose estimation and dense geometry reconstruction with only 16% of prior decoder parameters.

BA-T: An Iterative Transformer for Two-View Bundle Adjustment

Introduction and Motivation

Feed-forward deep learning models for two-view 3D reconstruction have utilized dense, deep cross-view attention decoders to propagate geometric information across images. However, these methods lack mechanisms for structured, iterative geometric refinement and therefore struggle with multi-view consistency and systematic error correction. In contrast, classical bundle adjustment (BA) performs jointly iterative optimization of camera poses and scene geometry, enforcing consistency but relying on explicit geometric representations difficult to integrate in deep latent token spaces. The paper "BA-T: An Iterative Transformer for Two-View Bundle Adjustment" (2606.03287) synthesizes these paradigms, proposing a compact iterative Transformer, BA-T, which embeds BA-style information propagation into the latent token space via a single reusable refinement layer.

Figure 1: Overview of BA-T: input images lead to iterative, latent-space camera and local geometry token updates with error decreasing across iterations.

Model Architecture and BA-Inspired Information Propagation

BA-T replaces traditional deep decoder stacks with a structured, lightweight iterative Transformer module. Its central abstraction treats both relative camera parameters and local geometric structure as latent tokens. The BA-T layer propagates and exchanges information between these tokens in a pattern analogous to classical BA:

• Camera-Conditioned Geometry Transform: Geometry tokens are projected into a common space via conditioning on the camera token using adaptive LayerNorm.
• Token-Level Correspondence Matching: For a geometry token in view $a$, cross-attention identifies the soft correspondence in view $b$, allowing differentiable, flexible association beyond rigid geometric priors.

Figure 2: BA-T pipeline: tokens iteratively refined by BA-inspired steps—geometry is transformed, correspondences matched, residials computed, and both camera and geometry tokens updated.

Figure 3: Token-level attention visualizations show correct cross-view responses in both ambiguous and distinctive regions.

• Latent Residual Computation: After correspondence, the difference in projected latent representations (via a linear+MLP head) yields a residual token, providing a learned analogue to the traditional reprojection error.
• Camera Update: The camera token is refined at each step by one-to-many cross-attention from the camera token to all residual tokens, mimicking Schur complement aggregation in classical BA.
• Local Geometry Update: Geometry tokens are refined via cross-attention with geometry tokens from the other view (projected to the current frame via the updated camera), enforcing cross-view consistency.

Figure 4: Magnitude of computed latent residuals decreases with each refinement, indicating progressive convergence.

Training and Iterative Supervision

BA-T employs iterative supervision, where the full loss (composed of local point map accuracy, relative pose error—supplemented with cycle-consistency, and cross-view geometric consistency) is computed and weighted at each refinement iteration. This encourages the module to perform improving, stepwise refinement, in contrast to single-pass predictors or unrolled attention stacks.

Empirical Results: Pose and Geometry Estimation

Experiments demonstrate that BA-T achieves state-of-the-art results for two-view relative pose estimation and competitive performance in dense geometry reconstruction, while using only 16% of the decoder parameters of prior models such as DUSt3R. With four iterative refinement steps, it matches or surpasses models trained with richer multi-view data, despite a substantially smaller architecture.

Key numerical results:

• On the 7Scenes and BundleFusion datasets, BA-T achieves the best relative pose AUCs among all two-view models, and remains within statistical margin of the strongest multi-view trained models on geometric metrics.
• Iterative refinement is critical; BA-T's structured updates consistently outperform models with equally-sized conventional stacked attention decoders, with rapid convergence in 3-4 steps.

Figure 5: Qualitative progression: local geometry and error maps improve markedly across BA-T iterations, correcting initial misalignments.

Ablations and Component Analysis

Ablation studies highlight the critical contributions of each design element:

• Iterative Supervision: Removing supervision at intermediate steps leads to degraded accuracy and consistency.
• Correspondence Matching: Disabling learnable cross-view matching reduces pose alignment and 3D consistency.
• Camera-Conditioned Transformation: Omitting camera-aware alignment undermines both pose and geometry recovery.

Each component, especially the explicit BA-inspired cross-token propagation and update structure, is essential for the observed performance gains.

Comparison with Classical BA Solvers and Runtime Considerations

BA-T outperforms classical solver-based BA-init methods like DROID-SLAM both in accuracy and efficiency. Notably, even when DROID-SLAM is given oracle ground-truth intrinsics, BA-T achieves greater reconstruction accuracy and requires only a fifth of the runtime and less than half the memory—attributable to implicit latent-space representation and learned priors. Direct latent-residual minimization in deep token-space grants robustness under conditions of limited co-visibility or ambiguous correspondences where explicit geometric BA struggles.

Figure 6: Runtime and inference efficiency comparison: BA-T converges faster and yields more accurate geometry than explicit classical BA-based approaches.

Multi-View Generalization and Future Prospects

The BA-T framework, though instantiated for two-view settings, naturally generalizes to multi-view configurations. Architecture remains unchanged; the refinement layer is shared across all views, and geometric updates are performed via cross-attention to all relevant view tokens as per a view-graph. This enables scalability to larger sequences and more complex SLAM pipelines.

Figure 7: 4-view qualitative results: multi-view refinement yields globally consistent reconstruction and pose estimation.

Further, the factorized token-based update approach opens avenues for integrating learned priors, memory augmentation, and hierarchical spatial reasoning, which could accelerate convergence and improve global consistency across larger scenes.

Conclusion

BA-T demonstrates that embedding bundle adjustment-style information propagation and consistency constraints within a compact, latent, iterative Transformer layer yields high-accuracy, robust, and efficient two-view 3D reconstruction. Iterative BA-like refinement in token space establishes stronger geometric consistency than previous deep attention stacks, achieves or surpasses state-of-the-art results with far fewer parameters, and provides a structural blueprint for scalable, real-time, and data-driven SLAM and multi-view reconstruction pipelines. The approach's architectural efficiency and empirical performance suggest promising directions for extending structured, optimization-inspired learning frameworks in geometric deep learning and vision.