CalibAnyView: Beyond Single-View Camera Calibration in the Wild

Abstract: Camera calibration is a fundamental prerequisite for reliable geometric perception, yet classical approaches rely on controlled acquisition setups that are impractical for in-the-wild imagery. Recent learning-based methods have shown promising results for single-view calibration, but inherently neglect geometric consistency across multiple views. We introduce CalibAnyView, a unified formulation that supports an arbitrary number of input views ($N \geq 1$) by explicitly modeling cross-view geometric consistency. To facilitate this, we construct a large-scale multi-view video dataset covering diverse real-world scenarios, including multiple camera models, dynamic scenes, realistic motion trajectories, and heterogeneous lens distortions. Building on this dataset, we develop a multi-view transformer that predicts dense perspective fields, which are further integrated into a geometric optimization framework to jointly estimate camera intrinsics and gravity direction. Extensive experiments demonstrate that CalibAnyView consistently outperforms state-of-the-art methods, achieves strong robustness under single-view settings, and further improves with multi-view inference, providing a reliable foundation for downstream tasks such as 3D reconstruction and robotic perception in the wild.

• The paper introduces a unified calibration method that leverages dense perspective fields and non-linear optimization to accurately estimate camera parameters from unconstrained imagery.
• It employs a Dense Prediction Transformer and alternating attention to fuse multi-scale features and enforce cross-view geometric consistency, reducing calibration errors in roll, pitch, and FoV.
• The method enables robust camera calibration for applications like 3D reconstruction and robotic navigation by overcoming the limitations of classical calibration techniques.

Unified Calibration in the Wild: CalibAny View

Motivation and Problem Statement

Camera calibration, the estimation of intrinsic parameters governing the image formation process (including focal length, principal point, and lens distortion), is foundational for geometric computer vision applications like 3D reconstruction, SLAM, and robotic navigation. Classical calibration paradigms depend on controlled setups with geometric targets and highly structured environments, rendering them ill-suited for increasingly prevalent unconstrained scenarios—such as handheld smartphone captures, mobile robotics, and drone videography—where visual data is acquired casually and under diverse camera models and dynamic conditions. Recent single-view, learning-based calibration methods address certain limitations but fundamentally neglect cross-view geometric consistency and suffer from intrinsic ambiguity due to the ill-posed nature of the problem.

CalibAny View: Methodological Advances

CalibAny View presents a unified, any-view calibration framework supporting both single-view and arbitrary multi-view inputs by explicitly modeling geometric consistency across perspectives.

Perspective Field Representation and Geometric Optimization

The approach leverages perspective field representations (dense per-pixel up-vector and latitude fields) to encode geometric cues derived from scene verticals and the horizon, which are robust to camera model agnosticism. A Dense Prediction Transformer (DPT) head generates these fields at reduced spatial resolution by fusing multi-scale features from deep transformer layers, filtering out local noise and capturing global structural geometry critical for calibration. Confidence maps are produced alongside perspective fields, enabling uncertainty-aware loss weighting.

Camera parameters—including shared intrinsics and per-view gravity direction—are estimated via a joint non-linear least-squares optimization that minimizes residuals between network-predicted and model-induced perspective fields. The optimization utilizes the iterative Levenberg-Marquardt algorithm on a spherical manifold, with confidence-weighting to prioritize robust geometric regions.

Multi-View Aggregation and Alternating Attention

To enable geometric aggregation across multiple views, CalibAny View employs an alternating attention mechanism inspired by 3D feed-forward transformers. Dense patch-level features extracted by DINOv2 are refined through intra-frame self-attention to encode structural priors (verticals, vanishing points, perspective grids), then aggregated via cross-frame global attention to enforce spatio-temporal consistency and resolve ambiguities. The resulting joint geometric latent is decoded into perspective fields and passed into the optimization stage, where shared intrinsic constraints further regularize calibration across the sequence.

Dataset Construction: Realistic Multi-View Video Ground Truth

Recognizing the shortcomings of single-image supervision and pinhole-only datasets, the authors construct a large-scale, gravity-aligned, multi-view video dataset by projecting diverse 360° panoramic video content onto augmented virtual trajectories and camera models (Unified Camera Model, Pinhole, Simple Radial), each sampled across varied field-of-view and distortion ranges. Realistic motion is transferred from CameraBench trajectories, augmented with rotation offset and sweeping. Quality control is enforced via VLM-based filtering (Qwen2.5-VL) to exclude clips with artifacts, overlays, or synthetic content, resulting in a collection of ~23.7K video clips, each 5 seconds at 16 fps, encompassing 1.9M frames.

Empirical Evaluation

Single-View Calibration

In the N=1 regime, CalibAny View outperforms or matches state-of-the-art baselines (DeepCalib, GeoCalib, AnyCalib, VGGT) across indoor, synthetic, and in-the-wild benchmarks (Stanford2D3D, TartanAir, MegaDepth, LaMAR) in roll, pitch, and field-of-view estimation, with improvement attributable to both architectural advances (transformer aggregation, DPT head) and multi-view training exposure. Strong robustness to lens distortion is demonstrated, with a single unified model achieving competitive or superior results to specialized variants trained for radial distortion.

Multi-View Calibration and Cross-View Consistency

As the input view count N increases, calibration error decreases monotonically—both for perspective field and parameter estimation—demonstrating the effectiveness of cross-view attention and shared intrinsic optimization in resolving single-view ambiguities. On challenging test sets and public benchmarks, CalibAny View outperforms learning-based baselines and traditional multi-view pipelines (COLMAP, DroidCalib), which frequently fail under sparse views and dynamic scenes due to their reliance on successful geometric reconstruction.

On Stanford2D3D (in-the-wild, distortion-rich), CalibAny View attains the lowest roll, pitch, and FoV errors among all tested methods, with COLMAP reconstructing only a fraction of sequences due to near-zero parallax. On TartanAir (synthetic, pinhole), performance remains competitive, achieving accurate gravity estimation absent in reconstruction-oriented models like VGGT.

Ablation and Efficiency

Architectural ablations (MLP vs. DPT head, transformer layer selection, sampling ratio) confirm the necessity of deep layer fusion and multi-scale feature aggregation for precise geometric field prediction, with the 1/4 downsampling offering the best accuracy-efficiency trade-off. Runtime analysis evidences moderate GPU memory consumption and fast inference relative to other transformer-based and learning-SLAM baselines.

Practical and Theoretical Implications

CalibAny View establishes a robust paradigm for calibration from unconstrained imagery, eliminating the requirement for controlled capture protocols and geometric patterns. Its unified formulation and multi-view optimization enable stable calibration across arbitrary sequences and camera models, directly supporting critical downstream tasks—3D reconstruction, visual localization, robotic navigation, AR deployment—where absolute orientation and calibrated intrinsics are prerequisites. The method further demonstrates strong generalization to in-the-wild scenes and challenging lens distortions, obviating the need for specialized variants.

Theoretically, the work advances geometric deep learning by integrating dense, model-agnostic intermediate representations with explicit physical optimization, leveraging data-driven priors and cross-view consensus to ameliorate the inherent ambiguities of single-view estimation. The constructed dataset also provides a resource for further research in camera modeling, video generation, and pose estimation.

Limitations and Future Directions

Current assumptions include fixed principal point (centered crop) and shared intrinsics across sequences, adequate for most consumer cameras but limiting for scenarios involving zoom or asymmetric cropping. Adapting the model for off-center principal points and zoom-varying sequences constitutes an immediate direction. Further integration with full 6-DoF extrinsics estimation and dynamic lens models would extend applicability to even more challenging environments.

Conclusion

CalibAny View delivers a technically rigorous solution for camera calibration in unconstrained conditions, unifying single- and multi-view inference with a geometric reasoning backbone and large-scale multi-view supervision. Quantitative and qualitative evaluations validate its superiority over existing approaches. Its methodological design and supporting dataset contribute a robust foundation for real-world 3D vision and robotics, and its architecture informs future exploration in generalizable geometric perception for AI systems.