Unlocking the Power of Critical Factors for 3D Visual Geometry Estimation

Abstract: Feed-forward visual geometry estimation has recently made rapid progress. However, an important gap remains: multi-frame models usually produce better cross-frame consistency, yet they often underperform strong per-frame methods on single-frame accuracy. This observation motivates our systematic investigation into the critical factors driving model performance through rigorous ablation studies, which reveals several key insights: 1) Scaling up data diversity and quality unlocks further performance gains even in state-of-the-art visual geometry estimation methods; 2) Commonly adopted confidence-aware loss and gradient-based loss mechanisms may unintentionally hinder performance; 3) Joint supervision through both per-sequence and per-frame alignment improves results, while local region alignment surprisingly degrades performance. Furthermore, we introduce two enhancements to integrate the advantages of optimization-based methods and high-resolution inputs: a consistency loss function that enforces alignment between depth maps, camera parameters, and point maps, and an efficient architectural design that leverages high-resolution information. We integrate these designs into CARVE, a resolution-enhanced model for feed-forward visual geometry estimation. Experiments on point cloud reconstruction, video depth estimation, and camera pose/intrinsic estimation show that CARVE achieves strong and robust performance across diverse benchmarks.

• The paper introduces the CARVE model that decomposes critical factors in 3D visual geometry estimation to enhance performance across tasks.
• It reveals that multi-frame models may underperform on single-frame accuracy due to counterintuitive loss weighting and supervision strategies.
• The study demonstrates that resolution-enhanced feature fusion and geometric consistency loss yield robust, state-of-the-art results on varied benchmarks.

Unlocking the Power of Critical Factors for 3D Visual Geometry Estimation

Introduction

"Unlocking the Power of Critical Factors for 3D Visual Geometry Estimation" (2604.21713) presents a rigorous analysis and architectural advancement for feed-forward 3D visual geometry estimation, focusing on decomposing and improving the performance of multi-frame models compared to strong per-frame baselines. The work thoroughly dissects critical components in model design, objective function formulation, and data scaling, identifying counterintuitive behaviors in common practices and proposing systematic remedies. The resulting architecture, CARVE, attains robust, state-of-the-art performance across a diverse suite of benchmarks encompassing point cloud, monocular & video depth, and camera intrinsic/pose estimation.

Analysis of Key Factors in 3D Visual Geometry Estimation

The investigation commences with the empirical observation that multi-frame models, despite superior temporal consistency, underperform on single-frame accuracy compared to specialized per-frame methods. Through exhaustive ablations, several findings emerge:

• Dataset scaling: Augmenting data diversity and quality yields further gains, even when beginning from large pretrained foundations. Larger, multi-domain, and moderate-quality datasets ("Data3") consistently produce performance lifts.
• Objective functions: Gradient-based loss terms (spatial gradient, temporal gradient) and learnable confidence weighting, frequently used to encourage local smoothness and handle outlier regions, are shown to degrade performance in both per-frame and multi-frame regimes. Instead, fixed weighting inversely proportional to depth amplifies stability and accuracy.
• Supervision schemes: Joint supervision through per-sequence and per-frame alignment (via global and per-frame scale matches) leads to measurable improvements; however, enforcing additional local region alignment—especially in a spherical 3D zone—yields contrary results, inducing a decline in fidelity.

The removal of the spatial gradient and confidence loss terms produces only minor qualitative differences in predictions, as the model implicitly learns structural priors via other components.

Figure 1: Removing the spatial gradient loss and confidence loss has minimal impact on qualitative results when continuing training from VGGT pretrained weights.

CARVE Model: Efficient High-Resolution Visual Geometry Estimation

Building on these insights, the CARVE model integrates two methodological advances:

1. Geometric Consistency Loss: A differentiable loss enforces strict photogeometric alignment among the predicted depth map, camera parameters, and 3D point cloud, incorporating explicit multi-view geometry constraints into the training objective.
2. Resolution-Enhanced Feature Fusion: The architecture processes high-resolution and low-resolution views in parallel. High-res features are fused into the main (low-res) branch using frame-wise cross-attention modules with zero-initialized residual gates, ensuring that pretraining priors are preserved while extracting fine-grained spatial details at minimal computational overhead.

   Figure 2: Network architecture of the CARVE model, employing dual-resolution feature extraction and residual cross-attention for efficient high-resolution geometry prediction.

The adaptation allows inference at substantially higher spatial scales with only a fraction of the expected increase in FLOPs and memory usage.

Quantitative and Qualitative Results

CARVE is evaluated on major benchmarks, including KITTI, 7-Scenes, TUM, HO3D, ETH3D, HAMMER, and Bonn. The results highlight several salient points:

• Point Cloud Estimation: CARVE exhibits the lowest Chamfer-L1 distances and highest F-scores, consistently outperforming baselines such as VGGT, Pi3, and Fast3R. Its robustness is evident under diverse scene and viewpoint variations.
• Video Depth and Pose Estimation: The model yields the best or competitive performance across absolute/relative pose error and depth error metrics, showing stable generalization on indoor/outdoor, static/dynamic datasets.
• Monocular Depth: Despite not being explicitly optimized for per-image depth, CARVE remains on par with or outperforms dedicated monocular methods such as MoGe v2.

  Figure 3: Qualitative results from CARVE on in-the-wild images, illustrating detailed reconstructions, and highlighting error modes such as poor pose estimation and scale inconsistencies.

  

  Figure 4: Additional quantitative results showcasing CARVE’s robust performance across datasets and configurations.

Numerically, CARVE demonstrates an average rank improvement over fine-tuned VGGT in both point cloud estimation and camera pose/intrinsics, sustaining its advantage even after controlling for training data scale and protocol.

Discussion, Implications, and Future Directions

This work demonstrates that several long-standing practices in loss weighting and supervision—particularly the use of adaptively learned pixel confidences and gradient-based regularization—may be detrimental in high-performance, cross-domain visual geometry models. Instead, simple, interpretable, and geometrically justified alternatives outperform more sophisticated but less robust design choices.

Furthermore, the integration of explicit geometric consistency into the optimization target echoes trends across geometric deep learning, suggesting renewed value in hybridizing data-driven feature learning with domain-theoretic priors. The cross-attentional dual-resolution fusion architecture establishes a paradigm whereby high-res detail is captured without prohibitive increases in computational complexity, a potentially general strategy for other tasks (e.g., multi-view fusion, SLAM).

Pragmatically, CARVE’s improvements in efficiency and robustness open the door for real-time, cross-device deployment in robotics, AR/VR, and large-scale scene modeling. The codebase and model weights are made available and can serve as a strong baseline for future work seeking to unify high fidelity and high efficiency in 3D perception.

Conclusion

This study provides a comprehensive, methodologically careful investigation of the components underlying strong feed-forward 3D visual geometry estimation. Through critical ablations and principled architectural modifications, the CARVE model advances both the performance and the efficiency frontiers of video-based and monocular 3D reconstruction. The results underscore the importance of scaling, objective rethinking, and efficient architectural design, and will likely inform future developments in geometry-centric visual learning frameworks.