Emergent Outlier View Rejection in Visual Geometry Grounded Transformers (2512.04012v1)

Abstract: Reliable 3D reconstruction from in-the-wild image collections is often hindered by "noisy" images-irrelevant inputs with little or no view overlap with others. While traditional Structure-from-Motion pipelines handle such cases through geometric verification and outlier rejection, feed-forward 3D reconstruction models lack these explicit mechanisms, leading to degraded performance under in-the-wild conditions. In this paper, we discover that the existing feed-forward reconstruction model, e.g., VGGT, despite lacking explicit outlier-rejection mechanisms or noise-aware training, can inherently distinguish distractor images. Through an in-depth analysis under varying proportions of synthetic distractors, we identify a specific layer that naturally exhibits outlier-suppressing behavior. Further probing reveals that this layer encodes discriminative internal representations that enable an effective noise-filtering capability, which we simply leverage to perform outlier-view rejection in feed-forward 3D reconstruction without any additional fine-tuning or supervision. Extensive experiments on both controlled and in-the-wild datasets demonstrate that this implicit filtering mechanism is consistent and generalizes well across diverse scenarios.

• The paper demonstrates that internal VGGT layers can inherently suppress distractor views for improved 3D reconstruction.
• It leverages cross-view attention and cosine feature similarity with a fixed threshold to reject outliers without retraining.
• The approach enhances pose and depth estimation across diverse benchmarks, outperforming traditional and descriptor-based methods.

Emergent Outlier View Rejection in Visual Geometry Grounded Transformers

Introduction and Motivation

This paper investigates the challenge of robust multi-view 3D reconstruction from unconstrained, "in-the-wild" image collections, where feed-forward models, specifically Visual Geometry Grounded Transformers (VGGT), often encounter noisy, irrelevant, or distractor images that compromise geometric estimation. The central thesis is the discovery of an emergent outlier-suppressing mechanism within the internal representations of VGGT, specifically in late attention and feature layers, which can be exploited for effective view selection without model modification, retraining, or supervision.

Conventional Structure-from-Motion (SfM) pipelines achieve resilience to outlier views through geometric verification and robust optimization, but are computationally heavy and not tightly integrated with end-to-end learned approaches. In contrast, recent transformer-based models typically lack explicit filtering and degenerate when faced with contaminated image sets. The work introduces "RobustVGGT"—a training-free, threshold-based view rejection scheme leveraging internal signals from VGGT, yielding substantial improvements on both pose and depth estimation across controlled and real-world benchmarks.

Figure 1: The need for outlier rejection is illustrated by contamination of the VGGT pipeline when distractors are present, producing noisy geometry and artifacts; RobustVGGT achieves cleaner results with simple view filtering.

Technical Analysis: Outlier Suppression Emergence

The methodological insight is based on probing internal layers of VGGT. By analyzing cross-view attention and intermediate feature similarity across the model's stack, the study observes that the discriminative gap between clean and distractor views intensifies with depth, reaching its maximum at the last layer. Both attention weights and cosine feature similarity metrics naturally downweight distractors, enabling scene-consistent view selection.

Figure 2: Layer-wise analysis shows the gap between clean and distractor views—measured in attention and feature similarity—peaking at the final layer, indicating emergent noise suppression.

Qualitative visualizations further corroborate this, revealing that context images with little geometric overlap receive negligible final-layer attention or feature correlation, while consistent images are strongly activated.

Figure 3: Cross-view attention and feature similarity maps from VGGT's final layer explicitly suppress distractors, with low scores for non-overlapping views.

RobustVGGT: Practical View Rejection Mechanism

The proposed mechanism computes per-view relevance using VGGT's final layer signals:

• Attention-based rejection: Scores are derived by averaging cross-view attention weights between a query and each context image.
• Feature similarity-based rejection: Mean cosine similarities of dense features between the query and contexts are aggregated.

A fixed global threshold, shared across datasets, is applied to reject distractors. The filtered image set is re-fed to the backbone for reconstruction. Two variants, RobustVGGT-$\mathcal{A}$ (attention) and RobustVGGT-$\mathcal{F}$ (feature), are instantiated, both yielding significant accuracy improvements.

Figure 4: Overview of RobustVGGT, which computes per-view relevance from internal attention and feature similarity and applies a global threshold for view filtering.

Experimental Evaluation

Experiments span Phototourism (internet images), On-the-Go (casual sequences), RobustNeRF (distractor augmentation), and ETH3D (high-fidelity multi-view). Quantitative and qualitative analysis demonstrates that the vanilla VGGT degrades as distractor count grows, whereas RobustVGGT maintains stable performance across all noise levels—attesting to the generalizability of the internal filtering mechanism.

Figure 5: Camera ATE (trajectory error) as a function of distractor prevalence; RobustVGGT prevents degradation as noise grows, while unfiltered VGGT performance deteriorates.

Camera pose estimation and multi-view depth metrics both show RobustVGGT surpasses baselines, including strong global descriptor and VPR-based pre-filtering methods—even those benefiting from oracle hyperparameter tuning.

Figure 6: Qualitative camera trajectory predictions highlight RobustVGGT's stability in challenging scenarios with many distractors.

Figure 7: Multi-view depth reconstructions remain consistent under RobustVGGT even with large amounts of noise.

Success rate analysis indicates that internal cross-view reasoning is more effective for outlier rejection than global descriptor similarity, as DINO-based approaches are not geometry-sensitive.

Figure 8: VGGT and RobustVGGT-$\mathcal{F}$ point map visualizations demonstrate filtered reconstructions suppress spurious geometry induced by distractors.

Figure 9: Internet-collected sets with heavy distractor contamination are robustly reconstructed with view filtering.

Ablations and Further Analyses

Threshold tuning ablations confirm that the view selection mechanism generalizes using a single hyperparameter across scene domains and noise levels. Comparison with variants using combined attention and feature signals yields additional robustness. Layer-wise analyses, extended to the Pi3 model, validate the emergence of filtering as a property generalizable across transformer architectures.

Implications and Future Directions

This work establishes that transformer-based feed-forward models for 3D reconstruction encode multi-view geometric consistency signals, which can be directly leveraged to reject outlier views—obviating the need for explicit geometric verification or iterative optimization. The findings suggest a new paradigm for robust, efficient scene reconstruction in uncontrolled environments, with practical ramifications for internet-scale SfM, AR/VR systems, and autonomous scene understanding.

Future directions include extending view-level rejection to finer granularity (token/patch-wise), improving throughput via hierarchical filtering or clustering, and integrating the approach into larger retrieval pipelines or scene graph models. A deeper theoretical understanding of emergent consistency signals in learned geometric transformer models may inform the development of more adaptive, self-filtering architectures with online scene adaptation capabilities.

Conclusion

The identification and quantitative characterization of emergent outlier suppression in VGGT's internal representations enables a simple, robust, and efficient view rejection module, "RobustVGGT," that consistently improves multi-view 3D reconstruction under diverse, noisy real-world conditions. The method is complementary to classical and VPR-based approaches, and highlights the capacity of self-supervised transformer backbones to encode geometric priors sufficient for outlier discrimination (2512.04012).