VGGT-$Ω$

Abstract: Recent feed-forward reconstruction models, such as VGGT, have proven competitive with traditional optimization-based reconstructors while also providing geometry-aware features useful for other tasks. Here, we show that the quality of these models scales predictably with model and data size. We do so by introducing VGGT-$Ω$, which substantially improves reconstruction accuracy, efficiency, and capabilities for both static and dynamic scenes. To enable training this model at an unprecedented scale, we introduce architectural changes that improve training efficiency, a high-quality data annotation pipeline that supports dynamic scenes, and a self-supervised learning protocol. We simplify VGGT's architecture by using a single dense prediction head with multi-task supervision and removing the expensive high-resolution convolutional layers. We also use registers to aggregate scene information into a compact representation and introduce register attention, which restricts inter-frame information exchange to these registers, in part replacing global attention. In this way, during training, VGGT-$Ω$ uses only about 30% of the GPU memory of its predecessor, allowing us to train with 15x more supervised data than prior work and to leverage vast amounts of unlabeled video data. VGGT-$Ω$ achieves strong results for reconstruction of static and dynamic scenes across multiple benchmarks, for example, improving over the previous best camera estimation accuracy on Sintel by 77%. We also show that the learned registers can improve vision-language-action models and support alignment with language, suggesting that reconstruction can be a powerful and scalable proxy task for spatial understanding. Project Page: http://vggt-omega.github.io/

• The paper introduces a feed-forward transformer with register attention that scales to 10B+ parameters and reduces GPU memory use by approximately 70%.
• It achieves state-of-the-art results on both static and dynamic scenes, demonstrating a 77% improvement in camera accuracy and 50× faster processing than prior methods.
• The model aligns geometric representations with language and robotics tasks, enabling unified spatial-linguistic reasoning and robust cross-modal performance.

VGGT-$\Omega$: Scalable Feed-Forward 3D Scene Reconstruction with Register Attention

Introduction and Motivation

VGGT-$\Omega$ introduces a new, highly scalable feed-forward approach to 3D scene reconstruction, capable of accurately estimating both camera parameters and depth maps for both static and dynamic scenes using multi-view images. The key objective is to systematically investigate and empirically verify whether scaling model size and data volume—an established paradigm in vision-language and language foundation models—also yields uniform benefits in 3D geometry, and to engineer a feed-forward framework conducive to such scaling. VGGT-$\Omega$ departs from previous transformer-based geometry models through distinctive architectural changes, efficient memory utilization, and a comprehensive, high-quality data annotation pipeline optimized for dynamic content.

Model Architecture and Efficiency

The VGGT-$\Omega$ architecture is built upon a permutation-equivariant feed-forward transformer, initialized from DINOv3, with several crucial enhancements over prior versions such as VGGT:

• Computation Bottleneck Reduction: Introduction of "register attention," which complements global attention by restricting inter-frame information exchange to a small set of learned "register" tokens per frame. This design eliminates substantial computation and memory overhead, while preserving global scene aggregation capabilities.
• Unified Dense Prediction Head: The architecture eschews multiple high-resolution dense heads, instead using a single dense head for depth prediction, with multi-task supervision (also on point maps and tracks).
• Lossless Convolution Replacement: High-resolution convolutional layers in the dense head are replaced (where feasible) with MLPs and pixel shuffle layers, further decreasing memory consumption during training.
• Scalable Tokenization: Camera and register tokens are appended to the sequence of image tokens for each frame, and processed via alternating global/register and frame-wise attention layers.

These choices jointly reduce training GPU memory by approximately 70%, enabling model scaling to 10B+ parameters and training with up to 15× more data than prior benchmarks.

Figure 1: The architectural block diagram showing the alternation between global/register attention and frame attention, with camera/register tokens appended to each image sequence.

Data Pipeline and Scaling Laws

Robustness and generalization in VGGT-$\Omega$ derive heavily from scaling both dataset size and coverage. The devised data pipeline:

• Integrates hundreds of public and internal datasets, yielding ~3M sequences, with significant representation of both rigid and dynamic videos.
• Employs a multi-stage annotation pipeline combining VLM-based pre-filtering, motion masking, feature tracking, multi-view optimization (COLMAP), and supervised geometric filtering.
• Applies stringent multi-view consistency checks to ensure annotation precision, at the cost of yield but with high data fidelity.

For additional unlabeled videos (~40M), a student-teacher self-supervised protocol inspired by momentum distillation is leveraged.

Empirical results display canonical power-law behavior: increasing data and capacity systematically reduces reconstruction error over orders of magnitude, mirroring scaling law trends observed in language and vision foundation models.

Figure 2: Systematic reduction in 3D point error with increasing model size and data scale, measured across six benchmarks.

Empirical Results and Robustness

VGGT-$\Omega$ achieves a new state of the art across both static and dynamic reconstruction benchmarks. Notably:

• On the challenging Sintel dataset, the model achieves camera AUC@$3^\circ$ of 40.0 and depth $\delta_{1.25}$ of 93.5, representing a 77% and 26% improvement, respectively, over MegaSaM—the top optimization-based method.
• The model processes over 1000-frame sequences in a single forward pass, at speeds 50× faster than prior optimization-based pipelines, and is significantly more memory efficient.

Qualitative results confirm robust handling of scenes ranging from structured traffic flows, complex human motion, to underwater coral reefs, and effective generalization to sparse and dynamic settings.

Figure 3: Examples demonstrating VGGT-$\Omega$ handling of both static and dynamic content with dense reconstructions.

Comparisons highlight that prior state-of-the-art methods (e.g., MegaSaM, Depth Anything 3) falter under geometric ambiguity, dynamic content, or extreme camera motion, while VGGT-$\Omega$ maintains consistent, drift-free scene structure.

Figure 4: MegaSaM often fails on global consistency under challenging conditions, while VGGT-$\Omega$0 provides geometrically plausible results.

Figure 5: VGGT-$\Omega$1 accurately recovers camera motion and depth in cases where DA3 demonstrates drift and ghosting.

Furthermore, inference-time memory use and throughput are highly competitive with, or superior to, other approaches.

Figure 6: Trade-offs between memory and runtime for VGGT, DA3, and VGGT-$\Omega$2 at a large batch/frame scale.

Analysis of Representation and Downstream Applications

A key insight of VGGT-$\Omega$3 is the use of register tokens not only as efficient aggregation mechanisms but also as stable, information-dense representations for downstream cross-modal alignment and control.

• Language Alignment: Registers can be projected into a feature space aligned with language embeddings using an InfoNCE objective, supporting CLIP-style retrieval tasks at the scene level. Transfer to held-out text-only LLM embeddings remains robust.

  Figure 7: Pipeline for aligning register tokens to VLM language embeddings using a simple contrastive protocol.

• Robotics Integration: Frozen VGGT-$\Omega$4 registers, when injected into VLA policy models (e.g., OpenVLA-OFT), consistently improve downstream robotic control performance across complex task suites.
• Motion Awareness: Intermediate token activations, when unsupervisedly clustered, localize motion without explicit supervision—indicating the emergence of spatiotemporal awareness solely from the geometric reconstruction task.

  Figure 8: Emergent motion sensitivity in representation space, with clusters isolating dynamic agents in the scene.

Registers thus serve as a flexible, high-level compact state representation, generalizing to tasks such as language retrieval, robotics, metric scale prediction, and gravity inference, without task-specific architectural changes.

Practical and Theoretical Implications

The results of VGGT-$\Omega$5 support several claims of practical and foundational interest:

• Uniform Scaling Law Benefits: Both data and parameter scaling, when coupled with a compute/memory-efficient architecture, quantitative and qualitative data filtering, and permutation-invariant design, yield consistent accuracy improvements, extending scaling law doctrine into 3D spatial representation learning.
• Feed-forward Models vs. Optimization-based Pipelines: Feed-forward architectures now match or exceed traditional SfM in both accuracy and throughput, are robust to dynamic and challenging real-world data, and naturally provide reusable geometric features.
• Registers as Generalized State: The register abstraction enables direct alignment of geometric representations with language, action, and other modalities, suggesting a pathway toward unified spatial-linguistic reasoning models.
• Open Directions in Self-Supervised Learning: While self-supervised pretraining with large-scale video currently yields modest improvements in standard metrics, it enhances out-of-distribution generalization and constitutes an open research area.

Performance gaps with optimization-based approaches are now largely confined to highly controlled benchmarks requiring extreme precision (e.g., NeRF initialization), but practical requirements in robotics, AR/VR, and general scene understanding are increasingly met by feed-forward approaches.

Future Directions

Future work may explore:

• Integration of VGGT-$\Omega$6-style representations into "omnimodal" unified models that span reconstruction, language, and world modeling, using registers as compositional, multi-task state.
• Improved self-supervised and cross-modal training paradigms for geometric tasks, leveraging joint training with language and action data.
• Transfer to generative 3D/4D modeling, video synthesis, and multimodal policy models, given the generality and alignment capability of register representations.
• Further architectural simplifications or specialization for on-device, low-memory, or streaming settings, with register-only inter-frame communication as a path toward real-time scalability.

Conclusion

VGGT-$\Omega$7 demonstrates that with careful architectural, loss, and data engineering, fully feed-forward transformers can match or significantly exceed traditional geometric pipelines in both accuracy and efficiency, and learn representations that transfer robustly to semantic, spatial, and action tasks. Register tokens emerge as a powerful primitive for scene- and frame-level information aggregation, compact state, and cross-modal alignment. This work sets the stage for a new generation of geometric models poised for integration and scaling within broader multimodal AI systems.