FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention

Abstract: 3D reconstruction from multi-view images is a core challenge in computer vision. Recently, feed-forward methods have emerged as efficient and robust alternatives to traditional per-scene optimization techniques. Among them, state-of-the-art models like the Visual Geometry Grounding Transformer (VGGT) leverage full self-attention over all image tokens to capture global relationships. However, this approach suffers from poor scalability due to the quadratic complexity of self-attention and the large number of tokens generated in long image sequences. In this work, we introduce FlashVGGT, an efficient alternative that addresses this bottleneck through a descriptor-based attention mechanism. Instead of applying dense global attention across all tokens, FlashVGGT compresses spatial information from each frame into a compact set of descriptor tokens. Global attention is then computed as cross-attention between the full set of image tokens and this smaller descriptor set, significantly reducing computational overhead. Moreover, the compactness of the descriptors enables online inference over long sequences via a chunk-recursive mechanism that reuses cached descriptors from previous chunks. Experimental results show that FlashVGGT achieves reconstruction accuracy competitive with VGGT while reducing inference time to just 9.3% of VGGT for 1,000 images, and scaling efficiently to sequences exceeding 3,000 images. Our project page is available at https://wzpscott.github.io/flashvggt_page/.

• The paper introduces a novel compressed descriptor attention mechanism that reduces computational complexity for 3D reconstruction from quadratic to O(N²/r²).
• It employs a DINO-based token encoding and a chunk-recursive mechanism to efficiently process long image sequences while maintaining robust reconstructions.
• Experimental results demonstrate comparable accuracy to traditional methods with over a 90% speedup in inference for sequences of up to 1,000 images.

FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention

FlashVGGT represents a significant advancement in the field of computer vision, specifically addressing the computational bottlenecks associated with 3D reconstruction from multi-view images. This paper introduces a novel approach to tackling the quadratic complexity endemic to traditional Visual Geometry Grounding Transformer (VGGT) methods by employing compressed descriptor attention.

Introduction to FlashVGGT

FlashVGGT emerges as a solution to the scalability challenges faced by VGGT in processing long sequences of images for 3D reconstruction. Central to its innovation is a descriptor-based attention mechanism that reformulates the global attention block, compressing spatial information from each frame into a compact set of descriptor tokens. This approach reduces the computational complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(N^2 / r^2)$, where $r$ is the compression ratio.

Figure 1: Computational Breakdown of VGGT

The framework achieves high accuracy akin to VGGT while significantly decreasing inference time and scaling efficiently to sequences surpassing 3,000 images. The reduction in complexity is achieved without compromising the model's ability to perform robust 3D reconstruction.

Architecture and Methodology

FlashVGGT employs a simplified architecture that uses DINO for encoding input images into tokens before passing them through alternating attention blocks—frame and global attention. Unlike VGGT, FlashVGGT compresses spatial information into descriptor tokens and computes cross-attention between these tokens and full image tokens.

Figure 2: Architecture Overview. Our framework encodes input images into tokens using DINO.

A chunk-recursive mechanism supports online inference by utilizing cached descriptors, thus enabling the model to maintain a global receptive field across extended sequences. This design greatly enhances memory efficiency, allowing FlashVGGT to scale beyond the limits of existing VGGT-like models.

Experimental Results

FlashVGGT was thoroughly evaluated against baseline methods like FastVGGT and VGGT across different benchmarks. The results demonstrated competitive accuracy and substantial reductions in inference time for sequences of varying lengths. Notably, FlashVGGT achieved over a 90% speedup in inference time for 1,000-image inputs compared to VGGT, while maintaining comparable accuracy.

Figure 3: Chunk Recursive Inference. For long input sequences, we process them in a chunked manner while retaining global reception by caching descriptor tokens.

Additionally, qualitative comparisons highlighted FlashVGGT's ability to produce more robust and complete reconstructions over long input sequences, evidencing the model's efficiency and scalability.

Figure 4: Qualitative comparison of long-sequence dense 3D reconstruction. FlashVGGT produces more robust reconstructions over long input sequences while being significantly faster.

Implications and Future Directions

The implementation of compressed descriptor attention in FlashVGGT marks a pivotal shift, enabling efficient 3D reconstruction of multi-view images. This approach could be integrated into other models, facilitating applications that necessitate processing large-scale data in real-time scenarios.

Future developments may explore more sophisticated token compression strategies, potentially further enhancing the efficiency and generalizability of FlashVGGT-like architectures. Moreover, continued integration with other attention-based models could yield versatile applications spanning various domains within AI.

Conclusion

FlashVGGT demonstrates significant progress in efficiently handling complex 3D reconstruction tasks using compressed descriptor attention. It achieves high accuracy with drastically reduced computational demands, opening pathways for scalable and practical deployment in resource-constrained environments. By removing the bottleneck of quadratic complexity prevalent in previous models, FlashVGGT stands as a robust, scalable solution in advancing the capabilities of visual geometry transformers.