InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams

Abstract: The grand vision of enabling persistent, large-scale 3D visual geometry understanding is shackled by the irreconcilable demands of scalability and long-term stability. While offline models like VGGT achieve inspiring geometry capability, their batch-based nature renders them irrelevant for live systems. Streaming architectures, though the intended solution for live operation, have proven inadequate. Existing methods either fail to support truly infinite-horizon inputs or suffer from catastrophic drift over long sequences. We shatter this long-standing dilemma with InfiniteVGGT, a causal visual geometry transformer that operationalizes the concept of a rolling memory through a bounded yet adaptive and perpetually expressive KV cache. Capitalizing on this, we devise a training-free, attention-agnostic pruning strategy that intelligently discards obsolete information, effectively ``rolling'' the memory forward with each new frame. Fully compatible with FlashAttention, InfiniteVGGT finally alleviates the compromise, enabling infinite-horizon streaming while outperforming existing streaming methods in long-term stability. The ultimate test for such a system is its performance over a truly infinite horizon, a capability that has been impossible to rigorously validate due to the lack of extremely long-term, continuous benchmarks. To address this critical gap, we introduce the Long3D benchmark, which, for the first time, enables a rigorous evaluation of continuous 3D geometry estimation on sequences about 10,000 frames. This provides the definitive evaluation platform for future research in long-term 3D geometry understanding. Code is available at: https://github.com/AutoLab-SAI-SJTU/InfiniteVGGT

• The paper introduces a rolling memory mechanism that leverages diversity-aware token pruning to maintain causal context in continuous 3D scene reconstruction.
• It reduces VRAM usage and error accumulation compared to existing batch and online methods, ensuring sustained geometric fidelity over long sequences.
• Empirical evaluations on the Long3D benchmark confirm superior accuracy and consistency, making it suitable for real-time applications in robotics and AR.

InfiniteVGGT: A Scalable Visual Geometry Transformer for Streaming 3D Scene Understanding

Introduction and Problem Formulation

InfiniteVGGT (2601.02281) introduces a transformer-based framework designed to address the challenge of persistent, long-horizon 3D visual geometry understanding from continuous 2D video streams. Classical methods (SfM, MVS) excel in geometric fidelity but are unsuitable for real-time or unbounded operation due to multi-stage pipelines and memory requirements. More recently, end-to-end vision transformers such as VGGT established a new standard for batch-based high-fidelity 3D scene inference, but their global attention mechanisms induce prohibitive VRAM consumption for long or infinite input streams, making them impractical for online applications.

Streaming architectures have emerged as a conceptual solution but, in practice, bifurcate into two unsatisfactory paradigms: those with explicit history accumulation (e.g., StreamVGGT) that scale in memory and computation with input length, or those that compress all history into a fixed-size state (e.g., CUT3R, TTT3R), sacrificing geometric consistency and accumulating drift. There is an unmet requirement for an online system that preserves long-term context without unbounded memory growth.

Figure 1: InfiniteVGGT achieves scalable 3D geometry understanding, resolving the limitations of both batch (offline) and existing online (streaming) paradigms.

Technical Contributions: Rolling Memory and Diversity-Aware Pruning

The key innovation of InfiniteVGGT is a training-free, attention-agnostic dynamic memory paradigm, termed "rolling memory," that maintains a causal KV cache with token-level pruning. The pruning mechanism leverages redundancy characteristics in contiguous camera trajectories: streaming video sequences exhibit high frame-to-frame redundancy at the token level, a fact empirically shown by cosine similarity analysis in the DINO backbone feature space. Instead of attention-weight-driven importance—which would require access to the absent or computationally expensive attention matrix when using FastAttention/FlashAttention kernels—the authors propose to quantify token salience directly in the key space, performing diversity estimation and selection by negative cosine similarity to the mean key direction.

Figure 2: Frame-to-frame attention maps are highly redundant and query/key feature spaces are clustered, supporting redundancy-driven pruning in online streams.

The memory is explicitly structured: (1) the initial frame's KV cache is set as immutable anchors to preserve geometric alignment, (2) subsequent token retention within each layer and head is performed dynamically, guided by diversity scores, and (3) per-layer budgets are adaptively allocated via softmax-normalized diversity, acknowledging that redundancy (and thus retention need) varies structurally within the network. This allows InfiniteVGGT to be compatible with FlashAttention and other hardware-optimized attention schemes without additional training or full attention computation.

Figure 3: The InfiniteVGGT rolling memory consists of immutable anchors, diversity-quantified token retention, and layer-wise adaptive budget, ensuring bounded-state while preserving informative context.

Evaluation: Long3D Benchmark and Downstream Performance

A major bottleneck in evaluating long-horizon visual geometry models is the absence of continuous, extreme-length benchmarks. The paper therefore introduces the Long3D benchmark, providing streaming RGB-D and IMU data with up to 10,000 frames per sequence, including ground-truth global point clouds for rigorous, metric evaluation.

Figure 4: Long3D data comprises scenes spanning indoor/outdoor domains with multi-thousand frame contiguous video and aggregated ground-truth geometry.

In experiments on Long3D and standard datasets, InfiniteVGGT demonstrates robust and stable 3D reconstruction across arbitrarily long inputs. Competing systems based on unbounded or simplistic memory (StreamVGGT, offline VGGT) run out of memory (OOM), while RNN-based compression approaches like CUT3R and TTT3R accumulate drift, especially on extended sequences. By contrast, InfiniteVGGT achieves lower Acc./Comp. error, higher NC, and lower Chamfer Distance, with error propagation tightly controlled as input length increases.

Figure 5: Qualitative point cloud reconstructions obtained by InfiniteVGGT on extended sequences show high geometric fidelity and alignment with ground-truth.

Figure 6: Quantitative comparison (CD/NC) with prior methods: InfiniteVGGT prevents error accumulation and maintains superior normal consistency on long-horizon NRGBD experiments.

Video depth estimation experiments on the Bonn dataset further demonstrate competitive depth accuracy and consistency, outperforming both RNN-based and pointer memory methods as sequence length grows.

Detailed Analysis and Ablation

Extensive ablation studies reinforce the core engineering choices:

• Cosine similarity-based diversity selection yields more precise and memory-efficient performance than attention weight-driven methods.
• Anchor tokens are essential: pruning the first-frame cache irreversibly degrades coordinate consistency.
• Layer-wise adaptive budgeting improves both accuracy and efficiency over uniform allocation.
• Memory budget per head reveals diminishing quality returns beyond moderate storage.

Importantly, the system is training-free and generalizes robustly to various sequences and domains without re-training or fine-tuning.

Implications, Limitations, and Future Directions

The introduction of a diversity-aware token eviction stratagem for long-horizon causal transformers directly addresses scalability/stability trade-offs in online 3D geometry estimation. Practically, this enables real-time 3D scene understanding for robotics, AR, and autonomous driving under true infinite-horizon constraints. The modularity of the method suggests straightforward adaptation to other perception tasks where informative context retention and bounded memory are critical (e.g., streaming visual SLAM, online segmentation).

One limitation identified is that InfiniteVGGT, while dominating accuracy/consistency metrics, can underperform on the mean Completion metric, highlighting an opportunity for further optimization in spatial recall, possibly via hybrid KV management or incorporation of dynamic attention windowing.

Theoretically, this work opens lines of investigation into more sophisticated token importance estimation (e.g., in the context of geometric/semantic cues) as well as cross-modal rolling memories (image-LiDAR fusion). The practical impact is amplified by the release of the Long3D benchmark, which is now the de facto testbed for benchmarking next-generation online geometry understanding systems.

Conclusion

InfiniteVGGT defines a scalable, hardware-aware, causally correct streaming transformer architecture for endless 3D geometry estimation. By leveraging redundancy in the key-token space and deploying a diversity-quantified, adaptive rolling memory, it circumvents the critical limitations of existing streaming and offline paradigms. Empirical evaluation confirms its state-of-the-art performance and robustness on extreme-length sequences, marking a significant step in general-purpose, online visual geometry understanding.

(2601.02281)