Multi-view Pyramid Transformer: Look Coarser to See Broader (2512.07806v1)

Abstract: We propose Multi-view Pyramid Transformer (MVP), a scalable multi-view transformer architecture that directly reconstructs large 3D scenes from tens to hundreds of images in a single forward pass. Drawing on the idea of ``looking broader to see the whole, looking finer to see the details," MVP is built on two core design principles: 1) a local-to-global inter-view hierarchy that gradually broadens the model's perspective from local views to groups and ultimately the full scene, and 2) a fine-to-coarse intra-view hierarchy that starts from detailed spatial representations and progressively aggregates them into compact, information-dense tokens. This dual hierarchy achieves both computational efficiency and representational richness, enabling fast reconstruction of large and complex scenes. We validate MVP on diverse datasets and show that, when coupled with 3D Gaussian Splatting as the underlying 3D representation, it achieves state-of-the-art generalizable reconstruction quality while maintaining high efficiency and scalability across a wide range of view configurations.

• The paper introduces a dual-hierarchy attention mechanism that scales self-attention efficiently across both inter-view and intra-view dimensions for robust 3D reconstruction.
• It integrates a pyramidal feature aggregation module that fuses multi-scale outputs, ensuring consistent scene-level reasoning and enhanced computational efficiency.
• Experimental results demonstrate state-of-the-art performance, achieving superior quality and inference speed across diverse benchmarks and view counts.

Multi-view Pyramid Transformer: Hierarchical Attention for Scalable Multi-view 3D Scene Reconstruction

Introduction

The Multi-view Pyramid Transformer (MVP) addresses a critical challenge in transformer-based multi-view 3D scene reconstruction: the quadratic complexity of self-attention prevents scalable and efficient processing of high-resolution images across many viewpoints. MVP introduces a dual-hierarchical approach—jointly expanding attention coverage across both inter-view and intra-view hierarchies—which enables fast, high-quality, and memory-efficient single-pass 3D Gaussian Splatting-based reconstruction from sparse to dense multi-view imagery. The architecture achieves state-of-the-art quality and scalability, generalizing across datasets and outperforming both feed-forward and optimization-based baselines.

Dual Attention Hierarchy Architecture

MVP's design rests on two complementary hierarchical mechanisms:

1. Inter-view Local-to-Global Hierarchy: Attention windows expand from local (per-frame) to group-wise and finally to global, encompassing all views. The local-to-global schedule ensures computational efficiency while retaining the capacity for consistent multi-view geometric reasoning across increasing view counts.
2. Intra-view Fine-to-Coarse Hierarchy: Tokenization of spatial image information proceeds from fine granularity to progressive spatial coarsening, reducing the per-image token length, increasing the effective receptive field, and boosting representational capacity to encode both fine structure and global context.

These hierarchies operate synergistically in a three-stage processing pipeline. The architecture employs frame-wise attention for initial local feature extraction, group-wise attention via a fixed partitioning of views (e.g., groups of four) to establish short-range geometric correspondences, and finally global attention at reduced spatial resolution for consistent scene-level reasoning.

Figure 1: The MVP architecture applies hierarchical attention—alternating self-attention blocks with different scopes and spatial resolutions, followed by pyramidal feature aggregation for dense prediction.

The pyramidal feature aggregation (PFA) module fuses multi-scale outputs from each attention stage, enabling the model to integrate local and global context for dense 3D prediction. The architectural choices are validated by ablations demonstrating significant degradation in both efficiency and reconstruction accuracy upon removing either hierarchy, reversing their order, or omitting the multi-scale aggregation.

Dense Feed-forward 3D Gaussian Splatting

MVP is instantiated in a feed-forward 3D Gaussian Splatting (3DGS) framework, with each pixel mapped to 3D Gaussian primitives parameterized by position, scale, rotation, opacity, and color (with SH basis for view dependence). The approach operates directly on sets of posed images, concatenating per-pixel Plücker ray encodings, and employs registers and relative positional encoding (PRoPE) for robust view conditioning.

The Gaussian parameters are post-processed by differentiable rasterization (using gsplat), enabling photometric losses to drive end-to-end training with standard rendering-based supervision.

Experimental Results

Quantitative and Qualitative Performance

MVP achieves consistent and significant improvements over recent transformer-based baselines (Long-LRM, iLRM) and matches or outperforms slow optimization-based methods (3D-GS with 30K iters) on core benchmarks (DL3DV, Tanks&Temples, Mip-NeRF360, RE10K), across a wide spectrum of input view counts—from extremely sparse (4–8 views) to dense (256 views):

• On DL3DV at 128 input views, MVP obtains PSNR 29.02, SSIM 0.903, and LPIPS 0.134 in 0.77s, outperforming Long-LRM (OOM at this scale), iLRM (worse metrics and >5x slower), and being >600× faster than optimization-based methods for similar quality.
• Generalization across previously unseen datasets and increased view counts demonstrates the stability of hierarchical attention: gains continue scaling with more views, and performance does not saturate as in global-attention baselines.
• On low-resolution tasks (RE10K, 4/8 views), MVP surpasses compact feed-forward models, with PSNR up to 33.40 and SSIM up to 0.952.

  Figure 2: MVP achieves strong visual fidelity and 3D consistency under sparse and dense input regimes across diverse datasets.

  

  Figure 3: Qualitative results on the RE10K dataset demonstrate robustness at low input views and improved per-pixel accuracy.

Attention Visualization and Scalability

Attention maps reveal that group-wise attention in fine-resolution layers captures local geometries, while global coarse-level attention enables aggregation of consistent object- or region-level correspondence across arbitrary camera baselines—without suffering the dilution and instability seen in pure global self-attention.

Scalability is documented by inference time curves and ablation studies: MVP maintains sharply sublinear growth in latency as view count increases, with out-of-memory or massive slowdowns avoided due to hierarchically limited context windows and progressive token reduction.

Figure 4: Attention visualization indicates focused attention on group-local and scene-global correspondences across stages.

Figure 5: Inference time remains low and scalable as input view count increases; global attention baselines suffer exponential slowdowns or OOM errors.

Architectural Ablation and Theoretical Insights

Ablation studies confirm that the performance/efficiency trade-off hinges on both the dual-hierarchy and pyramidal aggregation. Removing group-wise attention (favoring only frame-wise or only global attention) increases compute without recovering accuracy, indicating that intermediate-range interactions are crucial. Similarly, bypassing intra-view spatial hierarchy (constant token granularity) yields poor scalability and out-of-memory failures at high view counts.

Further experiments on the group size reveal diminishing returns for group sizes beyond four and instability for smaller groups due to insufficient inter-view context; four is optimal for balancing correspondence reasoning and compute. The design also demonstrates improved generalization in spatial cognition tasks (identifying pose-image mismatches in input sets), suggesting robust multi-view awareness.

Implications and Future Directions

MVP's dual-hierarchy design resolves core issues with attention dilution and quadratic scaling intrinsic to pure-transformer 3D reasoning at scale. Its integration with 3DGS provides a highly efficient and generalizable framework for both high-fidelity photo-realistic synthesis and explicit geometry estimation—even outperforming geometry-agnostic baselines despite using only photometric supervision. The architectural principles established are applicable to other structured scene modeling tasks (e.g., dynamic scenes, geometry-supervised learning).

Future work includes extending MVP to dynamic 3D scenes by incorporating explicit temporal hierarchies, geometric supervision for improved 3D accuracy, and applications in view-agnostic spatial reasoning and scene understanding.

Conclusion

MVP establishes a new standard in transformer-based large-scale multi-view 3D scene reasoning, leveraging a dual attention hierarchy to enable both computational and representational scalability. Hierarchical view and spatial token aggregation, combined with feature fusion, allow MVP to achieve state-of-the-art accuracy with strict improvements in inference speed and memory, robust generalization, and strong multi-view consistency. The dual-hierarchy approach provides a blueprint for addressing attention scaling challenges in large-context vision and geometric learning scenarios.