Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2312.09147v2)

Abstract: Recent advancements in 3D reconstruction from single images have been driven by the evolution of generative models. Prominent among these are methods based on Score Distillation Sampling (SDS) and the adaptation of diffusion models in the 3D domain. Despite their progress, these techniques often face limitations due to slow optimization or rendering processes, leading to extensive training and optimization times. In this paper, we introduce a novel approach for single-view reconstruction that efficiently generates a 3D model from a single image via feed-forward inference. Our method utilizes two transformer-based networks, namely a point decoder and a triplane decoder, to reconstruct 3D objects using a hybrid Triplane-Gaussian intermediate representation. This hybrid representation strikes a balance, achieving a faster rendering speed compared to implicit representations while simultaneously delivering superior rendering quality than explicit representations. The point decoder is designed for generating point clouds from single images, offering an explicit representation which is then utilized by the triplane decoder to query Gaussian features for each point. This design choice addresses the challenges associated with directly regressing explicit 3D Gaussian attributes characterized by their non-structural nature. Subsequently, the 3D Gaussians are decoded by an MLP to enable rapid rendering through splatting. Both decoders are built upon a scalable, transformer-based architecture and have been efficiently trained on large-scale 3D datasets. The evaluations conducted on both synthetic datasets and real-world images demonstrate that our method not only achieves higher quality but also ensures a faster runtime in comparison to previous state-of-the-art techniques. Please see our project page at https://zouzx.github.io/TriplaneGaussian/.

• The paper introduces a hybrid transformer-based model that combines explicit point clouds with implicit triplane features to accelerate 3D reconstruction.
• The methodology leverages a dual decoder system, integrating local image features to refine geometry and enhance rendering quality.
• Evaluations reveal significant improvements in accuracy and runtime efficiency, demonstrating robustness across diverse datasets.

Triplane Meets Gaussian Splatting: An Overview of Single-View 3D Reconstruction with Transformers

The paper "Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers" introduces a novel method for the rapid reconstruction of 3D models from single-view images leveraging transformer architectures. In the field of computer vision and graphics, this approach addresses the persistent challenge of effectively digitizing three-dimensional shapes based on minimal image data. This paper proposes a hybrid representation model that combines explicit point clouds with implicit triplane features, decoded to populate anisotropic Gaussian splats, enhancing rendering quality and drastically reducing computational time typically required by prior methods.

Methodology Overview

The proposed model integrates two primary transformer-based networks: a point decoder and a triplane decoder. The point cloud decoder initially reconstructs a coarse representation of 3D geometry using explicit points that serve as a foundation for further refinement by the triplane decoder. The triplane representation facilitates implicit spatial feature encoding, which optimizes Gaussian attributes decoding. This is particularly beneficial for addressing challenges associated with densely packed 3D Gaussian attributes, characterized by their unstructured nature across different dimensional data.

Furthermore, the utilization of explicit geometry from the point cloud aids in the projection and conditioning of image features—a key step for enhancing the overall fidelity of novel view synthesis. Local image features are incorporated into the point cloud expansion and Gaussian decoding processes, contributing to the accurate reconstruction of fine details and textures.

Evaluation and Results

The hybrid representation model is thoroughly validated on both synthetic and real-world datasets, demonstrating superior runtime efficiency and higher rendering quality compared to existing techniques. The empirical evaluations highlight impressive quantitative improvements: the proposed method achieves reduced Chamfer distance and improved Volume IoU in geometric reconstructions, as well as better PSNR, SSIM, and LPIPS scores in novel view synthesis benchmarks.

Importantly, the results are consistent across diverse object categories, affirming the method's generalizability and robustness. The rendering process, facilitated by efficient Gaussian splatting, further underscores the model's capability for real-time applications—a critical advantage over traditional volume rendering techniques.

Implications and Future Directions

The methodological advancements put forth in this paper offer significant potential for various applications in augmented and virtual reality, gaming, and automated design, where rapid and accurate 3D modeling from limited visual input is essential. The model's architecture, based on scalable transformer designs, also lends itself well to continued evolution and potential integration with emerging technologies in neural rendering and feature synthesis.

Looking ahead, further exploration into refining the hybrid Triplane-Gaussian representation could enhance texture prediction, especially on surfaces occluded from the input view. Additionally, integration with probabilistic models might improve generative robustness, offering sharper textures in challenging scenarios. The emphasis on transformer adaptability signifies a promising direction for broader AI applications in real-time vision-based modeling tasks.

In conclusion, this paper presents a technically sound and efficient approach to single-view 3D reconstruction, leveraging the synergistic potential of explicit and implicit representation frameworks. The proposed methodology successfully circumvents many of the limitations prevalent in existing techniques, delineating a pathway for future research and application development in the field of rapid 3D object reconstruction.