GStex: Per-Primitive Texturing of 2D Gaussian Splatting for Decoupled Appearance and Geometry Modeling (2409.12954v3)

Abstract: Gaussian splatting has demonstrated excellent performance for view synthesis and scene reconstruction. The representation achieves photorealistic quality by optimizing the position, scale, color, and opacity of thousands to millions of 2D or 3D Gaussian primitives within a scene. However, since each Gaussian primitive encodes both appearance and geometry, these attributes are strongly coupled--thus, high-fidelity appearance modeling requires a large number of Gaussian primitives, even when the scene geometry is simple (e.g., for a textured planar surface). We propose to texture each 2D Gaussian primitive so that even a single Gaussian can be used to capture appearance details. By employing per-primitive texturing, our appearance representation is agnostic to the topology and complexity of the scene's geometry. We show that our approach, GStex, yields improved visual quality over prior work in texturing Gaussian splats. Furthermore, we demonstrate that our decoupling enables improved novel view synthesis performance compared to 2D Gaussian splatting when reducing the number of Gaussian primitives, and that GStex can be used for scene appearance editing and re-texturing.

• The paper introduces GStex, a method that adds per-primitive texture maps to decouple appearance and geometry in 2D Gaussian Splatting.
• The approach optimizes Gaussian primitives and integrates texture maps by minimizing photometric loss to achieve high-fidelity scene reconstruction.
• GStex achieves superior PSNR, SSIM, and low LPIPS metrics using fewer primitives, enhancing efficiency in real-time rendering applications.

GStex: Per-Primitive Texturing of 2D Gaussian Splatting for Decoupled Appearance and Geometry Modeling

The paper "GStex: Per-Primitive Texturing of 2D Gaussian Splatting for Decoupled Appearance and Geometry Modeling" introduces a novel approach centered on enhancing the conventional models of 2D Gaussian Splatting (2DGS) for scene reconstruction and novel view synthesis tasks. The authors propose a new representation, GStex, which adds per-Gaussian texture maps to the 2D Gaussian primitives employed in the splatting process, effectively decoupling the appearance and geometry components.

Overview

Gaussian splatting has gained significant traction due to its efficacy in applications like view synthesis and photorealistic scene reconstruction. This method optimizes a multitude of Gaussian primitives within a scene by adjusting their spatial parameters (position, scale, orientation) and appearance characteristics (color and opacity). However, a notable limitation arises when the appearance and geometry are coupled, necessitating a high number of Gaussian primitives to capture high-fidelity textures even in geometrically simple scenes. GStex addresses this limitation by introducing texture maps that are associated with each Gaussian primitive, thereby creating a more flexible and powerful representation that decouples these two aspects.

Methodology

Fundamental Approach

The cornerstone of GStex is its ability to texture each Gaussian primitive independently, addressing the shortfall of the conventional 2DGS method where appearance and geometry are inherently coupled. The proposed method comprises two primary stages:

1. Initial Gaussian Representation:
   • An initial set of Gaussian primitives is generated using 2DGS.
   • This step involves optimizing the primitives' positions, scales, colors, and opacities based on input images.
2. Texture Map Integration:
   • Each Gaussian primitive is enhanced with its own texture map.
   • The texture maps are optimized to capture fine-detail appearance properties by minimizing the photometric loss between scene images and rendered views, taking into account ray-Gaussian intersections.

This approach leverages an efficient rendering model that maps intersection points of rays with Gaussians to coordinates in the texture map for color lookup, followed by alpha compositing the results.

Numerical Results

The paper presents compelling quantitative evidence showcasing the capabilities of GStex. For synthetic Blender and DTU datasets, GStex demonstrates significant improvements in visual metrics over the compared methods, including 3DGS, 2DGS, and the recent Texture-GS. Notably, GStex achieves higher PSNR and SSIM scores while maintaining low LPIPS values, indicative of superior perceptual and pixel-level consistency. Furthermore, GStex provides these improvements using fewer Gaussian primitives than needed by traditional methods, underscoring its efficiency.

Implications and Future Directions

Practical Implications

The introduction of GStex has profound implications in graphics and augmented reality fields. By decoupling the appearance and geometry, GStex can significantly reduce the computational load, enabling real-time applications and enhanced scalability for complex scenes. Particularly in dynamic or interactive environments, such as videogames or virtual tours, this decoupled model allows for more flexible and efficient texture editing without the need to adjust underlying geometry.

Theoretical Implications

From a theoretical standpoint, GStex opens avenues for exploring more complex texture and appearance representations within Gaussian splatting frameworks. It potentially supports the integration of higher-order texture representation techniques like mip-mapping for anti-aliasing, as alluded to in future work.

Conclusion

GStex advances the state-of-the-art in Gaussian splatting by efficiently decoupling appearance and geometry, achieving high fidelity in scene rendering while utilizing fewer primitives. The proposed method enhances both the theoretical underpinnings and practical applications of Gaussian splatting, paving the way for more detailed and computationally efficient scene reconstructions. Future research could further explore optimizations in texture handling and integration with conventional graphics techniques, enhancing the robustness and applicability of GStex in real-world scenarios.