GaussianBlock: Building Part-Aware Compositional and Editable 3D Scene by Primitives and Gaussians (2410.01535v3)

Abstract: Recently, with the development of Neural Radiance Fields and Gaussian Splatting, 3D reconstruction techniques have achieved remarkably high fidelity. However, the latent representations learnt by these methods are highly entangled and lack interpretability. In this paper, we propose a novel part-aware compositional reconstruction method, called GaussianBlock, that enables semantically coherent and disentangled representations, allowing for precise and physical editing akin to building blocks, while simultaneously maintaining high fidelity. Our GaussianBlock introduces a hybrid representation that leverages the advantages of both primitives, known for their flexible actionability and editability, and 3D Gaussians, which excel in reconstruction quality. Specifically, we achieve semantically coherent primitives through a novel attention-guided centering loss derived from 2D semantic priors, complemented by a dynamic splitting and fusion strategy. Furthermore, we utilize 3D Gaussians that hybridize with primitives to refine structural details and enhance fidelity. Additionally, a binding inheritance strategy is employed to strengthen and maintain the connection between the two. Our reconstructed scenes are evidenced to be disentangled, compositional, and compact across diverse benchmarks, enabling seamless, direct and precise editing while maintaining high quality.

• The paper introduces the GaussianBlock framework, which fuses primitives with 3D Gaussians to yield disentangled and editable scene representations.
• It employs an attention-guided centering loss using 2D semantic priors to align primitives for coherent scene decomposition.
• The dynamic splitting and binding inheritance strategies enable efficient, high-fidelity edits suitable for virtual reality, gaming, and digital content creation.

The paper "GaussianBlock: Building Part-Aware Compositional and Editable 3D Scene by Primitives and Gaussians" introduces an innovative approach to 3D scene reconstruction that enhances both the interpretability and editability of the results. The authors address a key limitation in existing methods, such as Neural Radiance Fields and Gaussian Splatting, which produce high-fidelity reconstructions but result in highly entangled latent representations that are difficult to interpret and modify.

Core Contributions:

1. GaussianBlock Framework: This novel method is designed to create semantically coherent and disentangled representations. It allows for precise, physical editing akin to handling building blocks while ensuring high fidelity in the reconstructed scenes.
2. Hybrid Representation:
   • The approach combines the strengths of primitives and 3D Gaussians.
   • Primitives are used for their flexibility and editability. The authors introduce semantically coherent primitives by using an attention-guided centering loss derived from 2D semantic priors.
   • 3D Gaussians are leveraged for their superior reconstruction quality, providing refined structural details and fidelity.
3. Attention-Guided Centering Loss: This innovation uses 2D semantic priors to ensure that the primitives are semantically aligned, aiding in achieving more coherent scene decomposition.
4. Dynamic Splitting and Fusion Strategy: This mechanism allows for adaptive modifications to the structure, enabling efficient and effective scene alteration without compromising fidelity.
5. Binding Inheritance Strategy: A mechanism to maintain strong and consistent connections between the primitives and Gaussians, ensuring that both components work cohesively to produce disentangled and compositional results.

Impact and Applications:

The method shows significant improvements across various benchmarks, demonstrating that scenes reconstructed with GaussianBlock can be seamlessly and precisely edited. This approach allows for compact and compositional representations, making it promising for applications needing detailed customization, such as virtual reality, game design, and digital content creation.

Overall, the paper presents a significant step forward in 3D reconstruction by providing a pathway toward more interpretable and editable scene representations while not sacrificing the quality and details provided by modern techniques.