EdgeGaussians -- 3D Edge Mapping via Gaussian Splatting (2409.12886v2)

Abstract: With their meaningful geometry and their omnipresence in the 3D world, edges are extremely useful primitives in computer vision. 3D edges comprise of lines and curves, and methods to reconstruct them use either multi-view images or point clouds as input. State-of-the-art image-based methods first learn a 3D edge point cloud then fit 3D edges to it. The edge point cloud is obtained by learning a 3D neural implicit edge field from which the 3D edge points are sampled on a specific level set (0 or 1). However, such methods present two important drawbacks: i) it is not realistic to sample points on exact level sets due to float imprecision and training inaccuracies. Instead, they are sampled within a range of levels so the points do not lie accurately on the 3D edges and require further processing. ii) Such implicit representations are computationally expensive and require long training times. In this paper, we address these two limitations and propose a 3D edge mapping that is simpler, more efficient, and preserves accuracy. Our method learns explicitly the 3D edge points and their edge direction hence bypassing the need for point sampling. It casts a 3D edge point as the center of a 3D Gaussian and the edge direction as the principal axis of the Gaussian. Such a representation has the advantage of being not only geometrically meaningful but also compatible with the efficient training optimization defined in Gaussian Splatting. Results show that the proposed method produces edges as accurate and complete as the state-of-the-art while being an order of magnitude faster. Code is released at https://github.com/kunalchelani/EdgeGaussians.

• The paper introduces a novel 3D Gaussian representation that replaces neural implicit fields for precise edge mapping.
• It leverages efficient training with Gaussian splatting and geometric constraints, reducing runtime by up to 30x compared to state-of-the-art methods.
• The method clusters and fits Gaussian-based edge points to achieve robust 3D edge reconstructions on diverse datasets.

EdgeGaussians - 3D Edge Mapping via Gaussian Splatting

The paper "EdgeGaussians - 3D Edge Mapping via Gaussian Splatting" presents a novel method for reconstructing 3D edges from multi-view images. This method addresses limitations associated with the current state-of-the-art (SotA) image-based 3D edge detection techniques, such as computation costs and sampling inaccuracies, by introducing a more efficient and accurate approach using Gaussian splatting.

Main Contributions

1. 3D Gaussian Representation: The method replaces traditional neural implicit fields with an explicit representation where 3D edge points are cast as the centers of 3D Gaussians, and the edge directions are represented as the principal axes of these Gaussians. This strategy bypasses point sampling on imprecise level sets and eliminates post-processing needs.
2. Efficient Training: Leveraging the geometrically meaningful representation of 3D Gaussians and adopting the training optimization defined in Gaussian splatting, the method ensures efficient and fast training while maintaining accurate 3D edge reconstructions.
3. Clustering and Edge Fitting: Post-training, the Gaussians' centers and orientations are used to cluster edge points based on spatial proximity and direction consistency. Parametric edges are then fitted to these clusters, resulting in accurate 3D edge models.

Implementation and Performance

The proposed method initializes the 3D Gaussians and trains their parameters (positions, scales, opacities, and orientations) using a rendering loss supervised by off-the-shelf 2D edge maps. By masking the $\mathcal{L}_1$ loss to focus on informative pixels and incorporating additional geometric constraints such as the alignment of Gaussians’ principal axes, the algorithm efficiently converges to a high-quality 3D edge representation.

Quantitative Evaluation

The method has been extensively evaluated on two datasets: ABC-NEF and DTU. On ABC-NEF, the proposed method demonstrates performance metrics nearly equivalent to SotA methods like EMAP and NEF, with the added benefit of computational efficiency. Specifically, the accuracy and completeness are competitive, and the runtime is significantly reduced (by a factor of 17-30 times). The results highlight the robustness of the method against computationally intensive neural implicit representations.

The evaluation also covers the DTU dataset, where pseudo-ground-truth points pose challenges due to inherent biases introduced during their generation. Despite these challenges, the method outperforms the considered baselines in both precision and recall under a 5mm error threshold.

Practical and Theoretical Implications

The proposed method's advantage lies in its ability to deliver high-accuracy 3D edge reconstructions much faster than competing methods, making it highly practical for real-world applications requiring rapid and reliable edge detection. The explicit representation using 3D Gaussians simplifies the post-processing pipeline and improves the quality of clustering and edge fitting.

Future Directions

Future work could explore further refining the supervisory signals to address biases and inaccuracies in the off-the-shelf 2D edge detectors. Another direction could involve scaling the method for large, unbounded scenes and enhancing its applicability to diverse environments. Lastly, integrating additional geometric information and optimizing the clustering and edge fitting processes could further improve the robustness and accuracy of the method in complex scenes.

In conclusion, "EdgeGaussians - 3D Edge Mapping via Gaussian Splatting" presents a compelling advancement in the field of 3D edge reconstruction, achieving a crucial balance between accuracy and computational efficiency. This work paves the way for more practical implementations of 3D edge detection in dynamic and resource-constrained environments.