2000 character limit reached
Neural Mesh Fusion: Unsupervised 3D Planar Surface Understanding (2402.16739v1)
Published 26 Feb 2024 in cs.CV
Abstract: This paper presents Neural Mesh Fusion (NMF), an efficient approach for joint optimization of polygon mesh from multi-view image observations and unsupervised 3D planar-surface parsing of the scene. In contrast to implicit neural representations, NMF directly learns to deform surface triangle mesh and generate an embedding for unsupervised 3D planar segmentation through gradient-based optimization directly on the surface mesh. The conducted experiments show that NMF obtains competitive results compared to state-of-the-art multi-view planar reconstruction, while not requiring any ground-truth 3D or planar supervision. Moreover, NMF is significantly more computationally efficient compared to implicit neural rendering-based scene reconstruction approaches.
- “PlaneNet: Piece-wise planar reconstruction from a single RGB image,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
- “PlaneRCNN: 3D plane detection and reconstruction from a single image,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
- “Single-image piece-wise planar 3d reconstruction via associative embedding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
- “PlaneTR: Structure-guided transformers for 3d plane recovery,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
- “PlaneFormers: From sparse view planes to 3D reconstruction,” in Proceedings of the European Conference on Computer Vision, 2022.
- “PlanarRecon: Real-time 3D plane detection and reconstruction from posed monocular videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
- “NeRF: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
- “Ref-NeRF: Structured view-dependent appearance for neural radiance fields,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2022.
- “Depth-supervised NeRF: Fewer views and faster training for free,” in Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2022.
- “Controlling neural level sets,” Advances in Neural Information Processing Systems, 2019.
- “Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” arXiv preprint arXiv:2106.10689, 2021.
- “Volume rendering of neural implicit surfaces,” Advances in Neural Information Processing Systems, vol. 34, pp. 4805–4815, 2021.
- “MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction,” Advances in neural information processing systems, 2022.
- “Marching cubes: A high resolution 3D surface construction algorithm,” ACM SIGGRAPH Computer Graphics, vol. 21, no. 4, pp. 163–169, 1987.
- “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981.
- “Learning deformable tetrahedral meshes for 3D reconstruction,” Advances In Neural Information Processing Systems, vol. 33, pp. 9936–9947, 2020.
- “Deep marching tetrahedra: a hybrid representation for high-resolution 3D shape synthesis,” Advances in Neural Information Processing Systems, 2021.
- “Extracting triangular 3D models, materials, and lighting from images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
- “MobileNeRF: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
- “Scannet: Richly-annotated 3D reconstructions of indoor scenes,” in Proc. of the IEEE conf. on computer vision and pattern recognition, 2017.
- “Recovering 3D planes from a single image via convolutional neural networks,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018.
- “Learning pairwise inter-plane relations for piecewise planar reconstruction,” in Proceedings of the European Conference on Computer Vision, 2020.
- “Planar surface reconstruction from sparse views,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
- “PlaneMVS: 3D plane reconstruction from multi-view stereo,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
- “Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3D scans,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
- “Fast, minimum storage ray/triangle intersection,” in ACM SIGGRAPH 2005 Courses. 2005.
- “NeuralRecon: Real-time coherent 3D reconstruction from monocular video,” in Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2021, pp. 15598–15607.
- “Neural 3D scene reconstruction with the manhattan-world assumption,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.