2000 character limit reached
RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction (2312.01215v2)
Published 2 Dec 2023 in cs.CV
Abstract: This paper introduces a versatile paradigm for integrating multi-view reflectance (optional) and normal maps acquired through photometric stereo. Our approach employs a pixel-wise joint re-parameterization of reflectance and normal, considering them as a vector of radiances rendered under simulated, varying illumination. This re-parameterization enables the seamless integration of reflectance and normal maps as input data in neural volume rendering-based 3D reconstruction while preserving a single optimization objective. In contrast, recent multi-view photometric stereo (MVPS) methods depend on multiple, potentially conflicting objectives. Despite its apparent simplicity, our proposed approach outperforms state-of-the-art approaches in MVPS benchmarks across F-score, Chamfer distance, and mean angular error metrics. Notably, it significantly improves the detailed 3D reconstruction of areas with high curvature or low visibility.
- VCGLib. https://github.com/cnr-isti-vclab/vcglib.
- Neural apparent BRDF fields for multiview photometric stereo. In Proceedings of the 19th ACM SIGGRAPH European Conference on Visual Media Production, pages 1–10, 2022.
- Meshlab: an open-source mesh processing tool. In Proceedings of the Eurographics Italian Chapter Conference, pages 129–136, 2008.
- On optimal light configurations in photometric stereo. In Proceedings of the 10th IEEE International Conference on Computer Vision, pages 1707–1712, 2005.
- Multi-view stereo: A tutorial. Foundations and Trends® in Computer Graphics and Vision, 9(1-2):1–148, 2015.
- Multiview Photometric Stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(3):548–554, 2008.
- Satoshi Ikehata. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In Proceedings of the European Conference on Computer Vision, pages 3–18, 2018.
- Satoshi Ikehata. Scalable, Detailed and Mask-Free Universal Photometric Stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13198–13207, 2023.
- Large scale multi-view stereopsis evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 406–413, 2014.
- Uncertainty-aware deep multi-view photometric stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12601–12611, 2022a.
- Neural radiance fields approach to deep multi-view photometric stereo. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1965–1977, 2022b.
- Multi-View Photometric Stereo Revisited. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3126–3135, 2023.
- Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic materials. IEEE Transactions on Image Processing, 29:4159–4173, 2020.
- Neuralangelo: High-Fidelity Neural Surface Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8456–8465, 2023.
- A differential volumetric approach to multi-view photometric stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1052–1061, 2019.
- Marching cubes: A high resolution 3D surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pages 347–353. 1998.
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Communications of the ACM, 65(1):99–106, 2021.
- Efficiently combining positions and normals for precise 3D geometry. ACM Tansactions on Graphics, 24(3):536–543, 2005.
- Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589–5599, 2021.
- Multiview photometric stereo using planar mesh parameterization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1161–1168, 2013.
- Robust multiview photometric stereo using planar mesh parameterization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(8):1591–1604, 2016.
- Patchmatchnet: Learned multi-view patchmatch stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14194–14203, 2021a.
- NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Proceedings of the Conference on Neural Information Processing Systems, 2021b.
- Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3295–3306, 2023.
- Robert J Woodham. Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1):139–144, 1980.
- Robust photometric stereo via low-rank matrix completion and recovery. In Proceedings of the 10th Asian Conference on Computer Vision, pages 703–717, 2011.
- Multi-scale geometric consistency guided and planar prior assisted multi-view stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4945–4963, 2022.
- PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo. In Proceedings of the European Conference on Computer Vision, pages 266–284, 2022.
- MVPSNet: Fast Generalizable Multi-view Photometric Stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12525–12536, 2023.