Den-SOFT: Dense Space-Oriented Light Field DataseT for 6-DOF Immersive Experience (2403.09973v1)
Abstract: We have built a custom mobile multi-camera large-space dense light field capture system, which provides a series of high-quality and sufficiently dense light field images for various scenarios. Our aim is to contribute to the development of popular 3D scene reconstruction algorithms such as IBRnet, NeRF, and 3D Gaussian splitting. More importantly, the collected dataset, which is much denser than existing datasets, may also inspire space-oriented light field reconstruction, which is potentially different from object-centric 3D reconstruction, for immersive VR/AR experiences. We utilized a total of 40 GoPro 10 cameras, capturing images of 5k resolution. The number of photos captured for each scene is no less than 1000, and the average density (view number within a unit sphere) is 134.68. It is also worth noting that our system is capable of efficiently capturing large outdoor scenes. Addressing the current lack of large-space and dense light field datasets, we made efforts to include elements such as sky, reflections, lights and shadows that are of interest to researchers in the field of 3D reconstruction during the data capture process. Finally, we validated the effectiveness of our provided dataset on three popular algorithms and also integrated the reconstructed 3DGS results into the Unity engine, demonstrating the potential of utilizing our datasets to enhance the realism of virtual reality (VR) and create feasible interactive spaces. The dataset is available at our project website.
- Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5470–5479.
- Zip-NeRF: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706 (2023).
- Omniphotos: casual 360 vr photography. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–12.
- Immersive Light Field Video with a Layered Mesh Representation. ACM Transactions on Graphics 39, 4 (Aug. 2020). https://doi.org/10.1145/3386569.3392485
- Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 (2017).
- Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5828–5839.
- DeepView: View Synthesis with Learned Gradient Descent. arXiv:1906.07316 [cs, eess]
- Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5152–5161.
- Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence 36, 7 (2013), 1325–1339.
- Large scale multi-view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 406–413.
- Panoptic studio: A massively multiview system for social motion capture. In Proceedings of the IEEE International Conference on Computer Vision. 3334–3342.
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics 42, 4 (Aug. 2023), 1–14. https://doi.org/10.1145/3592433
- Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG) 36, 4 (2017), 1–13.
- Motion-x: A large-scale 3d expressive whole-body human motion dataset. arXiv preprint arXiv:2307.00818 (2023).
- AMASS: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision. 5442–5451.
- Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 1 (2021), 99–106.
- Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics 41, 4 (July 2022), 1–15. https://doi.org/10.1145/3528223.3530127 arXiv:2201.05989 [cs]
- Free-viewpoint indoor neural relighting from multi-view stereo. ACM Transactions on Graphics (TOG) 40, 5 (2021), 1–18.
- BABEL: Bodies, action and behavior with english labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 722–731.
- Recognizing fine-grained and composite activities using hand-centric features and script data. International Journal of Computer Vision 119 (2016), 346–373.
- A Multi-View Stereo Benchmark with High-Resolution Images and Multi-Camera Videos. In Conference on Computer Vision and Pattern Recognition (CVPR).
- Multi-sensor large-scale dataset for multi-view 3D reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21392–21403.
- IBRNet: Learning Multi-View Image-Based Rendering. arXiv:2102.13090 [cs]
- Sun3d: A database of big spaces reconstructed using sfm and object labels. In Proceedings of the IEEE international conference on computer vision. 1625–1632.
- VR-NeRF: High-Fidelity Virtualized Walkable Spaces. In SIGGRAPH Asia 2023 Conference Papers. ACM, Sydney NSW Australia, 1–12. https://doi.org/10.1145/3610548.3618139
- Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5336–5345.