SANPO: A Scene Understanding, Accessibility and Human Navigation Dataset
Abstract: Vision is essential for human navigation. The World Health Organization (WHO) estimates that 43.3 million people were blind in 2020, and this number is projected to reach 61 million by 2050. Modern scene understanding models could empower these people by assisting them with navigation, obstacle avoidance and visual recognition capabilities. The research community needs high quality datasets for both training and evaluation to build these systems. While datasets for autonomous vehicles are abundant, there is a critical gap in datasets tailored for outdoor human navigation. This gap poses a major obstacle to the development of computer vision based Assistive Technologies. To overcome this obstacle, we present SANPO, a large-scale egocentric video dataset designed for dense prediction in outdoor human navigation environments. SANPO contains 701 stereo videos of 30+ seconds captured in diverse real-world outdoor environments across four geographic locations in the USA. Every frame has a high resolution depth map and 112K frames were annotated with temporally consistent dense video panoptic segmentation labels. The dataset also includes 1961 high-quality synthetic videos with pixel accurate depth and panoptic segmentation annotations to balance the noisy real world annotations with the high precision synthetic annotations. SANPO is already publicly available and is being used by mobile applications like Project Guideline to train mobile models that help low-vision users go running outdoors independently. To preserve anonymization during peer review, we will provide a link to our dataset upon acceptance. SANPO is available here: https://google-research-datasets.github.io/sanpo_dataset/
- Test your self-driving algorithm: An overview of publicly available driving datasets and virtual testing environments. IEEE Transactions on Intelligent Vehicles, 4(2):171–185, 2019.
- Dolphins: Dataset for collaborative perception enabled harmonious and interconnected self-driving. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 4361–4377, December 2022.
- Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493, 2023.
- Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation. IEEE Robotics and Automation Letters, 7(4):11807–11814, 2022.
- Toward human-like social robot navigation: A large-scale, multi-modal, social human navigation dataset. arXiv preprint arXiv:2303.14880, 2023.
- Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022.
- Motsynth: How can synthetic data help pedestrian detection and tracking? In International Conference on Computer Vision (ICCV), 2021.
- Deep learning-based perception systems for autonomous driving: A comprehensive survey. Neurocomputing, 489:255–270, 2022.
- Algorithm and hardware implementation for visual perception system in autonomous vehicle: A survey. Integration, 59:148–156, 2017.
- Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation. arXiv preprint arXiv:2012.05258, 2020.
- The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. Pattern Analysis and Machine Intelligence (PAMI), 2022.
- Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
- Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.
- The 2019 davis challenge on vos: Unsupervised multi-object segmentation. arXiv:1905.00737, 2019.
- Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters, 30(2):88–97, 2009. ISSN 0167-8655. doi: https://doi.org/10.1016/j.patrec.2008.04.005. URL https://www.sciencedirect.com/science/article/pii/S0167865508001220. Video-based Object and Event Analysis.
- nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.11027, 2019.
- Playing for benchmarks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2213–2222, 2017.
- Waymo open dataset: Panoramic video panoptic segmentation, 2022.
- A*3d dataset: Towards autonomous driving in challenging environments. In Proceedings of The International Conference in Robotics and Automation (ICRA), 2020.
- Practical stereo matching via cascaded recurrent network with adaptive correlation, 2022a.
- Associating objects with transformers for video object segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
- kmax-deeplab: k-means mask transformer, 2023.
- Zoedepth: Zero-shot transfer by combining relative and metric depth, 2023.
- Vision transformers for dense prediction, 2021.
- Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
- Segment anything, 2023.
- Reviving iterative training with mask guidance for interactive segmentation. In 2022 IEEE International Conference on Image Processing (ICIP), pages 3141–3145. IEEE, 2022.
- Binsformer: Revisiting adaptive bins for monocular depth estimation, 2022b.
- DeepLab2: A TensorFlow Library for Deep Labeling. arXiv: 2106.09748, 2021.
- The apolloscape dataset for autonomous driving. arXiv: 1803.06184, 2018.
- 3d packing for self-supervised monocular depth estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
