Get-A-Sense: Designing Spatial Context Awareness for Mobile AR Environment Understanding
Abstract: Physical environment understanding is vital in delivering immersive and interactive mobile augmented reality (AR) user experiences. Recently, we have witnessed a transition in the design of environment understanding systems, from visual data focused to centering on the concept of spatial context, including user, device, and environment information. Even though spatial context can benefit the environment understanding tasks, e.g., we demonstrate in a case study that lighting estimation performance can be improved by as much as 59%, not all environment understanding systems support spatial context. Furthermore, even for the environment understanding systems that support spatial context, not all useful spatial context has been leveraged; and the design and implementation can differ vastly. In this paper, we advocate for the design of a spatial context-aware and shared environment understanding system that can effectively support multiple tasks simultaneously. Fundamentally, there are many practical challenges in designing such a unified environment understanding system. We discuss the challenges in the context of three open questions, (i) how to fully leverage user mobility, (ii) how to design a unified data model, and (iii) finally how to build a shared system for multiple tasks.
- ARKitscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
- Midas v3.1 – a model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307.14460, 2023.
- Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
- Text2light: Zero-shot text-driven hdr panorama generation. ACM Transactions on Graphics (TOG), 2022.
- LocIn: Inferring semantic location from spatial maps in mixed reality. In 32nd USENIX Security Symposium (USENIX Security 23), 2023.
- YOLO by Ultralytics, Jan. 2023.
- Segment anything. arXiv:2304.02643, 2023.
- Adaptive resolution-based tradeoffs for energy-efficient visual computing systems. IEEE Pervasive Computing, 20(2):18–26, 2021.
- User friendly slam initialization. In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2013.
- P. Nowacki and M. Woda. Capabilities of arcore and arkit platforms for ar/vr applications. In Engineering in Dependability of Computer Systems and Networks: Proceedings of the Fourteenth International Conference on Dependability of Computer Systems DepCoS-RELCOMEX, July 1–5, 2019, Brunów, Poland, pages 358–370. Springer, 2020.
- Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975–10985, 2019.
- Deep depth estimation from visual-inertial slam. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
- Visual contextual awareness in wearable computing. In Digest of Papers. Second International Symposium on Wearable Computers (Cat. No. 98EX215).
- Retargetable ar: Context-aware augmented reality in indoor scenes based on 3d scene graph. In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pages 249–255. IEEE, 2020.
- Sparse and noisy lidar completion with rgb guidance and uncertainty. In 2019 16th International Conference on Machine Vision Applications (MVA).
- B. Wen and K. E. Bekris. Bundletrack: 6d pose tracking for novel objects without instance or category-level 3d models. In IEEE/RSJ International Conference on Intelligent Robots and Systems.
- Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. CVPR, 2023.
- Rcareworld: A human-centric simulation world for caregiving robots. IROS, 2022.
- J. Yi and Y. Lee. Heimdall: mobile gpu coordination platform for augmented reality applications. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, 2020.
- Litar: Visually coherent lighting for mobile augmented reality. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(3):1–29, 2022.
Paper Prompts
Sign up for free to create and run prompts on this paper.