2000 character limit reached
OCTO+: A Suite for Automatic Open-Vocabulary Object Placement in Mixed Reality
Published 17 Jan 2024 in cs.CV, cs.AI, and cs.CL | (2401.08973v1)
Abstract: One key challenge in Augmented Reality is the placement of virtual content in natural locations. Most existing automated techniques can only work with a closed-vocabulary, fixed set of objects. In this paper, we introduce and evaluate several methods for automatic object placement using recent advances in open-vocabulary vision-LLMs. Through a multifaceted evaluation, we identify a new state-of-the-art method, OCTO+. We also introduce a benchmark for automatically evaluating the placement of virtual objects in augmented reality, alleviating the need for costly user studies. Through this, in addition to human evaluations, we find that OCTO+ places objects in a valid region over 70% of the time, outperforming other methods on a range of metrics.
- L. Yoffe, A. Sharma, and T. Höllerer, “Octopus: Open-vocabulary content tracking and object placement using semantic understanding in mixed reality,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, 2023, pp. 587–588.
- B. Nuernberger, E. Ofek, H. Benko, and A. D. Wilson, “Snaptoreality: Aligning augmented reality to the real world,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2016, pp. 1233–1244.
- D. E. Breen, R. T. Whitaker, E. Rose, and M. Tuceryan, “Interactive occlusion and automatic object placement for augmented reality,” in Computer Graphics Forum, vol. 15, no. 3. Wiley Online Library, 1996, pp. 11–22.
- T. Rafi, X. Zhang, and X. Wang, “Predart: Towards automatic oracle prediction of object placements in augmented reality testing,” in 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–13.
- Y. Cheng, Y. Yan, X. Yi, Y. Shi, and D. Lindlbauer, “Semanticadapt: Optimization-based adaptation of mixed reality layouts leveraging virtual-physical semantic connections,” in The 34th Annual ACM Symposium on User Interface Software and Technology, 2021, pp. 282–297.
- Y. Lang, W. Liang, and L.-F. Yu, “Virtual agent positioning driven by scene semantics in mixed reality,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2019, pp. 767–775.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021.
- A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” 2023.
- F. Odom, “clip-text-decoder,” https://github.com/fkodom/clip-text-decoder, 2022.
- A. Akbik, D. Blythe, and R. Vollgraf, “Contextual string embeddings for sequence labeling,” in COLING 2018, 27th International Conference on Computational Linguistics, 2018, pp. 1638–1649.
- W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in International Conference on Machine Learning. PMLR, 2021, pp. 5583–5594.
- T. Lüddecke and A. S. Ecker, “Image segmentation using text and image prompts,” 2022.
- S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang, “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” 2023.
- X. Huang, Y.-J. Huang, Y. Zhang, W. Tian, R. Feng, Y. Zhang, Y. Xie, Y. Li, and L. Zhang, “Open-set image tagging with multi-grained text supervision,” 2023.
- OpenAI, “Gpt-4 technical report,” 2023.
- H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” 2023.
- J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023.
- A. Lampinen, I. Dasgupta, S. Chan, K. Mathewson, M. Tessler, A. Creswell, J. McClelland, J. Wang, and F. Hill, “Can language models learn from explanations in context?” in Findings of the Association for Computational Linguistics: EMNLP 2022, Y. Goldberg, Z. Kozareva, and Y. Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 537–563. [Online]. Available: https://aclanthology.org/2022.findings-emnlp.38
- IDEA-Research, “Grounded-segment-anything,” https://github.com/IDEA-Research/Grounded-Segment-Anything, 2023.
- T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” 2023.
- N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images.” ECCV (5), vol. 7576, pp. 746–760, 2012.
- J. Xiao, A. Owens, and A. Torralba, “Sun3d: A database of big spaces reconstructed using sfm and object labels,” in Proceedings of the IEEE international conference on computer vision, 2013, pp. 1625–1632.
- N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” 2019.
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019.
- Apple, “Arkit,” https://developer.apple.com/augmented-reality/arkit/.
- Apple., “Scenekit,” https://developer.apple.com/documentation/scenekit.
- H. Jun and A. Nichol, “Shap-e: Generating conditional 3d implicit functions,” 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
