Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MineXR: Mining Personalized Extended Reality Interfaces (2403.08057v1)

Published 12 Mar 2024 in cs.HC

Abstract: Extended Reality (XR) interfaces offer engaging user experiences, but their effective design requires a nuanced understanding of user behavior and preferences. This knowledge is challenging to obtain without the widespread adoption of XR devices. We introduce MineXR, a design mining workflow and data analysis platform for collecting and analyzing personalized XR user interaction and experience data. MineXR enables elicitation of personalized interfaces from participants of a data collection: for any particular context, participants create interface elements using application screenshots from their own smartphone, place them in the environment, and simultaneously preview the resulting XR layout on a headset. Using MineXR, we contribute a dataset of personalized XR interfaces collected from 31 participants, consisting of 695 XR widgets created from 178 unique applications. We provide insights for XR widget functionalities, categories, clusters, UI element types, and placement. Our open-source tools and data support researchers and designers in developing future XR interfaces.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (81)
  1. App-Based Task Shortcuts for Virtual Assistants. In The 34th Annual ACM Symposium on User Interface Software and Technology. ACM, Virtual Event USA, 1089–1099. https://doi.org/10.1145/3472749.3474808
  2. Ronald Azuma and Chris Furmanski. 2003. Evaluating label placement for augmented reality view management. In The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings. IEEE, 66–75.
  3. UIBert: Learning Generic Multimodal Representations for UI Understanding. arXiv:2107.13731 (Aug 2021). http://arxiv.org/abs/2107.13731 arXiv:2107.13731 [cs].
  4. View management for virtual and augmented reality. In Proceedings of the 14th annual ACM symposium on User interface software and technology. 101–110.
  5. João Belo and Mathias N Lystb. 2022. AUIT – the Adaptive User Interfaces Toolkit for Designing XR Applications. (2022), 17.
  6. Cross-device taxonomy: Survey, opportunities and challenges of interactions spanning across multiple devices. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow Scotland Uk). ACM, New York, NY, USA. https://doi.org/10.1145/3290605.3300792
  7. SemanticAdapt: Optimization-based Adaptation of Mixed Reality Layouts Leveraging Virtual-Physical Semantic Connections. In The 34th Annual ACM Symposium on User Interface Software and Technology. ACM, Virtual Event USA, 282–297. https://doi.org/10.1145/3472749.3474750
  8. Reflect, not regret: Understanding regretful smartphone use with app feature-level analysis. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–36.
  9. Validating the Benefits of Glanceable and Context-Aware Augmented Reality for Everyday Information Access Tasks. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Christchurch, New Zealand, 436–444. https://doi.org/10.1109/VR51125.2022.00063
  10. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, Québec City QC Canada, 845–854. https://doi.org/10.1145/3126594.3126651
  11. ZIPT: Zero-Integration Performance Testing of Mobile App Designs. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, Québec City QC Canada, 727–736. https://doi.org/10.1145/3126594.3126647
  12. ERICA: Interaction Mining Mobile Apps. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, Tokyo Japan, 767–776. https://doi.org/10.1145/2984511.2984581
  13. Level of detail interfaces. In Third IEEE and ACM International Symposium on Mixed and Augmented Reality. IEEE, 300–301.
  14. Tailored reality: Perception-aware scene restructuring for adaptive vr navigation. ACM Transactions on Graphics (TOG) 40, 5 (2021), 1–15.
  15. DepthLab: Real-time 3D interaction with depth maps for mobile augmented reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 829–843.
  16. Crowdsourcing design guidance for contextual adaptation of text content in augmented reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
  17. Ivy: Exploring Spatially Situated Visual Programming for Authoring and Understanding Intelligent Environments. In Proceedings of the 43rd Graphics Interface Conference (Edmonton, Alberta, Canada) (GI ’17). Canadian Human-Computer Communications Society, Waterloo, CAN, 156–162.
  18. Spatial Constancy of Surface-Embedded Layouts across Multiple Environments. In Proceedings of the 3rd ACM Symposium on Spatial User Interaction. ACM, Los Angeles California USA, 65–68. https://doi.org/10.1145/2788940.2788954
  19. Optispace: Automated placement of interactive 3d projection mapping content. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–11.
  20. Heatspace: Automatic placement of displays by empirical analysis of user behavior. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 611–621.
  21. Brassau: automatic generation of graphical user interfaces for virtual assistants. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, Barcelona Spain, 1–12. https://doi.org/10.1145/3229434.3229481
  22. Krzysztof Gajos and Daniel S Weld. 2004. SUPPLE: automatically generating user interfaces. In Proceedings of the 9th international conference on Intelligent user interfaces. 93–100.
  23. FLARE: Fast layout for augmented reality applications. In 2014 IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, 207–212.
  24. What You Sketch Is What You Get: Quick and Easy Augmented Reality Prototyping with PintAR. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1–6. https://doi.org/10.1145/3290607.3312847
  25. Adaptive augmented reality: plasticity of augmentations. In Proceedings of the 2014 Virtual Reality International Conference. 1–4.
  26. Image-driven view management for augmented reality browsers. In 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 177–186.
  27. Sinem Guven and Steven Feiner. 2003. Authoring 3D Hypermedia for Wearable Augmented and Virtual Reality. In Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings. IEEE, White Plains, NY, USA, 118–126. https://doi.org/10.1109/ISWC.2003.1241401
  28. Mobile Augmented Reality Interaction Techniques for Authoring Situated Media On-Site. In 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE, Santa Barbara, CA, USA, 235–236. https://doi.org/10.1109/ISMAR.2006.297821
  29. Live semantic 3d perception for immersive augmented reality. IEEE transactions on visualization and computer graphics 26, 5 (2020), 2012–2022.
  30. BlendMR: A Computational Method to Create Ambient Mixed Reality Interfaces. Proceedings of the ACM on Human-Computer Interaction 7, ISS (2023), 217–241.
  31. Immersive modular factory layout planning using augmented reality. Procedia CIRP 72 (2018), 1112–1117.
  32. XSpace: An Augmented Reality Toolkit for Enabling Spatially-Aware Distributed Collaboration. Proceedings of the ACM on Human-Computer Interaction 6, ISS (2022), 277–302.
  33. Scenariot: Spatially mapping smart things within augmented reality scenes. In Proceedings of the 2018 CHI Conference on human factors in computing systems. 1–13.
  34. Window-Shaping: 3D Design Ideation by Creating on, Borrowing from, and Looking at the Physical World. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction. ACM, Yokohama Japan, 37–45. https://doi.org/10.1145/3024969.3024995
  35. Layerable Apps: Comparing Concurrent and Exclusive Display of Augmented Reality Applications. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 857–863.
  36. “In VR, everything is possible!”: Sketching and Simulating Spatially-Aware Interactive Spaces in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–16. https://doi.org/10.1145/3313831.3376652
  37. Dongsik Jo and Gerard Jounghyun Kim. 2016. ARIoT: scalable augmented reality framework for interacting with Internet of Things appliances everywhere. IEEE Transactions on Consumer Electronics 62, 3 (2016), 334–340.
  38. Information filtering for mobile augmented reality. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000). IEEE, 3–11.
  39. Hirokazu Kato and Mark Billinghurst. 1999. Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System. In Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99). IEEE Comput. Soc, San Francisco, CA, USA, 85–94. https://doi.org/10.1109/IWAR.1999.803809
  40. X-droid: a quick and easy android prototyping framework with a single-app illusion. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 95–108.
  41. Webzeitgeist: design mining the web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris France, 3083–3092. https://doi.org/10.1145/2470654.2466420
  42. Bricolage: example-based retargeting for web design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Vancouver BC Canada, 2197–2206. https://doi.org/10.1145/1978942.1979262
  43. Kin Chung Kwan and Hongbo Fu. 2019. Mobi3dsketch: 3d sketching in mobile ar. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–11.
  44. Wallace S. Lages and Doug A. Bowman. 2019. Walking with adaptive augmented reality workspaces: design and usage patterns. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, Marina del Ray California, 356–366. https://doi.org/10.1145/3301275.3302278
  45. Virtual agent positioning driven by scene semantics in mixed reality. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 767–775.
  46. User Preference for Navigation Instructions in Mixed Reality. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 802–811. https://doi.org/10.1109/VR51125.2022.00102
  47. A-mash: providing single-app illusion for multi-app use through user-centric UI mashup. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. ACM, Sydney NSW Australia, 690–702. https://doi.org/10.1145/3495243.3560522
  48. Rapido: Prototyping Interactive AR Experiences through Programming by Demonstration. In The 34th Annual ACM Symposium on User Interface Software and Technology. ACM, Virtual Event USA, 626–637. https://doi.org/10.1145/3472749.3474774
  49. Enact: Reducing Designer–Developer Breakdowns When Prototyping Custom Interactions. ACM Transactions on Computer-Human Interaction 26, 3 (June 2019), 1–48. https://doi.org/10.1145/3310276
  50. Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–13. https://doi.org/10.1145/3313831.3376160
  51. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, Denver Colorado USA, 6038–6049. https://doi.org/10.1145/3025453.3025483
  52. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–15. https://doi.org/10.1145/3411764.3445049
  53. Toby Jia-Jun Li and Oriana Riva. 2018. Kite: Building Conversational Bots from Mobile Apps. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, Munich Germany, 96–109. https://doi.org/10.1145/3210240.3210339
  54. Scene-aware behavior synthesis for virtual pets in mixed reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12.
  55. Context-Aware Online Adaptation of Mixed Reality Interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, New Orleans LA USA, 147–160. https://doi.org/10.1145/3332165.3347945
  56. Feiyu Lu and Doug A. Bowman. 2021. Evaluating the Potential of Glanceable AR Interfaces for Authentic Everyday Uses. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, Lisboa, Portugal, 768–777. https://doi.org/10.1109/VR50410.2021.00104
  57. Glanceable AR: Evaluating Information Access Methods for Head-Worn Augmented Reality. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Atlanta, GA, USA, 930–939. https://doi.org/10.1109/VR46266.2020.00113
  58. Feiyu Lu and Yan Xu. 2022. Exploring Spatial UI Transition Mechanisms with Head-Worn Augmented Reality. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–16. https://doi.org/10.1145/3491102.3517723
  59. DART: a toolkit for rapid design exploration of augmented reality experiences. In Proceedings of the 17th annual ACM symposium on User interface software and technology. 197–206.
  60. Mark R Mine. 1995. ISAAC: A Virtual Environment Tool for the Interactive Construction of Virtual Worlds. (1995).
  61. ProtoAR: Rapid Physical-Digital Prototyping of Mobile Augmented Reality Applications. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–12. https://doi.org/10.1145/3173574.3173927
  62. Michael Nebeling and Maximilian Speicher. 2018. The trouble with augmented reality/virtual reality authoring tools. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE. https://doi.org/10.1109/ismar-adjunct.2018.00098
  63. MRAT: The mixed reality analytics toolkit. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.
  64. SnapToReality: Aligning Augmented Reality to the Real World. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, San Jose California USA, 1233–1244. https://doi.org/10.1145/2858036.2858250
  65. Corsican Twin: Authoring In Situ Augmented Reality Visualisations in Virtual Reality. In Proceedings of the International Conference on Advanced Visual Interfaces. ACM, Salerno Italy, 1–9. https://doi.org/10.1145/3399715.3399743
  66. ScalAR: Authoring Semantically Adaptive Augmented Reality Experiences in Virtual Reality. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–18. https://doi.org/10.1145/3491102.3517665
  67. Speeded up detection of squared fiducial markers. Image and vision Computing 76 (2018), 38–47.
  68. ComposAR: An intuitive tool for authoring AR applications. In 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE, Cambridge, UK, 177–178. https://doi.org/10.1109/ISMAR.2008.4637354
  69. Spatial. 2023. Create, share, and experience your creativity in 3D. http://www.spatial.io/
  70. Retargetable AR: Context-aware augmented reality in indoor scenes based on 3D scene graph. In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, 249–255.
  71. Adaptive information density for augmented reality displays. In 2016 IEEE Virtual Reality (VR). IEEE, 83–92.
  72. Daniel Wagner and Dieter Schmalstieg. 2007. Artoolkitplus for pose tracking on mobile devices. (2007).
  73. Enabling Conversational Interaction with Mobile UI Using Large Language Models. arXiv:arXiv:2209.08655
  74. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. In The 34th Annual ACM Symposium on User Interface Software and Technology. ACM, Virtual Event USA, 498–510. https://doi.org/10.1145/3472749.3474765
  75. CAPturAR: An Augmented Reality Tool for Authoring Human-Involved Context-Aware Applications. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, Virtual Event USA, 328–341. https://doi.org/10.1145/3379337.3415815
  76. DistanciAR: Authoring Site-Specific Augmented Reality Experiences for Remote Environments. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–12. https://doi.org/10.1145/3411764.3445552
  77. Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots. In The 34th Annual ACM Symposium on User Interface Software and Technology. ACM, Virtual Event USA, 470–483. https://doi.org/10.1145/3472749.3474763
  78. Spacetime: Enabling Fluid Individual and Collaborative Editing in Virtual Reality. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. ACM, Berlin Germany, 853–866. https://doi.org/10.1145/3242587.3242597
  79. SceneCtrl: Mixed Reality Enhancement via Efficient Scene Editing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, Québec City QC Canada, 427–436. https://doi.org/10.1145/3126594.3126601
  80. Lei Zhang and Steve Oney. 2020. FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, Virtual Event USA, 342–353. https://doi.org/10.1145/3379337.3415824
  81. Screen recognition: Creating accessibility metadata for mobile applications from pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now