Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PalmGazer: Unimanual Eye-hand Menus in Augmented Reality (2306.12402v1)

Published 21 Jun 2023 in cs.HC

Abstract: How can we design the user interfaces for augmented reality (AR) so that we can interact as simple, flexible and expressive as we can with smartphones in one hand? To explore this question, we propose PalmGazer as an interaction concept integrating eye-hand interaction to establish a singlehandedly operable menu system. In particular, PalmGazer is designed to support quick and spontaneous digital commands -- such as to play a music track, check notifications or browse visual media -- through our devised three-way interaction model: hand opening to summon the menu UI, eye-hand input for selection of items, and dragging gesture for navigation. A key aspect is that it remains always-accessible and movable to the user, as the menu supports meaningful hand and head based reference frames. We demonstrate the concept in practice through a prototypical personal UI with application probes, and describe technique designs specifically-tailored to the application UI. A qualitative evaluation highlights the system's design benefits and drawbacks, e.g., that common 2D scroll and selection tasks are simple to operate, but higher degrees of freedom may be reserved for two hands. Our work contributes interaction techniques and design insights to expand AR's uni-manual capabilities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. StickyPie: A Gaze-Based, Scale-Invariant Marking Menu Optimized for AR/VR. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 739, 16 pages. https://doi.org/10.1145/3411764.3445297
  2. AUIT - the Adaptive User Interfaces Toolkit for Designing XR Applications. In Proceedings of the 35nd Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3526113.3545651
  3. D.A. Bowman and C.A. Wingrave. 2001. Design and evaluation of menu systems for immersive virtual environments. In Proceedings IEEE Virtual Reality 2001. IEEE, 149–156. https://doi.org/10.1109/VR.2001.913781
  4. Two-handed input in human-computer interaction. Haptic Input 11 (2008).
  5. SemanticAdapt: Optimization-Based Adaptation of Mixed Reality Layouts Leveraging Virtual-Physical Semantic Connections. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 282–297. https://doi.org/10.1145/3472749.3474750
  6. Validating the Benefits of Glanceable and Context-Aware Augmented Reality for Everyday Information Access Tasks. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 436–444. https://doi.org/10.1109/VR51125.2022.00063
  7. Jack T. Dennerlein. 2015. The state of ergonomics for mobile computing technology. Work 52 (2015), 269–277. https://doi.org/10.3233/WOR-152159 2.
  8. Leveling the Playing Field: A Comparative Reevaluation of Unmodified Eye Tracking as an Input and Interaction Modality for VR. IEEE Transactions on Visualization and Computer Graphics 29, 5 (2023), 2269–2279. https://doi.org/10.1109/TVCG.2023.3247058
  9. Virtual Reality for Palmtop Computers. ACM Trans. Inf. Syst. 11, 3 (jul 1993), 197–218. https://doi.org/10.1145/159161.159160
  10. Ultraleap for developers. 2021. UI Panels. Website. Retrieved April 2, 2023 from https://docs.ultraleap.com/xr-guidelines/Components/ui-panels.html
  11. Chelsey Nicole Gobeli. 2018. The Effects of Reference Frames on 3D Menus in Virtual Environments. Ph. D. Dissertation.
  12. Yves Guiard. 1987. Asymmetric Division of Labor in Human Skilled Bimanual Action. Journal of Motor Behavior 19, 4, 486–517. https://doi.org/10.1080/00222895.1987.10735426
  13. RotoSwype: Word-Gesture Typing Using a Ring. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300244
  14. On-Body Interaction: Armed and Dangerous. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (Kingston, Ontario, Canada) (TEI ’12). Association for Computing Machinery, New York, NY, USA, 69–76. https://doi.org/10.1145/2148131.2148148
  15. Zhenyi He and Xubo Yang. 2014. Hand-Based Interaction for Object Manipulation with Augmented Reality Glasses. In Proceedings of the 13th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (Shenzhen, China) (VRCAI ’14). Association for Computing Machinery, New York, NY, USA, 227–230. https://doi.org/10.1145/2670473.2670505
  16. The Springboard: Multiple Modes in One Spring-Loaded Control. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada) (CHI ’06). Association for Computing Machinery, New York, NY, USA, 181–190. https://doi.org/10.1145/1124772.1124801
  17. Technology Probes: Inspiring Design for and with Families. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI ’03). Association for Computing Machinery, New York, NY, USA, 17–24. https://doi.org/10.1145/642611.642616
  18. PinchText: One-Handed Text Entry Technique Combining Pinch Gestures and Hand Positions for Head-Mounted Displays. International Journal of Human–Computer Interaction 0, 0 (2022), 1–17. https://doi.org/10.1080/10447318.2022.2115333
  19. Look & Pedal: Hands-Free Navigation in Zoomable Information Spaces through Gaze-Supported Foot Input. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (Seattle, Washington, USA) (ICMI ’15). Association for Computing Machinery, New York, NY, USA, 123–130. https://doi.org/10.1145/2818346.2820751
  20. Pinpointing: Precise Head- and Eye-Based Target Selection for Augmented Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3173574.3173655
  21. HiMER Lab. 2020. Demo: Eye Tracking Interaction Example - MRTK, Hololens 2. Video. Retrieved April 2, 2023 from https://youtu.be/UPH4lk1jAWs?t=21
  22. Exploration of Techniques for Rapid Activation of Glanceable Information in Head-Worn Augmented Reality. In Symposium on Spatial User Interaction (Virtual Event, USA) (SUI ’21). Association for Computing Machinery, New York, NY, USA, Article 14, 11 pages. https://doi.org/10.1145/3485279.3485286
  23. Glanceable AR: Evaluating Information Access Methods for Head-Worn Augmented Reality. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 930–939. https://doi.org/10.1109/VR46266.2020.00113
  24. Feiyu Lu and Yan Xu. 2022. Exploring Spatial UI Transition Mechanisms with Head-Worn Augmented Reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 550, 16 pages. https://doi.org/10.1145/3491102.3517723
  25. Exploring Gaze for Assisting Freehand Selection-Based Text Entry in AR. Proc. ACM Hum.-Comput. Interact. 6, ETRA, Article 141 (may 2022), 16 pages. https://doi.org/10.1145/3530882
  26. Gaze-Hand Alignment: Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality. Proceedings of the ACM on Human-Computer Interaction 6, ETRA (2022), 1–18.
  27. Daniel P. Mapes and J. Michael Moshell. 1995. A Two-Handed Interface for Object Manipulation in Virtual Environments. Presence: Teleoperators and Virtual Environments 4, 4 (11 1995), 403–416. https://doi.org/10.1162/pres.1995.4.4.403
  28. Navigating on Handheld Displays: Dynamic versus Static Peephole Navigation. ACM Trans. Comput.-Hum. Interact. 13, 4 (dec 2006), 448–457. https://doi.org/10.1145/1188816.1188818
  29. Microsoft. 2022a. Eye-supported navigation in MRTK. Website. Retrieved April 2, 2023 from https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/features/input/eye-tracking/eye-tracking-navigation?view=mrtkunity-2022-05#gaze-supported-pan-and-zoom
  30. Microsoft. 2022b. Hand menu. Webpage. Retrieved September 7, 2022 from https://docs.microsoft.com/en-us/windows/mixed-reality/design/hand-menu
  31. Microsoft. 2023. Gaze and commit. Website. Retrieved April 2, 2023 from https://learn.microsoft.com/en-us/windows/mixed-reality/design/gaze-and-commit
  32. Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co., USA, 19–26. https://doi.org/10.1145/258734.258747
  33. Pinch, Click, or Dwell: Comparing Different Selection Techniques for Eye-Gaze-Based Pointing in Virtual Reality. Association for Computing Machinery, New York, NY, USA, Chapter 15, 7. https://doi.org/10.1145/3448018.3457998
  34. Jan Obernolte. Master Thesis, Submitted 30-09-2022. Exploring Gaze and Gesture Interactions in Augmented Reality. Ludwig-Maximilians-Universität München, Munich, Germany.
  35. ARtention: A design space for gaze-adaptive user interfaces in augmented reality. Computers & Graphics 95 (2021), 1–12. https://doi.org/10.1016/j.cag.2021.01.001
  36. Gaze-Touch: Combining Gaze with Multi-Touch for Interaction on the Same Surface. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 509–518. https://doi.org/10.1145/2642918.2647397
  37. Partially-Indirect Bimanual Input with Gaze, Pen, and Touch for Pan, Zoom, and Ink Interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 2845–2856. https://doi.org/10.1145/2858036.2858201
  38. Ken Pfeuffer and Hans Gellersen. 2016. Gaze and Touch Interaction on Tablets. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for Computing Machinery, New York, NY, USA, 301–311. https://doi.org/10.1145/2984511.2984514
  39. Gaze + Pinch Interaction in Virtual Reality. In Proceedings of the 5th Symposium on Spatial User Interaction (Brighton, United Kingdom) (SUI ’17). Association for Computing Machinery, New York, NY, USA, 99–108. https://doi.org/10.1145/3131277.3132180
  40. Empirical Evaluation of Gaze-Enhanced Menus in Virtual Reality. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology (Virtual Event, Canada) (VRST ’20). Association for Computing Machinery, New York, NY, USA, Article 20, 11 pages. https://doi.org/10.1145/3385956.3418962
  41. Looking for info: Evaluation of gaze based information retrieval in augmented reality. In Human-Computer Interaction–INTERACT 2021: 18th IFIP TC 13 International Conference, Bari, Italy, August 30–September 3, 2021, Proceedings, Part I 18. Springer, 544–565. https://doi.org/10.1007/978-3-030-85623-6_32
  42. Jef Raskin. 2000. The humane interface: new directions for designing interactive systems. Addison-Wesley Professional.
  43. Look & Turn: One-Handed and Expressive Menu Interaction by Gaze and Arm Turns in VR. In 2022 Symposium on Eye Tracking Research and Applications (Seattle, WA, USA) (ETRA ’22). Association for Computing Machinery, New York, NY, USA, Article 66, 7 pages. https://doi.org/10.1145/3517031.3529233
  44. Squeezy-Feely: Investigating Lateral Thumb-Index Pinching as an Input Modality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 61, 15 pages. https://doi.org/10.1145/3491102.3501981
  45. The Prevention of Mode Errors Through Sensory Feedback. Human–Computer Interaction 7, 2 (1992), 141–164. https://doi.org/10.1207/s15327051hci0702_1
  46. Radi-Eye: Hands-Free Radial Interfaces for 3D Interaction Using Gaze-Activated Head-Crossing. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 740, 11 pages. https://doi.org/10.1145/3411764.3445697
  47. Sophie Stellmach and Raimund Dachselt. 2012. Investigating Gaze-Supported Multimodal Pan and Zoom. In Proceedings of the Symposium on Eye Tracking Research and Applications (Santa Barbara, California) (ETRA ’12). Association for Computing Machinery, New York, NY, USA, 357–360. https://doi.org/10.1145/2168556.2168636
  48. Virtual Reality on a WIM: Interactive Worlds in Miniature. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’95). ACM Press/Addison-Wesley Publishing Co., USA, 265–272. https://doi.org/10.1145/223904.223938
  49. Experimental Analysis of Barehand Mid-Air Mode-Switching Techniques in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300426
  50. Patricia Tegtmeier. 2018. A scoping review on smart mobile devices and physical strain. Work 59 (2018), 273–283. https://doi.org/10.3233/WOR-172678 2.
  51. A Fitts’ Law Study of Gaze-Hand Alignment for Selection in 3D User Interfaces. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, 15 pages. https://doi.org/10.1145/3544548.3581423
  52. DigiTouch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-mounted Displays. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 113 (Sept. 2017), 21 pages. https://doi.org/10.1145/3130978
  53. GazeDock: Gaze-Only Menu Selection in Virtual Reality using Auto-Triggering Peripheral Menu. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 832–842. https://doi.org/10.1109/VR51125.2022.00105
  54. Manual and Gaze Input Cascaded (MAGIC) Pointing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 246–253. https://doi.org/10.1145/302979.303053
  55. ActiTouch: Robust Touch Detection for On-Skin AR/VR Interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 1151–1159. https://doi.org/10.1145/3332165.3347869
Citations (13)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now