Objestures: Everyday Objects Meet Mid-Air Gestures for Expressive Interaction

Abstract: Everyday objects and mid-air gestures have been explored as input modalities, but each has its strengths and limitations - for example, objects offer tangibility but rely on their physical presence; gestures are convenient but lack haptic feedback. We introduce Objestures ("Obj" + "Gestures"), five interaction types that utilize both modalities for a design space of expressive and playful interaction. To evaluate its usefulness, we conducted a user study (N = 12) assessing whether it can effectively support basic 3D tasks such as rotation and scaling and found it has performance comparable to or better than the headset's native freehand manipulation. To understand its user experience, we conducted case studies on three example applications - Sound, Draw, and Shadow - with the same participants, who found it intuitive, engaging, and expressive, and were interested in its everyday use. We further illustrate 30 examples to showcase how Objestures can enrich everyday interactions and discuss its limitations and implications. https://www.zhuoyuelyu.com/objestures