Emerging Natural User Interfaces in Mobile Computing: A Bottoms-Up Survey (1911.04794v2)

Abstract: Mobile and wearable interfaces and interaction paradigms are highly constrained by the available screen real estate, and the computational and power resources. Although there exist many ways of displaying information to mobile users, inputting data to a mobile device is, usually, limited to a conventional touch based interaction, that distracts users from their ongoing activities. Furthermore, emerging applications, like augmented, mixed and virtual reality (AR/MR/VR), require new types of input methods in order to interact with complex virtual worlds, challenging the traditional techniques of Human-Computer Interaction (HCI). Leveraging of Natural User Interfaces (NUIs), as a paradigm of using natural intuitive actions to interact with computing systems, is one of many ways to meet these challenges in mobile computing and its modern applications. Brain-Machine Interfaces that enable thought-only hands-free interaction, Myoelectric input methods that track body gestures and gaze-tracking input interfaces - are the examples of NUIs applicable to mobile and wearable interactions. The wide adoption of wearable devices and the penetration of mobile technologies, alongside with the growing market of AR/MR/VR, motivates the exploration and implementation of new interaction paradigms. The concurrent development of bio-signal acquisition techniques and accompanying ecosystems offers a useful toolbox to address open challenges. In this survey, we present state-of-the-art bio-signal acquisition methods, summarize and evaluate recent developments in the area of NUIs and outline potential application in mobile scenarios. The survey will provide a bottoms-up overview starting from (i) underlying biological aspects and signal acquisition techniques, (ii) portable NUI hardware solutions, (iii) NUI-enabled applications, as well as (iv) research challenges and open problems.

• The paper surveys emerging Natural User Interfaces (NUIs) in mobile and wearable computing, highlighting challenges with traditional inputs and the growing need for intuitive interaction in AR/MR/VR contexts.
• It explores various NUI modalities like speech, gesture, gaze, and Brain-Computer Interfaces, detailing underlying biological signals (EEG, EOG, EMG) and their acquisition techniques.
• The survey discusses portable hardware solutions, practical applications in assistive technology, industry, and security, and identifies key open problems for future research, including signal fidelity and privacy.

The paper "Emerging Natural User Interfaces in Mobile Computing: A Bottoms-Up Survey" provides a thorough examination of Natural User Interfaces (NUIs) within the context of mobile computing and wearable technology. The authors explore the constraints of current input methods in mobile and wearable devices, emphasizing the need for new interaction paradigms, particularly with the increasing prevalence of applications in augmented, mixed, and virtual reality (AR/MR/VR).

Key Areas Discussed

Motivation and Challenges

The paper identifies several challenges in current mobile interactions, such as limited screen real estate and input bandwidth. Traditional touch-based inputs are often disruptive to users' ongoing activities and ill-suited for complex virtual environments, especially in AR/MR/VR contexts. The authors argue that leveraging NUIs can address these limitations by using intuitive, natural actions for user interaction.

Biological Foundations and Signal Acquisition

The biology underlying various NUI modalities includes electrical neural activities, muscle signals (myoelectricity), and ocular movements. These are captured via different signal acquisition techniques:

• Electroencephalography (EEG): Measures electrical activity in the brain with high temporal resolution.
• Electrooculography (EOG): Tracks eye movements by measuring the corneo-retinal standing potential.
• Electromyography (EMG): Quantifies muscular electric potentials, particularly useful in gesture recognition.

NUI Modality Types

The survey highlights several NUI modalities:

• Speech Interfaces: Includes silent speech interfaces to address privacy and usability concerns.
• Gesture Inputs: Utilizes motion-based inputs, although mobile applicability is fraught with interference challenges.
• Gaze Interaction: Relies on EOG and camera-based systems to track eye movements for input.
• Brain-Computer Interfaces (BCIs): Provides direct neural input, categorized into attention-based systems like Steady State Visually Evoked Potentials (SSVEPs) and endogenous systems like Motor Imagery (MI).

Hardware Solutions and Challenges

The paper explores portable NUI hardware, providing an in-depth overview of commercial and open-source devices for EEG, fNIRS, and EMG acquisitions. It discusses the limitations such as signal noise, spatial-temporal resolutions, and the ergonomic challenges of combining these devices with mobile outputs like AR headsets.

Applications and Use Cases

1. Assistive Technology: BCIs and EMG NUIs facilitate communication for individuals with disabilities.
2. Hybrid Interfaces: Combined modalities address limitations, augment usability, and increase Information Transfer Rates (ITRs).
3. Silent Speech Interfaces: Promises robust communication in noisy environments without audible speech.
4. Biometrics and Security: NUIs offer novel secure biometric authentication methods.
5. Industrial and Commercial Applications: Enhances hands-free operations in maintenance, training, and e-commerce.

Future Directions and Open Problems

The paper addresses several open research problems, including improving signal fidelity and classification accuracy, overcoming the intrinsic "Midas touch" problem of unintentional inputs, and addressing ethical concerns like privacy in bio-signal data usage. Future work is suggested in redefining user interfaces and leveraging deep learning for bio-signal classification to enhance user experiences and content presentation, particularly in immersive environments like AR/MR/VR.

Through comprehensive coverage of the current state and future potential, this survey serves as a valuable resource for researchers exploring the development and application of NUIs in mobile interactions.