Appearance-based gaze estimation enhanced with synthetic images using deep neural networks (2311.14175v2)
Abstract: Human eye gaze estimation is an important cognitive ingredient for successful human-robot interaction, enabling the robot to read and predict human behavior. We approach this problem using artificial neural networks and build a modular system estimating gaze from separately cropped eyes, taking advantage of existing well-functioning components for face detection (RetinaFace) and head pose estimation (6DRepNet). Our proposed method does not require any special hardware or infrared filters but uses a standard notebook-builtin RGB camera, as often approached with appearance-based methods. Using the MetaHuman tool, we also generated a large synthetic dataset of more than 57,000 human faces and made it publicly available. The inclusion of this dataset (with eye gaze and head pose information) on top of the standard Columbia Gaze dataset into training the model led to better accuracy with a mean average error below two degrees in eye pitch and yaw directions, which compares favourably to related methods. We also verified the feasibility of our model by its preliminary testing in real-world setting using the builtin 4K camera in NICO semi-humanoid robot's eye.
- H. Su, W. Qi, J. Chen, C. Yang, J. Sandoval, and M. Laribi, “Recent advancements in multimodal human–robot interaction,” Frontiers in Neurorobotics, 2023.
- A. Thomaz, G. Hoffman, and M. Cakmak, “Computational human-robot interaction,” New Foundations and Trends, vol. 4, no. 2-3, p. 105–223, 2013.
- A. Sciutti, M. Mara, V. Tagliasco, and G. Sandini, “Humanizing human-robot interaction: On the importance of mutual understanding,” IEEE Technology and Society Magazine, vol. 37, no. 1, pp. 22–29, 2018.
- H. Admoni and B. Scassellati, “Social eye gaze in human-robot interaction: A review,” Journal of Human-Robot Interaction, vol. 6, no. 1, p. 25–63, 2017. [Online]. Available: https://doi.org/10.5898/JHRI.6.1.Admoni
- O. Palinko, F. Rea, G. Sandini, and A. Sciutti, “Robot reading human gaze: Why eye tracking is better than head tracking for human-robot collaboration,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, pp. 5048–5054.
- A. Kar and P. Corcoran, “A review and analysis of eye-gaze estimation systems, algorithms and performance evaluation methods in consumer platforms,” IEEE Access, vol. 5, pp. 16 495–16 519, 2017.
- J. Oh, Y. Lee, J. Yoo, and S. Kwon, “Improved feature-based gaze estimation using self-attention module and synthetic eye images,” Sensors, vol. 22, no. 11, 2022. [Online]. Available: https://www.mdpi.com/1424-8220/22/11/4026
- S. Park, A. Spurr, and O. Hilliges, “Deep pictorial gaze estimation,” in European Conference on Computer Vision, 2018, pp. 741–757.
- X. Zhang, Y. Sugano, M. Fritz, and A. Bulling, “Appearance-based gaze estimation in the wild,” in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, pp. 4511–4520.
- B. Smith, Q. Yin, S. Feiner, and S. Nayar, “Gaze locking: Passive eye contact detection for human-object interaction,” in ACM Symposium on User Interface Software and Technology, 2013.
- E. Wood, T. Baltrušaitis, L.-P. Morency, P. Robinson, and A. Bulling, “Learning an appearance-based gaze estimator from one million synthesised images,” in Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research and Applications, 2016, pp. 131–138.
- “MetaHuman high-fidelity digital humans made easy,” https://www.unrealengine.com/en-US/metahuman, 2021.
- J. Deng et al., “RetinaFace: Single-stage dense face localisation in the wild,” in arXiv:1905.00641v2, 2019.
- “Face detection,” https://github.com/elliottzheng/face-detection.
- T. Hempel, A. A. Abdelrahman, and A. Al-Hamadi, “6D rotation representation for unconstrained head pose estimation,” in 2022 IEEE International Conference on Image Processing, 2022.
- D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in arXiv.1412.6980, 2017.
- M. Kerzel et al., “NICO – Neuro-Inspired COmpanion: A developmental humanoid robot platform for multimodal interaction,” in IEEE International Symposium on Robot and Human Interactive Communication, 2017.
- Dmytro Herashchenko (1 paper)
- Igor Farkaš (7 papers)