iCub Detecting Gazed Objects: A Pipeline Estimating Human Attention (2308.13318v2)
Abstract: This research report explores the role of eye gaze in human-robot interactions and proposes a learning system for detecting objects gazed at by humans using solely visual feedback. The system leverages face detection, human attention prediction, and online object detection, and it allows the robot to perceive and interpret human gaze accurately, paving the way for establishing joint attention with human partners. Additionally, a novel dataset collected with the humanoid robot iCub is introduced, comprising over 22,000 images from ten participants gazing at different annotated objects. This dataset serves as a benchmark for the field of human gaze estimation in table-top human-robot interaction (HRI) contexts. In this work, we use it to evaluate the performance of the proposed pipeline and examine the performance of each component. Furthermore, the developed system is deployed on the iCub, and a supplementary video showcases its functionality. The results demonstrate the potential of the proposed approach as a first step to enhance social awareness and responsiveness in social robotics, as well as improve assistance and support in collaborative scenarios, promoting efficient human-robot collaboration.
- Social perception from visual cues: role of the sts region. Trends in cognitive sciences 4, 267–278
- One eye is all you need: Lightweight ensembles for gaze estimation with single encoders. arXiv preprint arXiv:2211.11936
- Small talk with a robot? the impact of dialog content, talk initiative, and gaze behavior of a social robot on trust, acceptance, and proximity. International Journal of Social Robotics , 1–14
- The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR) (IEEE), 510–517
- Openpose: realtime multi-person 2d pose estimation using part affinity fields. IEEE transactions on pattern analysis and machine intelligence 43, 172–186
- Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7291–7299
- Fast object segmentation learning with kernel-based methods for robotics. In 2021 IEEE International Conference on Robotics and Automation (ICRA). 13581–13588. 10.1109/ICRA48506.2021.9561758
- Gaze estimation by exploring two-eye asymmetry. IEEE Transactions on Image Processing 29, 5259–5272
- Examining joint attention with the use of humanoid robots-a new approach to study fundamental mechanisms of social cognition. Psychonomic Bulletin & Review 27, 217–236
- Detecting attended visual targets in video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5396–5406
- Watch where you’re going! gaze and head orientation as predictors for social robot navigation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) (IEEE), 3553–3559
- Eye tracking for everyone. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2176–2184
- Using human eye gaze patterns as indicators of need for assistance from a socially assistive robot. In Social Robotics: 11th International Conference, ICSR 2019, Madrid, Spain, November 26–29, 2019, Proceedings 11 (Springer), 200–210
- Toward an attentive robotic architecture: Learning-based mutual gaze estimation in human–robot interaction. Frontiers in Robotics and AI 9, 770165
- icub knows where you look: Exploiting social cues for interactive object detection learning. In 2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids). 480–487. 10.1109/Humanoids53995.2022.10000163
- Maiettini, E. (2020). From constraints to opportunities: Efficient object detection learning for humanoid robots. Ph.D. thesis, University of Genoa
- Interactive data collection for deep learning object detectors on humanoid robots. In 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids). 862–868. 10.1109/HUMANOIDS.2017.8246973
- On-line object detection: a robotics challenge. Autonomous Robots 10.1007/s10514-019-09894-9
- A weakly supervised strategy for learning object detection on a humanoid robot. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids). 194–201. 10.1109/Humanoids43949.2019.9035067
- Weakly-supervised object detection learning through human-robot interaction. In 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids). 392–399. 10.1109/HUMANOIDS47582.2021.9555781
- Yarp: Yet another robot platform. International Journal of Advanced Robotic Systems 3, 8. 10.5772/5761
- The icub humanoid robot: an open-systems platform for research in cognitive development. Neural networks : the official journal of the International Neural Network Society 23, 1125–34. 10.1016/j.neunet.2010.08.010
- Brain activation evoked by perception of gaze shifts: the influence of context. Neuropsychologia 41, 156–170
- Following gaze in video. In Proceedings of the IEEE International Conference on Computer Vision. 1435–1443
- Human gaze following for human-robot interaction. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE), 8615–8621
- Attention flow: End-to-end joint attention estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3327–3336