2000 character limit reached
A Study on Domain Generalization for Failure Detection through Human Reactions in HRI (2403.06315v1)
Published 10 Mar 2024 in cs.RO, cs.HC, and cs.LG
Abstract: Machine learning models are commonly tested in-distribution (same dataset); performance almost always drops in out-of-distribution settings. For HRI research, the goal is often to develop generalized models. This makes domain generalization - retaining performance in different settings - a critical issue. In this study, we present a concise analysis of domain generalization in failure detection models trained on human facial expressions. Using two distinct datasets of humans reacting to videos where error occurs, one from a controlled lab setting and another collected online, we trained deep learning models on each dataset. When testing these models on the alternate dataset, we observed a significant performance drop. We reflect on the causes for the observed model behavior and leave recommendations. This work emphasizes the need for HRI research focusing on improving model robustness and real-life applicability.
- Reuben M. Aronson. 2018. Gaze for Error Detection During Human-Robot Shared Manipulation. In RSS Workshop: Towards a Framework for Joint Action.
- The Bystander Affect Detection (BAD) Dataset for Failure Detection in HRI. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 11443–11450. https://doi.org/10.1109/IROS55552.2023.10342442
- Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37–46. https://doi.org/10.1177/001316446002000104 arXiv:https://doi.org/10.1177/001316446002000104
- Chapter 19 - Affective facial computing: Generalizability across domains. In Multimodal Behavior Analysis in the Wild, Xavier Alameda-Pineda, Elisa Ricci, and Nicu Sebe (Eds.). Academic Press, 407–441. https://doi.org/10.1016/B978-0-12-814601-9.00026-2
- ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248–255. https://doi.org/10.1109/CVPR.2009.5206848
- Cross-Domain Facial Expression Recognition through Reliable Globa&Local Representation Learning and Dynamic Label Weighting. Electronics 12, 21 (2023). https://doi.org/10.3390/electronics12214553
- Systematic analysis of video data from different human–robot interaction studies: a categorization of social signals during error situations. Frontiers in psychology 6 (2015), 931.
- Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385 (2015).
- Reward shaping for reinforcement learning by emotion expressions. In 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, IEEE, New York, NY, USA, 1288–1293.
- Behavioural responses to robot conversational failures. In 2020 15th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, IEEE, New York, NY, USA, 53–62.
- A systematic cross-corpus analysis of human reactions to robot conversational failures. In Proceedings of the 2021 International Conference on Multimodal Interaction. 112–120.
- Human Action Recognition for Social Robots. In 2019 22nd International Conference on Control Systems and Computer Science (CSCS). 675–681. https://doi.org/10.1109/CSCS.2019.00121
- Toward Human-Robot Cooperation: Unsupervised Domain Adaptation for Egocentric Action Recognition. In Human-Friendly Robotics 2022, Pablo Borja, Cosimo Della Santina, Luka Peternel, and Elena Torta (Eds.). Springer International Publishing, Cham, 218–232.
- Maia Stiber. 2022. Effective Human-Robot Collaboration via Generalized Robot Error Management Using Natural Human Responses. In Proceedings of the 2022 International Conference on Multimodal Interaction (Bengaluru, India) (ICMI ’22). Association for Computing Machinery, New York, NY, USA, 673–678. https://doi.org/10.1145/3536221.3557028
- Maia Stiber and Chien-Ming Huang. 2021. Not All Errors Are Created Equal: Exploring Human Responses to Robot Errors with Varying Severity. In Companion Publication of the 2020 International Conference on Multimodal Interaction (Virtual Event, Netherlands) (ICMI ’20 Companion). Association for Computing Machinery, New York, NY, USA, 97–101. https://doi.org/10.1145/3395035.3425245
- On Using Social Signals to Enable Flexible Error-Aware HRI. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction (Stockholm, Sweden) (HRI ’23). Association for Computing Machinery, New York, NY, USA, 222–230. https://doi.org/10.1145/3568162.3576990
- Critically examining the Domain Generalizability of Facial Expression Recognition models. arXiv:2106.15453 [cs.CV]
- Head and shoulders: Automatic error detection in human-robot interaction. Proceedings of International Conference on Multimodal Interaction (2017).
- Recent Advances in Transfer Learning for Cross-Dataset Visual Recognition: A Problem-Oriented Perspective. ACM Comput. Surv. 52, 1, Article 7 (feb 2019), 38 pages. https://doi.org/10.1145/3291124