A Multimodal Handover Failure Detection Dataset and Baselines

Abstract: An object handover between a robot and a human is a coordinated action which is prone to failure for reasons such as miscommunication, incorrect actions and unexpected object properties. Existing works on handover failure detection and prevention focus on preventing failures due to object slip or external disturbances. However, there is a lack of datasets and evaluation methods that consider unpreventable failures caused by the human participant. To address this deficit, we present the multimodal Handover Failure Detection dataset, which consists of failures induced by the human participant, such as ignoring the robot or not releasing the object. We also present two baseline methods for handover failure detection: (i) a video classification method using 3D CNNs and (ii) a temporal action segmentation approach which jointly classifies the human action, robot action and overall outcome of the action. The results show that video is an important modality, but using force-torque data and gripper position help improve failure detection and action segmentation accuracy.

• The paper introduces a novel multimodal dataset capturing human-induced handover failures using video, force-torque, and joint sensors.
• The paper presents baseline methods such as modified I3D for video classification and MSTCNs for temporal action segmentation to enhance detection accuracy.
• The paper demonstrates that integrating multimodal data, especially force-torque signals, significantly improves failure detection in robotic handovers.

A Multimodal Handover Failure Detection Dataset and Baselines

Introduction

The paper "A Multimodal Handover Failure Detection Dataset and Baselines" presents a dataset and evaluation framework focused on the detection of failures in robot-to-human (R2H) and human-to-robot (H2R) handovers. Unlike previous works concentrating on preventable failures such as object slips or disturbances, this research addresses failures induced by human participants, providing essential data for realistic benchmarking in human-robot interaction (HRI) scenarios.

Dataset Overview

The Handover Failure Detection (HFD) dataset is a novel contribution that includes various failure modes caused by the human participant, such as ignoring the robot or failing to release an object during handover. The dataset is composed of multimodal data, including video captured from robots, force-torque (F-T) readings, and robot joint states.

Figure 1: The dataset consists of video, force-torque readings and robot joint states, and contains annotations for the robot and human actions.

The data collection involved 17 participants interacting with two robotic platforms under controlled conditions to simulate diverse handover scenarios. Annotations include both successful and failed trials, categorized into different failure modes as depicted in Table 1 from the paper.

Baseline Methods

The paper introduces two baseline methods for detecting handover failures:

1. Video Classification: Leveraging 3D CNNs, specifically modified variants of the Inflated 3D ConvNet (I3D), to classify the outcome of handovers using different modalities such as RGB video, F-T, and gripper position.
2. Temporal Action Segmentation: Utilizing Multi-Stage Temporal Convolutional Networks (MS-TCNs) to predict human actions during the handover, allowing for the detection of unexpected or unperformed actions as indicators of failure.

   Figure 2: Illustration of network architectures for video classification and temporal action segmentation used in the study.

Experimental Results

The experiments demonstrate that incorporating multimodal data enhances failure detection accuracy, with F-T data proving particularly valuable. The I3D-D model, which performs intermediate fusion of modalities, achieved superior classification performance. Similarly, the MSTCN-A model outperformed in action segmentation, underscoring the importance of integrating visual and physical signals for accurate failure detection.

Implications and Future Work

This dataset and the accompanying baselines form a crucial foundation for advancing research in HRI safety and reliability. The inclusion of human-induced failure scenarios in benchmarking aligns closely with real-world applications, particularly in sensitive fields such as healthcare. Future research could explore real-time failure detection, enabling robots to react immediately and appropriately to failure conditions. The development of causal temporal prediction models would be an advantageous expansion, further increasing the applicability of these findings in dynamic, interactive environments.

Conclusion

The research provides a structured approach to handling the complexities of multimodal failure detection in robotic handovers. The HFD dataset enriches the resources available for evaluating robotic systems in realistic conditions, emphasizing the critical role of multimodal integration in failure detection. As robotics continues to evolve, methodologies developed through this work will likely inform safer and more effective human-robot collaborations.