ERR@HRI 2.0 Challenge

• ERR@HRI 2.0 is a benchmark challenge that defines conversational failures in LLM-powered human-robot interactions through dual system and user perspectives.
• The challenge utilizes a privacy-preserving dataset with multimodal features from facial, audio, and transcribed-speech streams across 101 dyadic sessions.
• Methodological insights include temporal segmentation, dual-perspective annotation, and evaluation under severe class imbalance for real-time failure detection.

Searching arXiv for the ERR@HRI 2.0 challenge paper and closely related benchmark papers to ground the article in the cited literature. The ERR@HRI 2.0 Challenge is a benchmark challenge for multimodal detection of failures in human-robot conversations involving LLM-powered robots. It defines conversational failure as a temporally localized phenomenon that can be described from both a system-design perspective and a user-perception perspective, and it operationalizes that problem on a privacy-preserving dataset of dyadic interactions represented through facial, head-movement, audio, and transcribed-speech features. The challenge was organized in conjunction with ACM Multimedia 2025 and frames failure detection as a prerequisite for downstream mitigation of conversational breakdown, task disruption, and loss of user trust in embodied conversational systems (Cao et al., 17 Jul 2025).

1. Conceptual framing and scope

ERR@HRI 2.0 is centered on the claim that conversational robots remain failure-prone even when powered by LLMs. The challenge paper gives concrete examples of such failures: misunderstanding user intent, interrupting users prematurely, not responding to the user, responding with an error message, or producing an inappropriate utterance for what the user has just said. Rather than treating failure as a single undifferentiated label, the benchmark distinguishes two perspectives. A robot error from the system perspective is defined as a noticeable misalignment between robot behavior and its expected behavior by system design. A robot error from the user perspective is operationalized as observable user behavior signaling an intention to correct a mismatch between robot behavior and user expectation, especially user-initiated disruptive interruption (Cao et al., 17 Jul 2025).

This dual perspective is the benchmark’s main conceptual feature. The two labels are explicitly described as overlapping but non-identical: a system-design deviation may not be perceived as an error by the user, while behavior that is valid by system design may still violate the user’s mental model. Accordingly, the challenge defines two sub-challenges: detection of robot error from the system’s perspective, and detection of robot error from the user’s perspective. The formulation is temporal and segment-level rather than turn-level; predictions are aligned against annotated time intervals rather than single utterance labels.

The benchmark therefore situates failure detection as an interaction-analysis problem rather than a purely internal fault-monitoring problem. Social signal analysis is central because user reactions may reveal expectation mismatch, attempted repair, or conversational breakdown earlier than explicit reporting.

2. Dataset composition and multimodal representation

The ERR@HRI 2.0 dataset contains approximately 16 hours of dyadic human-robot conversational interaction, reported more precisely as 15 hours, 50 minutes, 48 seconds, with 101 sessions from 42 users, evenly split into 21 female and 21 male participants. The data combine two prior studies: a human-social robot conversation study and a human-voice assistant conversation study. Two embodiments are included: a social robot and a smart-speaker-style voice assistant. Across these embodiments, the dataset spans five task settings: social-robot survival, social-robot discussion, voice-assistant medical self-diagnosis, voice-assistant trip planning, and voice-assistant discussion (Cao et al., 17 Jul 2025).

At corpus level, Table 1 in the challenge paper reports 414 system-perspective robot errors and 224 user-perspective robot errors. The released data are not raw audio or video. Owing to privacy constraints, participants receive only derived multimodal behavioral features. The official feature release is stated as 581 multimodal features, distributed across facial/head-pose, audio, and transcribed-speech streams.

The facial and head-pose stream is extracted with OpenFace 2.2.0 and contains 43 features per frame at 30 fps, including 18 binary facial action unit presence features, 17 facial action unit intensity features, head location $(x,y,z)$, head rotation $(x,y,z)$, and OpenFace confidence and success outputs. The audio stream is extracted with openSMILE using eGeMAPSv02, with 25 features, 20 ms frame length, and 10 ms hop. The transcribed-speech stream is based on Google’s speaker diarization and includes speaking indicators for user and robot, word count, speech-to-text confidence, and a 512-dimensional embedding of transcribed speech using a CLIP ViT-B/16 Transformer architecture, for a total of 515 transcribed speech features.

The manuscript notes a minor dimensional inconsistency: $43 + 25 + 515 = 583$, whereas the official total is stated as 581. A similar reporting inconsistency appears in the duration of the social-robot discussion subset. These discrepancies do not alter the benchmark design, but they matter for exact archival description.

3. Annotation schema and official task formulation

Annotations were created by two coders using the Datavyu video annotation tool. The coders first familiarized themselves with the system design and expected robot behavior, then annotated three temporal phenomena: the start and end time of observable robot behavior deviations from system design, the start and end time of observable user reaction to system-perspective robot errors, and the start and end time of observable user verbal or non-verbal disruptive interruption intended to correct a mismatch between robot behavior and user expectation (Cao et al., 17 Jul 2025).

The challenge therefore contains three annotation layers, although only two are primary prediction targets. The first target is robot error from the system perspective. The second is robot error from the user perspective. The intermediate reaction to robot error label is auxiliary and is explicitly included to facilitate training, especially for system-side error detection through user reaction cues.

The annotation process involved independent coding of 10 sessions, specifically 2 sessions from each task, followed by conflict discussion and resolution. The paper reports 100% agreement after discussion, but it does not report a standard inter-annotator coefficient such as Cohen’s $\kappa$ or Krippendorff’s $\alpha$.

The official task constraints are summarized below.

Sub-challenge	Target label	Maximum window size
1	Robot error from the system perspective	12 s
2	Robot error from the user perspective	5 s

The problem is binary temporal detection in both offline and on-the-fly settings. In the baseline preprocessing, continuous streams are segmented into 3-second windows with 0.5-second step size, and a window is labeled positive if it overlaps an annotated error interval. After such windowing, positive prevalence is highly imbalanced: only 11.35% of windows are positive for sub-challenge 1, and only 3.33% for sub-challenge 2.

4. Evaluation protocol and organizers’ baseline

ERR@HRI 2.0 supports two evaluation modes. The first is offline window-based classification, scored with AUC, F1 score, and Accuracy. The second is on-the-fly / streaming detection, in which predicted windows are matched against ground-truth intervals by temporal overlap. To accommodate annotation uncertainty, the organizers add 1 second tolerance to the start and end labels. A predicted error is treated as correctly detected if the predicted window overlaps the tolerated ground-truth interval by any amount. The streaming evaluation reports percentage detected, number of false positives, and overall F1, and teams are ranked only by overall F1 from the on-the-fly evaluation (Cao et al., 17 Jul 2025).

The organizers explored Random Forest (RF), Explainable Boosting Machine (EBM), XGBoost, LSTM, and single-layer Transformers. Among these, Random Forest performed best in their exploration and was adopted as the official baseline. The baseline uses mean imputation for missing features and SMOTE to address class imbalance. Hyperparameter tuning uses leave-one-out cross-validation with AUC as the validation criterion. The paper does not report the final tuned numeric hyperparameters, but it does report threshold adjustment during the test-set feedback process: 0.55 for sub-challenge 1 and 0.45 for sub-challenge 2.

The official baseline results show a clear gap between offline discrimination and streaming utility. For window-based evaluation, sub-challenge 1 reaches AUC 0.59, F1 0.49, Accuracy 0.83; sub-challenge 2 reaches AUC 0.70, F1 0.51, Accuracy 0.97. For on-the-fly evaluation, sub-challenge 1 achieves 39.54% detected, 163 false positives, and overall F1 0.24; sub-challenge 2 achieves 29.03% detected, 97 false positives, and overall F1 0.13. The manuscript contains a prose-table inconsistency for sub-challenge 2, where one passage states 0.29 instead of the table’s 0.13. The tabulated value is the more internally structured report.

These results indicate that the principal benchmark difficulty is not merely binary separability on fixed windows, but stable real-time detection under severe imbalance and false-positive constraints.

5. Position within the ERR@HRI benchmark lineage

ERR@HRI 2.0 extends the earlier ERR@HRI 2024 Challenge, which was organized with ACM ICMI 2024 and focused on multimodal detection of Robot Mistake (RM), User Awkwardness (UA), and Interaction Rupture (IR) in interactions with a robotic positive psychology coach. That earlier benchmark used processed face, audio, and pose features from 23 users, 89 sessions, and about 700 minutes of interaction, and its organizers’ recurrent baselines reached test-set F1 scores of 0.54184 for RM, 0.56698 for UA, and 0.41964 for IR (Spitale et al., 2024). ERR@HRI 2.0 retains the multimodal, temporally annotated, privacy-preserving benchmark logic, but shifts the focus to LLM-powered conversational robots, dyadic conversation, and the distinction between system-perspective and user-perspective error.

The challenge also differs from HRI datasets whose main difficulty is acoustic degradation rather than conversational failure. The HRRE corpus isolates reverberation time and source-microphone distance in far-field ASR evaluation, while MChRSR targets robot noise, background noise, and time-varying acoustic channels caused by PR2 motion and head rotation (Escudero et al., 2018, Novoa et al., 2017). ERR@HRI 2.0 instead targets multimodal social evidence of conversational mismatch, not speech-recognition robustness as such.

A plausible interpretation is that the benchmark line is moving toward a broader failure lifecycle. REPAIR-Bench expands the problem from binary failure detection toward longitudinal detection over inter-dependent sessions, failure-type classification, and user-centered recovery prediction, reporting, among other results, strict F1 0.80 vs. 0.68 for hierarchical versus single-session failure detection and Hit@5 = 0.76, F1@5 = 0.32 for recovery prediction (Pioldi et al., 29 Jun 2026). This suggests a methodological progression from rupture detection, to dual-perspective conversational failure detection, to detect-classify-recover formulations.

6. Significance, limitations, and methodological implications

ERR@HRI 2.0 is significant as a benchmark because it formalizes conversational robot failure as a multimodal temporal-detection problem and makes that problem comparable across teams through shared features, shared splits, and a common evaluation script. The challenge was organized with a dedicated repository and submission protocol; nine teams registered, two teams submitted models to each sub-challenge, and the two submitted systems in each sub-challenge surpassed baseline performance (Cao et al., 17 Jul 2025).

Its design also exposes several limitations. First, the labels are explicitly operationalized and do not capture all system-perspective errors or all user-perspective errors. User-perspective error is inferred from observable corrective behavior rather than direct access to user state, so silent frustration or unexpressed confusion may be missed. Second, the release is feature-based rather than raw-media-based, which protects privacy but prevents re-extraction with alternative foundation models and limits end-to-end multimodal learning. Third, class imbalance is extreme, especially for the user-perspective task. Fourth, the paper leaves some procedural details underspecified, including a clear external-data policy and a detailed fixed validation partition. Fifth, the manuscript contains several minor reporting inconsistencies in durations, feature counts, and one baseline F1 value.

Methodologically, the challenge emphasizes that conversational failure is distributed across modalities and time scales. Facial action units, head pose, prosodic audio, diarization-derived turn structure, and transcript embeddings are all treated as potential carriers of expectation mismatch. The benchmark therefore favors models that can integrate asynchronous, privacy-preserving behavioral signals under low-latency constraints. At the same time, the modest baseline streaming performance shows that detection quality is constrained not only by representation learning but also by label sparsity, overlap-based evaluation, and the practical requirement to control false alarms.

In broader HRI terms, this kind of shared benchmark aligns with arguments that transferability in HRI depends on standardization, reusable methods, and open-science pathways rather than isolated prototypes alone (Johal, 2024). ERR@HRI 2.0 can thus be read both as a challenge problem and as an attempt to stabilize a research object: conversational robot failure as an observable, annotatable, and benchmarkable multimodal phenomenon.