---
title: ERR@HRI 2.0 Challenge
url: https://www.emergentmind.com/topics/err-hri-2-0-challenge
type: topic
---

# ERR@HRI 2.0 Challenge

Searching arXiv for the ERR@HRI 2.0 challenge paper and closely related benchmark papers to ground the article in the cited literature.
The ERR@HRI 2.0 Challenge is a benchmark challenge for multimodal detection of failures in human-robot conversations involving LLM-powered robots. It defines conversational failure as a temporally localized phenomenon that can be described from both a system-design perspective and a user-perception perspective, and it operationalizes that problem on a privacy-preserving dataset of dyadic interactions represented through facial, head-movement, audio, and transcribed-speech features. The challenge was organized in conjunction with ACM Multimedia 2025 and frames failure detection as a prerequisite for downstream mitigation of conversational breakdown, task disruption, and loss of user trust in embodied conversational systems [2507.13468].

## 1. Conceptual framing and scope

ERR@HRI 2.0 is centered on the claim that conversational robots remain failure-prone even when powered by large language models. The challenge paper gives concrete examples of such failures: misunderstanding user intent, interrupting users prematurely, not responding to the user, responding with an error message, or producing an inappropriate utterance for what the user has just said. Rather than treating failure as a single undifferentiated label, the benchmark distinguishes two perspectives. A **robot error from the system perspective** is defined as a noticeable misalignment between robot behavior and its expected behavior by system design. A **robot error from the user perspective** is operationalized as observable user behavior signaling an intention to correct a mismatch between robot behavior and user expectation, especially user-initiated disruptive interruption [2507.13468].

This dual perspective is the benchmark’s main conceptual feature. The two labels are explicitly described as overlapping but non-identical: a system-design deviation may not be perceived as an error by the user, while behavior that is valid by system design may still violate the user’s mental model. Accordingly, the challenge defines two sub-challenges: detection of robot error from the system’s perspective, and detection of robot error from the user’s perspective. The formulation is temporal and segment-level rather than turn-level; predictions are aligned against annotated time intervals rather than single utterance labels.

The benchmark therefore situates failure detection as an interaction-analysis problem rather than a purely internal fault-monitoring problem. Social signal analysis is central because user reactions may reveal expectation mismatch, attempted repair, or conversational breakdown earlier than explicit reporting.

## 2. Dataset composition and multimodal representation

The ERR@HRI 2.0 dataset contains approximately 16 hours of dyadic human-robot conversational interaction, reported more precisely as **15 hours, 50 minutes, 48 seconds**, with **101 sessions** from **42 users**, evenly split into **21 female** and **21 male** participants. The data combine two prior studies: a human-social robot conversation study and a human-voice assistant conversation study. Two embodiments are included: a social robot and a smart-speaker-style voice assistant. Across these embodiments, the dataset spans five task settings: social-robot survival, social-robot discussion, voice-assistant medical self-diagnosis, voice-assistant trip planning, and voice-assistant discussion [2507.13468].

At corpus level, Table 1 in the challenge paper reports **414 system-perspective robot errors** and **224 user-perspective robot errors**. The released data are not raw audio or video. Owing to privacy constraints, participants receive only derived multimodal behavioral features. The official feature release is stated as **581 multimodal features**, distributed across facial/head-pose, audio, and transcribed-speech streams.

The facial and head-pose stream is extracted with **OpenFace 2.2.0** and contains **43 features per frame** at **30 fps**, including **18 binary facial action unit presence features**, **17 facial action unit intensity features**, **head location** \((x,y,z)\), **head rotation** \((x,y,z)\), and OpenFace **confidence** and **success** outputs. The audio stream is extracted with **openSMILE** using **eGeMAPSv02**, with **25 features**, **20 ms** frame length, and **10 ms** hop. The transcribed-speech stream is based on **Google’s speaker diarization** and includes speaking indicators for user and robot, word count, speech-to-text confidence, and a **512-dimensional embedding** of transcribed speech using a **CLIP ViT-B/16 Transformer architecture**, for a total of **515 transcribed speech features**.

The manuscript notes a minor dimensional inconsistency: \(43 + 25 + 515 = 583\), whereas the official total is stated as **581**. A similar reporting inconsistency appears in the duration of the social-robot discussion subset. These discrepancies do not alter the benchmark design, but they matter for exact archival description.

## 3. Annotation schema and official task formulation

Annotations were created by **two coders** using the **Datavyu** video annotation tool. The coders first familiarized themselves with the system design and expected robot behavior, then annotated three temporal phenomena: the start and end time of observable robot behavior deviations from system design, the start and end time of observable user reaction to system-perspective robot errors, and the start and end time of observable user verbal or non-verbal disruptive interruption intended to correct a mismatch between robot behavior and user expectation [2507.13468].

The challenge therefore contains three annotation layers, although only two are primary prediction targets. The first target is **robot error from the system perspective**. The second is **robot error from the user perspective**. The intermediate **reaction to robot error** label is auxiliary and is explicitly included to facilitate training, especially for system-side error detection through user reaction cues.

The annotation process involved independent coding of **10 sessions**, specifically **2 sessions from each task**, followed by conflict discussion and resolution. The paper reports **100% agreement** after discussion, but it does not report a standard inter-annotator coefficient such as Cohen’s \(\kappa\) or Krippendorff’s \(\alpha\).

The official task constraints are summarized below.

| Sub-challenge | Target label | Maximum window size |
|---|---|---:|
| 1 | Robot error from the system perspective | 12 s |
| 2 | Robot error from the user perspective | 5 s |

The problem is binary temporal detection in both offline and on-the-fly settings. In the baseline preprocessing, continuous streams are segmented into **3-second** windows with **0.5-second** step size, and a window is labeled positive if it overlaps an annotated error interval. After such windowing, positive prevalence is highly imbalanced: only **11.35%** of windows are positive for sub-challenge 1, and only **3.33%** for sub-challenge 2.

## 4. Evaluation protocol and organizers’ baseline

ERR@HRI 2.0 supports two evaluation modes. The first is **offline window-based classification**, scored with **AUC**, **F1 score**, and **Accuracy**. The second is **on-the-fly / streaming detection**, in which predicted windows are matched against ground-truth intervals by temporal overlap. To accommodate annotation uncertainty, the organizers add **1 second tolerance** to the start and end labels. A predicted error is treated as correctly detected if the predicted window overlaps the tolerated ground-truth interval by any amount. The streaming evaluation reports **percentage detected**, **number of false positives**, and **overall F1**, and teams are ranked **only by overall F1** from the on-the-fly evaluation [2507.13468].

The organizers explored **Random Forest (RF)**, **Explainable Boosting Machine (EBM)**, **XGBoost**, **LSTM**, and **single-layer Transformers**. Among these, **Random Forest** performed best in their exploration and was adopted as the official baseline. The baseline uses **mean imputation** for missing features and **SMOTE** to address class imbalance. Hyperparameter tuning uses **leave-one-out cross-validation** with **AUC** as the validation criterion. The paper does not report the final tuned numeric hyperparameters, but it does report threshold adjustment during the test-set feedback process: **0.55** for sub-challenge 1 and **0.45** for sub-challenge 2.

The official baseline results show a clear gap between offline discrimination and streaming utility. For window-based evaluation, sub-challenge 1 reaches **AUC 0.59**, **F1 0.49**, **Accuracy 0.83**; sub-challenge 2 reaches **AUC 0.70**, **F1 0.51**, **Accuracy 0.97**. For on-the-fly evaluation, sub-challenge 1 achieves **39.54% detected**, **163 false positives**, and **overall F1 0.24**; sub-challenge 2 achieves **29.03% detected**, **97 false positives**, and **overall F1 0.13**. The manuscript contains a prose-table inconsistency for sub-challenge 2, where one passage states **0.29** instead of the table’s **0.13**. The tabulated value is the more internally structured report.

These results indicate that the principal benchmark difficulty is not merely binary separability on fixed windows, but stable real-time detection under severe imbalance and false-positive constraints.

## 5. Position within the ERR@HRI benchmark lineage

ERR@HRI 2.0 extends the earlier **ERR@HRI 2024 Challenge**, which was organized with ACM ICMI 2024 and focused on multimodal detection of **Robot Mistake (RM)**, **User Awkwardness (UA)**, and **Interaction Rupture (IR)** in interactions with a robotic positive psychology coach. That earlier benchmark used processed face, audio, and pose features from **23 users**, **89 sessions**, and about **700 minutes** of interaction, and its organizers’ recurrent baselines reached test-set F1 scores of **0.54184** for RM, **0.56698** for UA, and **0.41964** for IR [2407.06094]. ERR@HRI 2.0 retains the multimodal, temporally annotated, privacy-preserving benchmark logic, but shifts the focus to **LLM-powered conversational robots**, **dyadic conversation**, and the distinction between **system-perspective** and **user-perspective** error.

The challenge also differs from HRI datasets whose main difficulty is acoustic degradation rather than conversational failure. The **HRRE** corpus isolates reverberation time and source-microphone distance in far-field ASR evaluation, while **MChRSR** targets robot noise, background noise, and time-varying acoustic channels caused by PR2 motion and head rotation [1801.09651], [1801.00061]. ERR@HRI 2.0 instead targets multimodal social evidence of conversational mismatch, not speech-recognition robustness as such.

A plausible interpretation is that the benchmark line is moving toward a broader failure lifecycle. **REPAIR-Bench** expands the problem from binary failure detection toward longitudinal detection over inter-dependent sessions, failure-type classification, and user-centered recovery prediction, reporting, among other results, **strict F1 0.80 vs. 0.68** for hierarchical versus single-session failure detection and **Hit@5 = 0.76**, **F1@5 = 0.32** for recovery prediction [2606.29937]. This suggests a methodological progression from rupture detection, to dual-perspective conversational failure detection, to detect-classify-recover formulations.

## 6. Significance, limitations, and methodological implications

ERR@HRI 2.0 is significant as a benchmark because it formalizes conversational robot failure as a multimodal temporal-detection problem and makes that problem comparable across teams through shared features, shared splits, and a common evaluation script. The challenge was organized with a dedicated repository and submission protocol; **nine teams registered**, **two teams submitted models to each sub-challenge**, and the two submitted systems in each sub-challenge **surpassed baseline performance** [2507.13468].

Its design also exposes several limitations. First, the labels are explicitly operationalized and do **not capture all system-perspective errors** or **all user-perspective errors**. User-perspective error is inferred from observable corrective behavior rather than direct access to user state, so silent frustration or unexpressed confusion may be missed. Second, the release is feature-based rather than raw-media-based, which protects privacy but prevents re-extraction with alternative foundation models and limits end-to-end multimodal learning. Third, class imbalance is extreme, especially for the user-perspective task. Fourth, the paper leaves some procedural details underspecified, including a clear external-data policy and a detailed fixed validation partition. Fifth, the manuscript contains several minor reporting inconsistencies in durations, feature counts, and one baseline F1 value.

Methodologically, the challenge emphasizes that conversational failure is distributed across modalities and time scales. Facial action units, head pose, prosodic audio, diarization-derived turn structure, and transcript embeddings are all treated as potential carriers of expectation mismatch. The benchmark therefore favors models that can integrate asynchronous, privacy-preserving behavioral signals under low-latency constraints. At the same time, the modest baseline streaming performance shows that detection quality is constrained not only by representation learning but also by label sparsity, overlap-based evaluation, and the practical requirement to control false alarms.

In broader HRI terms, this kind of shared benchmark aligns with arguments that transferability in HRI depends on standardization, reusable methods, and open-science pathways rather than isolated prototypes alone [2401.05802]. ERR@HRI 2.0 can thus be read both as a challenge problem and as an attempt to stabilize a research object: conversational robot failure as an observable, annotatable, and benchmarkable multimodal phenomenon.

Source: https://www.emergentmind.com/topics/err-hri-2-0-challenge