Multimodal Rapport Estimation in Real-World HRI

Abstract: Evaluating interaction quality in real-world HRI is an important challenge. If interaction quality can be estimated reliably, the results can be used to improve dialogue strategies and ultimately enable robots to adapt their behavior autonomously. However, existing automatic evaluation methods have been developed primarily in controlled laboratory settings, and it remains unclear whether they can be directly applied to real-world environments, where users are free to disengage and multi-party participation may arise naturally. In this study, we investigate the automatic estimation of third-party-rated rapport scores using 62 sessions of multimodal recordings collected in a Japanese drugstore. We compare zero-shot LLMs, pretrained text, audio, and visual models, and their prediction-level fusion. The results show that, in real-world HRI, zero-shot LLMs achieve strong performance, while audio and visual models tend to provide complementary information. In particular, Gemini 2.5 Flash performs strongly as a single model, and a fusion model combining Gemini (text) with HuBERT and V-JEPA performs best overall. Further analyses showed that estimation performance varied across interaction-duration and group-size conditions. These findings suggest that rapport estimation in real-world HRI requires evaluation and model design that account for contextual variability beyond that assumed in laboratory settings.

• The paper demonstrates a robust multimodal framework for automatically estimating rapport in real-world Human-Robot Interaction (HRI), particularly using a combination of GPT gemini for text and HuBERT & V-JEPA for audio-visual data.
• Speech embeddings from short real-world exchanges show weak rapport signal, while zero-shot LLMs perform well for text-only inputs, achieving a PCC of 0.665.
• The best prediction results were from model fusion of Gemini text with HuBERT and V-JEPA audio-visual embeddings; demonstrating the power of using disjoint data sources.

Motivation and problem setting

Most automatic methods for evaluating interaction quality in human–robot interaction (HRI) have been developed and validated under laboratory conditions, where interaction onset, duration, and participant composition are experimentally controlled. In real-world deployments—retail spaces, public facilities—users freely initiate and abandon interactions, multi-party participation emerges spontaneously, and environmental contingencies shape the exchange. This paper addresses the resulting evaluation gap by constructing a multimodal dataset of Wizard-of-Oz (WoZ) robot interactions collected in an uncontrolled Japanese drugstore and establishing baselines for automatically estimating third-party-rated rapport scores (2608.18401). The authors state this is among the first studies to estimate third-party-rated rapport from multimodal recordings of WoZ-mediated real-world HRI in a retail environment.

Dataset and annotation

The corpus comprises 32 hours of operation over six days with Sota, a tabletop humanoid teleoperated via voice, gesture, and gaze. From 131 recorded sessions, filtering removed sessions with four or more participants (side conversations confound human–robot rapport), sessions involving preschool-aged children, and sessions with fewer than two user utterances. The final set contains 62 sessions and 97 analyzable participants; sessions average 54.23 seconds of video ($SD = 42.42$) and 11.85 utterances.

Rapport is annotated with the eight-item Connection-Coordination Rapport Scale (CCR-8), translated into Japanese, rated on 5-point Likert items by three third-party annotators at the individual level. Annotation quality is strong: Cronbach's $\alpha = .95$ for the overall score, ICC(2,3) $= .85$ (individual level) and $.86$ (group level). Individual-level scores average $M = 3.72$ ($SD = 0.80$). A caveat the authors acknowledge: the Japanese CCR-8 was not fully psychometrically validated, and the target is third-party-judged rapport rather than participants' self-reported experience.

Modeling approach

The task regresses a scalar rapport score per participant from modality-specific feature sequences, trained with a concordance correlation coefficient (CCC) loss to jointly capture linear association and mean/variance calibration. Two families of predictors are compared:

• Zero-shot LLMs: GPT-5.4, Claude Sonnet 4.6, and Gemini 2.5 Flash prompted as third-party annotators with item definitions and behavioral criteria, in text-only (T), text+audio (T+A), and text+audio+visual (T+A+V) conditions.
• Pretrained embedding models: Sentence-T5-large (text), HuBERT-large (audio), and V-JEPA 2.1 ViT-G (visual), each followed by additive attention pooling and an MLP prediction head trained on 30 session-level cross-validation folds.

Fusion is unweighted averaging of prediction scores; weighted averaging offered no substantial gain.

Main results

Zero-shot LLMs are strong standalone estimators: Gemini 2.5 Flash (T+A+V) attains the best single-model CCC (0.618), while Gemini (T) achieves the highest PCC (0.665)—notably using text alone. Among embedding-based single-modality models, HuBERT (audio) leads all metrics (CCC 0.460), while Sentence-T5 text embeddings perform poorly (PCC 0.327), indicating weak rapport signal in supervised text embeddings for short real-world exchanges.

Prediction-level fusion of Gemini (T) with the audio–visual embeddings produces the best overall configuration:

Model	Modality	MAE	PCC	CCC
Gemini 2.5 Flash	T+A+V	0.549	0.625	0.618
ST5+HuBERT+V-JEPA	T+A+V	0.540	0.567	0.444
Gemini (T) + HuBERT+V-JEPA	T+A+V	0.471	0.717	0.656

Complementarity analysis supports this design: relative to Gemini (T), HuBERT+V-JEPA has partial correlation 0.359 and incremental $\Delta R^2 = 0.072$, occupying the informative-and-non-redundant region of the utility–redundancy map. The implication is that LLMs and embedding-based audio–visual predictors capture partially disjoint rapport-relevant information and function better as complementary components than competitors.

Contextual variability analysis

Two analyses probe conditions unique to real-world deployment:

• Interaction duration: units range from 12 to 227 seconds (median 40 s). Splitting at the median, only the supervised text model ST5 shows a large performance gap (CCC 0.411 long vs. 0.168 short); Gemini (T) varies minimally (0.551 vs. 0.563).
• Group size: 69 of 97 samples come from multi-party interactions. Gemini (T) maintains high CCC across all group sizes and reaches its maximum in three-person interactions (0.721), whereas supervised visual predictions degrade sharply (V-JEPA CCC falls from 0.503 to 0.043 from one- to three-person groups). Ground-truth rapport was lower in three-person sessions ($M = 3.391$, $n = 13$), though the authors caution this subset is small.

A notable contrast with laboratory-based work emerges: unlike Speech-to-Joy, where text–audio late fusion outperformed LLMs and text features were nearly as informative as acoustic ones, here LLMs dominate and text embeddings are weak. The authors attribute this plausibly to much shorter interaction durations (~54 s vs. ~7 minutes) and multi-party dynamics, which starve supervised embedding models of signal while LLMs remain robust.

Limitations

The authors are explicit that findings concern WoZ-mediated interaction in one Japanese drugstore; ratings may partly reflect operator skill, and generalization to autonomous HRI, other cultures, or embodiments is untested. With 97 instances—and far smaller subsets for the duration and group-size analyses—the contextual analyses are exploratory. Targets exclude low-speech or early-disengagement encounters, so results cover only moderately engaged interactions. The zero-shot LLM and embedding pipelines were not evaluated under strictly identical input conditions, so the comparison should be read as a practical baseline rather than evidence of inherent LLM superiority; prompt sensitivity and model-version dependence further qualify LLM results. Finally, the cues underlying Gemini's estimates remain opaque—an open interpretability question the authors flag.

Conclusion

This study establishes a first baseline for third-party rapport estimation in uncontrolled retail HRI: text-only zero-shot LLMs perform strongly, and fusing Gemini (T) with HuBERT and V-JEPA yields the best results (MAE 0.471, PCC 0.717, CCC 0.656). The central empirical lesson is that real-world conditions—variable duration and spontaneous multi-party participation—demand evaluation protocols and model designs beyond laboratory assumptions, with supervised embedding models particularly fragile across group-size variation.