When to Say "Hi" - Learn to Open a Conversation with an in-the-wild Dataset

Abstract: The social capabilities of socially interactive agents (SIA) are a key to successful and smooth interactions between the user and the SIA. A successful start of the interaction is one of the essential factors for satisfying SIA interactions. For a service and information task in which the SIA helps with information, e.g. about the location, it is an important skill to master the opening of the conversation and to recognize which interlocutor opens the conversation and when. We are therefore investigating the extent to which the opening of the conversation can be trained using the user's body language as an input for machine learning to ensure smooth conversation starts for the interaction. In this paper we propose the Interaction Initiation System (IIS) which we developed, trained and validated using an in-the-wild data set. In a field test at the Deutsches Museum Bonn, a Furhat robot from Furhat Robotics was used as a service and information point. Over the period of use we collected the data of \textit{N} = 201 single user interactions for the training of the algorithms. We can show that the IIS, achieves a performance that allows the conclusion that this system is able to determine the greeting period and the opener of the interaction.

• The paper introduces the Interaction Initiation System (IIS) that autonomously predicts when and how a robot should initiate greetings using multimodal body language cues.
• The methodology combines BlockRNN-based pose forecasting with an SVM classifier, achieving 75.3% accuracy in real-time greeting decision-making.
• The study demonstrates that real-world, data-driven models improve human-robot engagement, though challenges remain with pose recognition and class imbalance.

Learning Autonomous Conversation Openings in Human-Robot Interaction: The Interaction Initiation System

Introduction

Initiating conversations in Human-Robot Interaction (HRI) is a critical determinant of interaction flow, user trust, and engagement. Existing literature shows that initial robot behavior and timing strongly shapes user perception and willingness to engage [Erel, Xu, Paetzel]. While prior work extensively explores timing and modality in lab contexts, there is a lack of in-the-wild studies capturing spontaneous, real-world human-robot encounters. Addressing this gap, the presented paper introduces the Interaction Initiation System (IIS), an architecture that autonomously predicts when and how a robot should greet users, using multimodal behavioral data collected during field deployment at the Deutsches Museum Bonn.

Figure 1: The in-the-wild experimental setup, with a Furhat robot as information point in a demarcated museum area.

Related Work and Motivations

Previous HRI research has used proxemics, gaze, and gesture cues to determine conversational opening strategies [Hall, Shi, Kahn, Heenan, SidnerLee]. Strategies include state-machine models for spatial formation, adaptation of speech volume to user distance, and nonverbal greetings such as waving or facial expressions. However, these were often engineered in controlled or Wizard-of-Oz settings, resulting in limited generalizability and scalability in unconstrained environments.

Incremental and context-aware opening actions improve perceived robot competence and engagement [Fischer, Pitsch, Brink, Janssens], but robust computational models for real-time decision-making using body language features from spontaneous interactions remain underexplored. The presented work seeks to generalize the extraction and use of nonverbal user signals, leveraging pose and facial features for machine-learned timing and modality prediction.

Experimental Setup and Data Acquisition

The study utilized a Furhat robot ("Mira"), instrumented with three cameras and placed in a designated interaction area near the museum entrance. Visitors consented to data capture before approaching the robot, enabling collection of video, audio, and interaction events. The backend employed YOLOv5 for person detection and MediaPipe for dense body and face landmark extraction, yielding high-resolution time series for machine learning. 201 single-user interactions (26,675 annotated time steps, 1,682 features per frame) were recorded, ensuring visibility and entrance conformity for data consistency.

System Architecture: The Interaction Initiation System

The IIS comprises two principal classifiers:

• Timing Classifier: Determines the optimal instant for greeting initiation by predicting user movement and intention.
• Type Classifier: Suggests the manner (e.g., verbal, gestural) of the greeting; this component is architected but not empirically evaluated in the current study.

IIS operates by first applying a BlockRNN-based pose forecasting model to anticipate user body and facial landmark positions over short horizons (0.5s with 10 fps input). Predicted pose data are then input into a Support Vector Classifier, assigning one of {wait, speak, listen} actions per time step, reflecting whether the robot should initiate, listen for user opening, or remain idle.

Figure 2: Flow diagram outlining IIS components for timing and type determination in conversation initiation.

Figure 3: Sequence of the Timing Classifier pipeline, including pose forecasting and action classification.

Feature Engineering and Annotation

MediaPipe extracted 543 key landmarks covering body, face, and hands, supplemented by 52 facial expression shape scores. Each landmark provided normalized $(x, y, z)$ coordinates and a visibility score. Annotation involved labeling every frame as "wait," "speak," or "listen," based on expert review of approach and utterance phases. Class distribution, though imbalanced (majority "wait," minority "listen"), reflects real-world interaction dynamics where users mostly exhibit ambiguous pre-greeting behavior.

Class assignments were primarily based on proximity and mouth movements: "listen" is narrowly distinguished by subtle facial cues, complicating classifier discrimination and impacting recall for this class.

Model Training and Evaluation

The BlockRNN pose forecasting achieved RMSE of 0.0426, indicating low prediction error for body landmarks over short-term prediction. The action classifier, optimized via extensive grid search (SVM and Random Forest), delivered 75.3% accuracy (SVM, all features), and 74% weighted F1-score on a held-out test set. Macro-F1 was lower (69%), confirming compromised performance on "listen" due to data scarcity and indistinct class boundaries.

Confusion matrices highlight robust discrimination for "wait" and "speak," but reduced true positive rate for "listen," indicating a need for higher frame rates or richer feature sets to capture rapid mouth/opening cues.

Implications and Limitations

The IIS demonstrates that data-driven, multimodal body language analysis enables automated, real-time decisions on conversation initiation timing in HRI. The principled focus on in-the-wild data enhances ecological validity, ensuring models reflect genuine user variability and natural interactivity. However, pose recognition instability—a function of camera angles, occlusions, and landmark ambiguity—remains a critical bottleneck. As identified, improvements in camera coverage, frame rates, and synthetic data augmentation may yield further performance gains. Current limitations include lack of demographic granularity and incomplete evaluation of greeting type selection.

For transferability, IIS is immediately applicable to stationary robots in fixed interaction zones where user orientation and trajectory are predictable. Extension to mobile or open-space contexts will require increased data variability and dynamic camera calibration.

Future Directions

Continued development should pursue:

• Balancing data across greeting behaviors, potentially via strategic user recruitment or synthetic augmentation.
• Integrating explicit spatial features such as user-robot distance.
• Deploying higher fps video capture to resolve fine-grained mouth and gesture cues.
• Systematic evaluation of the Type Classifier and user perception of greeting appropriateness.
• Exploring adaptive interaction initiation policy learning via reinforcement learning frameworks.

Conclusion

The presented IIS achieves robust prediction of user-initiated vs. robot-initiated greetings in a public HRI scenario, with empirically validated models operating over detailed body posture data. While current macro-average performance is constrained by class imbalance and fine-grained action recognition challenges, the approach offers a scalable, principled template for autonomous conversational opening in interactive service robots. With further improvements and broader data, IIS is poised for widespread deployment in both research and public-facing robotic systems.