Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning

Abstract: For a robot to be called socially intelligent, it must be able to infer users internal states from their current behaviour, predict the users future behaviour, and if required, respond appropriately. In this work, we investigate how robots can be endowed with such social intelligence by modelling the dynamic relationship between user's internal states (latent) and actions (observable state). Our premise is that these states arise from the same underlying socio-cognitive process and influence each other dynamically. Drawing inspiration from theories in Cognitive Science, we propose a novel multi-task learning framework, termed as \textbf{SocialLDG} that explicitly models the dynamic relationship among the states represent as six distinct tasks. Our framework uses a LLM to introduce lexical priors for each task and employs dynamic graph learning to model task affinity evolving with time. SocialLDG has three advantages: First, it achieves state-of-the-art performance on two challenging human-robot social interaction datasets available publicly. Second, it supports strong task scalability by learning new tasks seamlessly without catastrophic forgetting. Finally, benefiting from explicit modelling task affinity, it offers insights on how different interactions unfolds in time and how the internal states and observable actions influence each other in human decision making.

• The paper presents SocialLDG, a multi-task framework that combines lexical guidance and dynamic graph reasoning to jointly infer intent, attitude, and action from egocentric pose data.
• It employs a spatial GAT encoder and a transformer for temporal context modeling, achieving an average F1 score of 85.25% and outperforming state-of-the-art methods.
• The method’s modular design and interpretability through learned affinity matrices enable scalability and adaptability to new social perception tasks in human-robot interaction.

Lexically-Guided Dynamic Graph Learning for Social Interaction Interpretation in Robots

Introduction

The challenge of enabling social robots to robustly interpret and predict human internal states and observable behaviors necessitates a shift from purely reactive paradigms towards architectures equipped with sophisticated models of multi-faceted social dynamics. "Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning" (2604.10895) addresses this with SocialLDG, a multi-task learning framework that integrates lexical priors from LLMs and dynamic graph reasoning to jointly infer users’ intent, attitude, and physical actions from egocentric pose sequences. The framework models complex, temporally-evolving interdependencies between latent and observable states, supporting both interpretability and scalability across multiple downstream tasks in human-robot interaction (HRI) settings.

Figure 1: Dynamic reasoning in social HRI multitasking.

Methodology

SocialLDG formulates the problem as joint inference over six key social perception tasks: current and future physical contact, current intent, current attitude, current action, and future action. The framework operates on social signal representations extracted from whole-body pose sequences in egocentric videos. Pose estimation is achieved via AlphaPose, and a self-supervised autoencoder backbone combines a GAT-based spatial encoder with a Transformer for temporal aggregation and context modeling. To enhance representation robustness, a confidence-weighted reconstruction loss is employed, down-weighting low-confidence keypoints.

Each task is represented as a node in a dynamic, fully-connected graph with explicit attention-based edge weights—the task affinity matrix $\mathbf{A}$. Lexical priors are injected via task-specific tokens generated by a pretrained SciBERT encoder, providing rich semantic anchors to each task node. These are further fine-tuned during training for optimal alignment with the actual data distribution.

Figure 2: The SocialLDG pipeline, including spatio-temporal social signal encoding and lexically initialized, dynamically weighted task graph inference.

A semantic edge generator creates context-dependent, continuous adjacency (edge bias) matrices by modulating lexical similarities with input context derived from multimodal signals. The dynamic graph transformer then aggregates features for each task, using a sigmoid-based attention formulation over the combined semantic and data-driven edge affinities. Future-to-current label leakage is structurally prevented via directed edge masking.

Experimental Evaluation

Experiments are conducted using a union of JPL-Social and HARPER, two comprehensive HRI datasets annotated for the six primary tasks, and encompassing a variety of challenging, robot-centric interaction scenarios.

Figure 3: Example frames from JPL-Social, illustrating diverse egocentric HRI scenarios involving explicit and implicit social cues.

SocialLDG achieves an average F1 of 85.25%, consistently outperforming SOTA MTL approaches including PLE, MMoE, AssociationGraph, and UinT baselines. Removal of either the lexically-initialized node tokens or the learnable affinity biases produces significant performance drops, demonstrating the necessity of both semantic and context-sensitive modeling for effective multi-task inference. Ablations further reveal that transformer-based temporal aggregation combined with GAT spatial modeling yields superior downstream task performance, despite slightly higher reconstruction error compared to RNN decoders.

A salient property of SocialLDG is its task scalability: new perception tasks can be incorporated with minimal fine-tuning, without catastrophic forgetting or significant loss of accuracy on prior tasks. This property is directly attributed to the explicit factorization of inter-task dependencies in the dynamic affinity graph and the modular injection of semantic priors.

Lexical Token Analysis

Task token initialization experiments establish that semantic priors from domain-adapted LMs (e.g., SciBERT) confer measurable improvements over random or generic LLMs. Fine-tuning these embeddings during training is critical for aligning semantic similarities with functional task dependencies—which is directly evidenced by post-hoc visualization of the token similarity matrix before and after adaptation.

Figure 4: Cosine similarity heatmaps of task tokens before and after fine-tuning; functional relationships between intent/attitude and current/future tasks become pronounced after adaptation.

Dynamic Task Relationship Visualization

The dynamic evolution of the task affinity matrix $\mathbf{A}$ during interaction episodes is visualized using chord diagrams and affinity similarity metrics between consecutive frames. These reveal discrete contextual shifts—such as transitions from pre-interaction to post-interaction phases—which correspond closely to the natural temporal segmentation of social exchanges. Notably, affinity matrices automatically become sparser during high-certainty phases (e.g., explicit actions) and denser in ambiguous or transitional periods.

Figure 5: Chord diagrams illustrate dynamic reweighting of inter-task relationships across interaction stages; phase boundaries are reflected in similarity minima between adjacent affinity matrices.

Theoretical and Practical Implications

The work affirms that explicit, learned models of inter-task affinity are essential for coupling latent internal state inference with observable action recognition in HRI. By leveraging both structured semantic priors from LLMs and context-sensitive data-driven signals, SocialLDG models the complex, temporally varying topology of socio-cognitive dependencies.

The interpretability afforded by the affinity matrix and the modularity of the task-graph architecture present significant advantages for scalable, extensible, and theoretically analyzable implementations of real-world social robots. Practically, this supports rapid expansion to new task domains and facilitates continuous adaptation in dynamically evolving social environments. Theoretically, it offers a data-driven framework for probing the structure and temporal dynamics of human socio-cognitive processes as expressed through interaction.

Conclusion

The contribution demonstrates that lexically-guided, dynamically weighted graph-based multi-task learning is a viable and effective paradigm for social perception in HRI. SocialLDG achieves robust joint inference of intent, attitude, and action from egocentric pose data, exhibiting strong empirical performance, scalability, and interpretability. Future directions include tackling broader, unconstrained social settings, integrating longer sequence modeling for temporal affinity continuity, and extending the approach to additional social signal modalities.