RILI: Robustly Influencing Latent Intent (2203.12705v2)

Abstract: When robots interact with human partners, often these partners change their behavior in response to the robot. On the one hand this is challenging because the robot must learn to coordinate with a dynamic partner. But on the other hand -- if the robot understands these dynamics -- it can harness its own behavior, influence the human, and guide the team towards effective collaboration. Prior research enables robots to learn to influence other robots or simulated agents. In this paper we extend these learning approaches to now influence humans. What makes humans especially hard to influence is that -- not only do humans react to the robot -- but the way a single user reacts to the robot may change over time, and different humans will respond to the same robot behavior in different ways. We therefore propose a robust approach that learns to influence changing partner dynamics. Our method first trains with a set of partners across repeated interactions, and learns to predict the current partner's behavior based on the previous states, actions, and rewards. Next, we rapidly adapt to new partners by sampling trajectories the robot learned with the original partners, and then leveraging those existing behaviors to influence the new partner dynamics. We compare our resulting algorithm to state-of-the-art baselines across simulated environments and a user study where the robot and participants collaborate to build towers. We find that our approach outperforms the alternatives, even when the partner follows new or unexpected dynamics. Videos of the user study are available here: https://youtu.be/lYsWM8An18g

Overview of "RILI: Robustly Influencing Latent Intent"

This paper presents a novel methodology termed 'RILI' (Robustly Influencing Latent Intent), which addresses the challenge of human-robot interaction by focusing on the dynamic nature of human responses. Unlike previous approaches that mainly targeted interactions with static dynamics or interactions with simulated agents, RILI extends its applicability to real human environments where dynamics are inherently non-static and diverse.

Core Contributions

1. Influencing Dynamic Human Partners: The authors propose a method enabling robots to learn influential behaviors through training with a diverse set of simulated partners. The methodology is designed to account for changes in human dynamics rather than assuming consistency over time or across individuals. Specifically, RILI employs a history of interactions to predict and adapt to a human partner’s latent strategy, effectively handling the variations in behavior that arise from different or changing human dynamics. This adaptability is achieved by using a Gated Recurrent Unit (GRU) to infer latent strategies based on past interactions.
2. Rapid Adaptation to New Partners: As real-world applications necessitate adaptability to previously unseen partner dynamics, RILI introduces a mechanism for quickly transferring learned behaviors to new partners. This transfer is executed by freezing the learned policy and focusing on updating the model's understanding of partner dynamics, thereby accelerating the adaptation process necessary for effectively influencing new dynamics.
3. Empirical Validation: The authors validate RILI's effectiveness through comprehensive simulations and a user paper. The simulation environments, including scenarios like driving and robot manipulation, confirm the method’s ability to learn and adapt to several changeable partner dynamics. The user paper with human participants further reinforces the approach’s practical applicability, where RILI-Transfer demonstrated robust coordination and influenced tower-building tasks effectively within few interactions.

Implications and Future Directions

RILI's contribution is particularly significant in the field of collaborative human-robot environments where robot systems are expected to be proactive and effectively adapt to human partners. By allowing robots to adjust to evolving human strategies, this research can substantially enhance the efficiency and efficacy of human-robot collaborations in various sectors, including manufacturing, healthcare, and service robotics.

Moreover, the robust transfer of learned patterns to new partner dynamics without requiring extensive retraining suggests potential applications in multi-environment deployment, where a robot can maintain high performance levels in diverse settings. As the state of the art in human-robot interaction progresses, future developments could refine RILI with improved partner modeling to better handle even more noisy or unpredictable human behaviors.

Numerical Results and Experimental Insights

The results reveal RILI's superior performance in diverse partner scenarios when compared to existing baselines like SAC, LILI, and SILI. With metrics showing substantial improvements in expected cost across multiple interactions, RILI not only demonstrates efficacious learning but also exceptional performance generalization. In the user paper, despite the variability in participant behaviors, RILI-Transfer showed convergence to high-performance collaboration within 35 interactions, a notable contrast to more traditional models which exhibited inconsistent outcomes.

Conclusion

"RILI: Robustly Influencing Latent Intent" introduces an impactful advancement in adaptive human-robot collaboration frameworks. The method’s robustness against dynamic and disparate human response patterns represents a significant step forward in situational awareness and the dynamic adaptability of artificial agents. Future exploration should aim at expanding on the breadth of environments and partner types, potentially including even more complex human decision-making processes and longitudinal dynamics adaptation.