DOROTHIE: Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agents (2210.12511v1)

Abstract: In the real world, autonomous driving agents navigate in highly dynamic environments full of unexpected situations where pre-trained models are unreliable. In these situations, what is immediately available to vehicles is often only human operators. Empowering autonomous driving agents with the ability to navigate in a continuous and dynamic environment and to communicate with humans through sensorimotor-grounded dialogue becomes critical. To this end, we introduce Dialogue On the ROad To Handle Irregular Events (DOROTHIE), a novel interactive simulation platform that enables the creation of unexpected situations on the fly to support empirical studies on situated communication with autonomous driving agents. Based on this platform, we created the Situated Dialogue Navigation (SDN), a navigation benchmark of 183 trials with a total of 8415 utterances, around 18.7 hours of control streams, and 2.9 hours of trimmed audio. SDN is developed to evaluate the agent's ability to predict dialogue moves from humans as well as generate its own dialogue moves and physical navigation actions. We further developed a transformer-based baseline model for these SDN tasks. Our empirical results indicate that language guided-navigation in a highly dynamic environment is an extremely difficult task for end-to-end models. These results will provide insight towards future work on robust autonomous driving agents. The DOROTHIE platform, SDN benchmark, and code for the baseline model are available at https://github.com/sled-group/DOROTHIE.

• The paper introduces the DOROTHIE platform, enabling interactive spoken dialogue to manage unexpected events in autonomous driving.
• It employs a dual Wizard-of-Oz setup within the CARLA simulator to create dynamic, real-time irregular scenarios for empirical studies.
• The transformer-based TOTO model and SDN benchmark provide key insights into integrating sensorimotor dialogue for adaptive vehicle navigation.

Overview of DOROTHIE: Advances in Spoken Dialogue for Autonomous Driving

The paper presents a comprehensive exploration into spoken dialogue systems for autonomous driving agents, introducing a novel simulation platform named DOROTHIE (Dialogue On the ROad To Handle Irregular Events). This platform is poised to address the critical need for autonomous driving systems to navigate unexpected situations in dynamic environments while effectively communicating with human operators. The development and evaluation of DOROTHIE underscore the importance of equipping autonomous vehicles with sensorimotor-grounded dialogue capabilities to enhance their adaptability to real-world complexities.

Simulation Platform: DOROTHIE

The DOROTHIE platform integrates with the CARLA simulator, creating an interactive environment that can generate unexpected scenarios dynamically. Such capabilities are crucial for empirical studies focused on situated communication within autonomous driving contexts. DOROTHIE is distinguished by its dual Wizard-of-Oz setup, comprising a Collaborative Wizard (Co-Wizard) and an Adversarial Wizard (Ad-Wizard). The Co-Wizard collaborates with human participants to navigate tasks, whereas the Ad-Wizard introduces irregular events, such as roadblocks or weather changes, challenging the navigation tasks in real-time. This setup uniquely mirrors real-world situations where pre-trained models may falter, and adaptive communication becomes vital.

Situated Dialogue Navigation (SDN) Benchmark

The Situated Dialogue Navigation (SDN) benchmark is a product of DOROTHIE's capabilities, resulting in a dataset comprised of 183 trials, 8415 utterances, 18.7 hours of control streams, and 2.9 hours of audio. This extensive dataset encompasses multi-faceted, synchronized information, challenging agents to predict dialogue moves, generate communication responses, and formulate navigation actions. This benchmark provides a basis not only for analyzing dialogue behaviors but also for evaluating the efficacy of simulation-based interaction models within unpredictable driving contexts.

Transformer-based Baselines: Temporally-Ordered Task-Oriented Transformer (TOTO)

A significant contribution of the research is the development of a transformer-based model, the Temporally-Ordered Task-Oriented Transformer (TOTO), which serves as a baseline for the SDN tasks. TOTO leverages pre-trained transformer encoders and combines historical data with real-time perceptual inputs to predict human dialogue moves and generate agent responses. It demonstrates how end-to-end models might contend with the complexities of language-guided navigation tasks in dynamic settings. Despite TOTO's innovative architecture, results indicate that robust language-guided navigation remains a formidable challenge for end-to-end models, pointing to the need for further research and development.

Implications and Future Directions

The implications of this paper are both theoretical and practical. Theoretically, it enriches understanding of human-agent interaction by exploring unscripted dialogue management in the face of irregular events. Practically, it sets the stage for future enhancements in autonomous driving systems, emphasizing the necessity for adaptive communication frameworks that do not rely solely on pre-trained models. As the field progresses, more sophisticated machine learning models and dialogue systems need to evolve to bridge the gap between simulation and real-world application. Continuous improvements in sensorimotor integration and real-time response systems will be essential components of this development.

In conclusion, the research presented in this paper encapsulates significant strides in interactive communication within autonomous driving. The DOROTHIE platform and the SDN benchmark serve as instrumental resources for the research community, facilitating new explorations into dynamic, language-conditioned navigation tasks. This work lays a foundation for future innovations aiming to meet the challenges posed by complex, unpredictable driving environments through advanced AI and dialogue systems.