MUTEX: Learning Unified Policies from Multimodal Task Specifications (2309.14320v1)

Abstract: Humans use different modalities, such as speech, text, images, videos, etc., to communicate their intent and goals with teammates. For robots to become better assistants, we aim to endow them with the ability to follow instructions and understand tasks specified by their human partners. Most robotic policy learning methods have focused on one single modality of task specification while ignoring the rich cross-modal information. We present MUTEX, a unified approach to policy learning from multimodal task specifications. It trains a transformer-based architecture to facilitate cross-modal reasoning, combining masked modeling and cross-modal matching objectives in a two-stage training procedure. After training, MUTEX can follow a task specification in any of the six learned modalities (video demonstrations, goal images, text goal descriptions, text instructions, speech goal descriptions, and speech instructions) or a combination of them. We systematically evaluate the benefits of MUTEX in a newly designed dataset with 100 tasks in simulation and 50 tasks in the real world, annotated with multiple instances of task specifications in different modalities, and observe improved performance over methods trained specifically for any single modality. More information at https://ut-austin-rpl.github.io/MUTEX/

• The paper introduces Mutex, a framework that learns unified robotic policies from multimodal task specifications.
• It employs a two-stage transformer-based training process with masked modeling and cross-modal matching to enhance cross-modal representations.
• Empirical results demonstrate over 10% success rate improvement in simulations and 14% in real-world tasks, underscoring its practical potential.

Mutex: A Unified Approach to Multimodal Task Specification for Robots

The paper "Mutex: Learning Unified Policies from Multimodal Task Specifications" presents an innovative framework, Mutex, aimed at improving robotic policy learning by enabling robots to interpret and execute tasks specified in multiple modalities. This reflects an understanding of how humans naturally communicate through diverse modalities—text, speech, images, and video—and the intention to enable robots to function similarly within human environments. Unlike prior methods that often treat each modality in isolation, Mutex leverages cross-modal information, thereby enhancing the richness of the learned representations and improving task performance.

Methodology and Implementation

Mutex utilizes a two-stage training process built on transformer architecture to facilitate cross-modal reasoning, combining masked modeling and cross-modal matching. The first stage, masked modeling, improves cross-modal interaction by using features hidden from one modality that must be predicted using information from others. By masking elements within task specifications during training, the model learns to use complementary data from multiple modalities to fill in the gaps, thus enforcing deeper cross-modal learning. The second phase, cross-modal matching, enriches the representation of each modality by matching it with the richer representation derived from video demonstrations. This is achieved through an $\ell_2$ regression loss, which ensures that all modality representations are aligned with the richer video representations, leading to uniformly enhanced representations across modalities.

Mutex employs a transformer-based architecture, which is well-suited for such cross-modal applications. The architecture involves modality-specific encoders that incorporate embeddings from pre-trained models and consolidate them into single representations for each modality. These encoders then feed into a unified policy encoder, which combines these representations with robot observational data.

Numerical Results and Performance

The empirical assessments conducted through an extensive set of experiments in both simulated and real-world environments demonstrate Mutex's superior performance. When tasked with 100 simulation tasks and 50 real-world tasks, Mutex consistently outperformed single-modality-trained models. Simulation results showed an improvement of over 10.3% in success rates across all modalities when leveraging the unified, cross-modal approach of Mutex over baseline methods tailored for individual modalities. Real-world testing confirmed these findings, with Mutex displaying a notable 14.3% enhancement in performance over specific modality-trained methods.

Implications and Future Directions

The implications of this work are manifold, impacting both the practical deployment of robots in human-centered environments and theoretical advancements in multi-modal AI. By effectively using complementary cross-modal information, Mutex makes a strong case for improving robotic autonomy and efficiency in task execution. This has practical applications in service robotics, where robots could interpret and respond appropriately to a variety of human instructions communicated in diverse formats within domestic, healthcare, or industrial settings.

Future directions for this work may address several limitations in Mutex’s current design. Scaling Mutex to handle environmental diversity and dynamic real-world conditions remains a challenge. The reliance on synthesised and clean speech data in training may necessitate additional robustness against noise and variability inherent in real-world audio. Moreover, integrating interactive imitation or reinforcement learning strategies can potentially refine policy learning by reducing limitations associated with behaviour cloning, such as covariate shift and compounding errors.

In conclusion, Mutex represents progress in the domain of multimodal task instruction for robots, pushing the boundaries toward a more versatile, human-like interaction paradigm. By demonstrating improved task performance across varied modalities, Mutex provides a promising platform for future exploration and improvement in the field of robotic learning and interaction.