MOSAIC: A Modular System for Assistive and Interactive Cooking

Abstract: We present MOSAIC, a modular architecture for home robots to perform complex collaborative tasks, such as cooking with everyday users. MOSAIC tightly collaborates with humans, interacts with users using natural language, coordinates multiple robots, and manages an open vocabulary of everyday objects. At its core, MOSAIC employs modularity: it leverages multiple large-scale pre-trained models for general tasks like language and image recognition, while using streamlined modules designed for task-specific control. We extensively evaluate MOSAIC on 60 end-to-end trials where two robots collaborate with a human user to cook a combination of 6 recipes. We also extensively test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting, and 46 online user evaluations of the task planner. We show that MOSAIC is able to efficiently collaborate with humans by running the overall system end-to-end with a real human user, completing 68.3% (41/60) collaborative cooking trials of 6 different recipes with a subtask completion rate of 91.6%. Finally, we discuss the limitations of the current system and exciting open challenges in this domain. The project's website is at https://portal-cornell.github.io/MOSAIC/

• The paper introduces MOSAIC, a modular architecture that integrates large-scale language, vision, and motion models for collaborative cooking tasks.
• The paper employs an interactive task planner, visuo-motor skills, and human motion forecasting to achieve 68.3% overall trial success with 91.6% subtask completion.
• The paper demonstrates that a modular system design can enhance natural human-robot interaction and adaptability in dynamic home environments.

MOSAIC: A Modular System for Assistive and Interactive Cooking

The paper under review presents MOSAIC, a modular architecture devised to enable home robots to engage in complex collaborative tasks such as interactive cooking with human participants. This system intricately combines large-scale pre-trained models and task-specific modules, facilitating natural language interaction with users, coordinating two robots, and managing diverse everyday objects. The significance of MOSAIC lies in its ability to integrate LLMs, vision LLMs (VLMs), and motion forecasting models into a cohesive framework that enhances robot-home interaction.

MOSAIC's evaluation is comprehensive, encompassing 60 end-to-end trials across six distinct recipes, each with varying interaction complexities. The system successfully completes 68.3% of trials, demonstrating a subtask completion rate of 91.6%. This performance illustrates MOSAIC's efficiency in collaborative environments and highlights its potential application in autonomous cooking assistance.

Key Components and Methodology

MOSAIC comprises several core modules, each contributing to the system's overarching functionality:

1. Interactive Task Planner: This module uses LLMs embedded within a behavior tree to address high-complexity reasoning tasks, reducing potential error rates and providing reliable task allocation between robots and humans. The planner's design emphasizes modular scaling, allowing for seamless integration of new tasks and improved interaction with users.
2. Visuomotor Skills: To address the challenge of object detection and manipulation, MOSAIC employs OwlViT VLMs for identifying objects and leverages reinforcement learning (RL) in simulation for action execution. The separation between object identification and action execution enables specialization in each function, improving overall task performance without extensive in-field data collection.
3. Human Motion Forecasting: By incorporating pre-trained models on large-scale human motion data, MOSAIC forecasts human motion, ensuring safe and efficient collaboration. Reinforcement learning-based denoising techniques are applied to overcome input noise, enhancing real-time interactions.
4. System Flexibility and Adaptability: The modular design allows MOSAIC to adjust tasks dynamically based on user feedback and environmental changes. This flexibility ensures MOSAIC can handle wide-ranging scenarios and interactions, adhering to safety constraints and improving user experience.

Implications and Future Directions

The successful development and implementation of MOSAIC highlight several important theoretical and practical implications in the field of robotics and AI:

• Modular Architecture: By modularizing complex tasks, MOSAIC contributes to the broader application of modular systems in robotics, ensuring diverse and adaptive functionalities without overwhelming system complexity.
• Integration of Large-Scale Models: The system effectively consolidates LLMs, VLMs, and motion forecasting within a single framework, showcasing how large-scale pre-trained models can be composite parts of robust robotic solutions.
• Human-Robot Collaboration: MOSAIC paves the way for more intuitive and natural human-robot interactions, which are critical in augmenting the practical utility of robots in domestic settings.

Prospective research could explore the scalability of the MOSAIC architecture to broader applications beyond cooking, examining its integration with more advanced robotic skills and diverse environmental challenges. Further, enhancing the system's capacity for learning from real-time data and user feedback could significantly improve its adaptability and intelligence in complex, unstructured home environments.

In conclusion, MOSAIC presents a sophisticated, well-evaluated approach to assistive robotics in home settings, underscoring the potential of modular systems combined with advanced machine learning models. It establishes a promising foundation for future work in developing autonomous systems that are both user-friendly and capable of performing complex tasks in collaboration with humans.