WildLMa: Long Horizon Loco-Manipulation in the Wild (2411.15131v2)

Abstract: 'In-the-wild' mobile manipulation aims to deploy robots in diverse real-world environments, which requires the robot to (1) have skills that generalize across object configurations; (2) be capable of long-horizon task execution in diverse environments; and (3) perform complex manipulation beyond pick-and-place. Quadruped robots with manipulators hold promise for extending the workspace and enabling robust locomotion, but existing results do not investigate such a capability. This paper proposes WildLMa with three components to address these issues: (1) adaptation of learned low-level controller for VR-enabled whole-body teleoperation and traversability; (2) WildLMa-Skill -- a library of generalizable visuomotor skills acquired via imitation learning or heuristics and (3) WildLMa-Planner -- an interface of learned skills that allow LLM planners to coordinate skills for long-horizon tasks. We demonstrate the importance of high-quality training data by achieving higher grasping success rate over existing RL baselines using only tens of demonstrations. WildLMa exploits CLIP for language-conditioned imitation learning that empirically generalizes to objects unseen in training demonstrations. Besides extensive quantitative evaluation, we qualitatively demonstrate practical robot applications, such as cleaning up trash in university hallways or outdoor terrains, operating articulated objects, and rearranging items on a bookshelf.

• The paper introduces WildLMa, a framework that integrates whole-body control, language-conditioned imitation learning, and LLM-based planning for robust long-horizon loco-manipulation.
• It leverages VR-based teleoperation and cross-modal MaskedCLIP to generalize skills across varied, real-world object and task scenarios.
• Empirical evaluations demonstrate superior task success in out-of-distribution settings, highlighting its potential for autonomous service and collaborative robotics.

Summary of "WildLMa: Long Horizon Loco-Manipulation in the Wild"

In the field of robotics, the capability to perform long-horizon loco-manipulation tasks in varied real-world environments—referred to as "in-the-wild" scenarios—remains an essential yet complex challenge. This paper introduces WildLMa, a novel framework designed to address the demands of mobile manipulation using quadruped robots. The WildLMa framework integrates advanced imitation learning techniques with whole-body control to achieve robust skill generalization and long-horizon task execution.

Core Components

WildLMa consists of three primary components: a whole-body control mechanism, a subsystem for skill acquisition (WildLMa-Skill), and a planning interface (WildLMa-Planner) for sophisticated task execution.

1. Whole-Body Control and Teleoperation:
   • The system leverages a whole-body control policy adapted for VR-based teleoperation, facilitating coherent arm-base coordination and reducing the operational complexity for the human tele-operator. This adaptation enhances the robot's manipulability and reduces the effort required for demonstration, key factors for acquiring versatile locomotion and manipulation skills.
2. Skill Acquisition via WildLMa-Skill:
   • WildLMa-Skill employs language-conditioned imitation learning rooted in CLIP's visual and textual embeddings. By utilizing MaskedCLIP, the system enhances cross-modal attention, enabling the robot to generalize its skills across varied object conformations and environments. This component also integrates a mechanism for autonomous task termination, increasing the robustness of skill execution in volatile real-world settings.
3. Task Planning with WildLMa-Planner:
   • The framework provides an interface with LLM-based planners, facilitating the composition of various learned skills for executing complex, long-duration tasks. The use of a hierarchical graph and coarse-to-fine planning allows for translating high-level commands into structured actions, addressing the need for efficient task decomposition and skill coordination.

Empirical Evaluations and Implications

The empirical results underscore the efficacy of WildLMa, demonstrating notable improvements in task success rates over comparable reinforcement learning and zero-shot methods. For common loco-manipulation scenarios like object grasping and button pressing, WildLMa shows superior performance, especially in out-of-distribution environments, thereby indicating its potential for generalization to unseen challenges.

The integration of pre-trained foundational models (such as CLIP and DinoV2) within WildLMa-Skill suggests a viable pathway toward scaling the adaptability of learning-based methods in robotics. The results also emphasize the need for incorporating task-specific augmentations such as cross-attention with language input to further bolster model robustness.

Future Directions

Looking forward, the work on WildLMa opens several avenues for subsequent research. One promising direction involves experimenting with more complex language-conditioned task directives through LLM planners, potentially involving zero-shot or few-shot contexts for even broader generalization capabilities. Additionally, exploring the integration of adaptive and dynamically reconfigurable planners could enhance the efficiency of WildLMa-Planner in dealing with real-time environmental changes and uncertainties.

From a practical standpoint, the demonstrated ability to conduct tasks such as operating articulated objects or rearranging items illustrates the potential of WildLMa to contribute to service and collaborative robotics in human-centric environments. Future explorations could extend to industrial domains, where precision and adaptability in task execution are paramount.

In conclusion, WildLMa represents a significant step towards realizing sophisticated loco-manipulation capabilities for quadruped robots in the wild, addressing core challenges in generalization and long-horizon task execution while paving the way for the practical deployment of autonomous robotic systems.