Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations (2303.01497v1)

Abstract: While imitation learning provides us with an efficient toolkit to train robots, learning skills that are robust to environment variations remains a significant challenge. Current approaches address this challenge by relying either on large amounts of demonstrations that span environment variations or on handcrafted reward functions that require state estimates. Both directions are not scalable to fast imitation. In this work, we present Fast Imitation of Skills from Humans (FISH), a new imitation learning approach that can learn robust visual skills with less than a minute of human demonstrations. Given a weak base-policy trained by offline imitation of demonstrations, FISH computes rewards that correspond to the "match" between the robot's behavior and the demonstrations. These rewards are then used to adaptively update a residual policy that adds on to the base-policy. Across all tasks, FISH requires at most twenty minutes of interactive learning to imitate demonstrations on object configurations that were not seen in the demonstrations. Importantly, FISH is constructed to be versatile, which allows it to be used across robot morphologies (e.g. xArm, Allegro, Stretch) and camera configurations (e.g. third-person, eye-in-hand). Our experimental evaluations on 9 different tasks show that FISH achieves an average success rate of 93%, which is around 3.8x higher than prior state-of-the-art methods.

• The paper introduces a two-phase framework combining a non-parametric base policy with residual reinforcement learning to enable rapid skill acquisition.
• FISH achieves a 93% success rate across nine diverse tasks, outperforming traditional imitation learning methods.
• The approach utilizes optimal transport rewards and a fixed visual encoder to enhance stability and safety during online policy refinement.

Fast Imitation of Skills from Humans (FISH): An Overview

The paper “Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations” introduces a novel approach to imitation learning aimed at enabling robots to learn skilled tasks from a minimal amount of human demonstrations. This methodology, dubbed Fast Imitation of Skills from Humans (FISH), is distinct from existing approaches in its ability to adapt to variations in object configurations with less than a minute of initial demonstration data. The core innovation lies in combining a non-parametric base policy, which is derived from these demonstrations, with a residual reinforcement learning policy that learns correctional actions during online trials.

Methodological Framework

The FISH framework is structured into two primary phases:

1. Base Policy Learning: The initial phase involves training a weak base policy through offline imitation. This policy is non-parametric and relies on the few demonstrations provided, establishing a preliminary policy that can serve as a foundation for subsequent learning processes. The base policy is designed to be general enough to encompass various robot morphologies (e.g., xArm, Allegro, Stretch robots) and to interface with different camera setups. The strength of this approach lies in its ability to operate effectively with low data amounts, a significant advantage over traditional imitation learning strategies that often require extensive demonstration datasets.
2. Residual Policy Optimization: In the second phase, the robot undergoes an online learning process where a residual policy is developed. This policy refines the action outputs by the base policy through reinforcement learning, leveraging a reward system grounded in optimal transport (OT) computations. The OT approach assesses the match between current observations and the demonstration trajectory, translating these into reward signals that guide the learning of corrective actions. This technique circumvents the need for bespoke, task-specific reward functions, which are often complex and hard to derive for many real-world tasks.

Experimental Insights

The efficacy of FISH is empirically validated across nine tasks spanning diverse robotic platforms and tasks. Notably, FISH achieves substantial improvements in success rates—averaging a 93% success rate across tasks—substantially outperforming existing state-of-the-art methods. The experimental setup highlights FISH’s adaptability to different robot morphologies and tasks with varied difficulty levels, including manipulation and dexterous tasks.

Key insights from experiments include:

• Versatility and Adaptation: FISH's framework demonstrates significant generalization capabilities to new and unseen object configurations not covered in the initial demonstrations.
• Guided Exploration Benefits: The residual policy allows for guided exploration strategies, which focus on exploring significant subspaces of the action space. This guided exploration enhances safety during task learning and improves the sample efficiency of the learning process.
• Encoder Strategy: Fixing the visual encoder used during image processing stabilizes the reward signal and improves policy performance, contrasting with the flexible encoder strategies that often lead to unstable training dynamics.

Implications and Future Directions

FISH offers compelling implications for robotic learning, especially in scenarios with limited access to demonstration data. The combination of non-parametric base policies and residual learning presents a pathway towards achieving rapid learning and adaptation in robotic systems, opening avenues for practical implementations in dynamic, real-world environments.

Moving forward, expanding FISH to incorporate multi-modal sensory inputs, such as tactile feedback, may enhance its application scope—particularly for tasks requiring high precision or those where visual signals are inadequate. Moreover, integrating self-supervised or pre-trained representations from large datasets could further augment the learning efficiency, albeit the paper demonstrates that current pre-trained models offer limited benefits compared to in-domain learned representations.

In conclusion, FISH advances the field of robotics by addressing the scalability and adaptability challenges inherent in imitation learning, providing a robust framework for efficient skill acquisition with minimal data.