Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning (1703.02949v1)

Abstract: People can learn a wide range of tasks from their own experience, but can also learn from observing other creatures. This can accelerate acquisition of new skills even when the observed agent differs substantially from the learning agent in terms of morphology. In this paper, we examine how reinforcement learning algorithms can transfer knowledge between morphologically different agents (e.g., different robots). We introduce a problem formulation where two agents are tasked with learning multiple skills by sharing information. Our method uses the skills that were learned by both agents to train invariant feature spaces that can then be used to transfer other skills from one agent to another. The process of learning these invariant feature spaces can be viewed as a kind of "analogy making", or implicit learning of partial correspondences between two distinct domains. We evaluate our transfer learning algorithm in two simulated robotic manipulation skills, and illustrate that we can transfer knowledge between simulated robotic arms with different numbers of links, as well as simulated arms with different actuation mechanisms, where one robot is torque-driven while the other is tendon-driven.

• The paper introduces a framework that leverages invariant feature spaces to transfer skills between agents with different physical characteristics.
• It employs deep neural networks to map agent-specific states into a common space without requiring exact state or action correspondences.
• Experimental results demonstrate significant performance gains in robotic tasks, especially under sparse reward conditions, outperforming standard baselines.

Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning

The paper at hand addresses the challenge of skill transfer between morphologically disparate agents, particularly in the context of reinforcement learning (RL). The authors propose a novel method that leverages invariant feature spaces to facilitate skill transfer between agents with differing physical characteristics, such as robotic arms with various numbers of links or different actuation mechanisms.

Core Contributions

The primary contribution of the paper is the introduction of a framework for transferring skills between two agents by learning common invariant feature spaces. This framework involves two main components:

1. Multi-Skill Transfer Formulation: The authors formulate a setting where two agents learn multiple skills, using shared knowledge to create an invariant feature space. This allows one agent to transfer a new skill after projecting the executions into this shared space, tracking through its actions. Such invariant learning is akin to learning partial analogies across domains and can potentially exploit similarities even when the state or action spaces differ.
2. Algorithm for Invariant Feature Space Learning: The proposed algorithm employs deep neural networks to map agent-specific states into a common feature space. This process does not assume isomorphic state-space mappings, which are often impractical across disparate agents. The networks are trained on prior shared tasks, allowing them to maximize transferable information between agents while ignoring non-transferable aspects.

Experimental Evaluation

The paper demonstrates the efficacy of the proposed method through a series of simulated robotic manipulation tasks:

• Task with Different Robotic Arm Morphologies: The authors present an experiment transferring knowledge between a 3-link and a 4-link robotic arm, as well as transferring skills from a torque-driven to a tendon-driven arm. The tasks involve sparse rewards, highlighting the ability of the approach to effectively direct exploration and learning via structured transfer rewards.
• Transfer Through Image Features: Building upon the framework, the authors further applied their methodology to vision-based inputs, using pixel data as input for the feature mapping. This extension underscores the potential of invariant feature spaces in overcoming sensory modality differences, allowing transfer through learned image features.

Results

The experimental results indicate that the proposed approach achieves significant improvements in learning performance compared to several baselines, including no transfer and linear embedding methods like CCA and kernel CCA. Notably, the approach facilitates skill transfer across agents with non-trivial morphological differences and sparse reward signals, where learning from scratch is impractically slow or failed.

Theoretical and Practical Implications

The theoretical implications of this work lie in the abstraction of task-related knowledge into invariant feature spaces, enabling transfer learning without direct mappings between states or actions. Practically, this approach can radically improve the efficiency and efficacy of learning in robotics, where differences in morphology between robots can otherwise necessitate extensive retraining or complex reparameterization of learned skills.

Future Directions

The potential for future work is substantial, with several promising directions:

• Generalization to More Agents/Tasks: While the current work considers transfer between two agents, future extensions could address many agents/tasks. This will require methods to discern which skills shared across different agent pairs contribute to learning an effective shared feature space.
• Lifelong Learning and Cumulative Knowledge Construction: Real-world applications could benefit from a lifelong learning framework, where robots continually enhance their feature representations as they acquire new skills or observe tasks.

In conclusion, the paper establishes a robust foundation for transferring skills across morphologically distinct agents using invariant feature spaces, marking a significant step forward in the domain of reinforcement learning and robotic skill acquisition. Such advancements can enhance the adaptability and generalization capabilities of autonomous robots.