Adaptive Curriculum Generation from Demonstrations for Sim-to-Real Visuomotor Control (1910.07972v3)

Abstract: We propose Adaptive Curriculum Generation from Demonstrations (ACGD) for reinforcement learning in the presence of sparse rewards. Rather than designing shaped reward functions, ACGD adaptively sets the appropriate task difficulty for the learner by controlling where to sample from the demonstration trajectories and which set of simulation parameters to use. We show that training vision-based control policies in simulation while gradually increasing the difficulty of the task via ACGD improves the policy transfer to the real world. The degree of domain randomization is also gradually increased through the task difficulty. We demonstrate zero-shot transfer for two real-world manipulation tasks: pick-and-stow and block stacking. A video showing the results can be found at https://lmb.informatik.uni-freiburg.de/projects/curriculum/

• The paper introduces Adaptive Curriculum Generation from Demonstrations, dynamically adjusting task difficulty with reverse-trajectory sampling to overcome sparse reward challenges.
• The paper leverages dynamic task parameterization by varying simulation parameters, resulting in robust visuomotor policies that adapt to increasingly complex conditions.
• The paper demonstrates zero-shot transfer from simulation to real-world tasks, achieving success rates of 85% for pick-and-stow and 60% for block stacking.

Adaptive Curriculum Generation for Sim-to-Real Reinforcement Learning

The paper introduces an approach termed Adaptive Curriculum Generation from Demonstrations (ACGD) aimed at improving reinforcement learning (RL) in complex tasks characterized by sparse rewards. This method facilitates the training of vision-based control policies in simulated environments with the ultimate goal of executing these policies in real-world robotic manipulation tasks, namely, pick-and-stow and block stacking. ACGD addresses a notable challenge in RL, particularly the exploration issue due to sparse rewards and the potential hazards of random explorations in physical settings.

Key Methodologies and Contributions

ACGD employs a novel curriculum generation strategy that eschews traditional reward shaping techniques, which often require manual optimization and can bias learned policies. Instead, ACGD dynamically adjusts the task difficulty for the learner by manipulating two critical elements: the sampling of initial states from demonstration trajectories and the degree of domain randomization during training.

1. Reverse-Trajectory Sampling: The method initiates training from the end-states of demonstration trajectories and iteratively progresses towards the start of these trajectories. This approach ensures that tasks remain solvable in the initial stages, while gradually increasing complexity ensures agent sophistication over time.
2. Dynamic Task Parameterization: ACGD incorporates various task and environment parameters such as object bounciness, gripper speed, and visual domain randomization which are systematically varied. The adaptive curriculum adjusts these parameters in response to the agent's performance success rate, ensuring a constant learning challenge.
3. Zero-Shot Transfer Capability: Perhaps the most compelling aspect of ACGD is its ability to enable zero-shot transfer of policies trained in simulated environments to real-world scenarios. This is achieved by gradually enhancing the realism of training environments through increased domain randomization.

Experimental Evaluation

The proposed method was evaluated using two manipulation tasks executed in a physics simulator and subsequently transferred to real-world scenarios without further policy adjustments. Results indicate:

• ACGD significantly outperforms methods relying solely on behavior cloning or RL with either dense or sparse reward functions, achieving notable success rates in both simulated and real-world environments.
• The adaptive curriculum framework provided a comprehensive exploration-exploitation trade-off by maintaining the task difficulty within a defined success rate interval, thus ensuring focused and efficient training.
• The visuomotor policies developed through ACGD transferred effectively to a real robotic platform, with success rates of 85% for pick-and-stow and 60% for block stacking.

Implications and Future Directions

The research underscores the potential of ACGD for training complex robotic control tasks in simulation followed by direct application in real-world environments. The ability to adaptively calibrate difficulty and employ domain randomization effectively bridges the simulation-to-reality gap, highlighting a substantial step forward in practical RL applications in robotics.

Moving forward, expanding the range of robotic manipulation tasks and refining the parameters governing task difficulty could further enhance the effectiveness of ACGD. Moreover, integrating more sophisticated simulation techniques and real-world feedback mechanisms may push the boundaries of efficiency and accuracy for automated robotic control systems. Such developments will likely influence both the theoretical landscape of RL and its practical applications across various domains, from autonomous systems to industrial automation.