Automatic Curriculum Learning For Deep RL: A Short Survey (2003.04664v2)

Abstract: Automatic Curriculum Learning (ACL) has become a cornerstone of recent successes in Deep Reinforcement Learning (DRL).These methods shape the learning trajectories of agents by challenging them with tasks adapted to their capacities. In recent years, they have been used to improve sample efficiency and asymptotic performance, to organize exploration, to encourage generalization or to solve sparse reward problems, among others. The ambition of this work is dual: 1) to present a compact and accessible introduction to the Automatic Curriculum Learning literature and 2) to draw a bigger picture of the current state of the art in ACL to encourage the cross-breeding of existing concepts and the emergence of new ideas.

• The paper demonstrates ACL as a meta-learning approach that dynamically selects tasks to elevate agent performance in deep RL.
• It outlines diverse ACL strategies, including reward shaping and adaptive state modulation, to manage task difficulty and enhance learning.
• The survey emphasizes ACL’s role in improving sample efficiency and generalization, particularly in multi-task and Sim2Real scenarios.

Automatic Curriculum Learning in Deep Reinforcement Learning: An Overview

The paper "Automatic Curriculum Learning For Deep RL: A Short Survey" offers an in-depth examination of Automatic Curriculum Learning (ACL) as applied to Deep Reinforcement Learning (DRL). ACL serves as a pivotal methodology in DRL by adapting training scenarios to align with an agent's evolving capacities, thereby enhancing sample efficiency and performance, particularly in complex environments displaying sparse rewards. ACL's influence extends to multiple domains, including the optimization of domain randomization for smooth Sim2Real transitions and the orchestration of task sequences in multi-agent scenarios. The survey's dual objective is to provide an accessible introduction to ACL literature and to map out the state-of-the-art in ACL methodologies for fostering new concepts and advancements.

1. A Closer Examination of ACL and Its Application in DRL

Deep Reinforcement Learning algorithms, which leverage deep neural networks, address problems structured as Markov Decision Processes (MDPs). These processes require agents to learn effective policies that maximize cumulative rewards. In multi-task settings, where an agent must generalize across various tasks, the capacity to learn becomes even more complex. ACL mechanisms offer a strategic advantage by dynamically adjusting the distribution of training data, aligning learning trajectories with the agent's growing prowess. This paper formalizes ACL as a meta-learning approach where the task selection function is optimized to maximize agent performance on target tasks.

2. ACL Mechanisms and Their Underlying Objectives

The survey delineates ACL mechanisms into various categories based on their functional objectives:

• Improving Performance: In simpler reinforcement learning problems, ACL can enhance sample efficiency and optimize performance through tailored task sequences.
• Addressing Hard Tasks: When task complexity is a barrier, ACL introduces a sequence of progressively challenging auxiliary tasks, guiding the agent from simple to complex problem-solving.
• Facilitating Generalization: For multi-task and Sim2Real applications, ACL optimizes exploration, enhancing generalization to unforeseen but related tasks.
• Developing Multi-Goal Competence: ACL plays a critical role in training agents in multi-goal RL to establish a versatile learning repertoire.
• Promoting Open-Ended Exploration: In scenarios where tasks are open-ended, ACL organizes both exploration and skill acquisition processes.

3. Essential Targets of ACL

ACL mechanisms exert control over various dimensions to affect task difficulty and learning efficiency:

• Initial State Distribution: By modulating initial states, ACL regulates task difficulty for agents.
• Reward Functions: Automatic reward shaping through ACL enhances exploration by incentivizing uncertain areas within the state space.
• Goals: Goal-oriented ACL approaches increase efficiency in multi-goal RL through strategic goal selection.
• Environment Dynamics: With parametric PCG, ACL adjusts environmental parameters to balance difficulty and learning efficacy.
• Opponent Selection in Multi-Agent Settings: Self-play strategies under ACL pave the way for robust policy development against varied, challenging opponents.

4. Optimization Strategies for Surrogate Objectives in ACL

Drifting away from direct optimization of the ultimate learning objective, ACL methods deploy surrogate objectives that facilitate achievable learning trajectories:

• Intermediate Difficulty Objectives: By focusing on tasks of intermediate difficulty, ACL provides a controlled ascent in complexity.
• Learning Progress Maximization: Drawing from developmental robotics, ACL uses localized LP measures as an indicator for strategic task selection.
• Diversity and Novelty Seeking: Diverse task selection maintains a varied learning experience, avoiding local optima traps.
• Surprise-Driven Learning: Methods utilizing prediction error or model disagreement direct agents to novel state spaces.
• Energy Dynamics: Energetic transition prioritization ensures impactful trajectory learning in high-mobility environments.
• Adversarial Play: In self-play scenarios, ACL orchestrates opponent variability to discover robust policy strategies.

Conclusion and Research Outlook

The survey proposes a taxonomy and offers a comparative evaluation of ACL methods, shedding light on potential innovations and research intersections in the field. By unfurling the power of ACL through varied applications in DRL, the paper underscores the necessity of systematic studies to further theoretical and empirical comprehension. ACL's potential as a linchpin in fostering autonomous, open-ended learning aligns with the overarching aim of creating learning systems capable of complex adaptive behaviors, akin to those observed in human learning trajectories.