Exploration in Deep Reinforcement Learning: A Survey (2205.00824v1)

Abstract: This paper reviews exploration techniques in deep reinforcement learning. Exploration techniques are of primary importance when solving sparse reward problems. In sparse reward problems, the reward is rare, which means that the agent will not find the reward often by acting randomly. In such a scenario, it is challenging for reinforcement learning to learn rewards and actions association. Thus more sophisticated exploration methods need to be devised. This review provides a comprehensive overview of existing exploration approaches, which are categorized based on the key contributions as follows reward novel states, reward diverse behaviours, goal-based methods, probabilistic methods, imitation-based methods, safe exploration and random-based methods. Then, the unsolved challenges are discussed to provide valuable future research directions. Finally, the approaches of different categories are compared in terms of complexity, computational effort and overall performance.

• The paper categorizes exploration strategies in deep reinforcement learning into paradigms like novel state rewards, diverse behaviors, goal-based methods, probabilistic models, imitation, safe, and random techniques.
• It highlights innovative methods such as count-based and uncertainty approaches, with notable success like Agent57 surpassing human benchmarks in Atari games.
• The survey underscores enduring challenges, calling for improved evaluation metrics, scalable real-world applications, and a balanced exploitation-exploration trade-off.

Exploration in Deep Reinforcement Learning: A Survey

The paper "Exploration in Deep Reinforcement Learning: A Survey" by Ladosz et al. offers a comprehensive review of exploration strategies within the field of deep reinforcement learning (DRL). Exploration is a critical component in DRL, particularly when addressing sparse reward problems where agents receive infrequent feedback from the environment. This survey categorizes existing exploration approaches into several paradigms: reward for novel states, reward for diverse behaviors, goal-based methods, probabilistic methods, imitation-based methods, safe exploration, and random-based methods.

Each category revolves around distinct methodologies and insights:

• Reward Novel States: Approaches in this category encourage agents to explore new or less visited states in the environment by offering intrinsic rewards. This stream includes prediction error methods, count-based methods, and memory methods, each utilizing a different tactic to estimate state novelty or rarity. A key numerical result highlighted is Agent57's achievement in surpassing human benchmarks across all 57 Atari games.
• Reward Diverse Behaviors: This area focuses on encouraging agents to exhibit a variety of behaviors. It includes both evolutionary strategies and policy learning approaches that reward diversity in policy parameters and outputs. Such frameworks are adept at generating effective exploration strategies by diversifying the agent's experiences.
• Goal-Based Methods: Here, the exploration process is guided by setting explicit exploratory goals or identifying valuable states to explore next. This method often integrates planning mechanisms to determine which unexplored areas the agent should seek, thereby improving sample efficiency and directed exploration.
• Probabilistic Methods: These strategies involve constructing probabilistic models to manage exploration. They are subdivided into optimistic exploration, which uses an estimated upper confidence bound of rewards for action selection, and uncertainty methods, which infer policies based on uncertainty about values or transitions. This is crucial for balancing exploration with exploitation.
• Imitation-Based Methods: These approaches utilize demonstrations, often from expert agents, to guide initial exploration. This imitation can be integrated directly into experience replay mechanisms or combined with other exploration strategies to overcome challenging exploration environments.
• Safe Exploration: Safety in exploration involves ensuring agent actions do not lead to harmful or costly states. Techniques here include human intervention, auxiliary rewards, and predefined safety constraints that supervise the exploration process.
• Random-Based Methods: Despite their simplicity, random methods are enhanced in this paper with strategies that make random exploration more sample efficient, such as dynamically adjusting exploration parameters or adding noise to network parameters.

The paper identifies several enduring challenges in the field of exploration. Among them are the need for better evaluation metrics beyond cumulative rewards to assess exploratory efficiency, the scalability of these methods to real-world applications, and the optimal balance of exploration versus exploitation. Future developments are speculated to enhance adaptive exploration mechanisms, improve safe exploration frameworks, and integrate multi-task learning to transfer exploration strategies across diverse environments.

In summary, this survey highlights the criticality of exploration in DRL and identifies key areas for further research. By categorizing and discussing various exploration strategies, it serves as a foundational reference for ongoing innovation and application within this dynamic domain of artificial intelligence.