Deep Recurrent Q-Learning for Partially Observable MDPs (1507.06527v4)

Abstract: Deep Reinforcement Learning has yielded proficient controllers for complex tasks. However, these controllers have limited memory and rely on being able to perceive the complete game screen at each decision point. To address these shortcomings, this article investigates the effects of adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM. The resulting \textit{Deep Recurrent Q-Network} (DRQN), although capable of seeing only a single frame at each timestep, successfully integrates information through time and replicates DQN's performance on standard Atari games and partially observed equivalents featuring flickering game screens. Additionally, when trained with partial observations and evaluated with incrementally more complete observations, DRQN's performance scales as a function of observability. Conversely, when trained with full observations and evaluated with partial observations, DRQN's performance degrades less than DQN's. Thus, given the same length of history, recurrency is a viable alternative to stacking a history of frames in the DQN's input layer and while recurrency confers no systematic advantage when learning to play the game, the recurrent net can better adapt at evaluation time if the quality of observations changes.

• The paper introduces DRQN, a novel architecture that integrates LSTM with DQN to address challenges in partially observable environments.
• It employs bootstrapped sequential and random updates to preserve temporal dependencies while adapting learning in POMDP scenarios.
• Empirical results on flickering Atari games confirm DRQN's robustness in handling incomplete observations, highlighting its real-world potential.

Deep Recurrent Q-Learning for Partially Observable MDPs

The paper "Deep Recurrent Q-Learning for Partially Observable MDPs" by Matthew Hausknecht and Peter Stone addresses the limitations of Deep Q-Networks (DQNs) in environments where only partial observability is available. By interlacing DQNs with Long Short-Term Memory (LSTM) networks, the authors propose a novel architecture, the Deep Recurrent Q-Network (DRQN), aimed at overcoming these limitations.

Introduction and Motivations

Deep Q-Networks (DQNs) have demonstrated human-level control policies across various domains, particularly in Atari 2600 games. However, the sequential decision-making environment in Atari games is typically framed as a Markov Decision Process (MDP) where the state is fully observable. DQNs utilize the last four frames to estimate the state, leading to limitations in scenarios requiring a memory beyond this short history, thus rendering them inefficient in Partially Observable Markov Decision Processes (POMDPs).

The necessity arises from real-world applications where agents seldom have access to the complete state, making them inherently partial observable. This paper introduces DRQN to handle such POMDPs, integrating LSTMs within the DQN architecture to maintain temporal dependencies and long-term context.

DRQN Architecture

To amalgamate recurrency within the DQN framework, the architecture replaces the DQN's first post-convolutional fully connected layer with an LSTM layer. This architecture processes single frames at each timestep. The recurrent layer aggregates temporal information which is then passed to subsequent layers to predict Q-values for actions.

Training Mechanism:

• Bootstrapped Sequential Updates: Episodes are sampled starting from the beginning, carrying the LSTM's state forward, aligning more with temporal consistency but violating the DQN's random sampling methodology.
• Bootstrapped Random Updates: Episodes start at random points, zeroing the LSTM's initial state, maintaining the randomness of sampling at the cost of temporal context.

Evaluation and Results

Flickering Atari Games:

• Pong was used to demonstrate DRQN's capabilities in POMDPs, where frames are probabilistically obscured, simulating partial observability. DRQN succeeded in estimating state values and object velocities through temporal integration, compensating for the lack of visual continuity in the input.

Standard Atari Games:

• Nine Atari games were evaluated to compare DRQN's performance under full observability (MDPs). The results indicated similar performance between DRQN and DQN, signifying that while recurrency does not outperform frame stacking, it maintains a competitive edge.

Generalizing from MDP to POMDP:

• Trained on complete observations, DRQN's performance on POMDPs, induced by flickering screens, demonstrated robust adaptability, with performance scaling proportional to the observability. These findings corroborate that DRQN preserves efficacy under reduced information conditions better than DQN.

Implications and Future Directions

The implications of this research extend to any domain characterized by partially observable environments, such as autonomous driving and robotics, where agents operate with incomplete sensory data. DRQN can process temporal sequences efficiently to construct an implicit representation of the state.

Future Developments:

• Delving deeper into the attributes of tasks like Pong and Frostbite, which notably benefited from recurrent architectures, may yield insights into the conditions that favor recurrency over frame stacking.
• Enhanced architectures combining more sophisticated recurrent units or hybrid models may further improve adaptability in diverse and complex POMDP environments.

In conclusion, while the DRQN does not universally outperform DQN across MDPs, its robustness in handling partial observations underpins its potential applicability in real-world scenarios characterized by partial observability. This work underscores the necessity of adaptable learning frameworks in dynamic and uncertain environments, prompting continued exploration into sophisticated recurrent architectures for reinforcement learning.