Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions (1512.01124v2)

Abstract: Many real-world problems come with action spaces represented as feature vectors. Although high-dimensional control is a largely unsolved problem, there has recently been progress for modest dimensionalities. Here we report on a successful attempt at addressing problems of dimensionality as high as $2000$, of a particular form. Motivated by important applications such as recommendation systems that do not fit the standard reinforcement learning frameworks, we introduce Slate Markov Decision Processes (slate-MDPs). A Slate-MDP is an MDP with a combinatorial action space consisting of slates (tuples) of primitive actions of which one is executed in an underlying MDP. The agent does not control the choice of this executed action and the action might not even be from the slate, e.g., for recommendation systems for which all recommendations can be ignored. We use deep Q-learning based on feature representations of both the state and action to learn the value of whole slates. Unlike existing methods, we optimize for both the combinatorial and sequential aspects of our tasks. The new agent's superiority over agents that either ignore the combinatorial or sequential long-term value aspect is demonstrated on a range of environments with dynamics from a real-world recommendation system. Further, we use deep deterministic policy gradients to learn a policy that for each position of the slate, guides attention towards the part of the action space in which the value is the highest and we only evaluate actions in this area. The attention is used within a sequentially greedy procedure leveraging submodularity. Finally, we show how introducing risk-seeking can dramatically improve the agents performance and ability to discover more far reaching strategies.

• The paper presents a novel deep RL approach combining deep Q-learning and DDPG to learn the value of entire action slates.
• It leverages an attention mechanism to efficiently navigate the combinatorial action space in high-dimensional settings.
• Experimental results reveal that slate-based agents outperform traditional methods with superior scalability and robustness.

Deep Reinforcement Learning with Attention for Slate Markov Decision Processes

The paper presents a detailed exploration of using deep reinforcement learning techniques to solve complex decision-making problems involving high-dimensional states and actions, specifically focusing on Slate Markov Decision Processes (Slate-MDPs). This work is motivated by real-world applications such as recommendation systems, where a system proposes multiple options (a slate), but ultimately, only one action from the slate is executed based on user interaction or environmental feedback.

Overview and Methodology

Traditionally, reinforcement learning (RL) frameworks deal with single action decision spaces at each decision point. However, Slate-MDPs introduce combinatorial action spaces, where the decision involves selecting a slate of actions. The critical challenge is that the agent does not control which action from the slate is executed, complicating the estimation of action values due to incomplete observation of potential outcomes.

The authors propose a dual approach leveraging deep Q-learning and deep deterministic policy gradients (DDPG):

1. Deep Q-Learning for Slate-MDPs: Unlike traditional Q-learning that learns the value of individual actions, the proposed approach learns the value of entire slates. By doing so, the agent optimizes for both the combinatory nature of slates and the sequential decision-making process in the environment. This involves learning a value function that can assess the value of a slate as a single entity using deep neural networks.
2. Deep Deterministic Policy Gradient for Attention Mechanism: To efficiently navigate the high-dimensional action space, an attention mechanism guides the focus towards promising areas of the action space, optimizing the action selection process without evaluating the entire action space exhaustively. This involves a neural policy network that approximates which parts of the action space hold the potential for high value, directing the focus within the slate's dimensionality constraints.

Experimental Results

The experiments involve several environments modeled after real-world recommendation systems with varying sizes of state and action spaces, up to 13,138 dimensions. The results indicate that the slate-based agents significantly outperform baseline methods that ignore the combinatorial nature of the action space. Notably, full slate-based agents demonstrate superior scalability and robustness, particularly as the dimensionality and complexity of the slate increase.

The integration of a risk-seeking approach further enhances the exploration capability of the agents, allowing for discovery of high-reward strategies that are less apparent to traditional reward-driven learning mechanisms. By introducing non-linear transformations of reward signals, agents are shown to explore more effective strategic action paths.

Implications and Future Directions

This paper's framework expands the capacity of RL agents to operate in environments with sophisticated action structures and high dimensionality, such as those found in modern recommendation systems. The development and evaluation of Slate-MDPs provide a new ground for deploying RL in complex decision-making environments, facilitating applications beyond typical RL domains into any sector involving combinatorial action sets and decision-making under uncertainty.

Future research directions may involve extending this work to other forms of attention mechanisms, multi-agent settings where multiple agents propose and compete over slates, and integrating more sophisticated models for user choice prediction in recommendation systems. Moreover, further exploration of theoretical underpinnings in this context could provide valuable insights into optimizing policy guidance and learning efficiency in even larger and more complex action spaces.