Towards Deep Symbolic Reinforcement Learning (1609.05518v2)

Abstract: Deep reinforcement learning (DRL) brings the power of deep neural networks to bear on the generic task of trial-and-error learning, and its effectiveness has been convincingly demonstrated on tasks such as Atari video games and the game of Go. However, contemporary DRL systems inherit a number of shortcomings from the current generation of deep learning techniques. For example, they require very large datasets to work effectively, entailing that they are slow to learn even when such datasets are available. Moreover, they lack the ability to reason on an abstract level, which makes it difficult to implement high-level cognitive functions such as transfer learning, analogical reasoning, and hypothesis-based reasoning. Finally, their operation is largely opaque to humans, rendering them unsuitable for domains in which verifiability is important. In this paper, we propose an end-to-end reinforcement learning architecture comprising a neural back end and a symbolic front end with the potential to overcome each of these shortcomings. As proof-of-concept, we present a preliminary implementation of the architecture and apply it to several variants of a simple video game. We show that the resulting system -- though just a prototype -- learns effectively, and, by acquiring a set of symbolic rules that are easily comprehensible to humans, dramatically outperforms a conventional, fully neural DRL system on a stochastic variant of the game.

Towards Deep Symbolic Reinforcement Learning: An Evaluation

The paper "Towards Deep Symbolic Reinforcement Learning," authored by Marta Garnelo, Kai Arulkumaran, and Murray Shanahan, proposes a hybrid architecture that integrates neural network learning with symbolic reasoning, aiming to mitigate several limitations inherent in current deep reinforcement learning (DRL) systems. This paradigm shift seeks to synergistically combine the statistical power of neural networks with the abstract reasoning capabilities inherent in symbolic AI.

Paper Overview

Contemporary DRL, notably exemplified by achievements in games like Atari and Go, suffers from substantial data inefficiency, lacking transfer learning capacity, limited high-level reasoning, and opacity. The paper challenges these weaknesses by proposing an architecture composed of a neural back end and a symbolic front end, leveraging deep learning to automate the symbol grounding process without handcrafting. This system eschews the brittleness of traditional DRL strategies by fusing them with symbolic representations capable of capturing compositional structures and enabling conceptual abstraction.

The research introduces a proof-of-concept model applied to a simplified video game environment, demonstrating that it significantly outperforms traditional DRL implementations such as Deep Q-Networks (DQN) under similar conditions. The results particularly highlight superior performance in scenarios involving stochastic game environments, underlining the potential for data-efficient learning made possible by the proposed architecture.

Key Contributions

The architecture primarily hinges on four pillars: conceptual abstraction, compositional structure, common sense priors, and causal reasoning.

1. Conceptual Abstraction: By mapping high-dimensional data to lower-dimensional symbolic states, the system efficiently discerns high-level similarities, streamlining tasks such as transfer learning and planning.
2. Compositional Structure: A probabilistic, first-order logic forms the semantic backbone of the representations, intended to encapsulate the uncertainty of real-world data within a Bayesian framework.
3. Common Sense Priors: The symbolic representation is enriched with priors reflecting the typical structure of the world, such as persistence and stereotypical interactions among objects, alleviating the learning burden.
4. Causal Reasoning: The architecture seeks to uncover causal relations within domains, facilitating planning and off-line exploration, a challenge unaddressed by purely reactive, traditional DRL systems.

Experimental Evaluation and Results

In the experimental validation, the architecture was tested across four game variants. The results underscored that the prototype efficiently learned effective policies, achieving high data efficiency and markedly excelling over DQN, particularly in the stochastic environments where objects' initial placements were randomized. The results indicate the auspicious potential of symbolic reasoning to generalize across unseen situations, a decisive advantage over neural network reliance on pure statistical learning.

Implications and Future Work

The architectures' intrinsic ability to incorporate high-level reasoning processes presents salient implications for advancing artificial general intelligence (AGI). By enhancing data efficiency and interpretability, symbolic reinforcement learning can prove significantly advantageous in domains where rapid learning and human-comprehensible justifications are crucial.

Future research could involve:

• Extending the neural back end capabilities through advanced deep learning techniques focused on unsupervised learning.
• Incorporating inductive logic programming and analogical reasoning within the symbolic front end to bolster generalization and transfer learning further.
• Integrating symbolic planning mechanisms to exploit discovered domain causal structures for more strategic action exploration.

The paper invites future exploration into architectures that maintain the symbolic elements within a neural framework, promising a unified entity that could potentially balance the strengths of both neural networks and symbolic reasoning to push boundaries in AI research.