Evaluating Long-Term Memory in 3D Mazes (2210.13383v1)

Abstract: Intelligent agents need to remember salient information to reason in partially-observed environments. For example, agents with a first-person view should remember the positions of relevant objects even if they go out of view. Similarly, to effectively navigate through rooms agents need to remember the floor plan of how rooms are connected. However, most benchmark tasks in reinforcement learning do not test long-term memory in agents, slowing down progress in this important research direction. In this paper, we introduce the Memory Maze, a 3D domain of randomized mazes specifically designed for evaluating long-term memory in agents. Unlike existing benchmarks, Memory Maze measures long-term memory separate from confounding agent abilities and requires the agent to localize itself by integrating information over time. With Memory Maze, we propose an online reinforcement learning benchmark, a diverse offline dataset, and an offline probing evaluation. Recording a human player establishes a strong baseline and verifies the need to build up and retain memories, which is reflected in their gradually increasing rewards within each episode. We find that current algorithms benefit from training with truncated backpropagation through time and succeed on small mazes, but fall short of human performance on the large mazes, leaving room for future algorithmic designs to be evaluated on the Memory Maze.

• The paper introduces Memory Maze, a novel RL benchmark that isolates long-term memory challenges in partially observed 3D environments.
• The study establishes human baselines by comparing human performance with RL agents, highlighting memory limitations in current algorithms.
• Evaluations with algorithms like Dreamer and IMPALA reveal that truncated backpropagation aids small maze training but struggles with larger mazes.

Evaluation of Long-Term Memory in 3D Mazes: A Comprehensive Analysis

The paper "Evaluating Long-Term Memory in 3D Mazes" introduces a novel reinforcement learning (RL) benchmark called the Memory Maze. This benchmark specifically targets the evaluation of long-term memory capabilities in agents operating within partially-observed environments, a critical aspect of human intelligence and a challenge for contemporary AI systems. The research addresses gaps in current RL benchmarks that do not adequately test long-term memory by presenting a domain that demands precise localization and memorization skills from RL agents.

Core Contributions

The Memory Maze introduces a structured and controlled environment specifically designed to isolate memory challenges from other confounding agent abilities. The contributions of this paper can be outlined as follows:

1. Benchmark Design: The Memory Maze provides a platform for evaluating long-term memory separate from other RL challenges. It features randomized 3D mazes where agents must remember object positions and maze layouts to navigate efficiently.
2. Human Baseline: The researchers compared agent performance to that of human players. Human players were able to solve mazes through memory build-up, which set a benchmark for evaluating AI-based solutions, showing that human-like long-term memory still presents a significant challenge for current algorithms.
3. Evaluation Methodology: The paper proposes an online RL benchmark and an offline dataset complete with semantic information for probing evaluation. This allows for a comprehensive assessment of agent memory capabilities.
4. Exploration of Agent Capabilities: Current RL algorithms were tested, revealing that truncated backpropagation through time significantly benefits training, enabling agents to succeed with smaller mazes but still falling short in larger mazes compared to human performance.

Technical Results

Several RL algorithms were evaluated, with key findings highlighting the limitations and capabilities of existing approaches:

• Dreamer and IMPALA Algorithms: Dreamer's performance benefited from truncated backpropagation through time (TBTT), achieving notable success on smaller mazes like Memory 9x9. However, in larger mazes, IMPALA outperformed Dreamer, indicating the complexity and insufficiency of memory architectures in scaling through maze size.
• Human Baseline vs. AI Performance: Human players outperform RL agents on larger mazes. This showcases the deficiency in current algorithms' ability to replicate human-like memory retention and recall over extended episodes.
• Offline Probing Benchmarks: The offline experiments, employing RSSM models with variations, demonstrated incomplete but significant memory retention capabilities, falling notably short of a supervised oracle that could directly use probe information for training.

Theoretical and Practical Implications

The Memory Maze benchmark introduces a critical tool for the development of algorithms with enhanced memory capabilities. By isolating memory from other factors, the benchmark allows for precise measurement and analysis, steering future research toward improved memory mechanisms within AI systems. Specifically, the challenge remains to design algorithms capable of retaining and efficiently using past information to scaffold decision-making processes over extended time horizons, akin to human cognition.

Future Directions

The limitations identified during this paper pave the way for future research, aiming to bridge the performance gap between AI and human memory abilities. Future developments could involve the integration of advanced memory networks like transformers or episodic memory systems and their impact on long-term memory tasks in RL. Additionally, expanding the current experimental setup to incorporate diverse environmental conditions could yield insights into the adaptability of RL agents' memory faculties.

In conclusion, this paper significantly contributes to the field by emphasizing the importance of memory in RL and providing a structured environment to paper this facet. Memory Maze holds promise for shaping the future of AI, calling for algorithms that not only learn efficiently but also remember and act on experiences over long timescales.