ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games (1707.01067v2)

Abstract: In this paper, we propose ELF, an Extensive, Lightweight and Flexible platform for fundamental reinforcement learning research. Using ELF, we implement a highly customizable real-time strategy (RTS) engine with three game environments (Mini-RTS, Capture the Flag and Tower Defense). Mini-RTS, as a miniature version of StarCraft, captures key game dynamics and runs at 40K frame-per-second (FPS) per core on a Macbook Pro notebook. When coupled with modern reinforcement learning methods, the system can train a full-game bot against built-in AIs end-to-end in one day with 6 CPUs and 1 GPU. In addition, our platform is flexible in terms of environment-agent communication topologies, choices of RL methods, changes in game parameters, and can host existing C/C++-based game environments like Arcade Learning Environment. Using ELF, we thoroughly explore training parameters and show that a network with Leaky ReLU and Batch Normalization coupled with long-horizon training and progressive curriculum beats the rule-based built-in AI more than $70\%$ of the time in the full game of Mini-RTS. Strong performance is also achieved on the other two games. In game replays, we show our agents learn interesting strategies. ELF, along with its RL platform, is open-sourced at https://github.com/facebookresearch/ELF.

• The paper presents ELF, a platform that advances RL research by simulating complex RTS game dynamics with partial observability and delayed rewards.
• The paper demonstrates ELF's efficiency with a 70% win rate on Mini-RTS using just 6 CPUs and 1 GPU within a day.
• The platform’s open-source, flexible design enables rapid experimentation and seamless integration with various reinforcement learning methods.

Overview of ELF: A Platform for Reinforcement Learning in RTS Games

The paper "ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games" introduces a novel platform designed to advance research in reinforcement learning (RL) through a suite of customizable real-time strategy (RTS) game environments. Developed with a focus on extensibility, efficiency, and adaptability, ELF aims to address limitations present in existing game platforms utilized for RL research.

Key Features

ELF presents several distinguishing features:

• Extensiveness: The platform simulates complex game dynamics including partial observability, delayed rewards, and concurrent actions, creating a rich training ground for RL agents.
• Lightweight Design: Capable of running 40,000 frames-per-second per core on standard hardware, ELF significantly outpaces existing environments, facilitating rapid experimentation.
• Flexibility: Users can modify game parameters, customize environment-agent communication, and integrate various RL methods seamlessly. ELF supports integration with C/C++ game environments, broadening its utility.

Numerical Results

The authors demonstrate the platform's efficiency through extensive testing. In {Mini-RTS}, one of the provided environments, training a bot to outperform rule-based AI was achieved with over 70% success using only 6 CPUs and 1 GPU within a day. Such outcomes highlight ELF's capability to expedite computational research in AI.

Theoretical and Practical Implications

ELF serves as a bridge between abstract RL research and practical, real-world applications by providing a controlled, yet complex, testbed. Practically, its open-source nature and ease of use make it accessible for a wide range of educational and research purposes. Theoretically, ELF challenges current algorithms with its rich dynamics, paving the way for developments in hierarchical RL and multi-agent systems.

Future Directions

The potential for future work is substantial. ELF could house more diverse game environments and facilitate research into novel RL algorithms that leverage its high simulation speed and concurrency features. Moreover, the platform could explore advanced strategies in AI behavior, such as long-term planning and strategic deception.

In summary, ELF is a significant contribution to the reinforcement learning research infrastructure, providing an accessible, powerful, and versatile environment to test and improve intelligent agents in complex RTS scenarios.