On the Verge of Solving Rocket League using Deep Reinforcement Learning and Sim-to-sim Transfer (2205.05061v2)

Abstract: Autonomously trained agents that are supposed to play video games reasonably well rely either on fast simulation speeds or heavy parallelization across thousands of machines running concurrently. This work explores a third way that is established in robotics, namely sim-to-real transfer, or if the game is considered a simulation itself, sim-to-sim transfer. In the case of Rocket League, we demonstrate that single behaviors of goalies and strikers can be successfully learned using Deep Reinforcement Learning in the simulation environment and transferred back to the original game. Although the implemented training simulation is to some extent inaccurate, the goalkeeping agent saves nearly 100% of its faced shots once transferred, while the striking agent scores in about 75% of cases. Therefore, the trained agent is robust enough and able to generalize to the target domain of Rocket League.

• The paper introduces sim-to-sim transfer to train agents with deep reinforcement learning for proficient Rocket League gameplay.
• It employs a Unity-based simulation with mixed action spaces and proximal policy optimization to enhance goalkeeping and striking behaviors.
• Experimental results show nearly perfect saves for goalkeeping and a 75% scoring success rate for striking, emphasizing realistic simulation fidelity.

On the Verge of Solving Rocket League using Deep Reinforcement Learning and Sim-to-sim Transfer: A Summary

The paper under consideration explores the application of sim-to-sim transfer within the context of training agents to play Rocket League, an intricate multiplayer online game. Building on the prowess of Deep Reinforcement Learning (DRL) and proximal policy optimization (PPO), the paper offers insights into achieving performant agent behaviors in a virtual game environment that is difficult to parallelize or speed up.

Overview of Methodology

The paper embarks on a methodological journey well-grounded in robotics, adopting the sim-to-real transfer principle and aptly adapting it to a simulated environment (sim-to-sim) where the target platform remains virtual. The experiment leverages a Unity-based simulation of Rocket League's mechanics as a training ground, focusing specifically on learning single-agent tasks—goalkeeping and striking. Mixed action spaces and Unity’s ML-Agents Toolkit serve as frameworks to tackle the discrete and continuous action challenges innate to Rocket League.

Training Environment and Implementation

A noteworthy contribution is the detailed application of numerous physical mechanics of Rocket League—such as car acceleration, boosting, and ball dynamics— within a Unity simulation. This includes addressing simulation imperatives like car-to-car and ball interactions, air control, and physical properties deviation. While recording an impressive simulation speed of 950 steps per second, the paper endorses hardware settings that underline the practical integration measures for deploying DRL.

Experimental Results and Findings

Through a nuanced experimental design, the research evaluates the robustness of the sim-to-sim transfer via a series of scenarios that convey task performance pre- and post-transfer to the actual game. The numerical results are particularly notable—illustrating that, for the goalkeeping task, agents successfully save nearly all shots encountered despite simulation inaccuracies. Similarly, for striking tasks, agents achieve a goal-scoring success rate of approximately 75% once transferred to Rocket League.

The nuance is further highlighted through a constructed ablation paper revealing that turning off integral physics components results in noticeable performance declines. This underscores the critical need for thoughtful simulation environment alignment to ensure robust learning that transfers effectively.

Implications and Future Perspectives

This research has multiple implications for the development of game AI using reinforcement learning frameworks. It underscores the feasibility of achieving proficient agent behaviors in simulation environments before transfer back to highly complex, non-parallelizable games such as Rocket League. Beyond practical application, the theoretical implications manifest in leveraging sim-to-sim transfer strategies to bridge learning endeavors within virtual domains.

The paper anticipates the future of AI research to consider even larger scope challenges, such as multi-agent collaborative or competitive tasks in Rocket League. The introduction of partial observability and enhancements in handling dynamic, hybrid action spaces are emphasized as areas ripe for exploration. Additionally, the paper suggests the potential utility of curriculum learning and domain randomization as prospective routes to refine simulation fidelity and agent performance in more complex learning environments.

In essence, the paper adds a pivotal layer to the comprehension and capabilities of DRL within constrained simulation settings. It illuminates the trajectory of AI towards encompassing intricate virtual environments and contributing further to reinforcing manageable and scalable agents within computationally intensive gameplay scenarios.