R2D-RL: A RoboCup 2D Soccer Environment for Multi-Agent Reinforcement Learning

Abstract: Robot soccer is a challenging testbed for multi-agent reinforcement learning because it combines partial observability, cooperative and adversarial interaction, sparse rewards, and long-horizon tactical behavior. RoboCup 2D Soccer Simulation (RCSS2D) provides a mature robot-soccer platform, but its competition-oriented server-client architecture is difficult to use directly with modern Python-based MARL workflows. We introduce R2D-RL, a reinforcement learning environment that connects RCSS2D and HELIOS-based player clients to a Python MARL interface through shared-memory communication and cycle-level synchronization. R2D-RL supports full-field and scenario-based training with configurable opponents, Base discrete and Hybrid parameterized action spaces, action masks, expected possession value (EPV)-based reward shaping, and parallel execution. We provide front-goal scenarios and an 11-vs-11 full-field benchmark, together with baseline results.

• The paper introduces a step-synchronous MARL environment that preserves full RoboCup simulation semantics and integrates centralized state with local observations.
• It demonstrates robust parallel sampling—reaching up to 41.65M steps per day—and emphasizes the critical role of action masks for valid action selection.
• Through progressive scenario evaluations and full-field benchmarks, the study highlights the importance of reward shaping and synchronized control for tactical multi-agent coordination.

R2D-RL: A Step-Synchronous Multi-Agent RL Environment for 2D RoboCup Soccer

Motivation and Ecosystem Positioning

R2D-RL addresses the deficit of reinforcement-learning environments in robot soccer with step-synchronized multi-agent interactions, bridging the RoboCup 2D Soccer Simulation (RCSS2D) and the HELIOS base client ecosystem with a Python-based MARL interface. Unlike prior RCSS2D-derived platforms with hardcoded subtask environments (e.g., Keepaway, HFO), R2D-RL retains the full competition workflow semantics, exposes centralized state and local observations, and introduces action masks and scenario control for high-fidelity MARL benchmarking.

Figure 1: Positioning of R2D-RL in the RoboCup 2D ecosystem.

System Architecture and Synchronization Protocol

The environment implements shared-memory synchronization across the soccer server, modified HELIOS clients, and the Python MARL layer. Step synchronous communication is organized via sequence counters, guaranteeing that policy-driven actions, state transitions, and reward feedback are consistently mapped one-to-one with RCSS2D simulator cycles in the play_on phase. Scenario resets are efficiently handled in situ by the trainer, avoiding server restarts and enabling high-throughput parallel rollouts.

Figure 2: R2D-RL architecture and one-step synchronization, showing shared-memory exchange of actions, observations, masks, state, and reset signals between HELIOS clients and the Python MARL environment.

Parallel Sampling Throughput

R2D-RL demonstrates efficient scaling for parallel environment sampling, crucial for multi-agent exploration and learning. Throughput peaks at 14 parallel instances under a 32-CPU configuration, reaching up to 41.65M environment steps per day in the Hybrid action space. Beyond this, synchronization overhead and CPU resource saturation constrain additional scaling.

Figure 3: Parallel sampling throughput in 11-vs-11 full matches for Base and Hybrid action spaces.

Environment Semantics and Action Abstractions

R2D-RL exposes two action spaces:

• Base (HELIOS-derived): High-level discrete actions (e.g., pass, shoot, tackle, directional dribble/move) mapped to domain-specific behavior modules, supplemented by state-dependent action masks ensuring valid and semantically consistent decision points.
• Hybrid: Parameterized low-level commands (turn, dash, kick, catch) with select continuous parameters. HELIOS Fallback and empty are restricted to internal use.

Centralized state and local observations, episode control (goal, out-of-play, possession loss, timeout), reward shaping via MaxEPV grids, and scenario reset are fully supported. Episode steps are strictly aligned to synchronized play_on cycles.

Scenario-Based Evaluation

Front-Goal Scenarios

Five progressive front-goal scenarios are introduced, varying attacker/defender counts and tactical complexity, from solo finishing to compact defense coordination. Task complexity increases from basic ball convergence and finishing to coordinated multi-agent passing and attack under defensive pressure.

Figure 4: Initial frames for the five front-goal scenarios illustrating attacker/defender configurations.

Single-Attacker Results

Strong trainability is observed across all methods (MAPPO, QMIX, ParaDQN) in the Empty Goal scenario. In Blocked Shot, action masks are essential; QMIX fails without them, MAPPO is less sensitive, and ParaDQN achieves maximum $0.28\pm0.07$ only with MaxEPV shaping and masking. Action masks enforce valid action switches (e.g., shoot priority) under sparse tactical feedback.

Figure 5: Single-attacker front-goal scenario results, showing goal rates across reward/mask configurations.

Multi-Attacker Results

Difficulty increases substantially in multi-attacker (Support Option, Passing Lane, Compact Defense) scenarios. MAPPO attains the highest nonzero goal rates, while QMIX is highly seed-sensitive and ParaDQN is nearly nonviable. Without MaxEPV shaping and action masks, learning collapses in all multi-attacker settings, clearly demonstrating the necessity of environment-level reward shaping and valid action filtering for credit assignment and spatial-temporal coordination.

Figure 6: Multi-attacker front-goal scenario results with component-wise ablations on reward shaping and action masking.

Full-Field 11-vs-11 Benchmarks

Full-field experiments validate R2D-RL end-to-end synchronization and robust evaluation under complete RoboCup match dynamics. Policies are trained for 30M steps, with MAPPO outperforming QMIX and ParaDQN on both goal difference and MaxEPV attainment. However, all methods exhibit significant performance gaps compared to rule-based teams, especially in extended spatial coordination and off-ball movement. Qualitative analysis reveals emergent passing and shooting, but also fixed-direction movements and deficient goalkeeper-specific behaviors.

Practical and Theoretical Implications

• Action abstraction and masking: High-level soccer-primitive actions and dynamic masks improve convergence and feasibility in complex, credit-assignment-heavy domains.
• Reward shaping via MaxEPV: Integration of spatially grounded shaping rewards mitigates sparsity and enables tactical acceleration in long-horizon tasks.
• Synchronization protocols: Step-level cycle synchronization is a necessary architectural guarantee for MARL environments in distributed agent-control domains.
• Scenario resets and parallelization: Efficient in-place scenario control is critical for throughput and reproducibility in MARL benchmarking.

R2D-RL sets a new reproducible standard for robot soccer RL, compatible with the large RCSS2D ecosystem and HELIOS tactical modules, and lays the groundwork for curriculum learning, self-play, adaptive opponent modeling, and less hand-written action abstractions in future research.

Conclusion

R2D-RL delivers a robust, synchronized MARL environment for RoboCup 2D soccer simulation, supporting high-throughput parallel sampling, scenario and full-match tasking, centralized state, action masking, and reward shaping. Progressive scenario evaluation demonstrates its efficacy across increasing tactical complexity, with empirical data highlighting the critical role of action masks and spatial reward shaping. Theoretical and practical advances suggest promising directions for curriculum-based learning strategies, self-play, opponent diversification, and reduced reliance on hand-crafted action abstractions in robot soccer MARL.