Honor of Kings Arena: MOBA RL Benchmark

• Honor of Kings Arena is a reinforcement learning environment built on MOBA mechanics, featuring partial observability and high-dimensional action spaces.
• The platform supports advanced RL methods like PPO and DQN, facilitating studies in policy distillation, transfer learning, and multi-task learning.
• Empirical results demonstrate that scaled parallelization can achieve 100% win rates against behavior-tree baselines, highlighting efficiency and generalization.

Honor of Kings Arena is a reinforcement learning (RL) environment and evaluation suite based on the commercial Multiplayer Online Battle Arena (MOBA) game Honor of Kings. The platform is designed to foster research on competitive and multi-agent RL, with a particular emphasis on generalization, scalability, and high-dimensional control under partial observability. The released environment offers fine-grained control, flexible observation and reward design, and provides interfaces suitable for state-of-the-art RL methods. Honor of Kings Arena has become a critical benchmark for multi-agent RL and competitive deep learning systems, enabling rigorous study of sample efficiency, transfer, policy distillation, and multi-task learning within MOBA-style games (Wei et al., 2022).

1. Environment Specification and API

Honor of Kings Arena implements a symmetric, partially observable, 1v1 multi-agent system directly inspired by the mechanics of the commercial Honor of Kings MOBA. Each agent controls one hero and interacts at 133 ms intervals (the standard game frame rate), issuing actions to move, attack, cast up to four skills, recall, use items, and select fine-grained spatial or unit targets. Episodes conclude with base destruction (win/loss).

Observation Space: Each agent receives a set of structured NumPy arrays at every step:

• HeroStatePublic for both heroes (≈49×2 dims): status, position, HP, MP, cooldowns, etc.
• HeroStatePrivate (≈100 dims): hero-internal skills, buffs, and state flags.
• VecCreeps and VecTurrets (4×(12+6) dims each): closest neutral and structure entities.
• VecCampsWholeInfo (5 dims): time period, global map status.

Raw observable dimensionality is ≈347, all subject to visibility constraints (partial observability). Agents only perceive units within their vision range, requiring decision-making under significant uncertainty (Wei et al., 2022).

Action Space: Each action is a tuple:

• (Button, Move_X, Move_Z, Skill_X, Skill_Z, Target), with dependency masks for legality.
• Buttons: {None, Move, Attack, Skill1, Skill2, Skill3, (Skill4), Heal, Summoner, Recall, ItemSkill}, ≈11 choices.
• Move_X, Move_Z and Skill_X, Skill_Z: 16 discretized bins each, for spatial and skill aiming.
• Target: {None, Self, EnemyHero, up to 4 Creeps, Nearest Tower}.

A Boolean legal action mask is provided for every step to prune infeasible decisions. This compositional hierarchy enables sophisticated control over high-dimensional actions required by modern MOBA games (Wei et al., 2022).

The open-source environment (https://github.com/tencent-ailab/hok_env) exposes a Python API for full episode control, reset, action stepping, and auxiliary information (legal actions, game state, and custom logging) (Wei et al., 2022).

2. Reward Structure and Benchmarking Protocols

Reward design in Honor of Kings Arena is both flexible and critical for RL research. Standard reward functions combine both dense and sparse signals: $$R_t = w_{\mathrm{HP}}\cdot\Delta\mathrm{HP}_t + w_{\mathrm{tower}}\cdot\Delta\mathrm{TowerHP}_t + w_{\mathrm{gold}}\cdot\Delta\mathrm{Gold}_t + w_{\mathrm{exp}}\cdot\Delta\mathrm{Exp}_t + w_{\mathrm{kill}}\cdot\mathbb{I}_{\{\mathrm{kill}_t\}} + w_{\mathrm{death}}\cdot\mathbb{I}_{\{\mathrm{death}_t\}} + w_{\mathrm{win}}\cdot\mathbb{I}_{\{\mathrm{terminal=win}\}} + w_{\mathrm{lose}}\cdot\mathbb{I}_{\{\mathrm{terminal=lose}\}}$$ with default weights: $w_{\mathrm{HP}}=2.0$, $w_{\mathrm{tower}}=10.0$, $w_{\mathrm{gold}}=0.006$, $w_{\mathrm{exp}}=0.006$, $w_{\mathrm{kill}}=+0.6$, $w_{\mathrm{death}}=-1.0$, $w_{\mathrm{win}}=+10$, $w_{\mathrm{lose}}=-10$ (Wei et al., 2022).

Benchmark Protocols:

• Resource scaling studies: RL (PPO) can reach 100% win rate against a Behavior-Tree (BT) baseline within ≈2 hours when >512 CPU rollout workers are used.
• DQN requires ≈1.5x more samples and plateaus lower, highlighting the need for on-policy or actor-critic methods (Wei et al., 2022).
• Baseline agent performance is measured by aggregated win rate, average episode length, and cumulative in-game statistics over large samples.

3. Generalization and Multi-Task Learning Challenges

The environment is constructed to evaluate policy transfer and generalization across two axes:

• Across Opponent Heroes: agent is trained to play a fixed hero against multiple distinct opposing heroes.
• Across Controlled Targets: agent must generalize its policy when controlling different heroes against the same or varying opponents.

For a set $H$ of 20 released heroes, there are 400 possible tasks ($w_{\mathrm{HP}}=2.0$0); results are reported as averaged win rates across held-out pairs (Wei et al., 2022). Direct transfer from single-task policies yields <20% win rate on new tasks, exposing pronounced generalization gaps. Explicit multi-task training or policy distillation from multiple “teacher” agents raises cross-task win rate to 50–60%. Domain randomization over opponent hero identities further improves zero-shot robustness (Wei et al., 2022).

4. RL Algorithms and Technical Solutions

Honor of Kings Arena supports and motivates the study of high-performance RL algorithms capable of managing large action/state spaces and partial observability:

• DQN (Ape-X style off-policy replay): experiences are prioritized; target networks and ε-greedy exploration are standard.
• PPO (clipped policy gradient, GAE): state-of-the-art sample efficiency with hyperparameters tuned to the environment ($w_{\mathrm{HP}}=2.0$1, $w_{\mathrm{HP}}=2.0$2, $w_{\mathrm{HP}}=2.0$3, $w_{\mathrm{HP}}=2.0$4).
• Self-Play: Each episode samples opponent policies from a moving pool, encouraging continual learning and opponent adaptation (Wei et al., 2022).

The environment permits elaborate custom reward specification, full masking of legal actions, and frame-accurate control, making it a suitable testbed for policy distillation, transfer learning, and modular multi-agent control (Wei et al., 2022, Wei et al., 2021).

5. Empirical Results and Resource Requirements

Resource scaling studies demonstrate efficient parallelization: with 2048 CPU rollout workers, PPO-based agents master BT opponents in under one hour. PPO achieves 100% win rate against BT after 2×10⁹ environmental samples; DQN requires ≈3×10⁹ samples and underperforms on final win rate (Wei et al., 2022).

Resource scaling table (PPO vs BT baseline):

CPU Cores	Time to Beat BT Baseline
128	≈ 6.2 hours
256	≈ 1.7 hours
512	≈ 1.08 hours
2048	≈ 0.90 hours

Multi-task and policy distillation remedies substantially improve transfer and zero-shot generalization, and have become the focus of ongoing research using the platform (Wei et al., 2022).

6. Significance in the Multi-Agent RL Landscape

Honor of Kings Arena addresses generalization, scalability, and adversarial co-evolution, distinguishing it from previous environments like Atari, MuJoCo, and board-game simulators. Its compositional, partially observable state and high-dimensional action space instantiate many of the unresolved challenges in competitive RL, including:

• Sample efficiency in long-horizon, sparse-reward settings
• Credit assignment under adversarial nonstationarity
• Multi-agent transfer, curriculum, and distilled generalization

It has already facilitated research in action-conditioned offline RL (AQL) (Wei et al., 2021), curriculum learning, and large-scale policy distillation. The environment's open-source release and public leaderboard allow broad participation in reproducible, high-impact competitive RL research. Its architecture and features have influenced state representations, action space design, and generalization protocol in subsequent RL environments (Li et al., 3 Sep 2025).

7. Limitations and Future Directions

Honor of Kings Arena is currently limited to 1v1 play and 20 heroes, whereas the full commercial Honor of Kings game includes 5v5 matches and >100 heroes. Full multi-agent (5v5) and expanded hero pool settings remain an open challenge, as do issues of multi-agent credit assignment, long-term coordination, and partial observability. Ongoing work targets more sophisticated MARL benchmarks incorporating curriculum, larger hero sets, richer reward signals, and model-based planning components. Further integration with cloud training and online evaluation suites is anticipated (Wei et al., 2022, Li et al., 3 Sep 2025).