- The paper demonstrates that PPO self-play with sparse win/loss rewards, a spawn-distance curriculum, parameter EMA, and top-advantage filtering can produce superhuman Generals.io play without demonstrations, reward shaping, or population-based training.
- The agent reached the public #1 leaderboard after winning 815 of 1,000 ladder games and defeated the two top-ranked humans 199–70 overall, while beating the prior AI state of the art 20–0.
- The released JAX-native simulator reaches up to 50.7 million environment steps per second on one H200, showing that extreme simulation throughput can replace specialized training machinery by making terminal feedback sufficiently abundant.
Overview
This paper presents a superhuman agent for Generals.io, a real-time strategy game combining long-horizon planning, short-term tactics, and imperfect information under fog of war. The agent, trained for four days on 4× NVIDIA H200 GPUs using self-play reinforcement learning with sparse win/loss reward, reaches the #1 position on the public 1v1 leaderboard of over 5,000 human players and defeats the two top-ranked humans head-to-head with a combined 199–70 record across 269 ladder matches. The central claim is that a generic policy-gradient loop—without behavior cloning, reward shaping, or population-based outer loops—is sufficient at scale once the data bottleneck is removed by an extremely fast simulator. The paper also releases a JAX-native environment supporting 1v1, 2v2, and free-for-all modes through a shared interface.
The JAX-native simulator
The principal engineering contribution is a full reimplementation of Generals.io in JAX, with observations, rewards, fog-of-war masking, army production, combat resolution, and curriculum logic expressed as pure functions over a flat PyTree, jit-compiled end-to-end on GPU. Peak throughput is 50.7M environment steps per second on a single H200 (13.5M with 1,024 parallel environments), roughly four orders of magnitude above the prior NumPy-based CPU baseline (~3,500 steps/s) and a 32× speedup of the full training loop on identical hardware. This throughput is what makes the paper's methodological claims testable: enough games terminate per unit time that terminal win/loss signal alone can drive learning.
The action space is fixed at H×W×9 across all modes (pass, or send-all/send-half in each cardinal direction from any owned cell), and the observation is a 38-channel spatial tensor plus persistent memory channels and per-cell army-delta history.
Agent design
The policy is a 15.35M-parameter vision transformer: the spatial tensor is split into 3×3 patches embedded as tokens, augmented with two non-spatial tokens encoding 512-step sliding windows of the opponent's total army and land counts—allowing inference about opponent behavior behind the fog via scoreboard dynamics. A value token feeds an HL-Gauss categorical value head; the policy head projects grid-cell tokens into the 9-way action distribution.
Training uses PPO with an entropy bonus and HL-Gauss value loss, γ=1, GAE λ=0.9. Two ingredients stand out:
- Spawn-distance curriculum: training directly at the full minimum general separation of 17 BFS steps fails because a random policy almost never captures the opponent's base; the curriculum caps spawn distance at four cells and gradually raises it.
- Parameter EMA: gradients act on θ, but deployment evaluates the exponential moving average θˉt=τθˉt−1+(1−τ)θt with τ=0.999.
Notably, the authors omit the KL trust-region term toward the previous iterate that would recover magnetic mirror descent, saving one forward pass per minibatch with no visible instability—a deliberate deviation from the MMD recipe used for superhuman Stratego play (Sokota et al., 10 Nov 2025).
Evaluation against humans and prior bots
On the public 1v1 ladder, the agent won 815 of its first 1,000 rated games (81.5%) to finish as the top-rated player at 118 OpenSkill points, leading the strongest active human (101 points) by 17 points—the same margin separating that human from the player ranked 25th—and leading the prior learned state of the art, zero v3, by 44 points. Head-to-head results:
| Opponent |
W |
L |
Win rate |
| shimatetsu (rank-1 human) |
27 |
12 |
69.2% |
| Mithraaaa (rank-2 human) |
172 |
58 |
74.8% |
| Human.exe (strongest non-learning bot) |
20 |
0 |
100.0% |
| zero v3 (prior AI SOTA) |
20 |
0 |
100.0% |
The 199–70 record against the two top humans corresponds to p<0.001 on a one-sided binomial test. The matches were ordinary rated ladder games; both sides knew the opponent was a bot, and the agent had no interface advantage—one action per 0.5-second tick, same client view—with all replays public. The prior state of the art, zero v3, required behavior cloning from expert replays, potential-based reward shaping, and population-based self-play to reach only the top 25; this agent surpasses it decisively without any of those components.
Ablations
Reward shaping hurts at high throughput. Potential-based reward shaping, essential for zero v3 under tight compute, destabilizes late-stage training here, while sparse win/loss reward converges cleanly. The implication is direct: shaping was compensating for data scarcity, not adding information, and its bias toward material-rich states becomes harmful once sufficient terminal signal exists.
EMA consistently beats the last iterate. Across every development experiment, the EMA parameters outperformed the final iterate at inference, still leading by roughly 30 Elo after six days of training averaged over three runs. The authors flag this as striking given that parameter EMA is not identified as load-bearing in prior large-scale self-play work on StarCraft II, Dota 2, or Stratego, and leave open whether it is a generic stabilizer or regime-specific.
Top-advantage filtering improves efficiency. Retaining only the top 25% of transitions per rollout batch by critic-predicted advantage yields better wall-clock and sample efficiency than larger retention fractions. The 25% setting has the highest seed variance on a single GPU, but its best seed leads the next-best setting by 112 Elo after 24 hours; scaling from one to four GPUs stabilized it while preserving the relative advantage. The sample-efficiency gain indicates many collected samples are redundant—an observation the authors suggest motivates more principled data filtering.
Schedule robustness contradicts prior claims. Sokota et al. report that specific learning-rate and entropy-coefficient annealing schedules were critical for smooth training in Stratego. Here, combinations of power-law and linear decays for both schedules reach comparable local Elo, suggesting schedule range matters more than shape (annealing the learning rate from 32×0 to 32×1 was safe). The authors concede computational constraints prevented longer-term sensitivity experiments.
Interpretability. Most attention heads are not cleanly interpretable, and their apparent function shifts at qualitative game-state changes. One exception is head 4 of layer 6, whose value-token attention over spatial patches tracks a coherent posterior over the opponent's general location: a map-generation prior before contact, progressive narrowing through elimination reasoning ("the opponent has not attacked, so they cannot be at the closer candidates"), and collapse onto the true tile upon direct observation. This pattern recurs across many games.
Limitations and open questions
The evidence base for several ablations is thin: EMA comparisons average three runs, and the top-advantage result rests partly on best-seed performance whose variance the authors attribute to single-GPU training without fully resolving. The schedule-robustness finding explicitly lacks a broader sensitivity analysis. The KL trust-region omission is justified only by absence of visible degradation, not measurement. All headline results are confined to two-player zero-sum 1v1 play; the 2v2 and free-for-all modes are supported by the environment but no agent results are reported there, so whether the recipe transfers to team and general-sum settings remains untested. Finally, the mechanism behind parameter EMA's benefit in self-play RL is unexplained.
Conclusion
The paper demonstrates that superhuman play in a large partially observable real-time strategy game is achievable with a plain PPO self-play loop, sparse terminal reward, a spawn-distance curriculum, parameter EMA, and top-advantage sample filtering—no demonstrations, shaping, or population machinery—at modest compute cost enabled by a ~10,000× faster simulator. Its ablations sharpen the emerging picture from small-benchmark studies and Stratego that well-tuned policy gradient suffices where specialized game-theoretic pipelines were thought necessary, while its schedule-robustness result qualifies claims that precise annealing is critical. The released environment extends the testbed to cooperative and general-sum multi-agent settings that remain unexplored in this work.