FootsiesGym: A 2D Fighting Game RL Benchmark

• FootsiesGym is an open-source, two-player zero-sum reinforcement learning environment that simulates cyclic, non-transitive fighting game dynamics.
• It leverages vectorized headless simulation and efficient parallelism to enable reproducible experiments on strategic, adversarial learning without high computational demand.
• The benchmark highlights critical game-theoretic aspects like action delay, observation asymmetry, and policy exploitability through controlled experiments with PPO, EMAgnet, and PFSP.

FootsiesGym is an open-source reinforcement-learning benchmark built around HiFight’s minimalist 2D fighting game Footsies. It is designed as a tractable but strategically non-trivial environment for learning in a two-player, zero-sum, imperfect-information game, with particular emphasis on the cyclic, non-transitive interactions of fighting-game neutral play. The benchmark is intended to occupy a middle ground between analytically clean but overly simple small games and computationally demanding large real-time environments, while remaining accessible, reproducible, and efficient through vectorized headless simulation on standard hardware (McDonald et al., 7 Jul 2026).

1. Concept and benchmark rationale

FootsiesGym is positioned as a research environment for studying competitive multi-agent learning under conditions where strategic robustness matters more than absolute reward maximization against a fixed environment. Its stated purpose is to preserve the real-time, spatial, and imperfect-information structure of competitive games while stripping away enough extraneous complexity to enable efficient analysis and reproducible experimentation (McDonald et al., 7 Jul 2026).

The environment is built on HiFight’s open-source Footsies and isolates the “neutral game,” the phase in which neither player has secured a decisive advantage and both are engaged in maneuvering, spacing, baiting, and opening creation. In this formulation, the benchmark highlights cyclic and non-transitive strategic interactions analogous to rock-paper-scissors-like dynamics: no single option dominates globally, overcommitment is exploitable, and mixed-strategy behavior is consequential. This strategic profile makes the environment suitable for self-play, exploitability testing, and algorithmic comparison in game-theoretic RL (McDonald et al., 7 Jul 2026).

A central motivation is methodological. Small matrix and poker-style benchmarks are described as analytically clean but often too simple and short-horizon, whereas large real-time games are rich but expensive and difficult to reproduce. FootsiesGym is therefore presented as a “middle ground” that retains meaningful adversarial structure without requiring large-scale infrastructure (McDonald et al., 7 Jul 2026).

2. Formal game structure and core mechanics

FootsiesGym is formulated as a two-player zero-sum imperfect-information game. Rewards satisfy

$r^i = -r^{-i},$

where $i$ denotes the current player and $-i$ the opponent. The default terminal reward is a win/loss payoff of $\pm w$ with $w=10$, and a timeout yields zero reward for both players (McDonald et al., 7 Jul 2026).

The benchmark simulates the core mechanics of Footsies while excluding much of the complexity characteristic of commercial fighting games. The resulting design emphasizes neutral-game decision-making rather than combo-heavy execution. Several mechanics are explicitly highlighted in the environment description:

• There is no health bar in the usual sense; a round is lost when a player is hit by a special attack.
• Players can be blocked up to three times, tracked by a guard bar.
• After three blocks, attacks cause a guard break, leaving the player open to follow-up pressure.
• Players can dash forward or backward by double-pressing direction.
• Special attacks require charging attack for 60 frames and then releasing, optionally with direction (McDonald et al., 7 Jul 2026).

These mechanics preserve timing, spatial control, and tactical prediction, but reduce the extent to which long combo execution dominates outcomes. This suggests that the benchmark is intended to foreground strategic prediction and equilibrium-related phenomena rather than motor complexity.

A further parameter of theoretical importance is action delay, defined as the number of game frames between action selection and in-game execution. The paper states that at 0 delay, policies can become degenerate and purely reactive; as delay increases, more actions become unreactable; and at high delay, play can become near-random because response timing is too delayed. Action delay therefore functions as a controllable axis for shifting the environment’s emphasis between reflexive reaction and prediction (McDonald et al., 7 Jul 2026).

3. Action space, observations, and information asymmetry

By default, each agent operates in a discrete action space of size 6,

$|\mathcal{A}| = 6,$

with the actions: none, back, forward, attack, back+attack, and forward+attack. The environment can optionally enable a larger action space of size 9,

$|\mathcal{A}| = 9,$

by adding three special-charge actions: back-charge, neutral-charge, and forward-charge. These actions toggle the held-attack charge state required for special moves; the first press starts charging and the second press releases and executes the corresponding special move. This option is included because random exploration rarely discovers the charge-based attack sequence reliably (McDonald et al., 7 Jul 2026).

Observation design is explicitly asymmetric. Each agent receives an 85-dimensional observation by default, or 88-dimensional observations with special-charge enabled. The observation contains a common block, a shared block, and a privileged block. The common block includes inter-player distance. The shared block contains state features for self and opponent, including x-position, velocity, dead/alive flag, vital health, guard health, current action ID, action frame counters, hit stun, sprite shake, facing direction, and frame advantage. The privileged block contains self-only variables: whether the next input would cause a forward dash, whether the next input would cause a backward dash, special-attack charge progress, previous action, and whether special charge is currently held (McDonald et al., 7 Jul 2026).

This structure operationalizes imperfect information without hiding the entire state. Both players observe shared features, but each also receives a privileged self-information block that is not mirrored to the opponent. The paper compares this asymmetry to human fighting-game play, in which some opponent timing variables are not directly observable, particularly hidden internal state such as charge progress (McDonald et al., 7 Jul 2026).

Optional dense shaping is supported, including guard-break reward, but the main reported results use the default zero-sum reward design. This separation between baseline reward structure and optional shaping is important for benchmarking because it distinguishes the core task from auxiliary training aids (McDonald et al., 7 Jul 2026).

4. Simulator architecture and computational characteristics

FootsiesGym’s implementation is built in Unity with game logic in C#, but training runs headlessly without rendering. The simulator is thus decoupled from the visual game loop, enabling high-throughput training use cases (McDonald et al., 7 Jul 2026).

Its principal efficiency contribution is a vectorized simulator implemented entirely on the C# side. One headless process can step $K$ independent game instances in parallel, those instances are exposed through a single gRPC port, and Python clients access them through the PettingZoo API. The system therefore provides two distinct layers of parallelism: within-process parallelism, in which one server runs many environments concurrently, and across-process parallelism, in which multiple independent game server processes are stepped concurrently from Python threads (McDonald et al., 7 Jul 2026).

The reported throughput numbers are a central part of the benchmark’s practical identity. The paper reports roughly 19,000 environment steps per second at the single-process plateau and up to about 52,500 environment steps per second at $P=4$ processes and $K=128$ in-process environments, described as roughly 2.7 times the single-process peak. The environment also auto-resets terminated instances and uses headless optimizations including uncapped frame rate, disabled audio, and no rendering (McDonald et al., 7 Jul 2026).

These implementation choices are significant because they make the benchmark usable on standard workstation hardware rather than specialized clusters. A plausible implication is that the environment is intended not only for algorithmic comparison but also for experimental iteration, ablation work, and reproducible academic workflows where computational accessibility is a constraint.

5. Benchmarked algorithms and training protocol

The benchmark evaluates three main algorithmic families: PPO, EMAgnet, and Prioritized Fictitious Self-Play (PFSP) (McDonald et al., 7 Jul 2026).

Two PPO variants are used. “PPO (Sched.)” employs an entropy coefficient that starts high and anneals to zero, whereas “PPO” uses a fixed entropy coefficient. The paper motivates this design choice by reference to recent findings that carefully tuned entropy scheduling can be unexpectedly strong in cyclic zero-sum imperfect-information games (McDonald et al., 7 Jul 2026).

EMAgnet is described as a PPO variant regularized toward an exponential moving average of its own parameters. Its loss is given as

$i$0

with magnet parameters updated by

$i$1

The method is intended as a smoother form of policy regularization than entropy alone (McDonald et al., 7 Jul 2026).

PFSP is implemented with an opponent-sampling rule that favors snapshots against which the current policy struggles. Snapshot $i$2 is sampled with weight

$i$3

where $i$4 is the estimated win rate of the current policy against that snapshot. The implementation uses a bounded reservoir of past checkpoints and self-play with probability $i$5, with the remaining probability mass allocated to prioritized snapshots (McDonald et al., 7 Jul 2026).

All algorithms share a common training setup: a 2-layer MLP with 256 hidden units per layer and ReLU activations, 48 parallel game instances, 64-step rollouts, 8 update epochs, 8 minibatches per epoch, $i$6, $i$7, and PPO clip $i$8. Training length is

$i$9

and five random seeds are used for each algorithm (McDonald et al., 7 Jul 2026).

6. Empirical findings, interpretive value, and limitations

The reported results indicate that all benchmarked methods learn to defeat weak heuristic opponents. Against uniform random and no-op opponents, all policies improve substantially and reach roughly 85–95% win rates against random. At the same time, the paper notes a behavioral tendency: as training progresses, policies become increasingly reactive and less willing to initiate engagements, especially against a passive no-op opponent. This observation is used to argue that optimizing for win rate alone can produce strategically effective but potentially undesirable behavior (McDonald et al., 7 Jul 2026).

A major empirical theme is the distinction between head-to-head performance and exploitability. Because exact exploitability is intractable at this scale, the paper estimates strategic robustness by training approximate best responses against each learned policy. Under this procedure, PPO and PPO (Sched.) are reported as roughly similar in exploitability, EMAgnet performs worst under best-response attack, and PFSP exhibits a relatively high tie rate and is somewhat stronger in some pairwise comparisons. The paper highlights EMAgnet as an instructive case: it achieves positive head-to-head return against both PPO variants yet is also the most consistently exploited. This establishes FootsiesGym as a benchmark in which win rate and strategic robustness diverge in a measurable way (McDonald et al., 7 Jul 2026).

Another notable result concerns the discovery of rare but strategically important actions. The special attack B_SPECIAL is reported as rarely learned under standard training because it requires a long charge sequence, random exploration almost never discovers it, its success depends on opponent behavior, and entropy regularization can make reliable deployment even harder. B_SPECIAL usage is near zero across algorithms, with only a brief spike for fixed-entropy PPO. When the special-charge action mode is enabled, B_SPECIAL usage increases substantially without hurting performance against a random opponent (McDonald et al., 7 Jul 2026).

The benchmark’s broader significance follows from these results. It is proposed as a vehicle for studying game-theoretic learning, population-based training, policy regularization, exploitability estimation, play-style quality, human-like behavior, imitation learning, and human evaluation. Because the environment is built in Unity, it can also be exported to WebGL and used in-browser for human studies and data collection (McDonald et al., 7 Jul 2026).

The paper also states several limitations. Footsies remains much simpler than commercial fighting games; exploitability estimates are only approximate because they are derived from trained best responses and therefore lower-bound true exploitability; the baselines were “naïvely tuned,” so the results are not definitive algorithm rankings; hard-to-discover strategies may be suppressed by stabilizing regularization; and stronger policies may become overly passive or reactive, underscoring that skill is not identical to desirable behavior (McDonald et al., 7 Jul 2026).

Taken together, these characteristics define FootsiesGym as a benchmark whose value lies not merely in efficient self-play training, but in exposing tensions between reactivity and prediction, head-to-head strength and exploitability, and policy stability and strategic diversity. This suggests a research role broader than leaderboard comparison: FootsiesGym functions as an instrument for analyzing the structure of adversarial learning in compact real-time imperfect-information games.