PokeRL: Modular RL Benchmark for Pokémon Red

• PokeRL is a modular reinforcement-learning environment for Pokémon Red that integrates custom reward schemes and control adjustments.
• It wraps the PyBoy emulator in a Gymnasium interface, leveraging RAM-based event detection, map-aware observations, and anti-loop/spam mechanisms.
• Empirical results show improved exploration efficiency and a reduction in loop episodes, validating its practical benefits as an early-game RL benchmark.

PokeRL is a modular, emulator-based reinforcement-learning system for early-game Pokémon Red that was introduced to make the game more usable as a stable RL benchmark. Rather than proposing a new RL algorithm, it packages a custom Gymnasium environment wrapped around the PyBoy emulator, RAM-based event detection, map-aware observations, action-space restrictions, anti-loop and anti-spam controls, and a dense hierarchical reward scheme into a practical training stack for tasks such as exiting the player’s house, reaching tall grass, and winning the first rival battle (Mudireddy et al., 12 Apr 2026).

1. Benchmark setting and design rationale

PokeRL is motivated by the claim that Pokémon Red is difficult precisely because it is not a clean arcade-style control problem. Progress in the opening minutes already requires long-horizon navigation, interaction, scripted events, and turn-based combat under sparse rewards, partial observability, awkward game-specific controls, and hidden state variables such as HP, map ID, and party status. The paper emphasizes a particularly idiosyncratic control issue: directional movement uses a “turn first, then move” double-press mechanic, so a naïve action interface can cause an agent to spend much of its time turning in place rather than walking (Mudireddy et al., 12 Apr 2026).

The system is explicitly organized around failure modes observed in straightforward PPO training. These include infinite loops, where the agent paces between a few tiles to harvest shaping rewards; reward-instability from sparsity; button spam, especially on A, Start, Select, or no-op; incorrect movement semantics; and memoryless exploration under pure pixel observations, which leads to repeated revisiting of already seen areas. PokeRL therefore treats loop detection, spam mitigation, movement semantics, and exploration memory as first-class engineering concerns rather than incidental implementation details.

This framing is central to the project’s significance. PokeRL argues that practical systems which explicitly model such pathologies are a necessary intermediate step between toy benchmarks and agents that could eventually handle the broader opening of the game or later milestones.

2. Environment architecture and state-action interface

PokeRL is implemented as a custom Gymnasium environment wrapped around the PyBoy Game Boy emulator. Around that core it adds modules for memory reading, reward computation, loop and spam mitigation, and curriculum control. The code repository is given as https://github.com/reddheeraj/PokemonRL. The environment uses PyBoy to run Pokémon Red, reads selected RAM locations for event detection and termination logic, but does not expose those memory values directly to the policy (Mudireddy et al., 12 Apr 2026).

The observation pipeline combines raw visual input with engineered spatial memory. The emulator screen is downsampled to 72×80 grayscale. PokeRL then stacks four consecutive grayscale frames together with four corresponding visited-mask frames, producing an 8-channel input for a Stable-Baselines3 CnnPolicy actor-critic. The visited mask is maintained per map and centered on the map’s entry position so that it stays aligned to global map coordinates as the camera scrolls. The policy network consists of three convolutional layers, followed by a fully connected layer of 512 units, and outputs both policy logits and a scalar value estimate.

The action space is restricted to seven discrete actions: up, down, left, right, A, B, and no-op. Start and Select are removed because they are especially easy to exploit uselessly. To match the game’s movement semantics, every directional action is implemented as two press-release cycles, so one RL action corresponds to one intended grid-world step.

The RAM-reading layer is used only for environment logic. The paper lists the following addresses:

RAM address	Variable	Environment use
0xD361	player Y position	position/event detection
0xD362	player X position	position/event detection
0xD35E	map ID	map transitions/milestones
0xD163	party count	party-state checks
0xD057	battle state	battle detection
0xD16C	Pokémon HP	battle rewards/termination

This design is not fully end-to-end from pixels alone. The paper explicitly acknowledges that limitation and defends it as a practical compromise for a “messy” game environment.

3. Loop-aware control and anti-spam mechanisms

A major contribution of PokeRL is its loop-aware wrapper and multi-layer anti-loop system. The anti-loop mechanism has three parts. First, it tracks visits to each $(x,y)$ position within an episode: once a tile is visited more than three times, the agent receives a small penalty, and beyond five visits the penalty becomes larger. Second, it performs action-pattern detection over a sliding window of the last 20 actions and penalizes repetitive patterns such as alternating fixed button cycles; breaking out of such a cycle can produce a small positive bonus. Third, it performs position-loop detection by checking whether recent position history repeatedly returns to a small neighborhood, indicating circular wandering or local pacing (Mudireddy et al., 12 Apr 2026).

The paper operationalizes a “loop episode” as one in which either a single position is visited more than 10 times or the action-pattern detector triggers more than 20 times. In a 1,000-episode comparison, the baseline without anti-loop had 41.2% loop episodes and 58.8% normal episodes, whereas the anti-loop version reduced loop episodes to 4.7% and increased normal episodes to 95.3%.

The anti-spam mechanism targets repeated A/B/no-op behavior and menu abuse. Large spam penalties destabilized PPO, so PokeRL uses graduated streak penalties instead. After three consecutive presses of the same button, the environment applies -0.1; beyond five presses, it applies an additional -0.2. It also gives a tiny positive reward when recent action history contains at least four distinct actions. Combined with removing Start and Select from the action space, this changes the action distribution substantially: before anti-spam, roughly 32.1% of actions were A presses, 28.4% were no-ops, and only 27.2% were movement; after anti-spam, movement rose to 68.2%, A/B combined fell to 24.3%, and no-op fell to 7.5%.

Action diversity is quantified with Shannon entropy,

$H = - \sum_{a} p(a)\,\log_{2} p(a),$

where $p(a)$ is the probability of action $a$. The reported entropy rises from 1.21 bits before anti-spam to 1.82 bits after anti-spam. The paper interprets this as about a 50% relative improvement in exploration efficiency (Mudireddy et al., 12 Apr 2026).

4. Reward hierarchy and curriculum decomposition

PokeRL uses a dense hierarchical reward scheme divided into micro, meso, and macro rewards. Micro rewards encourage locomotion and local novelty: +1.0 for movement to a new tile, +0.2 × distance for Euclidean movement, and +0.5 for visiting a position for the first time. Meso rewards target subgoals: +10.0 for transitioning to a new map, +5.0 for entering a map for the first time in an episode, and +2.0 for reaching a large unexplored region. Macro rewards correspond to rarer milestones: +20.0 for entering tall grass, +10.0 for starting a battle, and +50.0 for catching a Pokémon or winning the Sequence 3 rival battle. Negative rewards are intentionally mild, typically -0.02 to -0.2, because harsh penalties were found to collapse training (Mudireddy et al., 12 Apr 2026).

Training is organized as a curriculum over three benchmark sequences, each with its own save state and reward emphasis.

Sequence	Start and end condition	Main emphasis
Sequence 1, House Exit	Starts in Red’s upstairs bedroom; ends when map ID changes to the outdoors or a step limit is hit	basic movement and map transitions
Sequence 2, Exploration to Grass	Starts outside the front door; ends on reaching tall grass, triggering Professor Oak’s event, or timing out	exploration coverage and reaching grass
Sequence 3, First Rival Battle	Starts at the beginning of the battle with a fixed starter Pokémon	offensive actions, fainting the opponent, winning

The paper argues that this decomposition is easier to debug and train than a monolithic full-game objective and better matches the natural progression of the early game. This suggests that PokeRL treats curriculum design as part of the environment definition, not merely as an optimization convenience.

5. Training regime and empirical performance

Algorithmically, PokeRL uses PPO from Stable-Baselines3 with the built-in CNN actor-critic policy. The reported hyperparameters are learning rate $3 * 10^{-4}$, $\gamma = 0.999$, $n\_steps = 2048$, batch size = 128, and 10 epochs per update. Training uses four parallel environments via DummyVecEnv, with one environment optionally rendered for inspection. Runs were performed for several hundred thousand timesteps per sequence or configuration on a single CPU/GPU setup, and metrics were logged to TensorBoard (Mudireddy et al., 12 Apr 2026).

One especially concrete ablation concerns the visited-mask observation in Sequence 2. Over 300k timesteps, the visited-mask variant is compared with a no-mask grayscale-only version using the metrics unique_positions, revisit_ratio, and exploration_ratio. With the mask, average unique_positions per episode increases from 34.2 to 48.1, a +40.6% gain. Pallet Town exploration coverage rises from 12% of tiles to 41%, and revisit ratio decreases from 4.8 to 3.1. The conclusion drawn in the paper is that the CNN uses the mask effectively as a form of memory, reducing redundant wandering.

Task-level performance remains early-game focused. For house exit, after 150k timesteps the agent exits the house in about 65% of episodes. For exploration to grass, the agent reaches tall grass and triggers the event in about 60% of episodes by 500k timesteps. For the first rival battle, win rates are around 50% after 500k timesteps with the fixed reward structure. The paper does not present these numbers as near-human performance; rather, they are used to show that the engineered system can learn nontrivial behaviors reliably.

6. Relation to prior Pokémon RL, limitations, and broader significance

PokeRL situates itself relative to earlier PPO-based Pokémon Red work, particularly the 2025 effort summarized as having used PPO in a PyBoy-based Gym environment, over 25 reward components, and a 72×80 visited mask to reach as far as the second gym in Cerulean City. PokeRL does not claim to exceed that level of game completion. Its narrower contribution is explicit anti-loop and anti-spam engineering, a per-map centered visited mask, and a curriculum over early-game sequences (Mudireddy et al., 12 Apr 2026).

The system’s limitations are stated clearly. It relies on direct memory access and handcrafted rewards, so it is not a pure pixels-to-actions solution. Training remains computationally heavy, time-expensive, and hyperparameter-sensitive. The paper also notes that manual human triggering may still be required in some text-dialog sequences during testing. The curriculum consists of separately trained sequences rather than one end-to-end agent that handles the full early game. Reported success rates therefore leave substantial room for improvement.

These limitations also define PokeRL’s scientific position. It is best understood as an environment-design and systems paper rather than a state-of-the-art game-completion result. Its broader argument is that Pokémon environments expose realistic failure modes—loops, menu abuse, exploration collapse, control-interface mismatch—that are often hidden in cleaner RL benchmarks. Later benchmark work on Pokémon, such as the two-track PokeAgent Challenge, would frame Pokémon RPG play as a standardized long-horizon decision-making problem at larger scale (Karten et al., 16 Mar 2026). This suggests that PokeRL occupies an earlier but important place in the same research trajectory: a practical intermediate step from brittle emulator control toward more general Pokémon agents.