---
title: 'PokeRL: Modular RL Benchmark for Pokémon Red'
url: https://www.emergentmind.com/topics/pokerl
type: topic
---

# PokeRL: Modular RL Benchmark for Pokémon Red

PokeRL is a modular, emulator-based reinforcement-learning system for early-game **Pokémon Red** that was introduced to make the game more usable as a stable RL benchmark. Rather than proposing a new RL algorithm, it packages a custom Gymnasium environment wrapped around the PyBoy emulator, RAM-based event detection, map-aware observations, action-space restrictions, anti-loop and anti-spam controls, and a dense hierarchical reward scheme into a practical training stack for tasks such as exiting the player’s house, reaching tall grass, and winning the first rival battle [2604.10812].

## 1. Benchmark setting and design rationale

PokeRL is motivated by the claim that **Pokémon Red** is difficult precisely because it is not a clean arcade-style control problem. Progress in the opening minutes already requires long-horizon navigation, interaction, scripted events, and turn-based combat under sparse rewards, partial observability, awkward game-specific controls, and hidden state variables such as HP, map ID, and party status. The paper emphasizes a particularly idiosyncratic control issue: directional movement uses a **“turn first, then move”** double-press mechanic, so a naïve action interface can cause an agent to spend much of its time turning in place rather than walking [2604.10812].

The system is explicitly organized around failure modes observed in straightforward PPO training. These include **infinite loops**, where the agent paces between a few tiles to harvest shaping rewards; **reward-instability from sparsity**; **button spam**, especially on A, Start, Select, or no-op; **incorrect movement semantics**; and **memoryless exploration** under pure pixel observations, which leads to repeated revisiting of already seen areas. PokeRL therefore treats loop detection, spam mitigation, movement semantics, and exploration memory as first-class engineering concerns rather than incidental implementation details.

This framing is central to the project’s significance. PokeRL argues that practical systems which explicitly model such pathologies are a necessary intermediate step between toy benchmarks and agents that could eventually handle the broader opening of the game or later milestones.

## 2. Environment architecture and state-action interface

PokeRL is implemented as a **custom Gymnasium environment wrapped around the PyBoy Game Boy emulator**. Around that core it adds modules for memory reading, reward computation, loop and spam mitigation, and curriculum control. The code repository is given as **https://github.com/reddheeraj/PokemonRL**. The environment uses PyBoy to run **Pokémon Red**, reads selected RAM locations for event detection and termination logic, but does **not** expose those memory values directly to the policy [2604.10812].

The observation pipeline combines raw visual input with engineered spatial memory. The emulator screen is downsampled to **72×80 grayscale**. PokeRL then stacks **four consecutive grayscale frames** together with **four corresponding visited-mask frames**, producing an **8-channel input** for a Stable-Baselines3 **CnnPolicy** actor-critic. The visited mask is maintained **per map** and centered on the map’s entry position so that it stays aligned to global map coordinates as the camera scrolls. The policy network consists of **three convolutional layers**, followed by a **fully connected layer of 512 units**, and outputs both **policy logits** and a **scalar value estimate**.

The action space is restricted to **seven discrete actions**: **up, down, left, right, A, B, and no-op**. **Start and Select are removed** because they are especially easy to exploit uselessly. To match the game’s movement semantics, every directional action is implemented as **two press-release cycles**, so one RL action corresponds to one intended grid-world step.

The RAM-reading layer is used only for environment logic. The paper lists the following addresses:

| RAM address | Variable | Environment use |
|---|---|---|
| `0xD361` | player Y position | position/event detection |
| `0xD362` | player X position | position/event detection |
| `0xD35E` | map ID | map transitions/milestones |
| `0xD163` | party count | party-state checks |
| `0xD057` | battle state | battle detection |
| `0xD16C` | Pokémon HP | battle rewards/termination |

This design is not fully end-to-end from pixels alone. The paper explicitly acknowledges that limitation and defends it as a practical compromise for a “messy” game environment.

## 3. Loop-aware control and anti-spam mechanisms

A major contribution of PokeRL is its **loop-aware wrapper** and **multi-layer anti-loop system**. The anti-loop mechanism has three parts. First, it tracks visits to each **\((x,y)\)** position within an episode: once a tile is visited more than three times, the agent receives a small penalty, and beyond five visits the penalty becomes larger. Second, it performs **action-pattern detection** over a sliding window of the last **20 actions** and penalizes repetitive patterns such as alternating fixed button cycles; breaking out of such a cycle can produce a small positive bonus. Third, it performs **position-loop detection** by checking whether recent position history repeatedly returns to a small neighborhood, indicating circular wandering or local pacing [2604.10812].

The paper operationalizes a **“loop episode”** as one in which either a single position is visited more than **10 times** or the action-pattern detector triggers more than **20 times**. In a **1,000-episode comparison**, the baseline without anti-loop had **41.2% loop episodes** and **58.8% normal episodes**, whereas the anti-loop version reduced loop episodes to **4.7%** and increased normal episodes to **95.3%**.

The **anti-spam mechanism** targets repeated A/B/no-op behavior and menu abuse. Large spam penalties destabilized PPO, so PokeRL uses **graduated streak penalties** instead. After **three consecutive presses** of the same button, the environment applies **-0.1**; beyond **five presses**, it applies an additional **-0.2**. It also gives a tiny positive reward when recent action history contains at least **four distinct actions**. Combined with removing Start and Select from the action space, this changes the action distribution substantially: before anti-spam, roughly **32.1%** of actions were A presses, **28.4%** were no-ops, and only **27.2%** were movement; after anti-spam, movement rose to **68.2%**, A/B combined fell to **24.3%**, and no-op fell to **7.5%**.

Action diversity is quantified with Shannon entropy,
$$
H = - \sum_{a} p(a)\,\log_{2} p(a),
$$
where \(p(a)\) is the probability of action \(a\). The reported entropy rises from **1.21 bits** before anti-spam to **1.82 bits** after anti-spam. The paper interprets this as about a **50% relative improvement in exploration efficiency** [2604.10812].

## 4. Reward hierarchy and curriculum decomposition

PokeRL uses a **dense hierarchical reward scheme** divided into **micro, meso, and macro rewards**. Micro rewards encourage locomotion and local novelty: **+1.0** for movement to a new tile, **+0.2 × distance** for Euclidean movement, and **+0.5** for visiting a position for the first time. Meso rewards target subgoals: **+10.0** for transitioning to a new map, **+5.0** for entering a map for the first time in an episode, and **+2.0** for reaching a large unexplored region. Macro rewards correspond to rarer milestones: **+20.0** for entering tall grass, **+10.0** for starting a battle, and **+50.0** for catching a Pokémon or winning the **Sequence 3** rival battle. Negative rewards are intentionally mild, typically **-0.02 to -0.2**, because harsh penalties were found to collapse training [2604.10812].

Training is organized as a curriculum over **three benchmark sequences**, each with its own save state and reward emphasis.

| Sequence | Start and end condition | Main emphasis |
|---|---|---|
| **Sequence 1, House Exit** | Starts in Red’s upstairs bedroom; ends when map ID changes to the outdoors or a step limit is hit | basic movement and map transitions |
| **Sequence 2, Exploration to Grass** | Starts outside the front door; ends on reaching tall grass, triggering Professor Oak’s event, or timing out | exploration coverage and reaching grass |
| **Sequence 3, First Rival Battle** | Starts at the beginning of the battle with a fixed starter Pokémon | offensive actions, fainting the opponent, winning |

The paper argues that this decomposition is easier to debug and train than a monolithic full-game objective and better matches the natural progression of the early game. This suggests that PokeRL treats curriculum design as part of the environment definition, not merely as an optimization convenience.

## 5. Training regime and empirical performance

Algorithmically, PokeRL uses **PPO from Stable-Baselines3** with the built-in CNN actor-critic policy. The reported hyperparameters are **learning rate \(3 * 10^{-4}\)**, **\(\gamma = 0.999\)**, **\(n\_steps = 2048\)**, **batch size = 128**, and **10 epochs per update**. Training uses **four parallel environments** via **DummyVecEnv**, with one environment optionally rendered for inspection. Runs were performed for **several hundred thousand timesteps** per sequence or configuration on a single CPU/GPU setup, and metrics were logged to TensorBoard [2604.10812].

One especially concrete ablation concerns the visited-mask observation in **Sequence 2**. Over **300k timesteps**, the **visited-mask** variant is compared with a **no-mask** grayscale-only version using the metrics **unique_positions**, **revisit_ratio**, and **exploration_ratio**. With the mask, average **unique_positions** per episode increases from **34.2** to **48.1**, a **+40.6%** gain. **Pallet Town exploration coverage** rises from **12%** of tiles to **41%**, and **revisit ratio** decreases from **4.8** to **3.1**. The conclusion drawn in the paper is that the CNN uses the mask effectively as a form of memory, reducing redundant wandering.

Task-level performance remains early-game focused. For **house exit**, after **150k timesteps** the agent exits the house in about **65%** of episodes. For **exploration to grass**, the agent reaches tall grass and triggers the event in about **60%** of episodes by **500k timesteps**. For the **first rival battle**, win rates are around **50%** after **500k timesteps** with the fixed reward structure. The paper does not present these numbers as near-human performance; rather, they are used to show that the engineered system can learn nontrivial behaviors reliably.

## 6. Relation to prior Pokémon RL, limitations, and broader significance

PokeRL situates itself relative to earlier PPO-based **Pokémon Red** work, particularly the 2025 effort summarized as having used PPO in a PyBoy-based Gym environment, **over 25 reward components**, and a **72×80 visited mask** to reach as far as the **second gym in Cerulean City**. PokeRL does **not** claim to exceed that level of game completion. Its narrower contribution is **explicit anti-loop and anti-spam engineering**, a **per-map centered visited mask**, and a **curriculum over early-game sequences** [2604.10812].

The system’s limitations are stated clearly. It relies on **direct memory access** and **handcrafted rewards**, so it is not a pure pixels-to-actions solution. Training remains **computationally heavy**, **time-expensive**, and **hyperparameter-sensitive**. The paper also notes that **manual human triggering may still be required in some text-dialog sequences** during testing. The curriculum consists of **separately trained sequences** rather than one end-to-end agent that handles the full early game. Reported success rates therefore leave substantial room for improvement.

These limitations also define PokeRL’s scientific position. It is best understood as an environment-design and systems paper rather than a state-of-the-art game-completion result. Its broader argument is that Pokémon environments expose realistic failure modes—loops, menu abuse, exploration collapse, control-interface mismatch—that are often hidden in cleaner RL benchmarks. Later benchmark work on Pokémon, such as the two-track **PokeAgent Challenge**, would frame Pokémon RPG play as a standardized long-horizon decision-making problem at larger scale [2603.15563]. This suggests that PokeRL occupies an earlier but important place in the same research trajectory: a practical intermediate step from brittle emulator control toward more general Pokémon agents.

Source: https://www.emergentmind.com/topics/pokerl