Progressive Procedural Content Generation (PPCG)

• Progressive Procedural Content Generation (PPCG) is an adaptive framework that dynamically generates game levels in real-time by adjusting difficulty based on reinforcement learning performance.
• It employs parameterized level generators across various games, modulating factors such as maze complexity and object distribution to align level structure with agent skill.
• Empirical results show that PPCG improves generalization and combats overfitting by evolving difficulty in step with the agent’s learning progress, leading to better performance on unseen levels.

Progressive Procedural Content Generation (PPCG) is an adaptive online framework for procedural level generation in deep reinforcement learning (RL), introduced by Justesen et al. in the context of improving generalization across levels in video game-like environments. PPCG not only synthesizes new levels in real-time but also continuously tunes their difficulty to closely match the agent’s learning progress. Its core design aims to counteract the well-documented overfitting of neural RL agents to fixed environments by ensuring persistent novelty and an evolving skill-based curriculum (Justesen et al., 2018).

1. Formal Definition and Mechanism of Progressive Level Generation

Let $L$ denote the set of all possible levels for a given GVGAI (General Video Game AI) game. A parameterized level generator, $G\colon [0,1]\to\mathrm{Dist}(L)$, maps a scalar difficulty $d\in[0,1]$ to a distribution over levels, $G_d$. For each episode $t$, an environment samples $\ell_t \sim G_{d_t}$ and the RL agent interacts with $\ell_t$ until episode termination (success or failure).

The mechanics of difficulty progression are defined as follows. After each episode, the system receives a binary performance signal $R_t\in\{0,1\}$, where $R_t=1$ denotes a win and $R_t=0$ otherwise. The difficulty is updated according to:

$G\colon [0,1]\to\mathrm{Dist}(L)$0

with $G\colon [0,1]\to\mathrm{Dist}(L)$1 in all experiments. Winning increases $G\colon [0,1]\to\mathrm{Dist}(L)$2, and losing decreases it, with clipping to enforce $G\colon [0,1]\to\mathrm{Dist}(L)$3. This one-dimensional curriculum scalar is shared among all parallel learners. As the agent’s mastery improves, $G\colon [0,1]\to\mathrm{Dist}(L)$4 rises—automatically generating a curriculum from easy to hard levels.

2. Parameterized Generators and Level Complexity

Each GVGAI game incorporates a compact, constructive procedural generator with game-specific logic. The generator consults the difficulty parameter $G\colon [0,1]\to\mathrm{Dist}(L)$5 to modulate level attributes such as active (reachable) area, number and type of objects (e.g., boulders, gems, cars, logs), and the degree of maze complexity or connectivity (e.g., additional branching at high $G\colon [0,1]\to\mathrm{Dist}(L)$6). For instance, Boulderdash employs a cellular-automaton cave system, while Zelda uses Prim’s algorithm to generate mazes and selectively remove walls. At the extremes, $G\colon [0,1]\to\mathrm{Dist}(L)$7 produces trivially easy layouts, and $G\colon [0,1]\to\mathrm{Dist}(L)$8 yields maximally challenging configurations.

The table below summarizes key generator controls per game, as implemented:

Game	Difficulty Controls	Generator Method
Boulderdash	Active area, object counts, cave complexity	Cellular automaton
Zelda	Maze connectivity, enemy density, wall removal	Prim’s maze + wall punch-out
Frogs	Vehicle/log density, river/lane complexity	Game-specific constructive steps
Solarfox	Gem variety/distribution, enemy layout	Custom rules

Each sampled level at a given $G\colon [0,1]\to\mathrm{Dist}(L)$9 is structurally distinct, promoting the acquisition of versatile policies.

3. RL Agent Structure and Curriculum Interface

RL agents in all experiments are instantiated using the Advantage Actor-Critic (A2C) architecture, following the conventions from Mnih et al. (2016). Observations are raw pixel frames, processed via three convolutional layers with ReLU activations and a fully connected layer, yielding a policy distribution $d\in[0,1]$0 and a state-value estimate $d\in[0,1]$1. The principal deviation from standard A2C is the continual resampling of levels: at each episode’s start, a fresh $d\in[0,1]$2 is used, requiring agents to internalize general strategies (e.g., navigation, avoidance, collection) rather than level-specific solutions.

4. Experimental Protocol and Comparative Regimes

The empirical evaluation spans four representative GVGAI games (Zelda, Frogs, Solarfox, Boulderdash), using the OpenAI Baselines A2C implementation with fixed hyperparameters: 12 parallel environments, rollout length $d\in[0,1]$3, constant learning rate 0.007 (RMSProp), and no gradient clipping. Training involves up to 100 million frames (Zelda), with lesser budgets for other games. Four regimes are compared:

• Lv X: Training exclusively on a single human-designed level.
• Lv 0–3: Uniform sampling from four human-crafted levels.
• PCG X: Each episode uses a new procedurally generated (PCG) level at fixed difficulty $d\in[0,1]$4.
• PPCG: Levels are procedurally generated with live difficulty adjustment as per agent success.

Evaluation is performed on held-out sets: 30 pre-generated levels at $d\in[0,1]$5, 30 at $d\in[0,1]$6, and five human-designed levels, with performance metrics based on episode-return and win rate per test level, averaged over four independent seeds.

5. Quantitative Results and Generalization Outcomes

Training on fixed levels ($d\in[0,1]$7) produces agents that overfit—achieving near-perfect scores in training but catastrophic generalization, often performing worse than random on unseen levels. Using PCG with fixed, maximal difficulty (PCG 1) improves generalization within the generator’s own distribution but remains brittle on distributional edges (e.g., very easy or hard test cases).

PPCG, by incrementally increasing difficulty in response to agent performance, matches or surpasses fixed-difficulty PCG—[e.g., in Zelda, PPCG achieves test returns of $d\in[0,1]$8 (on $d\in[0,1]$9) and $G_d$0 (on $G_d$1), outperforming PCG 1 (returns $G_d$2 and $G_d$3 respectively)]. In Frogs, PPCG attains considerably higher win rates on hard levels ($G_d$4 on $G_d$5 test and $G_d$6 on $G_d$7), unlike PCG 1 which fails completely on hard configurations ($G_d$8 win rate at $G_d$9).

Empirically, PPCG facilitates gradual competence acquisition, with the curriculum adapting in lockstep with performance plateaus observed in training curves. The approach enables learning of hard level variants that are otherwise impractical when training starts at high difficulty.

6. Distribution Analysis via Dimensionality Reduction and Clustering

To determine if procedural generators produce level distributions that align with human-designed levels, Justesen et al. perform the following analysis: 1,000 levels are sampled from $t$0 (maximum difficulty), converted into binary indicator arrays $t$1 (where $t$2 is the tile type count), then flattened and projected into two dimensions using Principal Component Analysis (PCA). Clustering is performed using DBSCAN ($t$3, minPts=10).

Findings reveal that, for some games, clusters of procedurally-generated levels do not overlap with human-designed levels—e.g., in Solarfox, human levels avoid “mixed gem” clusters, potentially explaining generalization failures on such levels. In Frogs, at least one human level is an outlier, featuring starting-row rivers not producible by the generator, rendering PPCG-trained policies ineffective on that instance. PCA + DBSCAN thus provides a principled diagnostic to assess coverage and uncover distributional mismatches.

7. Insights, Significance, and Limitations

Key insights include:

• Overfitting is prevalent in deep RL agents trained on fixed levels; procedural generation combats this by ensuring that agents must learn generalizable solutions within the distribution induced by the generator.
• Effective generalization to human-designed or otherwise out-of-distribution levels requires that these levels be within the support of the procedural generator; otherwise, a distribution mismatch impedes transfer.
• PPCG’s curriculum enables successful policy learning on hard instances by starting with easy tasks and advancing difficulty contingent on the agent’s wins, which is not achievable with fixed-difficulty settings.
• The tools of dimensionality reduction and clustering afford empirical visibility into generator coverage, supporting iterative refinement of generators toward target test distributions.

A plausible implication is that the utility of PPCG hinges on the generator’s expressivity: when human or otherwise relevant levels are structurally distinct from generated instances, even optimally-trained policies may fail to generalize. Consequently, generator design and coverage analysis are indispensable components of successful PPCG deployment (Justesen et al., 2018).