---
title: 'OpenGame-Bench: Game Generation Benchmark'
url: https://www.emergentmind.com/topics/opengame-bench
type: topic
---

# OpenGame-Bench: Game Generation Benchmark

OpenGame-Bench is a suite of automated benchmarking pipelines designed to evaluate the generation and creation of games and game content with a focus on procedural content generation (PCG) and end-to-end agentic coding. The benchmark is represented by two complementary paradigms: (1) a multi-task PCG testbed for evaluating generative algorithms across structurally diverse problems, and (2) an execution-driven assessment of full game implementations from high-level natural language prompts using automated build, play, and vision-language model (VLM) judgment. These frameworks enable standardized, scalable, and reproducible evaluation of algorithmic creativity, software integration, and design compliance in both content-centric and code-centric game generation scenarios [2604.18394] [2503.21474].

## 1. Evaluation Protocols and Metric Dimensions

OpenGame-Bench encompasses two principal evaluation suites, each with rigorous quantitative metrics tailored to its level of abstraction.

**A. Execution-Grounded Game Evaluation**

This protocol (introduced in [2604.18394]) targets agentic code generation for playable web-based games. All evaluation is conducted via a Node.js-driven pipeline, which performs build, deployment, and automated gameplay in a headless browser environment. The assessment consists of three dimensions, each mapped to a 0–100 scale:

| Metric                          | Definition                                                                                                       | Primary Assessment Modality                    |
|----------------------------------|-----------------------------------------------------------------------------------------------------------------|------------------------------------------------|
| Build Health (BH)                | Degree to which the project compiles, serves, runs in a headless browser, and produces at least one frame       | Node.js orchestration, error tracing           |
| Visual Usability (VU)            | Coherence, motion, and apparent interactivity of visuals                                                        | Pixel/entropy heuristics, VLM (few-shot prompts)|
| Intent Alignment (IA)            | Fidelity to structured requirements parsed from the prompt                                                       | VLM scoring of semantic evidence in frames     |

- **Build Health** captures graduated failure modes (e.g., partial builds, runtime exceptions, rendering stalls) rather than binary pass/fail, penalizing intermediate breakdowns.
- **Visual Usability** is a convex combination of image-difference heuristics (e.g., frame entropy, motion statistics) and a VLM’s structured judgment of screenshot series for playability.
- **Intent Alignment** relies on prompt-to-requirement extraction, with VLM judges determining the satisfaction of each artifact requirement through semantic analysis of execution frames.

**B. PCG Multi-Task Problem Suite**

As described in [2503.21474], OpenGame-Bench formalizes 12 generative environments. Evaluation across these is problem-agnostic and scored by:

- **Quality $q(c)\in [0,1]$:** Adherence to formal feasibility or playability constraints.
- **Diversity $d(c_i, c_j)\in [0,1]$:** Pairwise domain-specific dissimilarity (e.g., edit distances, structural metrics).
- **Controllability $t(c, p)\in [0,1]$:** Faithfulness to generator-control parameters.

This architecture supports scalable, objective comparisons across a heterogeneous set of content-generation tasks spanning layouts, rules, mechanics, and structures.

## 2. Problem Domains and Content Representations

The benchmark covers a spectrum of generative tasks:

- **Rule and mechanic specification:** E.g., Arcade Rules, which require dictionary-based mappings from game events to outcomes.
- **Level generation (grid-based, graph-based, or parametric):** Maze (Binary), Dungeon (Isaac), Platform (Dangerous Dave, Mario), Sokoban, MiniDungeons, Zelda, Lode Runner.
- **Pattern and structural synthesis:** 3D Building, Bullet Hell (Talakat), Letter Sequence (Elimination).

Each environment prescribes precise genotype representations (e.g., boolean matrices, voxel arrays, JSON scripts) together with control parameter spaces. Quality and controllability requirements specify playability, coverage, connectivity, and logic completeness (e.g., minimum path-length, tile counts, action constraints).

## 3. Automated Assessment and Judging Methodologies

Both OpenGame-Bench paradigms emphasize fully automated, reproducible evaluation.

### Execution-Grounded Pipeline ([2604.18394])

- **Build and Serve:** Project is built (via declared scripts), then served over localhost to a Chromium headless browser.
- **Frame-Driven Testing:** An instrumented JS harness advances a fixed number of simulation frames, logs exceptions, and captures periodic screenshots streamed to the evaluator.
- **VLM Judging:** Batched PNG screenshots are scored using structured few-shot prompts (two positive/negative exemplars per mechanic or requirement) to a vision-language model, which provides both coherence (for VU) and compliance (for IA) verdicts.

### PCG Evaluation Suite ([2503.21474])

- Each generator outputs $(c, p)$ pairs (content, parameters).
- For each population, the `evaluate` API computes $q, d, t$ aggregates.
- Visual renderers and auxiliary info routines support interactive or batch analysis.

## 4. Benchmark Algorithms and Empirical Results

In PCG benchmarking ([2503.21474]), search-based methods are tested:

- **Random Search:** Uninformed sampling, high diversity, but limited feasibility on complex tasks.
- **Evolutionary Strategy (μ+λ ES):** Population-based mutation and selection, fast improvement on quality but risk of diversity loss.
- **Genetic Algorithm (GA):** Tournament selection, crossover, elitism for sample robustness.

Performance varies with objective:

- **Quality-focused runs (Q):** Drive feasibility but collapse diversity.
- **Multi-objective (QTD):** Can balance feasibility, control, and uniqueness if weighted and managed.
- **Problem Difficulty:** Small/structured (Binary, Sokoban) are routinely solved. Large/complex (Mario, Lode Runner) consistently defy random and ES/GA using default variants.

In code generation for whole games ([2604.18394]):

- **State-of-the-Art Scores:** OpenGame (with Claude 4.6) reaches BH=72.4, VU=67.2, IA=65.1 where best baseline (Cursor, Claude 4.6) achieves BH=66.8, VU=61.4, IA=58.9.
- **Genre Sensitivity:** Highest IA for physics-intensive genres (platformers); lowest for abstract logic genres (strategy, puzzle/UI).
- **Repair Ablations:** Multiple automatic repair passes recover significant build and playability health, especially on cross-file and asset-key inconsistencies.

## 5. Experimental Procedures and Extensibility

**Prompt Curation and Runs ([2604.18394]):**

- 150 human-compiled prompts covering five balanced genres.
- Each prompt run with three random seeds; means reported over successful runs; pipeline errors logged but excluded from means.

**Extensibility ([2503.21474]):**

- Environments can be subclassed and extended; user defines info extraction, scoring, and rendering routines.
- Algorithms interface via a standard API—input pairs, return artifacts, and use `evaluate` for metrics.
- New tasks, control axes, and quality diversifiers can be introduced with minimal integration overhead.

**Repository:** https://github.com/amidos2006/pcg_benchmark

## 6. Limitations and Research Implications

The benchmark reveals:

- Standardized evaluation mitigates bespoke, non-reproducible experimentation prevalent in PCG.
- Diversity, controllability, and quality often exhibit trade-offs, and multi-objective balancing is nontrivial; objective drift and mode collapse are recurrent.
- Verifying interactive playability (not merely static syntax or semantics) demands execution-grounded or VLM-augmented methodologies, which may still miss latent control/logic errors (i.e., “silent inconsistencies”).

A plausible implication is that as VLMs and agentic pipelines mature, fully autonomous game synthesis and objective evaluation will become feasible not only for toy domains but for complex cross-file, real-time environments and creative-lead tasks [2604.18394] [2503.21474]. This suggests a future in which evaluation pipelines like OpenGame-Bench function both as scientific testbeds and as practical validation layers in mixed-initiative and autonomous content pipelines.

Source: https://www.emergentmind.com/topics/opengame-bench