OpenGame-Bench: Game Generation Benchmark

• OpenGame-Bench is a benchmark suite for automated game generation, combining multi-task PCG tests with execution-grounded evaluations.
• It employs clear metrics like Build Health, Visual Usability, and Intent Alignment to measure the quality, playability, and compliance of generated games.
• The framework supports extensible PCG tasks and comparative evaluations of algorithms, facilitating reproducible research in automated game content creation.

OpenGame-Bench is a suite of automated benchmarking pipelines designed to evaluate the generation and creation of games and game content with a focus on procedural content generation (PCG) and end-to-end agentic coding. The benchmark is represented by two complementary paradigms: (1) a multi-task PCG testbed for evaluating generative algorithms across structurally diverse problems, and (2) an execution-driven assessment of full game implementations from high-level natural language prompts using automated build, play, and vision-LLM (VLM) judgment. These frameworks enable standardized, scalable, and reproducible evaluation of algorithmic creativity, software integration, and design compliance in both content-centric and code-centric game generation scenarios (Jiang et al., 20 Apr 2026, Khalifa et al., 27 Mar 2025).

1. Evaluation Protocols and Metric Dimensions

OpenGame-Bench encompasses two principal evaluation suites, each with rigorous quantitative metrics tailored to its level of abstraction.

A. Execution-Grounded Game Evaluation

This protocol (introduced in (Jiang et al., 20 Apr 2026)) targets agentic code generation for playable web-based games. All evaluation is conducted via a Node.js-driven pipeline, which performs build, deployment, and automated gameplay in a headless browser environment. The assessment consists of three dimensions, each mapped to a 0–100 scale:

Metric	Definition	Primary Assessment Modality
Build Health (BH)	Degree to which the project compiles, serves, runs in a headless browser, and produces at least one frame	Node.js orchestration, error tracing
Visual Usability (VU)	Coherence, motion, and apparent interactivity of visuals	Pixel/entropy heuristics, VLM (few-shot prompts)
Intent Alignment (IA)	Fidelity to structured requirements parsed from the prompt	VLM scoring of semantic evidence in frames

• Build Health captures graduated failure modes (e.g., partial builds, runtime exceptions, rendering stalls) rather than binary pass/fail, penalizing intermediate breakdowns.
• Visual Usability is a convex combination of image-difference heuristics (e.g., frame entropy, motion statistics) and a VLM’s structured judgment of screenshot series for playability.
• Intent Alignment relies on prompt-to-requirement extraction, with VLM judges determining the satisfaction of each artifact requirement through semantic analysis of execution frames.

B. PCG Multi-Task Problem Suite

As described in (Khalifa et al., 27 Mar 2025), OpenGame-Bench formalizes 12 generative environments. Evaluation across these is problem-agnostic and scored by:

• Quality $q(c)\in [0,1]$: Adherence to formal feasibility or playability constraints.
• Diversity $d(c_i, c_j)\in [0,1]$: Pairwise domain-specific dissimilarity (e.g., edit distances, structural metrics).
• Controllability $t(c, p)\in [0,1]$: Faithfulness to generator-control parameters.

This architecture supports scalable, objective comparisons across a heterogeneous set of content-generation tasks spanning layouts, rules, mechanics, and structures.

2. Problem Domains and Content Representations

The benchmark covers a spectrum of generative tasks:

• Rule and mechanic specification: E.g., Arcade Rules, which require dictionary-based mappings from game events to outcomes.
• Level generation (grid-based, graph-based, or parametric): Maze (Binary), Dungeon (Isaac), Platform (Dangerous Dave, Mario), Sokoban, MiniDungeons, Zelda, Lode Runner.
• Pattern and structural synthesis: 3D Building, Bullet Hell (Talakat), Letter Sequence (Elimination).

Each environment prescribes precise genotype representations (e.g., boolean matrices, voxel arrays, JSON scripts) together with control parameter spaces. Quality and controllability requirements specify playability, coverage, connectivity, and logic completeness (e.g., minimum path-length, tile counts, action constraints).

3. Automated Assessment and Judging Methodologies

Both OpenGame-Bench paradigms emphasize fully automated, reproducible evaluation.

Execution-Grounded Pipeline (Jiang et al., 20 Apr 2026)

• Build and Serve: Project is built (via declared scripts), then served over localhost to a Chromium headless browser.
• Frame-Driven Testing: An instrumented JS harness advances a fixed number of simulation frames, logs exceptions, and captures periodic screenshots streamed to the evaluator.
• VLM Judging: Batched PNG screenshots are scored using structured few-shot prompts (two positive/negative exemplars per mechanic or requirement) to a vision-LLM, which provides both coherence (for VU) and compliance (for IA) verdicts.

PCG Evaluation Suite (Khalifa et al., 27 Mar 2025)

• Each generator outputs $(c, p)$ pairs (content, parameters).
• For each population, the evaluate API computes $q, d, t$ aggregates.
• Visual renderers and auxiliary info routines support interactive or batch analysis.

4. Benchmark Algorithms and Empirical Results

In PCG benchmarking (Khalifa et al., 27 Mar 2025), search-based methods are tested:

• Random Search: Uninformed sampling, high diversity, but limited feasibility on complex tasks.
• Evolutionary Strategy (μ+λ ES): Population-based mutation and selection, fast improvement on quality but risk of diversity loss.
• Genetic Algorithm (GA): Tournament selection, crossover, elitism for sample robustness.

Performance varies with objective:

• Quality-focused runs (Q): Drive feasibility but collapse diversity.
• Multi-objective (QTD): Can balance feasibility, control, and uniqueness if weighted and managed.
• Problem Difficulty: Small/structured (Binary, Sokoban) are routinely solved. Large/complex (Mario, Lode Runner) consistently defy random and ES/GA using default variants.

In code generation for whole games (Jiang et al., 20 Apr 2026):

• State-of-the-Art Scores: OpenGame (with Claude 4.6) reaches BH=72.4, VU=67.2, IA=65.1 where best baseline (Cursor, Claude 4.6) achieves BH=66.8, VU=61.4, IA=58.9.
• Genre Sensitivity: Highest IA for physics-intensive genres (platformers); lowest for abstract logic genres (strategy, puzzle/UI).
• Repair Ablations: Multiple automatic repair passes recover significant build and playability health, especially on cross-file and asset-key inconsistencies.

5. Experimental Procedures and Extensibility

Prompt Curation and Runs (Jiang et al., 20 Apr 2026):

• 150 human-compiled prompts covering five balanced genres.
• Each prompt run with three random seeds; means reported over successful runs; pipeline errors logged but excluded from means.

Extensibility (Khalifa et al., 27 Mar 2025):

• Environments can be subclassed and extended; user defines info extraction, scoring, and rendering routines.
• Algorithms interface via a standard API—input pairs, return artifacts, and use evaluate for metrics.
• New tasks, control axes, and quality diversifiers can be introduced with minimal integration overhead.

Repository: https://github.com/amidos2006/pcg_benchmark

6. Limitations and Research Implications

The benchmark reveals:

• Standardized evaluation mitigates bespoke, non-reproducible experimentation prevalent in PCG.
• Diversity, controllability, and quality often exhibit trade-offs, and multi-objective balancing is nontrivial; objective drift and mode collapse are recurrent.
• Verifying interactive playability (not merely static syntax or semantics) demands execution-grounded or VLM-augmented methodologies, which may still miss latent control/logic errors (i.e., “silent inconsistencies”).

A plausible implication is that as VLMs and agentic pipelines mature, fully autonomous game synthesis and objective evaluation will become feasible not only for toy domains but for complex cross-file, real-time environments and creative-lead tasks (Jiang et al., 20 Apr 2026, Khalifa et al., 27 Mar 2025). This suggests a future in which evaluation pipelines like OpenGame-Bench function both as scientific testbeds and as practical validation layers in mixed-initiative and autonomous content pipelines.