GameCraft-Bench: Game Synthesis Evaluation

• GameCraft-Bench is a benchmark suite that evaluates coding agents' ability to synthesize fully playable, commercial-grade game artifacts within real-world engines.
• It comprises two suites—the Godot End-to-End suite and the PCG suite—offering tasks that assess artifact completeness, interactivity, and design quality.
• The evaluation methodology uses multimodal replay traces and metric-driven scoring to distinguish genuine gameplay functionality from mere compile-pass success.

GameCraft-Bench is a comprehensive benchmark suite designed to evaluate the capability of coding agents and generative algorithms in synthesizing functional, playable games and game content within real-world environments. Unlike earlier procedural content generation testbeds, GameCraft-Bench aligns artifact quality, artifact completeness, and interactive verification with the semantics of commercial-grade game engines and heterogeneous task domains. Its scope spans fully executable 2D games in Godot, multi-environment procedural content challenges, and rigorous metric-driven evaluation pipelines (Luo et al., 16 Jun 2026, Khalifa et al., 27 Mar 2025).

1. Formal Problem Setting and Motivation

GameCraft-Bench formalizes end-to-end game generation as the function

$x = (s, E) \mapsto y = G$

where $s$ denotes a natural-language specification (including mechanics, core loop, content, and presentation), $E$ is the game engine environment (notably, a fully configured Godot workspace), and $G$ is a fully runnable game artifact. Evaluation is performed by launching $G$ under the engine’s runtime, driving interaction via replayable traces, and assessing correctness against a hidden rubric (Luo et al., 16 Jun 2026).

This approach isolates three evaluation desiderata:

• Engine Grounding: All development and runtime occur within a real engine (Godot 4.6.2), retaining scene structures, asset requirements, physics, rendering, and scripting lifecycles.
• Artifact Completeness: Only a launchable, correctly structured project—with all entry points, assets, UI, and configuration—can be scored.
• Interactive Verification: Correctness is established via observed gameplay from replayed inputs, eschewing static inspection in favor of multimodal evidence.

These principles explicitly separate GameCraft-Bench from prior “compile-pass” or code-generation benchmarks whose signals often fail to discriminate between syntactic compliance and genuine functionality, as detailed in analysis of scene synthesis for Unity (Liu et al., 8 May 2026).

2. Benchmark Composition and Task Taxonomy

GameCraft-Bench comprises two orthogonal but complementary suites:

• End-to-End Game Generation (Godot Suite): 140 distinct tasks across 15 families (e.g., Platformer, Strategy, Tycoon, Roguelike, Visual Novel, Puzzle, Shooter, Simulation), each defined by a public specification (instruction.md), a hidden rubric (rubric.json), and an “oracle” reference implementation.
• Procedural Content Generation (PCG) Suite: 12 modular content-generation tasks, each encapsulated as a “Gym-like” environment with explicit content representations and control parameters (e.g., 2D/3D maps, tile-based level layouts, arcade rule sets, bullet-hell patterns) (Khalifa et al., 27 Mar 2025).

Tasks in the Godot suite demand synthesis of a complete game, covering scene graphs, scripts, UI, assets, and replay files. PCG tasks focus on artifact generation given a control target, assessing quality, diversity, and controllability.

Suite/Domain	Tasks (#)	Scope
Godot End-to-End	140	Full Games
PCG Environments	12 (+vnts)	Content/Levels

Editor’s term: “Godot Suite” and “PCG Suite” for brevity.

3. Evaluation Methodology and Metrics

3.1 Godot End-to-End Suite

Benchmarking proceeds through an interaction-grounded pipeline:

1. Task Packaging: Agent receives $(s, E, \rho)$—the spec, engine workspace, and rubric.
2. Agent Generation: Agent scripts scenes, assets, and code, then generates replay traces.
3. Build Gate: Project must build successfully via godot --headless within a fixed timeout.
4. Replay: For each demo trace, the engine replays events, records video (854×480, 30 fps), and samples at 2 fps for up to 20 seconds.
5. Scoring: Each rubric item is scored in $[0,1]$ by a GPT-5.5 multimodal judge, with rubric-specific max/mean aggregation applied. Final score:

$$\text{Score} = \text{BUILD} \times (w_M M + w_D D + w_V V + w_A A)$$

with $M$ = Core Mechanics, $D$ = Content Depth, $s$0 = Functional Visuals, $s$1 = Art & Presentation, and weights $s$2, $s$3, $s$4, $s$5.

Rubrics include up to 24 items spanning core loop, scenario diversity, UI property, and presentation standards. For example, a strategy skirmish test requires a playable grid combat loop, three or more unit classes, custom sprites, and a coherent dark-fantasy art direction.

3.2 PCG Suite

Objects $s$6 are evaluated on:

• Quality $s$7: Artifact-specific heuristic (e.g., level solvability).
• Diversity $s$8: Dissimilarity between artifact pairs.
• Controllability $s$9: Fidelity to supplied control parameters $E$0.

Mean and pairwise metrics provide batch-level statistics. Fitness functions (Q, QT, QTD) promote quality, then controllability, then diversity in a staged reward structure.

4. Empirical Results: Performance of Coding Agents and PCG Algorithms

4.1 Godot Suite: Coding Agent Evaluation

Seven state-of-the-art agents were evaluated (2h/task, fixed environment). Performance:

Model	Overall Score (%)
Claude Code (Opus)	41.46
Codex (GPT-5.5)	39.49
Kimi Code (K2.6)	30.65
MiMo-V2.5-Pro	24.10
Code Buddy (GLM)	18.29
MiniMax-M2.7	10.95
DeepSeek-V4-Pro	2.15

Best category scores (Opus): Core Mechanics 55.34, Content Depth 39.48, Visuals 42.78, Art 36.86. Common failure modes included incomplete game loops, missing or invalid replay traces, lack of content depth, default/fragmented UI, and placeholder graphics. A critical finding is that successful builds without valid replay traces receive a zero score, emphasizing interaction-grounded correctness (Luo et al., 16 Jun 2026).

4.2 PCG Suite: Algorithmic Baselines

Random, $E$1 Evolution Strategy, and Genetic Algorithm baselines were compared:

Algorithm	Max. Quality (tasks)	Diversity	Controllability
GA	224/360*	Moderate	Moderate-High
ES	Slightly fewer	Highest overall	Highest
Random	Rarely improved	Maximal	Minimal

Across 360 runs, GA reached quality = 1 solutions most often. ES recurrently delivered more unique and controllable artifacts but with less convergence on very hard tasks. Fitness shaping (QT/QTD) increases control and diversity but hinders convergence in difficult domains (Khalifa et al., 27 Mar 2025).

5. Comparative Analysis with Prior Benchmarks

GameCraft-Bench directly addresses critical limitations in earlier code-generation and PCG benchmarks:

• Beyond Compile-Pass Rate: Demonstrated in Mage (Liu et al., 8 May 2026), “compile success” is often anti-correlated with gameplay fidelity—generated code may build and run but lack intended structure and behavior (e.g., $E$2 ≈ 0.12 in mechanism adherence). GameCraft-Bench enforces artifact completeness and observed interactivity.
• Multi-Axis Evaluation: Mage’s four-axis protocol (compile, runtime, structure, mechanism) inspires the multidimensional rubrics in GameCraft-Bench, which demand both functional correctness and presentation quality.
• Generality and Modularity: GameCraft-Bench is readily extensible by design (new environments and interfaces) and accommodates both search-based algorithms and LLM-driven agents for unified comparison.

6. Insights, Open Challenges, and Future Directions

Current coding agents cannot reliably produce games meeting all dimensions: the best agent achieves only 41.46% aggregate performance. While micro-mechanics and local loops (e.g., movement, input) are commonly realized, agents consistently underperform on content depth, robust visual feedback, and artistic presentation. Visual debugging and perception-driven iteration (i.e., tool-driven screenshot feedback) are necessary for progress. The need to supply valid and usable replay traces is non-optional; otherwise, otherwise functional builds receive no evaluable score.

A plausible implication is that closing the performance gap will require advances in agentic UI reasoning, content generation methods, procedural iteration over style and assets, and integrated pipelines treating code, assets, scenes, and interaction traces as inseparable components. Comparison with PCG baselines also indicates that diversity, controllability, and creative recombination remain inadequately addressed.

GameCraft-Bench thus operationalizes a scalable, extensible, and interaction-grounded framework for the next phase of game synthesis research, highlighting the divide between mere code correctness and holistic, player-facing, executable creativity (Luo et al., 16 Jun 2026, Liu et al., 8 May 2026, Khalifa et al., 27 Mar 2025).