GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

Abstract: Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.

• The paper presents GameCraft-Bench, a benchmark that evaluates autonomous agents generating fully playable game artifacts from natural language specifications in a real game engine.
• The paper details a five-stage evaluation process and shows that even top agents score only 41.46% overall, with notable deficiencies in visual coherence and content depth.
• The paper underscores the need for advances in multimodal perception and agent meta-control to achieve holistic, shippable game generation that integrates code, assets, and interactive verification.

GameCraft-Bench: A Benchmark for End-to-End Game Generation in Real Engines

Problem Motivation and Scope

GameCraft-Bench systematically investigates the challenge of generating fully-playable games from natural-language specifications using autonomous coding agents. Unlike traditional code synthesis or “game logic” challenges, this setting demands the automated production of complete, launchable game projects in a real game engine, integrating code, assets, UI, and runtime interaction. The benchmark asserts that valid evaluation of game generation requires three desiderata: (1) Engine Grounding—development occurs in a real, constraint-driven engine; (2) Artifact Completeness—output is an entirely runnable game artifact, not partial scripts; and (3) Interactive Verification—evaluation is predicated on observed player-game interactions, not static inspection.

Existing benchmarks (e.g., OpenGame-Bench (Jiang et al., 20 Apr 2026), GameDevBench (Chi et al., 11 Feb 2026)) either fail to ground development in mature engines, do not require artifact completeness, or omit interactive evaluation, thus leaving the central problem space of “can agents build shippable, play-verified games?” unaddressed.

Benchmark Design

Task Formalization and Pipeline

Each GameCraft-Bench task is defined by a tuple $(s, E, p)$, where $s$ is the game specification (natural-language, product-oriented, and open-ended), $E$ is the real Godot engine runtime and toolchain, and $p$ is a hidden multimodal rubric for behavioral evaluation. Agents receive $(s, E)$ and produce $(G, II)$: a complete Godot game project $G$ (including all necessary code, assets, configuration, and entrypoints) and a set $II$ of demonstration traces that encode player interaction. The verifier replays $G$ with $II$ to produce gameplay evidence, which is adjudicated with rubric-guided, interaction-grounded scoring.

The evaluation pipeline has five stages: (1) task packaging with precise rubric and scenario directives; (2) agent generation—full artifact construction plus trace authoring; (3) build gate—enforcing launchability; (4) replay—standardized demonstration in a fixed viewport via event traces; (5) scoring—rubric-driven analysis of mechanics, content depth, visual feedback, and art/presentation.

Task Suite and Annotation

GameCraft-Bench contains 140 tasks spanning 15 game families (platformer, strategy, tycoon, roguelike, simulation, etc.), deliberately emphasizing diversity in core gameplay loops, progression systems, and presentation. Each task is authored and validated by expert annotators, producing both player-centric specifications and granular evaluation rubrics, with validation via reference-oracle implementations to ensure internal consistency.

Evaluation of Coding Agents

Experimental Protocol

Seven leading coding agents (including Claude Code, MiMo, GPT-5.5, Kimi, DeepSeek, GLM, and MiniMax) are assessed across all benchmark tasks. Agents operate in a constrained workspace, with standardized environment and timeout enforcement. Build, replay trace, and demo counts are logged. The judge (GPT-5.5-based) is applied in a strict, rubric-bound mode to sampled gameplay frames and videos.

Key Results

• The top-scoring agent (Claude Code/Opus-4.7) achieves only 41.46% overall on the benchmark, with most other agents scoring well below this threshold. Notably, even the strongest agents do not consistently deliver complete, specification-satisfying game artifacts.
• Agents are more successful at implementing core mechanics (top: 55.34%) than at delivering meaningful content depth (39.48%), functional visuals (42.78%), or cohesive art/presentation (36.86%). The system completion gap is most apparent in content and polish, directly impacting playability and perceptual quality.
• Low or zero scores often emerge from: incomplete demonstration traces, minimal or disconnected in-game feedback, or presentation artifacts relying on placeholders rather than fully-authored assets.
• Diagnostic findings:
  • Agents leveraging perceptual feedback (inspecting rendered screenshots) more effectively debug visual/UI errors, supporting the value of visual interaction in agent iteration.
  • High tool usage or code churn is only weakly correlated with final game quality, indicating that success is constrained by the end-to-end integration and trace-completion, not mere coding volume.
  • Multimodal judge is stable and robust, with human calibration showing minor deviations (judge is slightly more permissive on e.g. art/presentation).
  • Game generation ability is not monolithic: only moderate coupling is found between mechanics and other dimensions (content, visuals, art), implying that scaling up interactive, perceptually-grounded evaluation is essential for capturing agent limitations on presentation and completeness.

Implications and Limitations

Practical and Theoretical Impact

GameCraft-Bench concretely establishes a rigorous, reproducible standard for measuring progress in code-to-game agentics. The empirical results reveal that, despite rapid advances in code LLMs and agentic frameworks, “build a shippable game” from high-level intent remains an open problem: current agents can prototype mechanics and partial interaction loops but rarely deliver fully-realized, polished, and robust game artifacts.

The findings shift research attention away from tool-centric or code-only breakthroughs toward holistic software artifact generation that respects engine semantics and player-facing interaction. Success will likely require new paradigms in multimodal perception (to close the code-render-debug loop), structured demonstration and trace generation, and integrated asset manipulation. The partial factorization of generation skills suggests that composite or collaborative agent designs—explicitly modularizing mechanics, art, UI, and test trace creation—may be fruitful directions.

Future Directions and Benchmark Constraints

GameCraft-Bench currently focuses on 2D games within Godot for practical headless evaluation and reproducibility. Extending the methodology to major 3D engines (Unity, Unreal) and to more complex genres (multiplayer, simulation-heavy, or large-scale projects) is a crucial next step. Audio-dependent gameplay is not directly scored; only visual evidence is used for evaluation.

Methodologically, robust and scalable human- or judge-in-the-loop frameworks for subjective dimensions (fun, creativity) remain an open research challenge. Finally, the requirement for agents to author deterministic, scenario-dividing demo traces is nontrivial and exposes the need for improved agent meta-control and self-evaluation.

Conclusion

GameCraft-Bench delivers an interaction-grounded, engine-anchored, artifact-complete benchmark for end-to-end game generation. The benchmark demonstrates that producing launchable, visually coherent, and behaviorally compliant games from natural-language specifications is well beyond the current capabilities of leading coding agents. These results underline that interactive system correctness—not simply code correctness—should be the core measured quantity in creative software agentics, and that substantial advances in perceptual iteration, agent coordination, and holistic software authorship are required to approach human-level game development.

Reference: "GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?" (2606.17861)