V-GameGym: Visual Game Generation for Code Large Language Models

Abstract: Code LLMs have demonstrated remarkable capabilities in programming tasks, yet current benchmarks primarily focus on single modality rather than visual game development. Most existing code-related benchmarks evaluate syntax correctness and execution accuracy, overlooking critical game-specific metrics such as playability, visual aesthetics, and user engagement that are essential for real-world deployment. To address the gap between current LLM capabilities in algorithmic problem-solving and competitive programming versus the comprehensive requirements of practical game development, we present V-GameGym, a comprehensive benchmark comprising 2,219 high-quality samples across 100 thematic clusters derived from real-world repositories, adopting a novel clustering-based curation methodology to ensure both diversity and structural completeness. Further, we introduce a multimodal evaluation framework with an automated LLM-driven pipeline for visual code synthesis using complete UI sandbox environments. Our extensive analysis reveals that V-GameGym effectively bridges the gap between code generation accuracy and practical game development workflows, providing quantifiable quality metrics for visual programming and interactive element generation.

• The paper introduces V-GameGym, a benchmark that evaluates code LLMs in visual game generation by integrating metrics for code, static visuals, and dynamic gameplay.
• It employs a two-stage methodology combining clustering-based curation and an LLM-driven pipeline to refine 2,219 high-quality Pygame samples.
• Experimental results reveal a trade-off between high code generation scores and lower visual and gameplay metrics, underscoring challenges in multimodal synthesis.

V-GameGym: A Multimodal Benchmark for Visual Game Generation with Code LLMs

Motivation and Problem Statement

The paper introduces V-GameGym, a comprehensive benchmark designed to evaluate the capabilities of code LLMs in the context of visual game generation. Existing code LLM benchmarks predominantly focus on unimodal tasks such as code completion, synthesis, and algorithmic problem-solving, neglecting the multimodal and interactive requirements of real-world game development. These requirements include not only code correctness and execution but also playability, visual aesthetics, and user engagement—dimensions critical for practical deployment in game development workflows.

Benchmark Construction and Methodology

V-GameGym is constructed from 2,219 high-quality, manually verified Pygame samples, curated from 2,190 unique repositories and organized into 100 thematic clusters. The curation process employs a two-stage methodology:

1. Clustering-Based Curation: The initial corpus is partitioned using high-dimensional feature vectors (capturing code size, structure, API usage, and semantics) via MiniBatchKMeans. Within each cluster, the most structurally complete program is selected using a quality heuristic that prioritizes code completeness and runnability.
2. Automated LLM-Driven Pipeline: The selected seed set is processed through an LLM pipeline that:
   • Analyzes code intent and infers game mechanics.
   • Refactors interactive programs into self-contained, autonomous demonstrations.
   • Verifies execution in sandboxed environments with automated error correction.
   • Generates high-level natural language requirements for each program.
   • Ensures correctness and quality via human validation in a UI sandbox.

     Figure 1: Overview of the V-GameGym framework from data collection to evaluation.

This pipeline ensures that each benchmark sample is paired with a precise, executable codebase and a corresponding natural language requirement, supporting robust evaluation of both code generation and multimodal synthesis.

Dataset Analysis

The dataset exhibits substantial diversity in both requirements and code complexity. Requirement texts display a right-skewed, log-normal length distribution, with significant heterogeneity across clusters, reflecting the spectrum of game development tasks.

Figure 2: Overall Requirement Length Distribution.

Figure 3: Linear-Scale Requirement Length Histogram.

Figure 4: Log-Scale Requirement Length Histogram.

Figure 5: Requirement Length Comparison by Cluster.

Reference code analysis reveals a median of 257 lines per game, with a 100% execution success rate and complete video coverage, ensuring the dataset's reliability for automated and human-in-the-loop evaluation.

Multimodal Evaluation Framework

V-GameGym introduces a multimodal evaluation protocol, aggregating scores from three modalities:

• Code Quality: Assessed for functionality, structure, logic, and technical implementation.
• Static Visuals: Evaluated via screenshots for UI completeness, design, and feature visibility.
• Dynamic Gameplay: Assessed via video for animation, interaction logic, and playability.

The final score is a weighted sum of these components, enabling granular analysis of model strengths and weaknesses across modalities.

Figure 6: Distribution of evaluation scores across three key dimensions: Code, Screenshot, and Video.

Experimental Results and Scaling Laws

A comprehensive evaluation of 70 models (proprietary and open-source) reveals several key findings:

• Performance Hierarchy: Proprietary models (e.g., GPT-5, O3, Gemini-2.5-Pro) dominate the leaderboard, with top open-source models (Qwen3-235B-A22B-Thinking-2507, gpt-oss-120b) closing the gap but still trailing in overall performance.
• Code-Visual Trade-off: All models achieve high code generation scores (70–97), but visual and dynamic evaluation scores remain low (typically <25), indicating a persistent gap in multimodal synthesis.
• Scaling Law: There is a clear, though logarithmic, correlation between model size and the number of games solved, with diminishing returns at larger scales. The empirical relationship is $M = 127.2 \cdot \log(N) + 135.6$, where $M$ is the number of solved games and $N$ is the parameter count.

  Figure 7: Correlation between model size and games solved.

• Difficulty Distribution: The benchmark is challenging, with most games unsolved by the majority of models, and only a small fraction reaching the "Excellent" quality band.

  Figure 8: Game difficulty distribution by number of solving models.

• Ranking Stability: Model rankings are robust across difficulty tiers and score thresholds, validating the discriminative power of the benchmark.

  Figure 9: Performance comparison of top 15 models across Easy, Medium, and Hard difficulty tiers, showing consistent ranking patterns and scaling challenges.

• Multimodal Correlation: There is a moderate-to-strong positive correlation between code, screenshot, and video evaluation dimensions, indicating that integrated multimodal understanding is required for high performance.

  Figure 10: Correlation matrix between Code, Screenshot, and Video evaluation dimensions, demonstrating the interdependence of multimodal capabilities in game development.

• Capability Specialization: Top models exhibit specialization, with some (e.g., GPT-5) excelling in code at the expense of visual quality, suggesting a trade-off at the performance frontier.

  Figure 11: Radar chart comparing the top 10 models across four key performance dimensions.

System Architecture and Pipeline Performance

The end-to-end pipeline is optimized for scalability and reliability, supporting parallel code generation, media capture, and multimodal evaluation. The system achieves an >80% end-to-end success rate, with robust error handling and automatic recovery mechanisms. Game code generation achieves an 85.3% compilation and execution success rate, with average generation and recording times of 2.3s and 1.2s per game, respectively.

Implications and Future Directions

V-GameGym exposes critical limitations in current code LLMs for real-world game development, particularly in visual and interactive synthesis. The persistent gap between code correctness and visual/gameplay quality underscores the need for:

• Enhanced multimodal pretraining and alignment strategies.
• Integration of visual and temporal reasoning modules.
• Improved evaluation protocols that reflect end-user experience and engagement.

The benchmark's diversity and rigor make it a valuable resource for driving progress in AI-assisted game development, multimodal code generation, and agentic reasoning. The observed scaling law suggests that further gains will require not only increased model capacity but also architectural and algorithmic innovations targeting multimodal integration.

Conclusion

V-GameGym establishes a new standard for evaluating code LLMs in the context of visual game generation, bridging the gap between unimodal code synthesis and the complex, multimodal requirements of practical game development. The benchmark's comprehensive methodology, rigorous evaluation, and detailed analysis provide actionable insights for both model development and deployment, highlighting the challenges and opportunities at the intersection of code intelligence and multimodal AI.