JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines

Abstract: Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine. Our key insight is that Game Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose. Building on the Godot engine's text-based format and headless execution mode, we design a deterministic verification pipeline from file integrity to runtime behavior collection, distilling 8,133 verified projects from over 240,000 repositories. Of these, 300 manually verified projects form JamBench; the rest constitute JamSet. JamBench defines theme-driven generation and code completion tasks, evaluated through a pipeline combining compilation pass rates, Structural Completeness Score (SCS), and Behavioral Alignment Score (BAS). Evaluation of 9 frontier models reveals a capability cliff as project scale increases, with runtime pass rates dropping from 80.4% on small projects to 5.7% on large ones (Task2a). Code Agents improve compilation rates yet yield no gains in runtime behavioral quality, indicating that the bottleneck lies in architectural design rather than syntactic correctness. Experiments validate JamSet as effective training data. All data and code are publicly available.

• The paper introduces JAMER, the first comprehensive dataset and benchmark for project-level game code generation in professional game engines using Godot.
• It details a multi-level, automated pipeline and robust evaluation metrics that distinguish between syntactic compilability and genuine behavioral fidelity.
• Empirical findings reveal a significant capability cliff in large projects and underscore the need for domain-specific fine-tuning to enhance performance.

JAMER: Project-Level Code Framework Dataset and Benchmark for Professional Game Engines

Motivation and Contribution

The JAMER framework introduces JamSet and JamBench, representing the first comprehensive dataset and benchmark for project-level game code framework generation situated within a professional game engine context, specifically Godot (2606.19830). Prior efforts in AI-driven game design predominantly focus on asset generation, procedural gameplay, or lightweight web-based coding frameworks. In contrast, real-world code engineering for complete game projects on professional engines remains largely absent due to the lack of scalable datasets and deterministic evaluation protocols. JAMER bridges this gap by systematically leveraging Game Jam repositories and the text-based, headless-executable properties of Godot to establish a deterministic pipeline for large-scale project curation and evaluation.

Dataset Construction Pipeline

JamSet and JamBench are curated using a multi-level, fully automated pipeline:

• Data Sources: Aggregation from Ludum Dare, itch.io, Global Game Jam, GMTK, Godot Wild Jam, and GitHub yields over 240,000 candidate repositories.
• Filtering Criteria: Quality thresholds ($\geq$1200 game code lines; $<$1000 plugin addon lines) are empirically established via correlation analysis with Ludum Dare ratings.
• Verification Levels:
  • L1 (File Integrity): Ensures valid project configuration, scene accessibility, and main scene designation for Godot 4.x.
  • L2 (Compilation): Detects syntax, type, and resource errors using headless Godot compilation.
  • L3a (Runtime): 30s headless execution for stability.
  • L3b (Behavior Collection): 60s deterministic input strategy drives behavioral data capture.
• Split: 8,133 projects pass all levels. 300 are manually verified to form JamBench; 7,833 constitute JamSet.

Structured annotation includes automatic manifest extraction (scene trees, script lists, input mappings, autoloads) and LLM/VLM-assisted asset labeling, facilitating downstream training and evaluation.

Benchmark Design and Evaluation Metrics

JamBench formulates two core tasks:

1. Theme-Driven Generation: Requires models to synthesize a game from a Jam theme, optionally supplemented with a gameplay description.
2. Multi-Granularity Completion: Models complete partial projects at function, script, or full-script levels.

Evaluation uses:

• Compilation Pass Rate (L1/L2/L3a): Objective binary checks based on headless execution.
• Structural Completeness Score (SCS): Static metric composited from script/scene/function counts, input actions, signal usage; normalized to dataset or ground truth.
• Behavioral Alignment Score (BAS): Quantifies dynamic similarity across runtime events, responsive actions, signal triggers, and node modifications.

Metrics are designed to capture the distinction between compilability and genuine engineering/behavioral fidelity—a necessity given the observed phenomena of minimal "shell" outputs passing compilation yet lacking meaningful gameplay.

Empirical Findings

Nine state-of-the-art LLMs and multiple Code Agent configurations are evaluated on JamBench:

• Capability Cliff: Runtime pass rates plummet from 80.4% on small projects to only 5.7% for large ones, echoing findings from repository-level code benchmarks (e.g., SWE-bench, RepoBench).
• Code Agents: Iterative tool-driven agents substantially improve compilation rates but fail to enhance SCS/BAS, highlighting architectural design, not syntactic repair, as the primary bottleneck.
• Fine-Tuning: JamSet-fine-tuned LLMs demonstrate improved structural completeness and integration of domain-specific engineering patterns (input abstractions, global state via autoloads, scene transitions)—features deeply lacking in baseline model outputs.
• Granularity Findings: Function-level completion is objectively harder than script-level, requiring tight consistency. Script-level allows internal rewrites, yielding higher compilation rates at the expense of preserved behavioral complexity.

Theoretical and Practical Implications

JAMER exposes key limitations of contemporary code LLMs in project-level design. Compilability does not imply behavioral correctness or adherence to real engineering paradigms. Current agents, while effective at syntactic repair, are insufficient for orchestrating cross-file logic and emergent interactive behaviors. Behavioral alignment scores, as introduced in JAMER, are imperative for quantifying this quality deficit.

Practically, JAMER provides a scalable methodology and resource for advancing AI-based game engineering. The deterministic, engine-driven evaluation enables reproducibility and objective comparison—properties lacking in prior asset- or gameplay-focused frameworks. Domain-specific curation and annotation strategies can be generalized to other engines for broader applicability.

Theoretically, JAMER establishes the need for new architectural priors, multi-modality learning, and agentic workflows that transcend simple patching. Fine-tuning on domain-aligned corpora demonstrably steers LLMs toward more human-like engineering practices. Ensuing research should incorporate domain knowledge and engine conventions within agentic repair loops, as well as scale behavioral evaluation across Unity and Unreal ecosystems.

Limitations and Future Directions

The dataset is currently Godot-centric, reflecting its rapid growth but omitting Unity/Unreal. Asset-focused evaluation is intentionally excluded due to subjectivity. JAMER prioritizes frameworks enabling flexible asset integration and design extension. Future expansions should include project variants, repaired projects that previously failed verification, and training of expert domain-specific models.

Conclusion

JAMER advances project-level code generation research by introducing deterministic, scalable curation and evaluation for professional game engines. It identifies a robust capability cliff and engineering paradigm gap, substantiates the necessity of structural and behavioral metrics, and validates the efficacy of domain-specific fine-tuning. JAMER’s methodology and dataset release set the stage for new research in automated game engineering, agentic workflows, and domain-adapted LLMs (2606.19830).