GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments

Abstract: Game engines provide real-time simulation, rendering, physics, interaction, networking, and asset pipelines, making them valuable not only for games but also for 3D applications in healthcare, robotics, architecture, manufacturing, and related domains. Because game development is where these systems are most mature and publicly available, it offers a practical testbed for evaluating coding agents that must modify C++ code within stateful, interactive, real-time systems. We present GameEngineBench, a benchmark for evaluating coding agents on scoped C++ implementation tasks inside Unreal Engine 5 projects, built from nine real-world game repositories. The evaluation set consists of 110 tasks spanning gameplay mechanics, multiplayer behavior, AI and world orchestration, animation and movement, UI and session code, loading behavior, online-service integration, persistence, data serialization, XR behavior, and rendering-oriented plugins. These tasks require models to make native C++ changes that compile and satisfy behavioral tests within executable Unreal Engine projects. Across twelve evaluated configurations, the strongest model reaches 55.5\% pass@1, while 31 tasks remain unsolved by every configuration. Our results demonstrate that frontier coding agents continue to struggle with deeply integrated C++ development for real-time interactive software, highlighting game-engine benchmarks as a valuable complement to existing software engineering evaluations.

• The paper introduces GameEngineBench, a novel benchmark evaluating coding agents’ ability to integrate C++ modifications in real Unreal Engine environments.
• It calibrates 110 tasks across multiple subsystems to ensure that agent-generated code meets rigorous runtime and behavioral correctness criteria.
• Experimental results highlight significant performance gaps among state-of-the-art LLM-based agents, emphasizing challenges in multi-system coordination.

GameEngineBench: Evaluation of Coding Agents in Real C++ Runtime Environments

Motivation and Context

Advances in LLM-driven coding agents have shifted benchmark development from isolated function synthesis toward complex, repository-scale, and system-integration tasks. However, most current benchmarks inadequately capture the idiosyncrasies of game engine integration, particularly for C++ code that must interact with real-time, stateful, and networked systems. "GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments" (2607.03525) addresses this gap by introducing an Unreal Engine 5 benchmark suite expressly designed to evaluate the integration capability of coding agents on tasks demanding native C++ modifications within substantial, non-trivial game projects.

GameEngineBench contextualizes the evaluation within a domain where runtime correctness depends not only on compilation but on faithful integration with engine subsystems, real-time simulation, multiplayer networking, asset pipelines, and user interaction. This focus on behavioral correctness surpasses the traditional evaluation of code synthesis, requiring that agent-generated code achieves effective, observable behavior within executable Unreal Engine environments.

Benchmark Construction and Task Calibration

The benchmark comprises 110 tasks, each derived from nine mature, publicly available Unreal Engine C++ repositories. Each task provides agents with a buildable project state, scoped source files for editing, and a natural language behavioral specification. Crucially, the test suite is withheld during code generation and injected post-hoc, preventing overfitting to test-specific signals and ensuring evaluation generalization.

Task calibration follows a rigorous process involving:

Figure 1: Pipeline for task authoring, test validation, LLM-as-a-Judge behavioral review, and iterative calibration.

• Test Automation: Ensures reference solutions pass behavioral tests and that editing states are consistent.
• LLM-as-a-Judge Audit: An LLM verifies the alignment between model-generated solutions, reference implementation, and test outcomes.
• Iterative Refinement: Calibration continues until tasks reliably distinguish between partial and total solution coverage by coding agents.

This evaluation design strictly targets native C++ changes (not blueprint scripting) within the constraints of each project’s architecture, ensuring the necessity of engine-level integration.

Task Distribution and Domain Coverage

The task distribution comprehensively spans gameplay mechanics, multiplayer logic, AI orchestration, animation, UI/session management, system initialization, persistent data handling, XR features, and Unreal plugin-based rendering. Each task typically requires coordinated C++ intervention across multiple subsystems. Task selection prioritizes areas with mature public repository coverage but acknowledges sparser coverage for domains such as audio, platform-specific features, or proprietary engine modifications.

Figure 2: Example tasks illustrating the behavioral and systems-level complexity required for benchmark tasks, as grounded in gameplay scenarios.

The benchmark’s categorical diversity (visible in Figure 3) enforces evaluation across orthogonal competencies, including state replication, authority handling, object lifecycle adherence, and subsystem orchestration.

Figure 3: The distribution of benchmark tasks across Unreal Engine development areas highlights the significant portion of integration-heavy and cross-cutting system requirements.

Evaluation Protocol and Metricization

Evaluation hinges on pass@1: the proportion of tasks solved on a single agent attempt. A successful solution must both pass the injected Unreal automation tests and satisfy an LLM-judge’s assessment of behavioral fidelity, not mere syntactic similarity to reference code. This protocol reflects the practical requirement that runtime behavior, not just code plausibility, determines utility in real environments.

Each model-agent is evaluated with a standardized command-line interface, ensuring deterministic input/output and preventing code/test interleaving. Execution occurs in isolated workspaces, and test scenarios utilize Unreal’s Play-in-Editor listen-server mode for accurate multiplayer/authority simulation.

Experimental Results and Model Performance

Twelve agent configurations are benchmarked, featuring major LLM architectures (Claude Fable, GPT-5.5, Gemini 3.1 Pro, DeepSeek, Qwen, Kimi) and various prompt/effort settings. Numerical results are definitive:

Figure 4: Success rates by model configuration, stratified by model/pretrained reasoning effort. Claude-fable-5 achieves the highest pass@1 rate of 55.5%.

• Top Configuration: Claude-fable-5 with maximal reasoning effort attains 55.5% pass@1.
• Model Gap: GPT-5.5 (xhigh) and Claude Opus-4-8 are next best, at 29.1% and 23.6% respectively. The performance differential is substantial; no other agent achieves competitive parity.
• Complementarity: No single model solves all solvable tasks, and solved-task sets between high-performing agents are only partially overlapping.

Efficiency profiling Figure 5 and configuration complementarity Figure 6 further dissect resource tradeoffs and solution diversity.

Failure Modes and Limit Analysis

Outcome decomposition reveals predominant failure classes:

Figure 7: Run-level outcome decomposition by agent configuration—highlighting where non-pass outcomes are predominantly behavioral/test rather than compilation failures for strong agents.

• Behavioral Mismatch: Most failures among strong agents occur during runtime (not compilation)—demonstrating the true challenge is not code syntax but cross-system integration.
• Multiplayer Authority and Replication: Frequent errors include incorrect ownership assignment, missing/incorrect replication logic, and insufficient event signaling across networked gameplay.
• Lifecycle Bugs: Agents commonly mishandle spawning, initialization, or actor destruction, resulting in resource leaks or broken gameplay loops.

Task-area analysis Figure 8 and outcome matrices Figure 9 corroborate that incomplete solutions frequently cluster around tasks demanding the highest degree of multi-system coordination (e.g., persistence, AI control, animation/event sequencing).

Implications and Future Directions

This work exposes concrete limitations in current coding agents for deeply integrated C++ systems:

• No Model Saturation: The strongest available agent leaves nearly half the benchmark unsolved; 31 tasks are beyond the current frontier.
• Non-nested Capability Profiles: Different agents excel at different classes of tasks, disallowing simple leaderboard-based selection for practical deployment.
• Runtime Integration Over Syntax: The fundamental barrier for agent performance is sophisticated, engine-level reasoning, not C++ code synthesis.

Practical implications extend to any context where LLMs are expected to operate on large, stateful, event-driven C++ systems with complex, high-integrity runtime requirements.

Future directions include:

• Extending task coverage to underrepresented gameplay domains and proprietary/system-level code,
• Standardizing agent execution wrappers for fair comparison,
• Enlarging the benchmark to include more cross-system orchestration and long-horizon tasks,
• Increasing test and judge generalization to further decouple evaluation from particular solution strategies.

Conclusion

GameEngineBench provides an authoritative assessment platform for the next generation of coding agents operating in authentic C++ game engine environments. The methodology requires not just plausible code edits but also robust behavioral integration within real, complex simulations. Current frontier agents achieve moderate pass@1, but their limitations are exposed by tasks necessitating rigorous, multi-system coordination. The benchmark establishes a high bar for general coding-agent research and identifies critical bottlenecks on the path to effective AI-augmented development for interactive, networked, stateful C++ applications.