AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems

Abstract: Automatically generating 3D games in commercial game engines remains a non-trivial challenge, as it involves complex engine-related workflows for generating assets such as scenes, blueprints, and code. To address this challenge, we propose a novel multi-agent system, AutoUE, which coordinates multiple agents to end-to-end generate 3D games, covering model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation. In order to mitigate tool-use hallucinations in LLMs, we introduce a retrieval-augmented generation mechanism that grounds agents with relevant UE tool documentation. Additionally, we incorporate game design patterns and engine constraints into the code generation process to ensure the generation of correct and robust code. Furthermore, we design an automated play-testing pipeline that generates and executes runtime test commands, enabling systematic evaluation of dynamic behaviors. Finally, we construct a game generation dataset and conduct a series of experiments that demonstrate AutoUE's ability to generate 3D games end-to-end, and validate the effectiveness of these designs.

• The paper presents AutoUE, a five-agent system that converts natural-language descriptions into playable Unreal Engine 5 games by coordinating asset retrieval, scene generation, code synthesis, object interaction, and play-testing.
• AutoUE achieves a game score of 8.68, a scene-aesthetic score of 7.8, and 100% node, parameter, and pin success in PCG graph generation across the PlayGen-20 benchmark.
• The ablations show that predefined PCG patterns, documentation-grounded retrieval, and engine-specific code constraints are essential, while scalability beyond 20 tasks and richer behavioral testing remain open challenges.

Overview

AutoUE is a multi-agent system that generates complete, playable 3D games end-to-end in Unreal Engine 5 (UE5) from natural-language game descriptions (2603.07106). The system coordinates five LLM-driven agents — model retrieval, scene generation, gameplay code synthesis, interactive object implementation, and automated play-testing — connected through structured specifications and a Model Context Protocol (MCP) integration layer with the engine. The work addresses a gap in prior research: existing LLM-based approaches target isolated pipeline stages (scene generation, gameplay logic, or engine code), whereas AutoUE composes these stages into a single workflow aligned with commercial game-engine practices.

System architecture

The input description $x$ is first decomposed into a scene description $Desc_S$ and a gameplay description $Desc_G$, separating what should be built from what should happen. The model retrieval agent builds an embedding database over TexVerse, a repository of over 858K text-described Sketchfab models, applying category pre-filtering via Sketchfab's 18 categories followed by cosine-similarity retrieval and LLM re-ranking to select the top-1 model per scene object.

The scene generation agent avoids hard-coded coordinates by leveraging UE's Procedural Content Generation (PCG) framework. It plans one PCG node chain per object under two canonical placement patterns: large objects (e.g., buildings) use pruning and bounds control before direct spawning to prevent overlaps, while small scatter objects (trees, rocks) add exclusion nodes subtracting forbidden regions around major actors. To mitigate tool-use hallucination, node parameters and pin connections are grounded through retrieval-augmented generation (RAG) over segmented PCG documentation, retrieving only the minimal chunks needed per node rather than maintaining an exhaustive index.

The gameplay code agent plans functional modules (e.g., InventoryModule, DialogueModule), topologically sorts them by dependency, and generates UE C++ implementations conditioned on code templates and engine constraints such as export/registration macros, include whitelists, logging conventions, and cross-module access rules. A generic module management framework exposes modules for controlled invocation. The interactive object agent then infers interaction flows — which modules to call, in what order, and whether external plugins are needed — and generates per-object C++ code consistent with those flows. Finally, the automated play-testing agent generates two command types (move-to-object and perform-interaction), executes them at runtime via MCP, and collects screenshots, logs, and intermediate specifications for evaluation.

Evaluation

Experiments use PlayGen-20, a benchmark of 20 game-generation tasks split into Easy (5), Medium (7), and Hard (8) categories by scene and interaction complexity, with Qwen-Plus as the default backbone and GPT-4o as judge for aesthetic comparisons.

Evaluation	Metric	Result
Overall game quality	Game score (1–10)	8.68 overall
Scene aesthetics	GAS	7.8 (best among baselines)
PCG graph generation	Node/Param/Pin success	100% across all difficulties
Gameplay code	GCS	7.38 vs. ~2.4–2.5 for ablations

On overall quality, AutoUE scores 9.90 on Scene, 8.25 on Gameplay, and 7.78 on Visual, aggregated into a Game score of 8.68. Notably, Hard games score higher than Medium games on the Scene dimension (9.94 vs. 9.79), which the authors attribute to more complex PCG graphs yielding higher build-success credit; Visual is lowest for Hard tasks because deviations of interactions from intended behavior degrade visual coherence.

For scene aesthetics, following UnrealLLM's GPT Aesthetic Score protocol, AutoUE achieves 7.8, surpassing UnrealLLM (7.71), SceneX (7.31), WonderJourney (7.38), Magic3D (6.39), DreamFusion (4.83), Infinigen (6.61), and 3D-GPT (6.76). The authors concede this margin is small but argue GAS exhibits pronounced saturation — ratings above 8 were empirically very difficult to obtain — so scores near 8 indicate high artistic quality. This is a subjective-judge metric, and the claim rests partly on that saturation argument rather than a large quantitative gap.

Ablation findings

Two ablation studies carry the strongest claims in the paper. For PCG graph generation, removing the predefined node-chain patterns drops node-creation success to 79.6% and pin-connection success to 64.0%; removing parameter documentation collapses parameter-filling success to 55.2%; removing connection documentation reduces pin success to 80.4%. Full AutoUE attains 100% on all three success rates and the highest PCG quality score (8.42). These results indicate that both structural priors (patterns) and RAG-grounded semantics (parameters, connections) are individually necessary for reliable PCG graph construction.

For gameplay code, the ablations yield a stark dichotomy: removing module dependencies, code templates, or engine constraints leaves module-analysis scores largely intact (~8.1–8.3) but drives compilation failure universally, zeroing MCS and IIS and collapsing GCS from 7.38 to roughly 2.44–2.48. The implication is that compilable UE C++ generation depends critically on grounding in engine-specific conventions; planning quality alone does not produce working code.

Limitations and open questions

The paper states two limitations plainly. First, no head-to-head comparison with DreamGarden was possible because it has not been open-sourced, so the claim of superior end-to-end capability relative to the closest comparable system rests on qualitative characterization rather than measured results. Second, PlayGen-20's scope may not capture the complexity of real-world game development; scalability to broader task distributions remains unverified. Additionally, all primary evaluations rely on LLM-as-a-judge scoring, whose reliability for assessing dynamic gameplay behavior is itself an open question, and the play-testing command space is limited to movement and interaction execution, leaving richer behavioral verification unaddressed.

Conclusion

AutoUE demonstrates that a multi-agent system grounded in RAG-supplied tool documentation, predefined PCG patterns, and engine-constrained code templates can generate complete, playable UE5 games end-to-end, achieving perfect PCG graph construction rates and the highest reported aesthetic score among compared baselines. Its automated play-testing pipeline and PlayGen-20 benchmark provide reusable infrastructure, though validation against non-open-source competitors and at real-world task complexity remains outstanding.