Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds (2511.08892v1)

Abstract: We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-LLM. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments.

• The paper introduces a VLM-driven framework that integrates human-like perception, reasoning, and fine-grained action control to complete long-horizon missions in 3D open worlds.
• The paper details a three-stage curriculum learning pipeline—from extensive pre-training to human-annotated reasoning fine-tuning—enabling robust performance and cross-domain generalization.
• The paper demonstrates human-comparable efficiency across varied tasks and games, highlighting the impact of hybrid reasoning and context-based memory on real-time autonomous control.

Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds

Introduction and Motivation

Lumine establishes an end-to-end, vision-language-model (VLM)-driven framework for generalist agents capable of long-horizon, real-time completion of complex missions in open-world games. The paper identifies six key challenges in scalable generalist agent development: environment richness and diversity, multimodal perception, high-level planning, low-level control, multi-timescale memory management, and real-time inference. Existing systems excel in single-domain, closed-loop tasks but fail to demonstrate robust abstraction, transfer, and adaptability under the ambiguity and scale of open-world scenarios.

Figure 1: Lumine, the first AI agent to complete hours-long missions in real time within expansive 3D open worlds.

Lumine addresses these challenges by unifying human-like perception, reasoning, and action through a VLM, operating directly on raw pixels and outputting keyboard-mouse actions at interactive frequencies, leveraging a hybrid reasoning strategy that enables adaptive monologue generation only when necessary.

Environment Selection and Complexity

Genshin Impact is selected as the principal environment, presenting high-fidelity graphics, compositional 3D spaces, multi-level reasoning puzzles, and asynchronous interactions without API access. Its open world features intricate navigation, a diverse catalogue of NPCs and enemies, and richly structured missions that demand hours of engagement—a considerably more demanding setting than conventional benchmarks. Lumine’s approach is general and scalable, explicitly engineered to be applicable across commercial AAA titles and, by extension, to other arbitrary software with human-level interfaces.

Figure 2: Genshin Impact is a large-scale open world requiring complex exploration, reasoning, and interaction.

Agent Architecture and Action Modeling

Lumine’s architecture builds upon Qwen2-VL-7B-Base, extending it with fine-grained action chunking, explicit reasoning, and natural language grounding. The agent operates autoregressively on 5 Hz pixel inputs, maintaining short-term and reasoning-based long-term memory in context. Critically, Lumine outputs raw keyboard and mouse actions represented in a concise, language-native format, covering mouse movements, scroll, keypress sequences, and combination states (modifiers, holds, multi-key actions) at 30 Hz granularity.

Figure 3: Lumine’s model combines VLM perception, thought generation, and action output in an autoregressive loop.

A hybrid reasoning mode is fundamental: the agent alternates between latent reasoning and direct action output based on contextual necessity, avoiding inefficiencies and hallucinations inherent to stepwise ReAct-style agents.

Data Curation and Curriculum

The training pipeline comprises three stages:

1. Pre-training on 1731 hours of filtered image-action data for action primitive acquisition.
2. Instruction-following stage with 200 hours of natural language-goal annotated gameplay for control alignment.
3. Reasoning-stage fine-tuning utilizing 15 hours of human-annotated inner monologues, explicitly grounding situational adaptation and trajectory planning.

   Figure 4: The three-stage pipeline incrementally instils action skill, language understanding, and reasoning/planning.

   

   Figure 5: Data processing pipeline for raw gameplay into structured datasets for multi-stage curriculum learning.

This multi-stage curriculum is essential for scalable learning, sidestepping the bottlenecks of large-scale human annotation through selective filtering, classifier-aided auto-labeling, and LLM-based verification.

Memory, Context, and Real-Time Optimization

Short-term memory is established via a 20-frame context window, while long-term memory is managed as the last reasoning step within the dialogue. The inference system is heavily optimized for latency, enabling real-time control through streaming output, speculative decoding, mixed-precision quantization (W8A8), tensor parallelism, and strict kernel/graph-level tuning.

Figure 6: Context management strategy during inference, blending sliding window for short-term memory and monologue for long-term planning.

Figure 7: Latency breakdown showing a 25.3$\times$ speedup over baseline via multi-stage technical and infrastructure optimization.

With these optimizations, action chunk latency never exceeds the 33 ms window, supporting seamless real-time interaction.

Benchmark and Evaluation Protocol

A novel benchmark of 141 tasks across four classes—Collection, Combat, NPC Interaction, Puzzle—tests agents under simple, hard, and unseen conditions, including out-of-distribution regions and tasks.

Figure 8: Benchmark tasks span Collection, Combat, NPC Interaction, and Puzzle-solving; each tests multimodal and reasoning abilities.

Empirical Analysis

Scaling and Emergence: Pre-training directly produces agentic skills: basic object interaction emerges immediately; navigation/puzzle-solving requires ~1800 hours; 7B scale shows markedly higher capacity and robustness than 2B.

Figure 9: Scaling analysis—success rate and loss curves reveal distinct phase transitions for skill emergence.

Instruction Following: Lumine-Instruct-NonHis achieves >80% success on all simple tasks; instruction fine-tuning yields substantial improvement in guided behaviors and generalization, especially for combat and puzzles, over both base and foundation models.

Figure 10: Lumine-Instruct-NonHis outperforms baselines and its pre-trained counterpart across all task categories.

Figure 11: Ablation shows pretraining essential for generalization; instruction-following aligns specific skills.

In-context Learning: Richer instruction context further boosts performance on previously low-performing tasks.

Figure 12: Semantic enrichment and stepwise guidance in context enhance success rates in complex objectives.

History and Memory: Multi-frame history yields clear gains in performance, peaking at 10 frames.

Figure 13: Performance as a function of historical context length validates importance of dynamic memory.

Figure 14: History setting consistently outperforms single-frame setting in navigation and manipulation.

Error analysis highlights that multimodal perception failures remain prevalent, despite marked improvement from historical context.

Figure 15: Model errors stem from perception, instruction following, spatial understanding, and control—history mitigates, but not eliminates, partial observability.

Long-horizon Reasoning and Generalization: In the main storyline (Prologue: Act I), Lumine-Thinking equals expert human efficiency (56 min vs. 53 min), vastly outperforming instruction-only models (6.6% vs. 93.4% subtask success).

Figure 16: Lumine-Thinking’s geographical trajectory and timescale in completing hour-long story missions.

Error analysis shows history models produce significantly fewer and better quality reasoning traces.

Figure 17: History-enabled reasoning minimizes misrepresentation and contradiction in planning.

Cross-mission generalization is strong: Lumine completes the next two acts (not covered in reasoning dataset) within 4.7 hours (vs. 3.6 for humans), leveraging previously acquired knowledge, context, and reasoning skills.

Figure 18: Lumine’s round-trip trajectory in unseen Acts II and III, extending to Liyue, an entirely novel region/task.

Examples demonstrate that learned mechanisms generalize to OOD contexts, e.g., moving platforms and wind-currents.

Figure 19: Lumine exploits learned mechanisms on moving platforms in both in-domain and OOD scenarios.

Memory and Planning Limitations: Multiple simultaneous quest markers and blocked routes reveal that limited long-term memory constrains efficiency and causes oscillatory or stuck behaviors.

Figure 20: Multiple quest targets induce oscillation, underlying short memory bottleneck.

Cross-game Generalization: Lumine achieves human-like performance in Wuthering Waves (107 min vs. 101 min for humans) and Honkai: Star Rail (full chapter in 7 hours, vs 4.7 for humans), despite unfamiliar control schemes and mechanics.

Figure 21: Successful navigation and UI manipulation demonstrated across four distinct game titles.

Implications, Theoretical Considerations, and Future Directions

Lumine demonstrates that scalable, data-driven curriculum learning and explicit, dialogic monologue planning in a VLM agent can yield robust autonomous control, cross-domain generalization, and competitive efficiency in high-dimensional, multi-modal environments. The decision to represent actions as natural language sequences—rather than dedicated heads or domain-specific structures—offers architecturally elegant integration and zero-shot portability. Strong empirical results underscore the viability of hybrid reasoning for efficient planning, and context-based memory for maintaining temporal coherence.

Key limitations remain:

• Memory: Even with 20-frame history and monologue, consistent long-term strategy is not guaranteed, especially in tasks with multi-step detours and distractors.
• Data scope: Reasoning annotations are limited; further scaling across domains and games is needed to test robustness.
• Online adaptation: The model learns fully offline; incorporating RL and continual learning would potentially enable superhuman proficiency.
• Real-time deployment: Despite extensive optimizations, GPU-bound inference still bottlenecks commercial application and high-throughput rollout; further system-level innovations are warranted.

In addition to its research utility, Lumine’s framework promises substantial impact for automated game testing, debug and QA, and interactive entertainment, demonstrating meaningful progress toward general decision foundation models.

Conclusion

Lumine provides an open, reproducible pipeline for end-to-end generalist agent development, integrating scalable data curation, naturalistic action modeling, hybrid reasoning, and real-time system deployment in challenging commercial environments. The agent achieves competitive, human-level long-horizon task completion, strong out-of-domain adaptation, and high cross-title transfer, all with modest model scale and data requirements. The results advance understanding of the curriculum, architectural, and optimization strategies necessary for artificial general agent intelligence, while highlighting open avenues in memory, online learning, and further scaling.