Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse

Abstract: The real world unfolds along a single set of physics laws, yet human intelligence demonstrates a remarkable capacity to generalize experiences from this singular physical existence into a multiverse of games, each governed by entirely different rules, aesthetics, physics, and objectives. This omni-reality adaptability is a hallmark of general intelligence. As Artificial Intelligence progresses towards Artificial General Intelligence, the multiverse of games has evolved from mere entertainment into the ultimate ground for training and evaluating AGI. The pursuit of this generality has unfolded across four eras: from environment-specific symbolic and reinforcement learning agents, to current large foundation models acting as generalist players, and toward a future creator stage where agent both creates new game worlds and continually evolves within them. We trace the full lifecycle of a generalist game player along four interdependent pillars: Dataset, Model, Harness, and Benchmark. Every advance across these pillars can be read as an attempt to break one of five fundamental trade-offs that currently bound the whole system. Building on this end-to-end view, we chart a five-level roadmap, progressing from single-game mastery to the ultimate creator stage in which the agent simultaneously creates and evolves within theoretical game multiverse. Taken together, our work offers a unified lens onto a rapidly shifting field,and a principled path toward the omnipotent generalist agent capable of seamlessly mastering any challenge within the multiverse of games, thereby paving the way for AGI.

• The paper presents a framework for building generalist game players using foundation models and outlines a five-level progression from single-game mastery to creator agents.
• It systematically analyzes the pillars of datasets, models, harnesses, and benchmarks, detailing trade-offs in scale, fidelity, depth, and modular integration.
• Numerical results from systems like NitroGen and OpenP2P highlight challenges in real-time control, sample efficiency, and cross-game transfer.

Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse

Introduction and Motivation

This work presents a comprehensive, systematized study of the landscape, advances, and persistent challenges involved in constructing generalist game-playing agents using Large Foundation Models (LFMs) spanning LLMs, VLMs, VLAs, and emerging WAMs, with the long-term objective of bridging the gap between brittle specialists and AI generalists. The core motivation stems from the observation that while humans, with a singular physical grounding, can fluidly transfer competence to a diverse "multiverse" of games with radically different rules and goals, AI agents have historically remained myopic specialists, unable to generalize beyond the narrow boundaries of a single environment. The study theoretically analyzes, empirically surveys, and procedurally benchmarks the whole lifecycle of generalist agents along four tightly interdependent pillars—Dataset, Model, Harness, and Benchmark—addressing five fundamental trade-offs that bound current progress and charting a roadmap through five developmental levels toward AGI-like agentic generality.

Figure 1: A holistic overview of the research landscape toward generalist game players, organizing relevant works along a temporal axis and four key interdependent pillars: Dataset, Model, Harness, and Benchmark.

Paradigm Shift in Game-Playing AI

The paper formalizes the evolution of game AI as a sequence of paradigm shifts regarding how agents interact with, perceive, and control elements of the POMDP tuple $$\mathcal{M} = \langle G, S, A, T, R, \Omega, O, \gamma \rangle$$. This trajectory comprises:

1. Symbolic Systems: Agents with direct state access, hard-coded rules, and zero perceptual load, producing brittle, environment-bound policies.
2. Deep RL Era: Specialists using end-to-end learning (from pixels or structured state) but isolated to a single game's interface and goals, requiring tabula rasa learning per environment, achieving superhuman but fragile mastery.
3. Foundation Model Era: Models leveraging internet-scale, multimodal pretraining, and open-ended goal conditioning; these models serve as the first plausible generalist players capable of zero-shot transfer, multimodal reasoning, and flexible action in the diverse "human-crafted multiverse".
4. Creator/Demiurge Era (Anticipated): Agents not only act within games, but simulate, generate, and evolve new game multiverses and their underlying rules, reward functions, and interfaces.

   Figure 2: The evolution of game-playing AI paradigms: from symbolic systems, through deep RL, foundation models, toward the anticipated Creator era.

The present frontier lies at the junction of the Deep RL and Foundation Model eras, where modular architectures are replaced by unified, multimodal, closed-loop pipelines, but the challenges of transfer, abstraction, and sample efficiency are unresolved.

Datasets: Trilemma of Scale, Fidelity, and Diversity

The data pillar is the structural bottleneck that circumscribes all downstream modeling, evaluation, and agent deployment. The work dissects the evolution of game datasets through three eras and a prospective coda:

• Act I: Learning from human play—expensive, precisely labeled but narrow datasets (e.g., MineRL, Atari-HEAD), yielding high-fidelity demonstration but at prohibitive cost.
• Act II: Mining the competitive archive—exploiting platform/game replays, battle logs, and agent-generated trajectories (e.g., SC2LE, Metamon, RL-Unplugged) to sidestep manual annotation and encode implicit reward/skill gradients. Quality is high, but diversity remains constrained.
• Act III: Scaling to multi-game, multi-task corpora—broad coverage via automatic collection and annotation (e.g., ALE, Genie, NitroGen), illuminating the action-space heterogeneity barrier and the challenge of unifying multimodal, multi-format behaviors for generalist pretraining.
• Coda: The world model as data engine—emergent paradigm where generative models (e.g., Genie, GameFactory, Matrix-Game) synthesize data and interactive worlds unconstrained by static code engines, aiming for a self-amplifying data-model flywheel, but with open issues in temporal/spatial consistency, reward calibration, and multi-agent interactivity.

  Figure 3: Evolution of game datasets from isolated single-game collections to multi-game corpora and toward world-model driven data generation.

Despite significant advances, there is no current corpus that simultaneously achieves high scale, annotation quality, and cross-game diversity—a foundational limitation for universal agents.

Models: Toward Unified Perception–Reasoning–Action

The Model pillar captures the architectural progression from perception-disconnected, text-only reasoning to predictive world-aware decision-making:

1. LLMs: Serve as the strategic brain for text-centric games, leveraging world knowledge and adaptable reasoning but lack direct perception or fine-grained control, suffering most in visually complex/temporal domains.
2. VLMs: Integrate visual encoders, enabling agents to ground semantic understanding in pixel-space but typically remain limited to high-level, semantic outputs requiring further translation for real-time control.
3. VLAs: Close the perception-action loop by directly mapping multimodal observations to low-level controls (e.g., keyboard/mouse)—realizing end-to-end sensorimotor pipelines and cross-game transfer, but bottlenecked by the challenge of real-time latency and action abstraction.
4. WAMs: Embody transition dynamics and support predictive/planning-based action generation. This nascent line shows potential for model-based policy improvement, future anticipation, and cross-environment transfer but faces challenges with high-fidelity dynamics in complex, modern games.

   Figure 4: Schematic of model evolution in game-playing AI, from LLMs to VLMs to VLAs and predictive WAMs, showing increasing closure of the perception-action loop and internalization of world dynamics.

Despite progress, all current models face the reasoning–reactivity dilemma: autoregressive architectures cannot simultaneously support deep planning (hundreds of tokens per decision) and millisecond-latency motor control, a critical bottleneck in complex, real-time environments.

Harness: Modular Augmentation and its Costs

The harness functions as the agent's nervous system, enabling memory, multi-modal perception, action abstraction, and adaptive learning since foundation models lack statefulness and persistence by default. This modularity, however, introduces a paradox—while necessary to engineer complex behaviors and close performance gaps, it segments the agent and risks brittle interfaces, poor closed-loop optimization, and non-generalizable scaffolding.

Figure 5: Overview of functional harness architectures, encompassing perception, action, reasoning, reactivity, memory, and adaptive self-improvement modules.

Key systems such as Cradle and Voyager exemplify sophisticated harnesses but at the cost of extensive engineering effort, fragile integration, and semantic–motor execution gaps. Eliminating reliance on modular harnesses while retaining generality remains an open challenge.

Benchmarking: Frameworks for Measuring Agentic Competence

Benchmarking migrates from static knowledge and one-shot reasoning to closed-loop, interactive, agentic evaluation. The paper proposes a unified, multi-dimensional benchmarking taxonomy integrating:

• Environment Organization: An ascending five-level evolutionary hierarchy—from rule compliance, through strategic and social reasoning, visual agency, to cross-game adaptation and open-ended generality.
• Capability Probing: Benchmarks are interpreted as capability probes, with environment affordances, abstraction, and protocol design determining what skills are genuinely measured.
• Interaction/Evaluation Paradigms: From high-privilege interfaces (text APIs, semantic actions) for reasoning analysis to low-privilege (raw visual input, native control) for full agentic assessment, including process-level evaluations, adversarial protocols, and cross-environment calibration.

  Figure 6: Conceptual framework for agent benchmarks with explicit environmental structure, capability focus, and evaluation pipeline.

The study highlights the risk of overinterpreting scores due to privilege abstraction, interface scaffolding, or measurement misalignment, calling for standardized, explicit reporting of interaction and evaluation contracts.

Fundamental Trade-Offs

The interlocking trade-offs that define the horizon of generalist agents are:

• Scale vs. Fidelity vs. Diversity: No existing dataset achieves all three; attempts to increase scale (e.g., NitroGen) sacrifice annotation, while high-fidelity (e.g., OpenP2P) is limited in diversity and cost.
• Breadth vs. Depth: Transfer across games with heterogenous semantics and control breaks down, action abstraction loses domain-specific performance, and cross-game scaling results in competence dilution.
• Reasoning vs. Reactivity: The latency–intelligence ceiling: deep deliberative models cannot act in real-time; chunk-based or parallel methods lose strategic tractability.
• Modular Workflow vs. Model-as-Whole: Current end-to-end agents underperform without engineered harnesses, but such harnesses defeat the goal of universal, plug-and-play agent architectures by imposing extrinsic segmentation and operational disparities.
• Code Engine vs. World Model: Static code engines provide reliable, calibrated evaluation and data generation, but are inflexible and costly per game; contemporary world models offer promise but lack long-term temporal coherence, verifiable reward signals, and multi-game, open-ended generation.

Roadmap to Generalist Game Players

Building on these axes, the study proposes a five-level progression for the development and assessment of truly generalist agents:

1. Single-Game Mastery: Full competence within all tasks of a single, complex environment (e.g., Minecraft), including long-horizon objectives and out-of-distribution generalization.
2. Cross-Task Transfer/Within-Genre Generalization: Transfer learning and zero-shot generalization across games with similar UI/mechanics, but distinct structure/content.
3. Cross-Genre Generalization: Agents that function effectively across structurally and mechanically diverse games; breadth with maintained depth.
4. Lifelong Adaptation: On-the-fly, online learning to rapidly adapt to completely novel environments, rules, and objectives amid sparse feedback and non-stationary task distributions.
5. Demiurge—Agent as Creator: Agents that dynamically generate, simulate, and evolve entire game worlds; the agent is the environment designer and governor, making rules, goals, and physics part of its internal generative process.

   Figure 7: Five-level roadmap toward generalist game players, charting the spectrum from narrow task-solving agents to the 'Demiurge' that creates, simulates, and evolves virtual worlds.

Numerical Results and Strong Claims

• NitroGen achieves 40,000 hours of gameplay for 1,000+ games using automatic annotation, but incurs a reported discrete action error of 4% and continuous control $R^2$ of 0.84, indicating significant compound error over trajectories.
• OpenP2P demonstrates emergent causal reasoning as both model and data scale increase, but with annotation effort confined to 8,300 human hours and 45 games.
• Lumine achieves robust zero-shot transfer for multi-hour story arc completion within visually and structurally similar anime RPGs, but fails on genre-disjoint titles, highlighting the hard boundary for Level-2 generalization.
• SIMA 2 demonstrates doubled performance on held-out environments and successful adaptation to new game settings via self-improvement, but remains dependent on RL post-training and cannot replicate rapid in-game strategic adaptation akin to humans.
• No state-of-the-art agent achieves even basic proficiency in multi-hour, long-horizon objectives in open-ended games like Minecraft's Ender Dragon challenge, nor sustained competence across genre-diverse action and strategy games.
• World models (e.g., Genie 3, GameFactory) can generate novel, interactive environments, and SIMA 2 shows positive transfer inside world-model-generated training loops, but rollouts are limited (e.g., 60s in Genie 3) and long-term consistency is not yet attainable.

Theoretical and Practical Implications

This study formalizes, with unprecedented breadth and rigor, the constraints and development axes for AI as a generalist game player. Practically, it demonstrates robust scaling laws and emergent capabilities for internet-scale behavior cloning and end-to-end sensorimotor pretraining, identifies tractable paths toward unified action representation, and calls for explicit privilege reporting in benchmarks. Theoretically, it places game-playing AI as a microcosm for AGI research, requiring the coupling of reasoning, perception, memory, self-improvement, and the agency to generate novel environments.

Future research should address the engineering of world-model-based data engines that are temporally, spatially, and reward consistent, tackle efficient and reliable unified action abstractions, develop hybrid model architectures that decouple latency from inference quality, and design benchmarks with first-contact novelty and unsolved open-endedness.

Conclusion

This work presents both an exhaustive account and a constructive theoretical program for transitioning from narrow, static specialists to agents capable of true generalization across the game multiverse. It highlights both the progress achieved—particularly through foundation model architectures, scaling of datasets, and harness advances—and the structural bottlenecks that delimit current frontiers: the trilemmas of data, the privilege gaps of modularity, and the operational ceilings imposed by static code environments. The architecture of agentic AI will likely be shaped by the resolution of these trade-offs, and the developmental roadmap outlined here provides a principled guide for future innovation in AI generality through the lens of games.

---

Reference: (2605.09965)