Game-TARS: A Generalist AI Agent

• Game-TARS is defined as a generalist AI agent that employs a unified keyboard and mouse control system to operate seamlessly across digital interfaces.
• It leverages extensive pretraining on over 500 billion tokens, integrating vision-language models and autoregressive policies to mimic human cognitive processes.
• Experimental results show it achieves twice the success rate in Minecraft and demonstrates robust generalization across diverse digital tasks.

Game-TARS: Overview and Purpose

Game-TARS represents an innovative paradigm in the development of AI agents capable of operating across diverse digital environments, including video games, web interfaces, and desktop GUIs. It is designed as a generalist game agent with a unified action space anchored to human-aligned keyboard and mouse inputs, contrasting with the traditional domain-specific approaches that limit scalability and generalization. Game-TARS leverages a substantial pretraining dataset to build multimodal competencies, thus providing an unprecedented breadth of computer-use abilities.

Architecture and Core Components

Autoregressive Model Design: Game-TARS employs an autoregressive policy $\pi_\theta$, which integrates vision-LLM backbones, such as vision transformers (ViT) and LLMs, including both Mixture-of-Experts (MoE) and dense variants. This allows it to generate actions and internal reasoning steps in a manner similar to human cognitive processes, following the ReAct paradigm where thoughts and actions alternate.

Unified Action Space: The main architectural advancement is grounding the agent’s actions in native human input devices—keyboard and mouse—facilitating universal applicability across digital interfaces. This human-native interaction approach enables seamless transition between game environments, operating systems, and web navigation without the need for environment-specific adjustments.

Pretraining Methodology

Game-TARS benefits from extensive pretraining on a dataset exceeding 500 billion tokens, composing:

• Image Tokens: 208 billion from screen captures.
• Text Tokens: 326 billion from various language-based interactions.
• Diverse Trajectories: Over 20,000 hours covering approximately 500 games.

Data Collection and Labeling: The use of online think-aloud protocols allows annotators to verbalize reasoning, synchronizing actions with video frames. Sparse reasoning steps ensure efficient cognitive modeling, reducing unnecessary computational overhead.

Technical Innovations

Decaying Continual Loss Function: Addressing causal confusion, Game-TARS introduces a loss weighting scheme that strategically reduces the influence of repetitive actions on the training signal, enhancing focus on transitional points and meaningful decisions.

Sparse Thinking Strategy: The agent applies sparse reasoning to balance inferential depth against computational cost, only generating thoughts at critical decision points. This approach mimics human deliberative processes, optimizing task performance and sample efficiency.

Experimental Results and Performance

Minecraft Benchmark: Game-TARS demonstrates approximately twice the success rate over existing state-of-the-art models in open-world Minecraft tasks, and exhibits adept performance in unseen web 3D games and FPS benchmarks. It surpasses other notable models like GPT-5 and Claude-4-Sonnet in competitive environments, achieving high success rates across multiple task categories.

Scale and Generalization: The agent’s performance improves with increased data diversity, validating the potential of its unified action space for scalable cross-domain application. The real-world implications are profound, pointing to its capability to serve as a generalist digital agent with broad operational proficiency.

Ablation Studies and Analysis

Sparse Thinking Ablation: Empirical results underscore the efficacy of sparse cognitive modeling, demonstrating optimal performance and improved efficiency compared to other reasoning strategies.

Loss Function Ablation: Analysis reveals that the decaying loss function substantially enhances model robustness, operational diversity, and efficiency in non-trivial action scenarios.

Implications and Broader Impact

Game-TARS sets a pivotal benchmark for generalist agents, laying foundational methodologies that integrate scalable pretraining, human-aligned interaction frameworks, and sophisticated cognitive modeling techniques. Its ability to seamlessly navigate and excel across varied digital interfaces marks a significant leap in AI agent research, promising a future where cross-domain operability and generalization are commonplace.

Conclusion

Game-TARS exemplifies the next evolutionary step in AI agent design, combining efficiency in multimodal pretraining with human-aligned actions to deliver unmatched generalization and adaptability in digital environments. Its achievements pave the way for expansive applications in AI, supporting diverse interactions in computer-use scenarios while setting a new standard for generalist digital agents. Further exploration and refinement of its architecture and methodologies will likely drive continued advancements in AI-enabled integration into everyday digital tasks.