- The paper introduces TextArena, an open-source framework that evaluates and trains LLMs via competitive text-based games.
- It integrates a Gymnasium-like API with over 57 diverse game environments, using TrueSkill for real-time, dynamic performance tracking.
- The framework assesses soft skills like strategic planning and theory of mind, providing granular insights and training data for advanced LLMs.
TextArena is introduced as an open-source framework designed for evaluating and training LLMs in agentic behavior through competitive text-based games (2504.11442). It addresses the limitations of traditional static benchmarks like MMLU and HumanEval, which are becoming saturated and fail to assess dynamic social skills such as negotiation, theory of mind, deception, and long-term planning.
The core components of TextArena include:
- Diverse Game Environments: A collection of over 57 unique text-based games at the time of publication (since expanded to 74+), covering single-player, two-player, and multi-player scenarios. These games test a wide array of cognitive and social skills.
- Unified Framework: An interface similar to OpenAI Gym (Gymnasium), facilitating ease of use for researchers familiar with reinforcement learning (RL) frameworks and enabling RL training on these environments.
- Online Evaluation Platform: A system for real-time online competition between different LLMs and between LLMs and human players.
- TrueSkill Leaderboard: Performance is tracked using the TrueSkill rating system, providing dynamic, relative rankings for models and a collective "Humanity" baseline. TrueSkill is chosen for its ability to handle varying player numbers, team vs. individual play, and faster convergence compared to Elo.
- Soft-Skill Profiling: Games are tagged with specific skills (e.g., Strategic Planning, Theory of Mind, Bluffing). Model performance across games provides a weighted average score for each skill, offering granular insights into strengths and weaknesses.
- Extensibility and Community Focus: Designed for easy addition of new games and scenarios, encouraging community contributions via GitHub and Discord.
The motivation behind TextArena stems from the need for more challenging and dynamic evaluation methods as LLMs approach human-level performance on existing benchmarks. Static benchmarks are prone to saturation and don't capture interactive capabilities. TextArena offers a relative performance measure through competition, avoiding a fixed upper performance limit. It also aims to provide a potentially vast source of training data for RL, particularly for developing agentic reasoning models, aligning with recent trends highlighted by models like DeepSeek-R1.
TextArena adopts a Gymnasium-like API for its environments, making it straightforward for developers to integrate LLMs as agents. The provided code example demonstrates initializing agents (e.g., OpenRouterAgent) and an environment (ta.make), interacting through standard reset, get_observation, step, and close methods. An LLMObservationWrapper is shown to format observations suitably for LLMs. Adding models for online play is also streamlined.
Compared to related work like Clembench, LMRL-Gym, GameBench, and others, TextArena offers a significantly larger and more diverse set of environments across single, two, and multi-player formats. Crucially, it uniquely combines a Gym-compatible API, online evaluation, model-vs-model play, and model-vs-human play within a single platform.
Future plans include explicitly focusing on RL training using TextArena environments, fostering community engagement, releasing gameplay data (including interactions with frontier models like OpenAI o1), and extending the concept to frame-based environments (VideoGameArena).