TextWorld: Generative RL in Text Games

• TextWorld is a Python-based, generative platform that formalizes text-based games as MDPs or POMDPs, enabling research in RL and language understanding.
• It features a robust procedural generator and game engine that emulate interactive fiction through controlled complexity, reward configurations, and linguistic variety.
• The platform facilitates benchmarking and advanced research in planning, representation learning, and multimodal extensions by integrating multi-level game design and evaluation protocols.

TextWorld is a Python-based, generative environment for developing, training, and evaluating reinforcement learning (RL) agents in text-based games. Unlike traditional visual or gridworld simulators, TextWorld provides a flexible platform for procedurally generating interactive fiction games formalized as Markov Decision Processes (MDPs) or Partially Observable MDPs (POMDPs), with fine-grained control over environmental complexity, reward structure, and language surface form. This system serves as a testbed for a range of research on planning, grounded language understanding, generalization, exploration, and representation learning in high-dimensional, combinatorial action and observation spaces.

1. System Architecture and Formal Model

TextWorld comprises two principal modules: a procedural Game Generator and a Game Engine/Interpreter. The Generator parameterizes and synthesizes new games via Inform 7, outputting Glulx or Z-machine binaries and full metadata, while the Engine launches the compiled game, tracks game state as a multiset of ground atoms (facts), and computes rewards. The backend separates world simulation (linear-logic rule execution) from the observation function, supporting environments of varying visibility and granularity (Côté et al., 2018). The core RL interface follows the OpenAI Gym standard:

$S$0

The fundamental formalism is either an MDP:

$(S,A,T,R,\gamma)$

where:

• $S$: state space as multisets of ground atoms (linear logic)
• $A$: parameterized action space (grounded rules; e.g. $\text{open}(C)$)
• $T$: transition (deterministic in generated games)
• $R$: reward function, with terminal or intermediate options
• $\gamma$: discount

or a POMDP:

$(S,T,A,\Omega,O,R,\gamma)$

with a textual observation space $\Omega$ and emission function $O$.

2. Procedural Content Generation and Game Design

TextWorld's generator operates in three stages:

• World/Map Generation: Places rooms on a configurable grid with adjustable topology (loops, doors, locks), distributing objects (containers, portables, supporters) throughout (Côté et al., 2018).
• Quest Generation: Constructs action sequences (quests) with forward or backward chaining, ensuring dependency satisfaction and avoiding cycles. Winning conditions can be goal-driven ("retrieve key," "eat apple") or fixed-length.
• Text Generation: Applies context-free grammars for entity naming, description, and instruction; lexical/thematic templates control language diversity, synonym use, and reading difficulty.

Researchers can specify all environment parameters—enabling scalable experiments in curriculum learning, transfer, zero-shot generalization, and difficulty calibration.

3. Benchmarking, Evaluation, and Agent Baselines

TextWorld is equipped with curated suites for rigorous benchmarking:

• Curated List: 50 Infocom-style commercial and custom games analyzed for structural features; used to highlight low agent performance due to task difficulty. Baseline agents include random actors, LSTM-DQN (BYU), and rule-based (Golovin) (Côté et al., 2018).
• Treasure Hunter: Synthetic mazes of controlled complexity; metrics include average score, success rate, and trajectory length in "one-life" settings.

Standard evaluation protocols emphasize:

• Normalized score (relative to max in-game score)
• Success rate (full task completion)
• Step efficiency

Recent RL architectures include attention-based Q-learners, policy gradient methods (PPO, RLOO, GRPO), and agents integrating supervised pretraining, context pruning (CREST), or LLM priors for improved sample efficiency and generalization (Chaudhury et al., 2020, Shihab et al., 12 May 2025, Wang et al., 1 Oct 2025).

4. Representation Learning, World Models, and Memory Architectures

TextWorld drives advances in structured representation learning, state abstraction, and internal world modeling:

• Strict Latent State Mediation: Textual belief states (predicates in JSON) learned under strict mediation enable interpretable, compressive, and empirically testable internal representations, sharply improving rollout stability and generalization with growth in task complexity and planning horizon (Gao et al., 26 Jun 2026).
• Spatio-Temporal Memory Agents: Agents employing both summarizing temporal memory and dynamic knowledge graph spatial memory (as in STMA) achieve substantial improvements in long-horizon planning and robustness over competitive LLM-based baselines (Lei et al., 14 Feb 2025).
• Graph-Based State Extraction: Persistent knowledge graphs built from OpenIE triples and domain rules encode agent-internal world state, supporting action pruning and more efficient deep RL (e.g., KG-DQN) (Ammanabrolu et al., 2018).
• Affordance and Commonsense Augmentation: Context-driven affordance extraction from resources like ConceptNet expands the admissible action space and integrates explicit commonsense, yielding more efficient policy learning (Murugesan et al., 2020, Gelhausen et al., 2022).

5. Language Understanding, Instruction Following, and Multimodal Extensions

TextWorld's expressive text interface enables direct study of natural language processing in RL:

• Instruction Following: Linear Temporal Logic (LTL) formalisms map free-form instructions into temporally-extended, compositional subgoals, enhancing both learning and measurement of progress across complex tasks (e.g., cooking) (Tuli et al., 2022).
• Multimodal Extensions: VisualHints augments TextWorld games with visual map-based hints, supporting RL research in visual-linguistic grounding and navigation under partial observability (Carta et al., 2020).
• Action Generation: Adaptive command generators (pointer-softmax models, hierarchical decoders) achieve near-perfect F1 on command-set prediction, making downstream RL tractable even in highly combinatorial spaces (Tao et al., 2018).

6. Transfer, Generalization, and Embodied Learning

Integrated with embodied agent research, TextWorld serves as an abstract backbone for cross-domain transfer:

• ALFWorld: Links symbolic (TextWorld) and physically-grounded (ALFRED/AI2-THOR) environments via shared PDDL world definitions. Modular agents (e.g., BUTLER) transfer abstract plans learned in TextWorld to visual execution, achieving improved sample efficiency and generalization versus vision-only pipelines (Shridhar et al., 2020).
• OPEx and LLM-Centric Systems: Decoupling perception, planning, and execution in pure-text domains like ALFWorld sharpens the role of reasoning, highlights bottlenecks due to low-level control and perception, and motivates multi-agent architectures for scalable instruction following (Shi et al., 2024).
• Text World Models: Recent research formalizes and expands the use of LLM-driven transition models for simulation, planning, and agent evaluation, grounding environment design choices in principled theory and comparative benchmarks (Li et al., 8 Jun 2026).

7. Open Challenges and Research Directions

Open problems in TextWorld–driven research include:

• Scaling Complexity and Diversity: While procedural generators automate world construction, bridging the gap to human-authored complexity and long-horizon causality remains unsolved (Jansen, 2021).
• Robust Generalization: Agents often overfit to particular vocabularies, entities, or quest structures; methods like CREST partially address this, but zero-shot transfer to new domains and instruction styles is limited (Chaudhury et al., 2020).
• Continual and Multi-agent Learning: Extensions such as LIGHT (social dialogue), multi-agent planning, or curriculum learning are at early stages.
• Unified Evaluation: Standardized complexity metrics, comprehensive benchmarks, and multi-modal extensions (vision, reasoning, social) are needed for ecosystem maturation (Jansen, 2021, Li et al., 8 Jun 2026).

TextWorld’s open, reproducible, and fully generative infrastructure has established it as a foundational platform for studying language-grounded RL, representation learning, planning, and embodied intelligence across both academia and industry (Côté et al., 2018, Jansen, 2021, Shridhar et al., 2020, Gao et al., 26 Jun 2026).