Interactive Fiction Environments

• Interactive Fiction environments are text-based simulation platforms where agents issue natural language commands to interact with hidden world states.
• They integrate reinforcement learning, natural language understanding, and planning to address challenges like partial observability and combinatorial action spaces.
• These environments serve as robust benchmarks for testing sample efficiency and strategic planning, with applications ranging from classic parser games to dynamic narrative systems.

Interactive Fiction (IF) environments are text-based simulation environments in which an agent interacts with a hidden world state exclusively through natural language: issuing free-form commands and receiving purely textual feedback that describes observations, narrative events, or state changes. These environments underlie a significant branch of AI research at the intersection of reinforcement learning, natural language understanding, planning, and commonsense reasoning. IF environments serve as both challenging benchmarks and generative frameworks for investigating the sample efficiency, generalization, and hierarchical reasoning capacities of autonomous agents (Hausknecht et al., 2019, Osborne et al., 2021, Phan et al., 31 Jul 2025).

1. Formal Structure and Core Characteristics

IF environments formalize as (often deterministic) partially observable Markov decision processes (POMDPs) or, in simplified settings, finite-horizon Markov decision processes (MDPs). The canonical specification is:

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, T, \Omega, O, R, \gamma)$$

• State space $\mathcal{S}$: Combinatorial configurations of rooms, objects, NPCs, inventory, and world flags. States are latent and only indirectly accessible through language.
• Action space $\mathcal{A}$: Unbounded, generally comprising all natural-language strings interpretable as commands. Practical implementations restrict this via templates $T$ and vocabulary $V$ (Hausknecht et al., 2019, Osborne et al., 2021).
• Transition function $T(s', s, a)$: (Typically) deterministic update based on command parsing and narrative logic (e.g., Z-machine semantics for Infocom games).
• Observation model $\Omega: S \to O$: Textual descriptions, including current room or scene, object lists, and narrative cues; partial observability is intrinsic, as essential state information is discursively embedded.
• Reward function $R(s, a)$: Sparse, event- or goal-driven; often score increases tied to puzzles/quest completion.
• Discount factor $\gamma$: Typically close to 1, reflecting the long-horizon planning required in extended IF games.

Partial observability, combinatorial action space (e.g., $|V|^4 \sim 10^8$ for 4-token commands), and linguistic variability (paraphrase, ambiguity, affordances) create a complex RL/NLU substrate (Hausknecht et al., 2019, Phan et al., 31 Jul 2025, Osborne et al., 2021).

2. Environment Design: Genres, Benchmarks, and Extensions

IF platforms span a continuum from highly-authored fictional worlds to procedural real-world task environments:

• Classic IF: Handcrafted, parser-based games (e.g., Zork, Anchorhead) wrapped by environments such as Jericho (Hausknecht et al., 2019), presenting open action spaces, rich object hierarchies, and multiple genres (fantasy, mystery, horror).
• Procedural/Synthetic IF: Logic-based engines (e.g., TextWorld, STARLING) generate synthetic games with controlled complexity, facilitating scaling, skill isolation, and curriculum learning (Osborne et al., 2021, Basavatia et al., 2024).
• Real-world Task IF: ScriptWorld grounds each scenario in daily human activities (e.g., "baking a cake") constructed from gold-aligned script datasets (DeScript), yielding real-world task graphs with paraphrastic variability (Joshi et al., 2023).
• Branching/Imaginative IF: WHAT-IF exploits LLM meta-prompting for the generation of dynamically branching narrative structures from pre-existing linear plots, supporting massive combinatorial exploration of "alternate timelines" (Huang et al., 2024).

Scenario generation pipelines exploit aligned event structures, paraphrase expansion, and action-distractor sampling strategies, resulting in highly variable environments for both gameplay and research (Joshi et al., 2023, Chen et al., 2023, Basavatia et al., 2024, Huang et al., 2024).

3. Technical Challenges: Action Space, State Representation, and Language

3.1 Combinatorial Action Spaces

• The natural-language command space is intractably large. Template-based pruning (e.g., choosing from context-sensitive verbs and argument slots) or candidate enumeration (using valid-action oracles) is essential (Hausknecht et al., 2019, Osborne et al., 2021).
• Recent systems employ external commonsense KBs (e.g., ConceptNet) and affordance extraction to augment command generation, though domain coverage and ambiguity persist (Gelhausen et al., 2022).

3.2 State and World Modeling

• Symbolic knowledge graphs—tracking locations, entities, states, and relations—enable systematic exploration, long-term planning, and action validation (Hausknecht et al., 2019, Ammanabrolu et al., 2021).
• State-update functions may involve rule-based extraction, QA-based extraction, or sequence-to-sequence modeling to capture the dynamic world graph, supporting navigation, inventory management, and causal reasoning (Ammanabrolu et al., 2021, Ammanabrolu et al., 2020).

3.3 Language Understanding and Feedback

• Observations are free-form, context-dependent, and require both surface parsing and commonsense inference (involving spatial, causal, and object-relational reasoning) (Yu et al., 2022).
• Multi-hop reasoning over past observations and integrating object-centric retrieval mechanisms (e.g., multi-paragraph reading comprehension) is necessary to resolve the partial observability (Guo et al., 2020).

4. Agent Architectures and Learning Paradigms

Agents operating in IF environments integrate NLU, structured memory, and planning:

Approach	Features	Representative Work
Value-based RL (DQN, DRRN)	Q-value over action/state	(Hausknecht et al., 2019, Guo et al., 2020)
Policy-gradient/Actor-Critic	Policy/value splits	(Joshi et al., 2023, Basavatia et al., 2024)
Choice-based RL with LM Encoders	Textual action embeddings	(Joshi et al., 2023, Osborne et al., 2021)
Memory-augmented, KG-based	Dynamic world/SLAM graphs	(Hausknecht et al., 2019, Ammanabrolu et al., 2021)
Cognitive-inspired frameworks	Map-building, action learning, feedback-driven adaptation	(Zhang et al., 18 May 2025)
LLM-driven imitation/zero-shot	Prompt-chained decisions	(Zhao et al., 2023, Huang et al., 2024, Yuan et al., 9 May 2025)

Key technical innovations include:

• Integrating pretrained LLM representations (e.g., SBERT, GPT-3, ALBERT) for both observation and command encoding (Joshi et al., 2023, Ammanabrolu et al., 2020).
• Structured memory: Explicit symbolic KGs or episodic memory libraries supporting experience retrieval and reflection (Hausknecht et al., 2019, Zhang et al., 18 May 2025).
• Modular hierarchical control (e.g., NAIL) using domain-specialized sub-policies (exploration, combat, inventory management) with symbolic arbitration (Hausknecht et al., 2019).

5. Evaluation Protocols, Benchmarks, and Metrics

Evaluation in IF environments employs multiple modalities:

• Normalized Score: Average agent score divided by game maximum (e.g., 1.8% for random agent, 10.7% for DRRN in Jericho) (Hausknecht et al., 2019).
• Game Progress: Fraction of expert-labeled checkpoints reached in long-horizon benchmarks (e.g., TextQuests) (Phan et al., 31 Jul 2025).
• Step Efficiency: Number of actions to completion or first sub-goal (Basavatia et al., 2024, Zhang et al., 18 May 2025).
• Human Baseline: Sample efficiency and coverage compared to human players (Basavatia et al., 2024).
• Functional Commonsense: Multi-choice next-observation or action prediction accuracy, focusing on functional rather than factual knowledge (Yu et al., 2022).

Benchmarks:

• Jericho: Over 30 classic parser-based IF games; unified Gym API; valid-action detection; world-object tree extraction (Hausknecht et al., 2019).
• TextWorld, STARLING: Synthetic task/environment generators supporting skill isolation, procedural curriculum generation, and RL diagnostic tasks (Osborne et al., 2021, Basavatia et al., 2024).
• ScriptWorld: 10 daily real-world tasks with paraphrase variability; metrics: average episode reward, learning curve, cross-scenario transfer (Joshi et al., 2023).
• TextQuests: Infocom suite; emphasis on long-horizon reasoning, trial-and-error in single-shot settings; "Game Progress" and "Average Harm" as novel metrics (Phan et al., 31 Jul 2025).

6. Research Directions, Applications, and Practical Extensibility

Research Frontiers:

• Transfer and generalization—pretraining on synthetic games (TextWorld, STARLING) to human-authored games (Jericho, Infocom, ScriptWorld) (Basavatia et al., 2024, Osborne et al., 2021, Joshi et al., 2023).
• Continual, curriculum and meta-RL—exploring unsupervised, skill-compositional exploration over families of IF tasks (Osborne et al., 2021, Basavatia et al., 2024).
• Hierarchical RL—learning options/macro-actions for decomposing long horizon quests (Osborne et al., 2021).
• Commonsense and multi-hop reasoning—core focus of JECC commonsense datasets derived from IF walkthroughs (Yu et al., 2022).

Practical Usage and Extensibility:

• Open-source frameworks: Jericho, ScriptWorld, STARLING, and modeling datasets (JerichoWorld) enable rapid environment extension and standardized evaluation (Joshi et al., 2023, Ammanabrolu et al., 2021, Basavatia et al., 2024).
• Parser-based/free-form as well as choice-based interfaces; option for switching between distractor-based choices and natural language command input (Joshi et al., 2023, Hausknecht et al., 2019).
• Integration of external KBs (ConceptNet), LLMs for command parsing, hint generation, paraphrase alignment, and dynamic narrative extension (Gelhausen et al., 2022, Joshi et al., 2023, Zhao et al., 2023).

Emergent Applications:

• Empathy and role-taking in social and occupational settings with LLM-based perspective-taking IF (Yuan et al., 9 May 2025).
• Multimodal and immersive branching narrative systems (WHAT-IF, NarrativePlay) that leverage LLMs for meta-prompted non-linear storytelling, proactive character modeling, and dynamic user interaction (Huang et al., 2024, Zhao et al., 2023).
• VR, physiological, and real-world-knowledge extensions to IF (VIF, interactive narrative VR tools, ScriptWorld) (Frey, 2016, Ostrin et al., 2019, Joshi et al., 2023).

7. Outlook: Open Issues and General Principles

Although LLMs and RL agents have substantially improved sample efficiency and gameplay robustness in IF environments, fundamental challenges persist:

• Long-horizon credit assignment and efficient exploration under partial observability and sparse rewards (Osborne et al., 2021, Phan et al., 31 Jul 2025).
• Compositional generalization: robust handling of paraphrase, synonymity, and object affordances across tasks, scenarios, and genres (Joshi et al., 2023, Ammanabrolu et al., 2021).
• Interpretability and explainability: developing modular, human-readable world models (KGs, episodic experience libraries) that facilitate debugging and transparent policy improvement (Hausknecht et al., 2019, Zhang et al., 18 May 2025).
• Safe and meaningful narrative control for branching LLM-based IF, ensuring narrative coherence, thematic alignment, and content moderation at scale (Huang et al., 2024, Zhao et al., 2023).

Generalizable principles for IF environment research converge on modular memory structures, retrieval-augmented or feedback-driven prompting, curated benchmarks with functional commonsense, and scaffolding agents with both symbolic (KGs, explicit mapping) and neural representations (Zhang et al., 18 May 2025, Hausknecht et al., 2019, Ammanabrolu et al., 2021). These environments continue to provide a comprehensive testbed for the study of grounded language understanding, adaptive reasoning, and interactive narrative generation in AI.