AI-Powered Interactive Storytelling

• AI-powered interactive storytelling is a system that combines narrative structures and interactive mechanisms to allow user-guided story evolution.
• It employs modular architectures that integrate neural generation, symbolic filtering, and explicit control schemes to maintain coherence and safety.
• Applications span AI-native games, immersive installations, and accessibility tools, demonstrating practical benefits in dynamic narrative generation.

AI-powered interactive storytelling denotes a family of systems in which AI mediates narrative progression, world construction, or story interpretation while users retain meaningful influence over the unfolding experience. Across recent work, the topic encompasses conversational co-authorship for children, AI-native games, multimodal installations in Unreal Engine and CAVE environments, social robots, interactive video and image authoring tools, writer-facing narrative planners, and accessibility systems for blind and low vision audiences. A recurring technical concern is how to preserve narrative coherence while supporting user agency, whether the interaction is implemented as dialogue, branching navigation, embodied play, or timeline-based media editing (Xu, 5 Aug 2025, Burtenshaw, 2020, Sun et al., 2023, Lu et al., 25 Feb 2025).

1. Conceptual foundations and domain boundaries

A useful general formulation appears in work on accessibility for blind and low vision users: AI-powered interactive storytelling combines a narrative structure with an interaction mechanism. The narrative structure organizes images, video, scenes, or events into a storyline or branching graph, while the interaction mechanism lets participants steer that structure through follow-up questions, branch choices, comments, or exploration without breaking narrative continuity (Xu, 5 Aug 2025). This formulation also fits dialogue systems such as AI Stories, where a child’s utterance seeds the next line of a jointly created narrative world, and systems such as StoryBuddy or ELLA, where AI scaffolds story-centered interaction while preserving parental or child agency (Burtenshaw, 2020, Zhang et al., 2022, Antony et al., 12 Mar 2026).

The literature does not restrict the topic to chatbot dialogue. Some systems are explicitly framed as AI-native, meaning that generative AI is not a bolt-on feature but the core mechanic without which the experience cannot exist; “1001 Nights” is the clearest example, since spoken or written keywords materialize as battle equipment and alter the game world (Sun et al., 2023). Other systems treat AI as an authoring collaborator rather than as an in-world actor. ImageTeller converts uploaded images into genre-conditioned chapters and illustrations, SportsBuddy turns sports footage into context-driven highlight stories through tracking and captions, DiaryPlay converts a single-branch diary-like text into an interactive vignette, and Fabula exposes hierarchical scene–beat plans for fiction writers (Lima et al., 2024, Lin et al., 12 Feb 2025, Xu et al., 15 Jul 2025, Mirowski et al., 12 Jun 2026).

A second conceptual boundary concerns the relation between authored and emergent narrative. WhatELSE formalizes a narrative possibility space as the set of story instances authors intend players might experience, derived from example “pivot” stories through summarization into outlines and recompilation into variants (Lu et al., 25 Feb 2025). NarrativeLoom, by contrast, grounds its design in Campbell’s Blind Variation and Selective Retention theory, explicitly separating diverse AI generation from human curation (Ma et al., 7 Mar 2026). Taken together, these systems suggest that AI-powered interactive storytelling is not a single genre but an umbrella for computational narrative systems that vary in how much of the narrative space is pre-bounded, procedurally unfolded, or negotiated at run time.

2. Architectural patterns and computational organization

A strikingly consistent property across the literature is modularity. AI Stories is organized as a collection of loosely coupled modules: an NLU/input handler, three subsystem responders, a dialogue manager/selector, and a surface realizer. The selector is formulated as a POMDP $(S,A,R)$ over dialogue state, candidate replies, and a reward measuring transition quality, with Q-learning used to choose which subsystem output to surface:

$$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$

This architecture explicitly separates retrieval, neural generation, template generation, and symbolic filtering rather than collapsing narrative control into a single model (Burtenshaw, 2020).

Comparable orchestration appears in more recent multimodal systems. ImageTeller uses a Plot Manager to coordinate three specialized agents: a Visual Analyzer based on GPT-4o Vision, a Storywriter based on GPT-4o, and an Illustrator based on Stable Diffusion XL (“Juggernaut XL”). The final prompt for the Storywriter is assembled compositionally from general instructions, story-driven or data-driven instructions, optional genre instructions, and ordered image descriptions, rather than from raw multimodal input alone (Lima et al., 2024). Storycaster similarly distributes work across geometry capture, cube-map rendering, cylindrical reprojection, diffusion-based image synthesis, audio generation, TTS, and a Narrator Agent that elicits scene parameters and coordinates subservices via a server running the Model Context Protocol (Agarwal et al., 26 Oct 2025).

Other systems embed the same modular logic in domain-specific production pipelines. SportsBuddy uses a cloud-deployed distributed backend with Python Flask, Kafka, and Azure GPU nodes, processing uploaded video in parallel branches for player detection and tracking, segmentation, and AI captioning; processed outputs are then exposed to a React editor with an interactive timeline of render objects (Lin et al., 12 Feb 2025). Digital Einstein divides the problem into input sensing, conversational AI, retrieval-based memory, persona integrity, emotion modeling, voice synthesis, facial animation, body animation, and environment integration, explicitly presenting character interaction as a systems-integration problem rather than a pure language-modeling problem (Wampfler et al., 3 Jan 2026). ELLA follows the same principle in a home robot setting, decomposing the pipeline into input sensing, content and dialogue generation, multimodal behavior planning, and robot actuation (Antony et al., 12 Mar 2026).

This repeated architectural choice corrects a common oversimplification: the literature does not treat interactive storytelling as synonymous with end-to-end text generation. Instead, it repeatedly introduces state machines, retrieval, heuristics, symbolic buffers, ranking modules, render timelines, animation controllers, and safety filters alongside LLM calls.

3. Narrative generation, planning, and control of story space

The central algorithmic problem is not merely generating fluent text but constraining generation so that interaction remains legible, coherent, and responsive. AI Stories exemplifies a hybrid approach. Its context-based subsystem is a seq2seq model that conditions next-token prediction on a sliding window of the last $k$ turns,

$P(w_t \mid w_{<t},x),$

while the language-model objective is standard cross-entropy over ground-truth next tokens. Yet this neural continuation is only one candidate among factual QA and template-based poetry/humor, and the selector biases among them according to dialogue state, explicit cues, and lexical rules (Burtenshaw, 2020).

Many later systems formalize control through explicit intermediate structures. In “1001 Nights,” keyword extraction is computed against a whitelist $$K=\{\text{sword, shield, dagger, knife, blade, wand}\}$$, and keyword presence drives both asset acquisition and narrative branching. Once enough assets accumulate, the experience transitions from storytelling to battle, so the story state is not only semantic but gameplay-operational (Sun et al., 2023). DiaryPlay converts a natural-language story into a branch-and-bottleneck graph: bottleneck nodes preserve the author’s intended key-event sequence, while branching nodes allow NPCs to react differently when the player character deviates. WhatELSE similarly exposes an outline-level event sequence and variant-level scatterplot, using authorial-intent distance and emergence distance to help authors inspect how far generated variants move from a pivot story (Xu et al., 15 Jul 2025, Lu et al., 25 Feb 2025).

A separate line of work uses multi-candidate ideation rather than explicit branching graphs. NarrativeLoom’s BVSR formulation generates ten parallel beat candidates, each from a distinct GPT-4o persona, then ranks them with a soft coherence check before presenting them to the user for selection and editing. Its formal selection model

$$b^*=\arg\max_{b\in B}\bigl(\alpha\,c(b)+(1-\alpha)\,U(b)\bigr)$$

makes human judgment an explicit part of the optimization target rather than a downstream editorial afterthought (Ma et al., 7 Mar 2026). Fabula adopts a different hierarchical control mechanism, decomposing a story into scenes and beats, then using generate–evaluate–select loops and an auto-rater based on 63 guideline statements. Its overall auto-rating is

$$Q_{\text{auto}}(d)=\frac{1}{63}\sum_{k=1}^{63} r_k(d),$$

with $r_k(d)\in\{1,\dots,5\}$ (Mirowski et al., 12 Jun 2026).

The same control logic extends beyond text. Toyteller introduces a shared semantic space in which motions and text are projected onto action embeddings plus an active-agent flag, enabling motion-steered text generation and text-steered motion generation (Chung et al., 23 Jan 2025). Tinker Tales uses a pointer-based finite-state machine and prompt templates to enforce a four-stage story arc while allowing children to define characters, places, items, and emotions through NFC scans and speech (Choi et al., 17 Apr 2025). These systems indicate that “interactive storytelling” is less about unconstrained generation than about the design of intermediate narrative representations—keywords, beats, outlines, flags, semantic embeddings, or scene/beat plans—that regulate how user input becomes story progression.

4. Interaction modalities, embodiment, and multimodal realization

AI-powered interactive storytelling is increasingly multimodal in both input and output. ImageTeller begins from single images or image sequences, optionally augmented with captions, then generates chaptered narratives and corresponding illustrations. Its genre control is prompt-based rather than classifier-based: Comedy, Romance, Tragedy, Satire, Mystery, Data, or None alter how GPT-4o interprets the ordered image descriptions, and each chapter is paired with a “single significant event description” that drives Stable Diffusion XL (Lima et al., 2024). Memory Remedy uses a different multimodal pipeline—story concept to hypertext novel to storyboard to Unreal Engine—with Skybox AI generating 360° HDRI panoramas, AI voice-over for scene narration, and hypertext buttons for deterministic branching across nonlinear flashbacks (Han et al., 2024).

Embodied systems push the same idea further by placing AI characters in immersive environments. “The Dream Within Huang Long Cave” uses a GPT-based model accessed via the Inworld AI plugin, a Metahuman avatar named YELL, multilingual speech recognition, and a CAVE installation. Dialogue history and life-stage flags are maintained in the LLM session context, while puzzle conditions and Boolean flags in Unreal Engine determine chapter progression across YELL’s ages 11, 24, and 36 (Huang et al., 7 Apr 2025). Storycaster also operates in a CAVE-like room-scale setting but emphasizes physical-room augmentation: four Azure Kinect cameras reconstruct the room, six projectors cover four walls, SDXL with Depth-ControlNet and 360°-LoRA generates environment imagery conditioned on cylindrical depth, and object-level editing maps real furniture to story-themed virtual counterparts through mask-based inpainting (Agarwal et al., 26 Oct 2025).

Other embodiments are smaller-scale but equally structured. ELLA is an autonomous social robot for early language development whose storytelling behaviors are generated from GPT-5 outputs, then translated into timestamped face animations and joint trajectories compiled into anime.js facial motions and servo commands (Antony et al., 12 Mar 2026). Tinker Tales blends physical and digital interaction through an accordion board, NFC-tagged pawns and tokens, STT, TTS, and GPT-4o prompt templates (Choi et al., 17 Apr 2025). Toyteller reduces embodiment to abstract triangular symbols whose trajectories become both an expressive input modality and a visual output format, showing that anthropomorphized motion alone can convey socially meaningful narrative cues (Chung et al., 23 Jan 2025).

Multimodality also serves accessibility and professional media practice. The BLV systems Memory Reviver, DanmuA11y, and Branch Explorer use hierarchical narration, time-synced comments with spatial audio, and branching 360° navigation to keep exploration inside the main narrative flow rather than in detached assistive menus (Xu, 5 Aug 2025). SportsBuddy integrates object tracking, pose estimation, segmentation, captions, spatial overlays, and timeline linking so that video stories become editable at the level of tracked players and tactical annotations rather than through conventional nonlinear editing alone (Lin et al., 12 Feb 2025). The cumulative picture is that AI-powered interactive storytelling increasingly involves coupled transformations across text, image, sound, motion, and spatial context.

5. Evaluation practices and empirical findings

The empirical base is heterogeneous. Some systems remain at the stage of planned studies or illustrative case reports. AI Stories proposed pediatric-ward evaluation using Engagement Rate, Story Diversity Score, qualitative Likert feedback, paired $t$-tests against a baseline storyteller, and ANOVA by age group, but reported no formal user data and only early simulated chats showing approximately 15% longer dialogues than a single-subsystem baseline (Burtenshaw, 2020). ImageTeller likewise presented examples and implementation details while leaving user studies and baseline comparisons as future work (Lima et al., 2024). Memory Remedy reported a small pilot with $N=13$ participants and uniformly positive qualitative feedback, but no formal quantitative metrics (Han et al., 2024). “The Dream Within Huang Long Cave” relied on museum and festival feedback, anonymous comment cards, and qualitative observations rather than surveys or physiological measures (Huang et al., 7 Apr 2025).

By contrast, several systems report controlled studies, field deployments, or large usage logs. ELLA was deployed with ten children for eight days: children engaged on 5.9/8 days ($$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$0), averaged 2.56 stories per active day, spent an average of 5.8 min per session ($$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$1), and produced a total of 14.0 h of story time. Average words per turn rose from approximately 3.6 in days 1–4 to approximately 4.5 in days 5–8 (Wilcoxon $$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$2), and a custom PPVT-style pre/post test showed a median gain of 3 words (Wilcoxon $$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$3) (Antony et al., 12 Mar 2026). StoryBuddy’s study with 12 parent–child dyads found average session durations of 18 min for co-reading and 17 min for bot-reading, with SUS above 80 and unanimous parent reports of decreased cognitive load and improved engagement (Zhang et al., 2022). Tinker Tales evaluated 30 simulated sessions and reported mean human ratings of 5.00 for Elements Relevancy, 4.77 for Narrative Coherence, and 4.97 for Educational Value, while moderation and Perspective scores were lower than those of a baseline children’s-stories corpus in every reported safety category (Choi et al., 17 Apr 2025).

Creativity-support tools have adopted more formal comparative designs. NarrativeLoom’s within-subjects study with $$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$4 compared the system against a single-persona GPT-4o chatbot baseline and found higher user-rated diversity (4.08 vs. 3.66; $$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$5, $$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$6), much higher word count (3803 vs. 1908; $$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$7, $$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$8), higher location count and dialogue ratio, and expert preference for NarrativeLoom in 38 of 40 forced-choice pairs. Expert-rated Torrance Test creativity scores improved across fluency, flexibility, originality, and elaboration, with overall creativity 9.72/14 versus 5.00 (Ma et al., 7 Mar 2026). DiaryPlay’s preliminary technical evaluation found its Controlled Divergence module significantly better than a random baseline and not significantly different from human-authored NPC activities ($$Q(s,a)\leftarrow Q(s,a)+\alpha\bigl(R(s,a)+\gamma\max_{a'}Q(s',a')-Q(s,a)\bigr).$$9), while its user study reported vignette creation in 12–19 min ($k$0 min), 100% post-view recall of intended key events, and replay by 15/16 viewers to explore divergent paths (Xu et al., 15 Jul 2025). WhatELSE’s within-subject user study with $k$1 reported higher perceived control over the outline, higher satisfaction editing it, better alignment of the generated game to the intended moral, higher overall plot satisfaction, and less need for post-editing than a ChatGPT-style baseline (Lu et al., 25 Feb 2025).

Large-scale deployments are also present. “1001 Nights” logged approximately 2,000 gameplay sessions with an average of 12 story turns before phase switch, more than 85% of sessions producing at least two valid keywords, a mean agency score of 4.2/5 from a post-survey of roughly 200 players, and 0% success across more than 200 jailbreak attempts at Gamescom (Sun et al., 2023). SportsBuddy reported 163 registered users three months after launch, 1,021 uploaded videos with a 90.8% upload success rate, 814 exported highlights with an 87.9% export success rate, average time-to-create below 5 min for a full highlight, and inter-rater agreement on usability improvement with mean 4.3 and $k$2 on a 5-point Likert scale (Lin et al., 12 Feb 2025). Storycaster’s $k$3 study found the narrator most impactful (mean approximately 4.2, 92.3% positive), audio next (mean approximately 4.0, 69.3% positive), and 84.7% positive ratings on immersion, ease of use, and adoption intent, while also revealing that about 8/13 participants perceived multisecond image transitions as breaking narrative flow (Agarwal et al., 26 Oct 2025). For BLV users, each of the three accessibility systems was evaluated with twelve participants, and all three reported significant improvements on their target measures, including comprehension, immersion, smoothness, engagement, and sense of agency (Xu, 5 Aug 2025).

6. Recurring tensions, misconceptions, and likely research directions

A persistent misconception is that interactivity is best served by maximal openness. The surveyed systems more often constrain interaction through finite-state machines, bottleneck nodes, chapter gates, branch thresholds, memory buffers, or author-facing abstraction controls. AI Stories filters replies against stored entity facts and lexical rules, DiaryPlay reconverges divergent interaction onto bottleneck events, WhatELSE lets authors shift an outline up or down an abstraction ladder, and Fabula exposes scene–beat decompositions instead of only free-form script generation (Burtenshaw, 2020, Xu et al., 15 Jul 2025, Lu et al., 25 Feb 2025, Mirowski et al., 12 Jun 2026). This suggests that, in practice, agency is usually engineered through bounded possibility spaces rather than unrestricted continuation.

Safety, reliability, and breakdown repair form a second recurrent tension. AI Stories recommends hybrid RL plus symbolic filters because hand-coded lexical rules catch safety and style issues that an end-to-end neural RL system alone often misses (Burtenshaw, 2020). Tinker Tales evaluates outputs with both human judges and two industry moderation services (Choi et al., 17 Apr 2025). StoryBuddy reports speech-recognition errors and therefore advocates multimodal repair through GUI controls as well as voice (Zhang et al., 2022). “1001 Nights” treats jailbreak resistance as a first-class evaluation dimension through structured JSON outputs, role prompting, and guardrails (Sun et al., 2023). These examples undermine the assumption that narrative quality alone is the relevant criterion; operational safety and interaction robustness are integral to the design space.

A third tension concerns latency, fidelity, and authorship. Storycaster achieves an end-to-end act latency of approximately 12–14 s, enough for many participants to notice a break in flow (Agarwal et al., 26 Oct 2025). Fabula’s writers valued structural coherence but criticized generated dialogue as generic or formulaic, while its adversarial-design process surfaced concerns about authorship, deskilling, and “AI ghostwriting,” particularly among experts and theatre practitioners (Mirowski et al., 12 Jun 2026). The same paper also records criticism that prevailing narratological assumptions can privilege Western, interiority-driven story models and surface “white, cis-male, middle-class defaults.” A plausible implication is that future systems will need not only better models but also more plural narrative theories, more editable planning layers, and clearer divisions of labor between human and machine.

The future directions named in the literature are notably concrete. “1001 Nights” proposes richer multimodal worlds through text-to-3D, adaptive combat difficulty, multi-agent dialogue, and persistent world state in natural language (Sun et al., 2023). Memory Remedy suggests adding truly generative LLMs for on-the-fly continuation, conversational interfaces, and AR/VR deployment (Han et al., 2024). Storycaster points to gesture-based triggers, animated objects, dynamic audio tied to user movement, diffusion-video models, and neural super-resolution (Agarwal et al., 26 Oct 2025). The BLV agenda prioritizes real-time narration generation, more expressive spatial audio and haptics, and extension to games, VR, and AR (Xu, 5 Aug 2025). ELLA’s design opportunities emphasize caregiver summaries, multimodal readiness estimation, sibling participation, and flexible “quick mode” sessions (Antony et al., 12 Mar 2026). Fabula proposes pedagogical tutoring, script-upload with branching continuations, and live-performance modes where actors’ improvisations feed on-the-fly scene–beat generation (Mirowski et al., 12 Jun 2026).

Taken together, the literature presents AI-powered interactive storytelling as a technically hybrid field defined less by any single model family than by the orchestration of narrative representations, interaction mechanisms, and multimodal interfaces. Its mature problems are no longer limited to text generation. They include authorial control over possibility spaces, state tracking across long horizons, alignment between user intention and generated action, safe and legible collaboration, cultural localization of narrative theory, and the real-time integration of language, image, sound, motion, and physical space.