Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
98 tokens/sec
GPT-4o
61 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Large Language Models and Video Games: A Preliminary Scoping Review (2403.02613v1)

Published 5 Mar 2024 in cs.HC and cs.AI

Abstract: LLMs hold interesting potential for the design, development, and research of video games. Building on the decades of prior research on generative AI in games, many researchers have sped to investigate the power and potential of LLMs for games. Given the recent spike in LLM-related research in games, there is already a wealth of relevant research to survey. In order to capture a snapshot of the state of LLM research in games, and to help lay the foundation for future work, we carried out an initial scoping review of relevant papers published so far. In this paper, we review 76 papers published between 2022 to early 2024 on LLMs and video games, with key focus areas in game AI, game development, narrative, and game research and reviews. Our paper provides an early state of the field and lays the groundwork for future research and reviews on this topic.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. Evaluating Multi-Agent Coordination Abilities in Large Language Models. arXiv preprint arXiv:2310.03903 (2023).
  2. Towards Grounded Dialogue Generation in Video Game Environments. (2023).
  3. A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games. In Findings of the Association for Computational Linguistics: EMNLP 2023. 2295–2311.
  4. Ashish Amresh. 2023. Integrating Reinforcement AI Into the Design of Educational Games. In Proceedings of the 17th European Conference on Game-Based Learning: ECGBL 2023. Academic Conferences and publishing limited.
  5. Personalized Quest and Dialogue Generation in Role-Playing Games: A Knowledge Graph- and Language Model-Based Approach. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (¡conf-loc¿, ¡city¿Hamburg¡/city¿, ¡country¿Germany¡/country¿, ¡/conf-loc¿) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 290, 20 pages. https://doi.org/10.1145/3544548.3581441
  6. Wor (l) d-GAN: Towards Natural Language Based PCG in Minecraft. IEEE Transactions on Games (2022).
  7. A bi-step grounding paradigm for large language models in recommendation systems. arXiv preprint arXiv:2308.08434 (2023).
  8. Jubileo: An Immersive Simulation Framework for Social Robot Design. Journal of Intelligent & Robotic Systems 109, 4 (2023), 91.
  9. Towards A Natural Language Interface for Flexible Multi-Agent Task Assignment. arXiv preprint arXiv:2311.00153 (2023).
  10. A Preliminary Study on a Conceptual Game Feature Generation and Recommendation System. arXiv preprint arXiv:2308.13538 (2023).
  11. Gamegpt: Multi-agent collaborative framework for game development. arXiv preprint arXiv:2310.08067 (2023).
  12. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.10848 (2023).
  13. Using new AI-driven techniques to ease serious games authoring. In 2023 IEEE Frontiers in Education Conference (FIE). IEEE, 1–9.
  14. Augmenting Autotelic Agents with Large Language Models. arXiv preprint arXiv:2305.12487 (2023).
  15. An Appraisal-Based Chain-Of-Emotion Architecture for Affective Language Model Game Agents. arXiv preprint arXiv:2309.05076 (2023).
  16. Lajos Matyas Csepregi. 2021. The Effect of Context-aware LLM-based NPC Conversations on Player Engagement in Role-playing Video Games. Unpublished manuscript (2021).
  17. Stefano De Paoli. 2023. Performing an Inductive Thematic Analysis of Semi-Structured Interviews With a Large Language Model: An Exploration and Provocation on the Limits of the Approach. Social Science Computer Review (2023), 08944393231220483.
  18. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.06692 (2023).
  19. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.09971 (2023).
  20. Skill Check: Some Considerations on the Evaluation of Gamemastering Models for Role-Playing Games. In International Conference on Games and Learning Alliance. Springer, 277–288.
  21. Efficient Human-AI Coordination via Preparatory Language-based Convention. arXiv preprint arXiv:2311.00416 (2023).
  22. The Chronicles of ChatGPT: Generating and Evaluating Visual Novel Narratives on Climate Change Through ChatGPT. In International Conference on Interactive Digital Storytelling. Springer, 181–194.
  23. Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects. (Nov. 2023). https://doi.org/10.36227/techrxiv.23589741.v4
  24. Neural Language Models as What If?-Engines for HCI Research. In 27th International Conference on Intelligent User Interfaces. 77–80.
  25. Evaluating large language models in generating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–19.
  26. Procedural content generation for games: A survey. ACM Trans. Multimedia Comput. Commun. Appl. 9, 1, Article 1 (feb 2013), 22 pages. https://doi.org/10.1145/2422956.2422957
  27. Lei Huang and Xing Sun. 2023. Create Ice Cream: Real-time Creative Element Synthesis Framework Based on GPT3. 0. In 2023 IEEE Conference on Games (CoG). IEEE, 1–4.
  28. How ChatGPT can inspire and improve serious board game design. International Journal of Serious Games 10, 4 (2023), 33–54.
  29. Lyfe Agents: Generative agents for low-cost real-time social interactions. arXiv preprint arXiv:2310.02172 (2023).
  30. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:2310.00166 (2023).
  31. End-to-End Procedural Level Generation in Educational Games with Natural Language Instruction. In 2023 IEEE Conference on Games (CoG). IEEE, 1–8.
  32. SCENECRAFT: automating interactive narrative scene generation in digital games with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 19. 86–96.
  33. Michael Lankes and Andreas Stöckl. 2023. Game Reviews Reviewed: A Game Designer’s Perspective on AI-generated Game Review Analyses. In 2023 IEEE Conference on Games (CoG). IEEE, 1–8.
  34. Pier Luca Lanzi and Daniele Loiacono. 2023. Chatgpt and other large language models as evolutionary engines for online interactive collaborative game design. arXiv preprint arXiv:2303.02155 (2023).
  35. Generating video game scripts with style. In Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023). 129–139.
  36. Lane Lawley and Christopher J MacLellan. 2023. VAL: Interactive Task Learning with GPT Dialog Parsing. arXiv preprint arXiv:2310.01627 (2023).
  37. Juyong Lee and Doohyun Lee. [n. d.]. Can Language Models Postprocess Rewards for Reinforcement Learning? ([n. d.]).
  38. RecExplainer: Aligning Large Language Models for Recommendation Model Interpretability. arXiv preprint arXiv:2311.10947 (2023).
  39. Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study. arXiv preprint arXiv:2311.07387 (2023).
  40. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint arXiv:2310.19773 (2023).
  41. LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination. arXiv preprint arXiv:2312.15224 (2023).
  42. Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach. arXiv preprint arXiv:2312.11865 (2023).
  43. Chatter Generation through Language Models. In 2023 IEEE Conference on Games (CoG). IEEE, 1–6.
  44. Muhammad U Nasir and Julian Togelius. 2023. Practical PCG Through Large Language Models. arXiv preprint arXiv:2305.18243 (2023).
  45. Am I Fighting Well? Fighting Game Commentary Generation With ChatGPT. In Proceedings of the 13th International Conference on Advances in Information Technology. 1–7.
  46. Mark J. Nelson Noor Shaker, Julian Togelius. 2016. Procedural Content Generation in Games. Springer Cham. https://doi.org/10.1007/978-3-319-42716-4
  47. Selective Perception: Learning Concise State Descriptions for Language Model Actors. In NeurIPS 2023 Foundation Models for Decision Making Workshop.
  48. Selective perception: Optimizing state descriptions with reinforcement learning for language model actors. arXiv preprint arXiv:2307.11922 (2023).
  49. Conversational Agents for Simulation Applications and Video Games. In Proceedings of the 18th International Conference on Software Technologies - ICSOFT. INSTICC, SciTePress, 27–36. https://doi.org/10.5220/0012060500003538
  50. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–22.
  51. diff History for Long-Context Language Agents. arXiv preprint arXiv:2312.07540 (2023).
  52. Controlling Personality Style in Dialogue with Zero-Shot Prompt-Based Learning. arXiv preprint arXiv:2302.03848 (2023).
  53. Make-A-Character: High Quality Text-to-3D Character Generation within Minutes. arXiv preprint arXiv:2312.15430 (2023).
  54. Steps towards prompt-based creation of virtual worlds. arXiv preprint arXiv:2211.05875 (2022).
  55. Surreal vr pong: Llm approach to game design. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022). https://www. microsoft. com/en-us/research/publication/surreal-vr-pong-llm-approach-to-gamedesign.
  56. Double Impact: Children’s Serious RPG Generation/Play with a Large Language Model for Their Deeper Engagement in Social Issues. In Joint International Conference on Serious Games. Springer, 274–289.
  57. Towards a Holodeck-style Simulation Game. arXiv preprint arXiv:2308.13548 (2023).
  58. RAH! RecSys-Assistant-Human: A Human-Central Recommendation Framework with Large Language Models. arXiv preprint arXiv:2308.09904 (2023).
  59. Prompt-Guided Level Generation. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation. 179–182.
  60. MarioGPT: Open-Ended Text2Level Generation through Large Language Models. arXiv:2302.05981 [cs.AI]
  61. Language as reality: a co-creative storytelling game experience in 1001 nights using generative AI. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 19. 425–434.
  62. Fictional Worlds, Real Connections: Developing Community Storytelling Social Chatbots through LLMs. arXiv preprint arXiv:2309.11478 (2023).
  63. Large Language Models for Intent-Driven Session Recommendations. arXiv preprint arXiv:2312.07552 (2023).
  64. Arina Svetasheva and Keeheon Lee. 2024. Harnessing Large Language Models for Effective and Efficient Hate Speech Detection. (2024).
  65. GlitchBench: Can large multimodal models detect video game glitches? arXiv preprint arXiv:2312.05291 (2023).
  66. Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectors. arXiv preprint arXiv:2210.02506 (2022).
  67. ChatGPT4PCG Competition: Character-like Level Generation for Science Birds. arXiv:2303.15662 [cs.AI]
  68. What Is Waiting for Us at the End? Inherent Biases of Game Story Endings in Large Language Models. In International Conference on Interactive Digital Storytelling. Springer, 274–284.
  69. Journey of ChatGPT from Prompts to Stories in Games: the Positive, the Negative, and the Neutral. In 2023 IEEE 13th International Conference on Consumer Electronics-Berlin (ICCE-Berlin). IEEE, 202–203.
  70. AI in board Game-Based Learning. In CEUR WORKSHOP PROCEEDINGS.
  71. Level Generation Through Large Language Models. In Proceedings of the 18th International Conference on the Foundations of Digital Games. 1–8.
  72. Generating role-playing game quests with gpt language models. IEEE Transactions on Games (2022).
  73. Preference Proxies: Evaluating Large Language Models in capturing Human Preferences in Human-AI Tasks. In ICML 2023 Workshop The Many Facets of Preference-Based Learning.
  74. Markos Viggiato and Cor-Paul Bezemer. 2023. Leveraging the OPT Large Language Model for Sentiment Analysis of Game Reviews. IEEE Transactions on Games (2023).
  75. Craft an iron sword: Dynamically generating interactive game characters by prompting large language models tuned on code. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022). 25–43.
  76. Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291 [cs.AI]
  77. SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning. arXiv preprint arXiv:2305.15486 (2023).
  78. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940 (2023).
  79. Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Models. arXiv preprint arXiv:2310.18127 (2023).
  80. Octopus: Embodied vision-language programmer from environmental feedback. arXiv preprint arXiv:2310.08588 (2023).
  81. Large language model can interpret latent space of sequential recommender. arXiv preprint arXiv:2310.20487 (2023).
  82. Qing Ru Yong and Alex Mitchell. 2023. From Playing the Story to Gaming the System: Repeat Experiences of a Large Language Model-Based Interactive Story. In International Conference on Interactive Digital Storytelling. Springer, 395–409.
  83. Lei Yue and Liang Guo. 2023. Combine DGBL With AI System: A Technical Guidance to Reduce Teacher’s Burden in Digital Game-Based Learning. In European Conference on Games Based Learning. Academic Conferences International Limited, 826–XXV.
  84. LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking. arXiv preprint arXiv:2311.02089 (2023).
  85. Recommendation as instruction following: A large language model empowered recommendation approach. arXiv preprint arXiv:2305.07001 (2023).
  86. Adapting large language models by integrating collaborative semantics for recommendation. arXiv preprint arXiv:2311.09049 (2023).
  87. Fireball: A dataset of dungeons and dragons actual-play with structured game state information. arXiv preprint arXiv:2305.01528 (2023).
  88. Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory. arXiv:2305.17144 [cs.AI]
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (1)
  1. Penny Sweetser (4 papers)
Citations (8)
View on Semantic Scholar