Papers
Topics
Authors
Recent
2000 character limit reached

On the Evaluation of Procedural Level Generation Systems (2404.18657v1)

Published 29 Apr 2024 in cs.HC

Abstract: The evaluation of procedural content generation (PCG) systems for generating video game levels is a complex and contested topic. Ideally, the field would have access to robust, generalisable and widely accepted evaluation approaches that can be used to compare novel PCG systems to prior work, but consensus on how to evaluate novel systems is currently limited. We argue that the field can benefit from a structured analysis of how procedural level generation systems can be evaluated, and how these techniques are currently used by researchers. This analysis can then be used to both inform on the current state of affairs, and to provide data to justify changes to this practice. This work aims to provide this by first developing a novel taxonomy of PCG evaluation approaches, and then presenting the results of a survey of recent work in the field through the lens of this taxonomy. The results of this survey highlight several important weaknesses in current practice which we argue could be substantially mitigated by 1) promoting use of evaluation free system descriptions where appropriate, 2) promoting the development of diverse research frameworks, 3) promoting reuse of code and methodology wherever possible.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. Tarn Adams and Zach Adams. 2022. Dwarf Fortress.
  2. OpenAI Gym. (2016). https://doi.org/10.48550/ARXIV.1606.01540 Publisher: arXiv Version Number: 1.
  3. Alessandro Canossa and Gillian Smith. 2015. Towards a procedural evaluation technique: Metrics for level design. In International conference on foundations of digital games. https://api.semanticscholar.org/CorpusID:16957692
  4. Danesh: Interactive Tools For Understanding Procedural Content Generators. IEEE Transactions on Games (2021). https://ieeexplore.ieee.org/document/9426419
  5. Blizard Entertainment. 2023. Diablo 4.
  6. Surrogate Infeasible Fitness Acquirement FI-2Pop for Procedural Content Generation. In 2022 IEEE Conference on Games (CoG). IEEE, Beijing, China, 500–503. https://doi.org/10.1109/CoG51982.2022.9893592
  7. Mario Level Generation From Mechanics Using Scene Stitching. arXiv:2002.02992 [cs] (Feb. 2020). http://arxiv.org/abs/2002.02992 arXiv: 2002.02992.
  8. Procedural content generation for games: A survey. ACM Transactions on Multimedia Computing, Communications, and Applications 9, 1 (Feb. 2013), 1–22. https://doi.org/10.1145/2422956.2422957
  9. A Procedural Content Generation-Based Framework for Educational Games: Toward a Tailored Data-Driven Game for Developing Early English Reading Skills. Journal of Educational Computing Research 56, 2 (April 2018), 293–310. https://doi.org/10.1177/0735633117706909
  10. Video game play is positively correlated with well-being. Royal Society Open Science 8, 2 (Feb. 2021), rsos.202049, 202049. https://doi.org/10.1098/rsos.202049
  11. Intentional Computational Level Design. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’19). Association for Computing Machinery, New York, NY, USA, 796–803. https://doi.org/10.1145/3321707.3321849 event-place: Prague, Czech Republic.
  12. Damian Kutzias and Sebastian Von Mammen. 2023. Recent Advances in Procedural Generation of Buildings: From Diversity to Integration. IEEE Transactions on Games (2023), 1–20. https://doi.org/10.1109/TG.2023.3262507
  13. On Mixed-Initiative Content Creation for Video Games. IEEE Transactions on Games (2022), 1–1. https://doi.org/10.1109/TG.2022.3176215
  14. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324. https://doi.org/10.1109/5.726791
  15. Antonios Liapis. 2020. 10 Years of the PCG workshop: Past and Future Trends. In International Conference on the Foundations of Digital Games. ACM, Bugibba Malta, 1–10. https://doi.org/10.1145/3402942.3409598
  16. Constrained Novelty Search: A Study on Game Content Generation. Evol. Comput. 23, 1 (March 2015), 101–129. https://doi.org/10.1162/EVCO_a_00123 Place: Cambridge, MA, USA Publisher: MIT Press.
  17. Deep Learning for Procedural Content Generation. Neural Computing and Applications 33, 1 (Jan. 2021), 19–37. https://doi.org/10.1007/s00521-020-05383-8 arXiv: 2010.04548.
  18. Simon M. Lucas and Vanessa Volz. 2019. Tile Pattern KL-Divergence for Analysing and Evolving Game Levels. Proceedings of the Genetic and Evolutionary Computation Conference (July 2019), 170–178. https://doi.org/10.1145/3321707.3321781 arXiv: 1905.05077.
  19. Improving Human Players’ T-Spin Skills in Tetris with Procedural Problem Generation. In Advances in Computer Games, Tristan Cazenave, Jaap Van Den Herik, Abdallah Saffidine, and I-Chen Wu (Eds.). Vol. 12516. Springer International Publishing, Cham, 41–52. https://doi.org/10.1007/978-3-030-65883-0_4 Series Title: Lecture Notes in Computer Science.
  20. The 2014 General Video Game Playing Competition. IEEE Transactions on Computational Intelligence and AI in Games 8, 3 (2016), 229–243. https://doi.org/10.1109/TCIAIG.2015.2402393
  21. Impressions of the GDMC AI Settlement Generation Challenge in Minecraft. arXiv:2108.02955 [cs] (Aug. 2021). http://arxiv.org/abs/2108.02955 arXiv: 2108.02955.
  22. Anurag Sarkar and Seth Cooper. 2021. Generating and Blending Game Levels via Quality-Diversity in the Latent Space of a Variational Autoencoder. In The 16th International Conference on the Foundations of Digital Games (FDG) 2021 (FDG’21). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3472538.3472545 event-place: Montreal, QC, Canada.
  23. Procedural Content Generation in Games. Springer International Publishing, Cham. https://doi.org/10.1007/978-3-319-42716-4
  24. Gillian Smith and Jim Whitehead. 2010. Analyzing the expressive range of a level generator. In Proceedings of the 2010 Workshop on Procedural Content Generation in Games - PCGames ’10. ACM Press, Monterey, California, 1–7. https://doi.org/10.1145/1814256.1814260
  25. Iron Gate Studio. 2021. Valheim.
  26. Bethesda Game Studios. 2023. Starfield.
  27. MarioGPT: Open-Ended Text2Level Generation through Large Language Models. http://arxiv.org/abs/2302.05981 arXiv:2302.05981 [cs].
  28. Understanding mario: an evaluation of design metrics for platformers. In Proceedings of the 12th International Conference on the Foundations of Digital Games. ACM, Hyannis Massachusetts, 1–10. https://doi.org/10.1145/3102071.3102080
  29. Adam Summerville and Michael Mateas. 2016. Super Mario as a String: Platformer Level Generation Via LSTMs. arXiv:1603.00930 [cs] (March 2016). http://arxiv.org/abs/1603.00930 arXiv: 1603.00930.
  30. Procedural Content Generation via Machine Learning (PCGML). IEEE Transactions on Games 10, 3 (Sept. 2018), 257–270. https://doi.org/10.1109/TG.2018.2846639
  31. The 2009 Mario AI Competition. In IEEE Congress on Evolutionary Computation. IEEE, Barcelona, Spain, 1–8. https://doi.org/10.1109/CEC.2010.5586133
  32. Search-Based Procedural Content Generation: A Taxonomy and Survey. IEEE Transactions on Computational Intelligence and AI in Games 3, 3 (2011), 172–186. https://doi.org/10.1109/TCIAIG.2011.2148116
  33. Tools for Landscape Analysis of Optimisation Problems in Procedural Content Generation for Games. Applied Soft Computing 136 (March 2023), 110121. https://doi.org/10.1016/j.asoc.2023.110121
  34. Oliver Withington. 2020. Illuminating super mario bros: quality-diversity within platformer level generation. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion. ACM, Cancún Mexico, 223–224. https://doi.org/10.1145/3377929.3390043

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 2 tweets with 28 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free