Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Predicting Player Engagement in Tom Clancy's The Division 2: A Multimodal Approach via Pixels and Gamepad Actions (2310.06136v1)

Published 9 Oct 2023 in cs.HC

Abstract: This paper introduces a large scale multimodal corpus collected for the purpose of analysing and predicting player engagement in commercial-standard games. The corpus is solicited from 25 players of the action role-playing game Tom Clancy's The Division 2, who annotated their level of engagement using a time-continuous annotation tool. The cleaned and processed corpus presented in this paper consists of nearly 20 hours of annotated gameplay videos accompanied by logged gamepad actions. We report preliminary results on predicting long-term player engagement based on in-game footage and game controller actions using Convolutional Neural Network architectures. Results obtained suggest we can predict the player engagement with up to 72% accuracy on average (88% at best) when we fuse information from the game footage and the player's controller input. Our findings validate the hypothesis that long-term (i.e. 1 hour of play) engagement can be predicted efficiently solely from pixels and gamepad actions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Deep learning techniques for speech emotion recognition, from databases to models. Sensors 21, 4 (2021), 1249.
  2. Multimodal emotion recognition using deep learning. Journal of Applied Science and Technology Trends 2, 02 (2021), 52–58.
  3. Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of school psychology 44, 5 (2006), 427–445.
  4. The FUNii database: A physiological, behavioral, demographic and subjective video game database for affective gaming and player experience research. In Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1–7.
  5. Uta K Bindl and Sharon K Parker. 2010. 32 Feeling good and performing well? Psychological engagement and positive behaviors at work. Handbook of employee engagement: Perspectives, issues, research and practice 385 (2010).
  6. Ran Breuer and Ron Kimmel. 2017. A deep learning perspective on the origin of facial expressions. arXiv preprint arXiv:1705.01842 (2017).
  7. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems 34 (2021), 15084–15097.
  8. Mohamed Dahmane and Jean Meunier. 2011. Emotion recognition using dynamic grid-based HoG features. In Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition (FG). IEEE, 884–888.
  9. A data-driven approach to reverse engineering customer engagement models: Towards functional constructs. PLOS ONE 9, 7 (2014).
  10. Soumia Dermouche and Catherine Pelachaud. 2019. Engagement modeling in dyadic interaction. In Proceedings of the International Conference on Multimodal Interaction. 440–445.
  11. MUMBAI: multi-person, multimodal board game affect and interaction analysis dataset. Journal on Multimodal User Interfaces (2021), 1–19.
  12. Oliver Dunn. 2012. Multiple comparisons among means. J. Amer. Statist. Assoc. 56 (2012), 52–64. Issue 293.
  13. A robotic coach architecture for elder care (ROCARE) based on multi-user engagement models. IEEE Transactions on Neural Systems and Rehabilitation Engineering 25, 8 (2016), 1153–1163.
  14. A survey on concept drift adaptation. ACM computing surveys 46, 4 (2014), 1–37.
  15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
  16. “Level up”: Leveraging skill and engagement to maximize player game-play in online video games. Information Systems Research 30, 3 (2019), 927–947.
  17. The platformer experience dataset. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII). 712–718.
  18. Michael Kearns and Dana Ron. 1999. Algorithmic Stability and Sanity-Check Bounds for Leave-One-out Cross-Validation. Neural Computation 11, 6 (1999).
  19. Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing 3, 1 (2011), 18–31.
  20. AFEW-VA database for valence and arousal estimation in-the-wild. Image and Vision Computing 65 (2017), 23–36.
  21. Sewa db: A rich database for audio-visual emotion and sentiment research in the wild. IEEE transactions on pattern analysis and machine intelligence 43, 3 (2019), 1022–1040.
  22. Love, hate, arousal and engagement: exploring audience responses to performing arts. In Proceedings of the SIGCHI conference on human factors in computing systems. 1845–1854.
  23. Joseph LeDoux. 1998. The emotional brain: The mysterious underpinnings of emotional life. Simon and Schuster.
  24. A review on speech emotion recognition using deep learning and attention mechanism. Electronics 10, 10 (2021), 1163.
  25. Ranktrace: Relative and unbounded affect annotation. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction. 158–163.
  26. From pixels to affect: A study on games and player experience. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction.
  27. The pixels and sounds of emotion: General-purpose representations of arousal in games. IEEE Transactions on Affective Computing (2021).
  28. Privileged information for modeling affect in the wild. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction. IEEE.
  29. The Invariant Ground Truth of Affect. In Proceedings of the ACII Workshop on What’s Next in Affect Modeling?
  30. Learning deep physiological models of affect. IEEE Computational intelligence magazine 8, 2 (2013), 20–33.
  31. Héctor P Martínez and Georgios N Yannakakis. 2014. Deep multimodal fusion: Combining discrete events and continuous signals. In Proceedings of the 16th International conference on multimodal interaction. 34–41.
  32. Multiplayer Tension In the Wild: A Hearthstone Case. In Proceedings of the International Conference on the Foundations of Digital Games.
  33. Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch. In Proceedings of the 15th International Conference on the Foundations of Digital Games. 1–10.
  34. PAGAN: Video affect annotation made easy. In Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 130–136.
  35. The arousal video game annotation (AGAIN) dataset. IEEE Transactions on Affective Computing 13, 4 (2022), 2171–2184.
  36. The Ethics of AI in Games. IEEE Transactions on Affective Computing (2023).
  37. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing 10, 1 (2017), 18–31.
  38. Deep learning for emotion recognition on small datasets using transfer learning. In Proceedings of the 2015 ACM on international conference on multimodal interaction. 443–449.
  39. An analysis of state-of-the-art activation functions for supervised deep neural network. In Proceedings of the International Conference on System Science and Engineering (ICSSE). IEEE, 215–220.
  40. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations).
  41. Rosalind W Picard. 2000. Affective computing. MIT press.
  42. RankNEAT: outperforming stochastic gradient search in preference learning tasks. In Proceedings of the Genetic and Evolutionary Computation Conference. 1084–1092.
  43. Supervised contrastive learning for affect modelling. In Proceedings of the International Conference on Multimodal Interaction. 531–539.
  44. Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. In Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture recognition (FG). 1–8.
  45. Imagenet large scale visual recognition challenge. International Journal of computer vision 115, 3 (2015).
  46. James A Russell. 1980. A circumplex model of affect. Journal of personality and social psychology 39, 6 (1980), 1161.
  47. Multimodal emotion recognition. In Handbook of pattern recognition and computer vision. World Scientific, 387–409.
  48. A multimodal database for affect recognition and implicit tagging. IEEE transactions on affective computing 3, 1 (2011), 42–55.
  49. Measuring Visual Fatigue and Cognitive Load via Eye Tracking while Learning with Virtual Reality Head-Mounted Displays: A Review. International Journal of Human–Computer Interaction 38, 9 (2022), 801–824.
  50. Student engagement modeling using bayesian networks. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 2939–2944.
  51. Attention is all you need. Advances in neural information processing systems 30 (2017).
  52. Masked conditional video diffusion for prediction, generation, and interpolation. arXiv preprint arXiv:2205.09853 (2022).
  53. Dynamic difficulty adjustment for maximized engagement in digital games. In Proceedings of the 26th International Conference on World Wide Web Companion. 465–471.
  54. The ordinal nature of emotions. In Proceedings of the 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 248–255.
  55. The ordinal nature of emotions: An emerging approach. IEEE Transactions on Affective Computing 12, 1 (2018), 16–35.
  56. Towards affective camera control in games. User Modeling and User-Adapted Interaction 20 (2010), 313–340.
  57. Georgios N Yannakakis and Ana Paiva. 2014. Emotion in games. Handbook on affective computing 2014 (2014), 459–471.
  58. Aff-wild: valence and arousal’In-the-Wild’challenge. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 34–41.
  59. DeepVANet: A Deep End-to-End Network for Multi-Modal Emotion Recognition. In Procceedings of the 18th International Conference on Human-Computer Interaction (INTERACT). 227–237.
  60. Emotion Recognition from Arbitrary View Facial Images. In Proceedings of the 11th European Conference on Computer Vision: Part VI. Springer-Verlag, Berlin, Heidelberg, 490–503.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now